block/vpc optimizations
v2 cover block/vpc optimizations
This series covers VPC format changes discussed during the last weeks.
Peter
v1->v2:
Patch 1: added comment why we can exit after the first allocated sector [Max] Patch 2: Fixed gemotry in commit msg [Max]
Patch 3: Fixed comparision from > to >= [Max]
Fixed exit in case of -EFBIG [Max]
Removed DIV_ROUND_UP [Max]
Patch 5: added
Kevin Wolf (1):
vpc: Ignore geometry for large images
Peter Lieven (4):
block/vpc: optimize vpc_co_get_block_status block/vpc: make calculate_geometry spec conform block/vpc: rename footer->size -> footer->current_size block/vpc: remove disabled code from get_sector_offset
block/vpc.c | 106 +++++++++++++++++++++--------------------------------------- 1 file changed, 37 insertions(+), 69 deletions(-)
Signature · 1 line
1.9.1
PATCH v2 1/5 block/vpc: optimize vpc_co_get_block_status
*pnum can't be greater than s->block_size / BDRV_SECTOR_SIZE for allocated sectors since there is always a bitmap in between.
--- block/vpc.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-)
block/vpc.c +8 -10
@@ -602,7 +602,7 @@ static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs, {BDRVVPCState *s = bs->opaque;VHDFooter *footer = (VHDFooter *) s->footer_buf; -int64_t start, offset, next; +int64_t start, offset;bool allocated;int n;
@@ -626,20 +626,18 @@ static int64_t coroutine_fn vpc_co_get_block_status(BlockDriverState *bs, *pnum += n; sector_num += n; nb_sectors -= n; - next = start + (*pnum *BDRV_SECTOR_SIZE ); - +/* *pnum can't be greater than one block for allocated +* sectors since there is always a bitmap in between. */ +if (allocated) { +return BDRV_BLOCK_DATA |BDRV_BLOCK_OFFSET_VALID | start; + }if (nb_sectors ==0 ) {break ; } - offset =get_sector_offset (bs, sector_num,0 ); - }while ((allocated && offset == next) || (!allocated && offset ==-1 )); + }while (offset ==-1 ); -if (allocated) { -return BDRV_BLOCK_DATA |BDRV_BLOCK_OFFSET_VALID | start; - }else { -return 0 ; - } +return 0 ; }/*
Signature · 1 line
1.9.1
PATCH v2 2/5 vpc: Ignore geometry for large images
In reply to Peter LievenFrom: Kevin Wolf <[email protected]>
The CHS calculation as done per the VHD spec imposes a maximum image size of ~127 GB. Real VHD images exist that are larger than that.
Apparently there are two separate non-standard ways to achieve this:
You could use more heads than the spec does - this is the option that qemu-img create chooses.
However, other images exist where the geometry is set to the maximum (65535/16/255), but the actual image size is larger. Until now, such images are truncated at 127 GB when opening them with qemu.
This patch changes the vpc driver to ignore geometry in this case and only trust the size field in the header.
--- block/vpc.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
block/vpc.c +4 -6
@@ -215,12 +215,10 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags, bs->total_sectors = (int64_t )be16_to_cpu (footer->cyls) * footer->heads * footer->secs_per_cyl; -/* images created with disk2vhd report a far higher virtual size -* than expected with the cyls * heads * sectors_per_cyl formula. -* use the footer->size instead if the image was created with -* disk2vhd. -*/ -if (!strncmp (footer->creator_app,"d2v" ,4 )) { +/* Images that have exactly the maximum geometry are probably bigger and +* would be truncated if we adhered to the geometry for them. Rely on +* footer->size for them. */ +if (bs->total_sectors ==65535ULL *16 *255 ) { bs->total_sectors =be64_to_cpu (footer->size) /BDRV_SECTOR_SIZE ; }
Signature · 1 line
1.9.1
Re: [Qemu-devel] [PATCH v2 2/5] vpc: Ignore geometry for large images
Quoted 16 lines · Peter Lieven
From: Kevin Wolf <[email protected]>
The CHS calculation as done per the VHD spec imposes a maximum image size of ~127 GB. Real VHD images exist that are larger than that.
Apparently there are two separate non-standard ways to achieve this:
You could use more heads than the spec does - this is the option that qemu-img create chooses.However, other images exist where the geometry is set to the maximum (65535/16/255), but the actual image size is larger. Until now, such images are truncated at 127 GB when opening them with qemu.
This patch changes the vpc driver to ignore geometry in this case and only trust the size field in the header.
Quoted 6 lines
Signed-off-by Kevin Wolf <[email protected]>[PL: Fixed maximum geometry in the commit msg]Signed-off-by Peter Lieven <[email protected]>--- block/vpc.c | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-)
PATCH v2 3/5 block/vpc: make calculate_geometry spec conform
In reply to Peter LievenThe VHD spec [1] allows for total_sectors of 65535 x 16 x 255 (~127GB) represented by a CHS geometry. If total_sectors is greater than 65535 x 16 x 255 this geometry is set as a maximum.
Qemu, Hyper-V and disk2vhd use this special geometry as an indicator to use the image current size from the footer as disk size.
This patch changes vpc_create to effectively calculate a CxHxS geometry for the given image size if possible while rounding up if necessary. If the image size is to big to be represented in CHS we set the maximum and write the exact requested image size into the footer.
This partly reverts commit 258d2edb, but leaves support for >127G disks intact.
--- block/vpc.c | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-)
block/vpc.c +22 -19
@@ -46,6 +46,7 @@ enum vhd_type {#define VHD_TIMESTAMP_BASE 946684800#define VHD_MAX_SECTORS (65535LL * 255 * 255) +#define VHD_MAX_GEOMETRY (65535LL * 16 * 255)// always big-endian typedef struct vhd_footer {
@@ -218,7 +219,7 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,/* Images that have exactly the maximum geometry are probably bigger and * would be truncated if we adhered to the geometry for them. Rely on * footer->size for them. */ -if (bs->total_sectors ==65535ULL *16 *255 ) { +if (bs->total_sectors ==VHD_MAX_GEOMETRY ) { bs->total_sectors =be64_to_cpu (footer->size) /BDRV_SECTOR_SIZE ; }
@@ -655,26 +656,20 @@ static int calculate_geometry(int64_t total_sectors, uint16_t* cyls, {uint32_t cyls_times_heads; -/* Allow a maximum disk size of approximately 2 TB */ -if (total_sectors >65535LL *255 *255 ) { -return -EFBIG ; - } + total_sectors =MIN (total_sectors,VHD_MAX_GEOMETRY ); -if (total_sectors >65535 *16 *63 ) { +if (total_sectors >=65535LL *16 *63 ) { *secs_per_cyl =255 ; -if (total_sectors >65535 *16 *255 ) { - *heads =255 ; - }else { - *heads =16 ; - } + *heads =16 ; cyls_times_heads = total_sectors / *secs_per_cyl; }else { *secs_per_cyl =17 ; cyls_times_heads = total_sectors / *secs_per_cyl; *heads = (cyls_times_heads +1023 ) /1024 ; -if (*heads <4 ) +if (*heads <4 ) { *heads =4 ; + }if (cyls_times_heads >= (*heads *1024 ) || *heads >16 ) { *secs_per_cyl =31 ;
@@ -830,20 +825,28 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp)* Calculate matching total_size and geometry. Increase the number of * sectors requested until we get enough (or fail). This ensures that * qemu-img convert doesn't truncate images, but rather rounds up. +* +* If the image size can't be represented by a spec conform CHS geometry, +* we set the geometry to 65535 x 16 x 255 (CxHxS) sectors and use +* the image size from the VHD footer to calculate total_sectors. */ - total_sectors = total_size /BDRV_SECTOR_SIZE ; + total_sectors =MIN (VHD_MAX_GEOMETRY , total_size /BDRV_SECTOR_SIZE );for (i =0 ; total_sectors > (int64_t )cyls * heads * secs_per_cyl; i++) { -if (calculate_geometry (total_sectors + i, &cyls, &heads, - &secs_per_cyl)) - { +calculate_geometry (total_sectors + i, &cyls, &heads, &secs_per_cyl); + } + +if ((int64_t )cyls * heads * secs_per_cyl ==VHD_MAX_GEOMETRY ) { + total_sectors = total_size /BDRV_SECTOR_SIZE ; +/* Allow a maximum disk size of approximately 2 TB */ +if (total_sectors >VHD_MAX_SECTORS ) { ret = -EFBIG ; goto out; } + }else { + total_sectors = (int64_t )cyls * heads * secs_per_cyl; + total_size = total_sectors *BDRV_SECTOR_SIZE ; } - total_sectors = (int64_t ) cyls * heads * secs_per_cyl; - total_size = total_sectors *BDRV_SECTOR_SIZE ; -/* Prepare the Hard Disk Footer */ memset (buf,0 ,1024 );
Signature · 1 line
1.9.1
Re: [Qemu-devel] [PATCH v2 3/5] block/vpc: make calculate_geometry spec conform
Quoted 10 lines · Peter Lieven
The VHD spec [1] allows for
total_sectorsof 65535 x 16 x 255 (~127GB) represented by a CHS geometry. Iftotal_sectorsis greater than 65535 x 16 x 255 this geometry is set as a maximum.Qemu, Hyper-V and disk2vhd use this special geometry as an indicator to use the image current size from the footer as disk size.
This patch changes
vpc_createto effectively calculate a CxHxS geometry for the given image size if possible while rounding up if necessary. If the image size is to big to be represented in CHS we set the maximum
"too big" (as it was in v1) was correct, no need to make it wrong. :-)
Quoted 11 lines
and write the exact requested image size into the footer.
This partly reverts commit
258d2edb, but leaves support for >127G disks intact.Signed-off-by Peter Lieven <[email protected]>--- block/vpc.c | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-)
With s/to big/too big/:
PATCH v2 4/5 block/vpc: rename footer->size -> footer->current_size
In reply to Peter Lieventhe field is named current size in the spec. Name it accordingly.
--- block/vpc.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
block/vpc.c +5 -4
@@ -66,7 +66,7 @@ typedef struct vhd_footer {char creator_os[4 ];// "Wi2k" uint64_t orig_size; -uint64_t size; +uint64_t current_size;uint16_t cyls;uint8_t heads;
@@ -218,9 +218,10 @@ static int vpc_open(BlockDriverState *bs, QDict *options, int flags,/* Images that have exactly the maximum geometry are probably bigger and * would be truncated if we adhered to the geometry for them. Rely on -* footer->size for them. */ +* footer->current_size for them. */ if (bs->total_sectors ==VHD_MAX_GEOMETRY ) { - bs->total_sectors =be64_to_cpu (footer->size) /BDRV_SECTOR_SIZE ; + bs->total_sectors =be64_to_cpu (footer->current_size) / +BDRV_SECTOR_SIZE ; }/* Allow a maximum disk size of approximately 2 TB */
@@ -868,7 +869,7 @@ static int vpc_create(const char *filename, QemuOpts *opts, Error **errp) footer->major =cpu_to_be16 (0x0005 ); footer->minor =cpu_to_be16 (0x0003 ); footer->orig_size =cpu_to_be64 (total_size); - footer->size =cpu_to_be64 (total_size); + footer->current_size =cpu_to_be64 (total_size); footer->cyls =cpu_to_be16 (cyls); footer->heads = heads; footer->secs_per_cyl = secs_per_cyl;
Signature · 1 line
1.9.1
PATCH v2 5/5 block/vpc: remove disabled code from get_sector_offset
In reply to Peter LievenThe code to check the bitmap for the allocation status of each sector has been "disabled by reason" ever since the vpc driver existed.
The reason might be that we might end up reading sector by sector in vpc_read if we really used it. This would be a performance desaster.
The current code would furthermore not work if the disabled parts get reactivated since vpc_read and vpc_write only use get_sector_offset to check the allocation status of the first sector of a read/write operation. This might lead to sectors incorrectly treated as zero in vpc_read and to sectors getting allocated twice in vpc_write.
--- block/vpc.c | 32 -------------------------------- 1 file changed, 32 deletions(-)
block/vpc.c +0 -32
@@ -376,38 +376,6 @@ static inline int64_t get_sector_offset(BlockDriverState *bs,bdrv_pwrite_sync (bs->file, bitmap_offset, bitmap, s->bitmap_size); } -// printf("sector: %" PRIx64 ", index: %x, offset: %x, bioff: %" PRIx64 ", bloff: %" PRIx64 "\n", -// sector_num, pagetable_index, pageentry_index, -// bitmap_offset, block_offset); - -// disabled by reason -#if 0 -#ifdef CACHE -if (bitmap_offset != s->last_bitmap) - { -lseek (s->fd, bitmap_offset,SEEK_SET ); - - s->last_bitmap = bitmap_offset; - -// Scary! Bitmap is stored as big endian 32bit entries, -// while we used to look it up byte by byte -read (s->fd, s->pageentry_u8,512 ); -for (i =0 ; i <128 ; i++) -be32_to_cpus (&s->pageentry_u32[i]); - } - -if ((s->pageentry_u8[pageentry_index /8 ] >> (pageentry_index %8 )) &1 ) -return -1 ; -#else -lseek (s->fd, bitmap_offset + (pageentry_index /8 ),SEEK_SET ); - -read (s->fd, &bitmap_entry,1 ); - -if ((bitmap_entry >> (pageentry_index %8 )) &1 ) -return -1 ;// not allocated -#endif -#endif -return block_offset; }
Signature · 1 line
1.9.1
Re: [Qemu-devel] [PATCH v2 5/5] block/vpc: remove disabled code from get_sector_offset
Quoted 12 lines · Peter Lieven
The code to check the bitmap for the allocation status of each sector has been "disabled by reason" ever since the vpc driver existed.
The reason might be that we might end up reading sector by sector in
vpc_readif we really used it. This would be a performance desaster.The current code would furthermore not work if the disabled parts get reactivated since
vpc_readandvpc_writeonly useget_sector_offsetto check the allocation status of the first sector of a read/write operation. This might lead to sectors incorrectly treated as zero invpc_readand to sectors getting allocated twice invpc_write.
Quoted 4 lines
Signed-off-by Peter Lieven <[email protected]>--- block/vpc.c | 32 -------------------------------- 1 file changed, 32 deletions(-)
Awww, that code nearly had its ten-year anniversary. Too bad it won't live to see it (I hope...).
Max Reitz
AppliedQuoted 15 lines · Peter Lieven
This series covers VPC format changes discussed during the last weeks.
Peter
v1->v2:
Patch 1: added comment why we can exit after the first allocated sector [Max] Patch 2: Fixed gemotry in commit msg [Max]
Patch 3: Fixed comparision from > to >= [Max]
Fixed exit in case of-EFBIG[Max]
RemovedDIV_ROUND_UP[Max]
Patch 5: addedKevin Wolf (1):
vpc: Ignore geometry for large images
Quoted 8 lines
Peter Lieven (4):
block/vpc: optimizevpc_co_get_block_statusblock/vpc: makecalculate_geometryspec conform block/vpc: renamefooter->size->footer->current_sizeblock/vpc: remove disabled code fromget_sector_offsetblock/vpc.c | 106 +++++++++++++++++++++--------------------------------------- 1 file changed, 37 insertions(+), 69 deletions(-)
Thanks, applied to my block-next tree:
Quoted 23 lines · Max Reitz
Quoted 22 lines · Peter Lieven
This series covers VPC format changes discussed during the last weeks.
Peter
v1->v2:
Patch 1: added comment why we can exit after the first allocated
sector [Max]
Patch 2: Fixed gemotry in commit msg [Max]
Patch 3: Fixed comparision from > to >= [Max]
Fixed exit in case of-EFBIG[Max]
RemovedDIV_ROUND_UP[Max]
Patch 5: addedKevin Wolf (1):
vpc: Ignore geometry for large imagesPeter Lieven (4):
block/vpc: optimizevpc_co_get_block_statusblock/vpc: makecalculate_geometryspec conform block/vpc: renamefooter->size->footer->current_sizeblock/vpc: remove disabled code fromget_sector_offset
Quoted 7 lines
Quoted 3 lines
block/vpc.c | 106 +++++++++++++++++++++--------------------------------------- 1 file changed, 37 insertions(+), 69 deletions(-)Thanks, applied to my block-next tree:
For the sake of completeness: Moved to the block tree, now merged to master.
Max