qemu-img convert: Rewrite copying logic
cover qemu-img convert: Rewrite copying logic
See the commit message of patch 1 for the background.
Kevin Wolf (2):
qemu-img convert: Rewrite copying logic
qemu-iotests: Some qemu-img convert tests
qemu-img.c | 511 +++++++++++++++++++++++++++------------------ tests/qemu-iotests/122 | 83 ++++++++ tests/qemu-iotests/122.out | 30 +++ tests/qemu-iotests/group | 1 + 4 files changed, 419 insertions(+), 206 deletions(-) create mode 100755 tests/qemu-iotests/122 create mode 100644 tests/qemu-iotests/122.out
Signature · 1 line
1.8.3.1
PATCH 1/2 qemu-img convert: Rewrite copying logic
The implementation of qemu-img convert is (a) messy, (b) buggy, and
(c) less efficient than possible. The changes required to beat some sense into it are massive enough that incremental changes would only make my and the reviewers' life harder. So throw it away and reimplement it from scratch.
Let me give some examples what I mean by messy, buggy and inefficient:
(a) The copying logic of qemu-img convert has two separate branches for compressed and normal target images, which roughly do the same - except for a little code that handles actual differences between compressed and uncompressed images, and much more code that implements just a different set of optimisations and bugs. This is unnecessary code duplication, and makes the code for compressed output (unsurprisingly) suffer from bitrot.
The code for uncompressed ouput is run twice to count the the total length for the progress bar. In the first run it just takes a shortcut and runs only half the loop, and when it's done, it toggles a boolean, jumps out of the loop with a backwards goto and starts over. Works, but pretty is something different.
(b) Converting while keeping a backing file (-B option) is broken in several ways. This includes not writing to the image file if the input has zero clusters or data filled with zeros (ignoring that the backing file will be visible instead).
It also doesn't correctly limit every iteration of the copy loop to sectors of the same status so that too many sectors may be copied to in the target image. For -B this gives an unexpected result, for other images it just does more work than necessary.
Conversion with a compressed target completely ignores any target backing file.
(c) qemu-img convert skips reading and writing an area if it knows from metadata that copying isn't needed (except for the bug mentioned above that ignores a status change in some cases). It does, however, read from the source even if it knows that it will read zeros, and then search for non-zero bytes in the read buffer, if it's possible that a write might be needed.
This reimplementation of the copying core reorganises the code to remove the duplication and have a much more obvious code flow, by essentially splitting the copy iteration loop into three parts:
1. Find the number of contiguous sectors of the same status at the current offset (This can also be called in a separate loop for the copying loop in order to determine the total sectors for the progress bar.)
2. Read sectors. If the status implies that there is no data there to read (zero or unallocated cluster), don't do anything.
3. Write sectors depending on the status. If it's data, write it. If we want the backing file to be visible (with -B), don't write it. If it's zeroed, skip it if you can, otherwise use bdrv_write_zeroes() to optimise the write at least where possible.
--- qemu-img.c | 511 ++++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 305 insertions(+), 206 deletions(-)
qemu-img.c +305 -206
@@ -1306,20 +1306,307 @@ out3:return ret; } +enum ImgConvertBlockStatus { +BLK_DATA , +BLK_ZERO , +BLK_BACKING_FILE , +}; + +typedef struct ImgConvertState { +BlockDriverState **src; +int64_t *src_sectors; +int src_cur, src_num; +int64_t src_cur_offset; +int64_t total_sectors; +int64_t allocated_sectors; +enum ImgConvertBlockStatus status; +int64_t sector_next_status; +BlockDriverState *target; +bool has_zero_init; +bool compressed; +bool out_backing; +int min_sparse; +size_t cluster_sectors; +size_t buf_sectors; +}ImgConvertState ; + +static void convert_select_part (ImgConvertState *s,int64_t sector_num) +{ +while (sector_num - s->src_cur_offset >= s->src_sectors[s->src_cur]) { + s->src_cur_offset += s->src_sectors[s->src_cur]; + s->src_cur++; +assert (s->src_cur < s->src_num); + } +} + +static int convert_iteration_sectors (ImgConvertState *s,int64_t sector_num) +{ +int64_t ret; +int n; + +convert_select_part (s, sector_num); + +assert (s->total_sectors > sector_num); + n =MIN (s->total_sectors - sector_num,INT_MAX ); + +if (s->sector_next_status <= sector_num) { + ret =bdrv_get_block_status (s->src[s->src_cur], + sector_num - s->src_cur_offset, + n, &n); +if (ret <0 ) { +return ret; + } + +if (ret &BDRV_BLOCK_ZERO ) { + s->status =BLK_ZERO ; + }else if (ret &BDRV_BLOCK_DATA ) { + s->status =BLK_DATA ; + }else if (!s->out_backing) { +/* Without a target backing file we must copy over the contents of +* the backing file as well. */ +/* TODO Check block status of the backing file chain to avoid +* needlessly reading zeroes and limiting the iteration to the +* buffer size */ + s->status =BLK_DATA ; + }else { + s->status =BLK_BACKING_FILE ; + } + + s->sector_next_status = sector_num + n; + } + + n =MIN (n, s->sector_next_status - sector_num); +if (s->status ==BLK_DATA ) { + n =MIN (n, s->buf_sectors); + } + +/* We need to write complete clusters for compressed images, so if an +* unallocated area is shorter than that, we must consider the whole +* cluster allocated. */ +if (s->compressed) { +if (n < s->cluster_sectors) { + n =MIN (s->cluster_sectors, s->total_sectors - sector_num); + s->status =BLK_DATA ; + }else { + n =QEMU_ALIGN_DOWN (n, s->cluster_sectors); + } + } + +return n; +} + +static int convert_read (ImgConvertState *s,int64_t sector_num,int nb_sectors, +uint8_t *buf) +{ +int n; +int ret; + +if (s->status ==BLK_ZERO || s->status ==BLK_BACKING_FILE ) { +return 0 ; + } + +assert (nb_sectors <= s->buf_sectors); +while (nb_sectors >0 ) { +BlockDriverState *bs; +int64_t bs_sectors; + +/* In the case of compression with multiple source files, we can get a +* nb_sectors that spreads into the next part. So we must be able to +* read across multiple BDSes for one convert_read() call. */ +convert_select_part (s, sector_num); + bs = s->src[s->src_cur]; + bs_sectors = s->src_sectors[s->src_cur]; + + n =MIN (nb_sectors, bs_sectors - (sector_num - s->src_cur_offset)); + ret =bdrv_read (bs, sector_num, buf, n); +if (ret <0 ) { +return ret; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_write (ImgConvertState *s,int64_t sector_num,int nb_sectors, +const uint8_t *buf) +{ +int ret; + +while (nb_sectors >0 ) { + +int n = nb_sectors; + +switch (s->status) { +case BLK_BACKING_FILE : +/* If we have a backing file, leave clusters unallocated that are +* unallocated in the source image, so that the backing file is +* visible at the respective offset. */ +assert (s->out_backing); +break ; + +case BLK_DATA : +/* We must always write compressed clusters as a whole, so don't +* try to find zeroed parts in the buffer. We can only save the +* write if the buffer is completely zeroed. */ +if (s->compressed) { +if (s->has_zero_init && +buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) + { +assert (!s->out_backing); +break ; + } + + ret =bdrv_write_compressed (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } + +/* If there is real non-zero data, we must write it. Otherwise we +* can treat it as zero sectors. */ +if (is_allocated_sectors_min (buf, n, &n, s->min_sparse)) { + ret =bdrv_write (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } +/* fall-through */ + +case BLK_ZERO : +if (s->has_zero_init) { +break ; + } +/* TODO Should this use BDRV_REQ_MAY_UNMAP? */ + ret =bdrv_write_zeroes (s->target, sector_num, n,0 ); +if (ret <0 ) { +return ret; + } +break ; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_do_copy (ImgConvertState *s) +{ +uint8_t *buf =NULL ; +int64_t sector_num, allocated_done; +int ret; +int n; + +/* Check whether we have zero initialisation or can get it efficiently */ + s->has_zero_init = s->min_sparse && !s->out_backing + ?bdrv_has_zero_init (s->target) + : false; + +if (!s->has_zero_init && !s->out_backing && +bdrv_can_write_zeroes_with_unmap (s->target)) + { + ret =bdrv_make_zero (s->target,BDRV_REQ_MAY_UNMAP ); +if (ret <0 ) { + goto fail; + } + s->has_zero_init =1 ; + } + +/* Allocate buffer for copied data. For compressed images, only one cluster +* can be copied at a time. */ +if (s->compressed) { +if (s->cluster_sectors <=0 || s->cluster_sectors > s->buf_sectors) { +error_report ("invalid cluster size" ); + ret = -EINVAL ; + goto fail; + } + s->buf_sectors = s->cluster_sectors; + } + buf =qemu_blockalign (s->target, s->buf_sectors *BDRV_SECTOR_SIZE ); + +/* Calculate allocated sectors for progress */ + s->allocated_sectors =0 ; + sector_num =0 ; +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + s->allocated_sectors += n; + } + sector_num += n; + } + +/* Do the copy */ + s->src_cur =0 ; + s->src_cur_offset =0 ; + s->sector_next_status =0 ; + + sector_num =0 ; + allocated_done =0 ; + +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + allocated_done += n; +qemu_progress_print (100.0 * allocated_done / s->allocated_sectors, +0 ); + } + + ret =convert_read (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while reading sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + ret =convert_write (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while writing sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + sector_num += n; + } + +if (s->compressed) { +/* signal EOF to align */ +bdrv_write_compressed (s->target,0 ,NULL ,0 ); + } + + ret =0 ; +fail: +qemu_vfree (buf); +return ret; +} +static int img_convert (int argc,char **argv) { -int c, n, n1, bs_n, bs_i, compress, cluster_sectors, skip_create; +int c, bs_n, bs_i, compress, cluster_sectors, skip_create;int64_t ret =0 ;int progress =0 , flags, src_flags;const char *fmt, *out_fmt, *cache, *src_cache, *out_baseimg, *out_filename;BlockDriver *drv, *proto_drv;BlockBackend **blk =NULL , *out_blk =NULL ;BlockDriverState **bs =NULL , *out_bs =NULL ; -int64_t total_sectors, nb_sectors, sector_num, bs_offset; +int64_t total_sectors;int64_t *bs_sectors =NULL ; -uint8_t * buf =NULL ;size_t bufsectors =IO_BUF_SIZE /BDRV_SECTOR_SIZE ; -const uint8_t *buf1;BlockDriverInfo bdi;QemuOpts *opts =NULL ;QemuOptsList *create_opts =NULL ;
@@ -1330,6 +1617,7 @@ static int img_convert(int argc, char **argv)bool quiet = false;Error *local_err =NULL ;QemuOpts *sn_opts =NULL ; +ImgConvertState state; fmt =NULL ; out_fmt ="raw" ;
@@ -1622,9 +1910,6 @@ static int img_convert(int argc, char **argv) } out_bs =blk_bs (out_blk); - bs_i =0 ; - bs_offset =0 ; -/* increase bufsectors from the default 4096 (2M) if opt_transfer_length * or discard_alignment of the out_bs is greater. Limit to 32768 (16MB) * as maximum. */
@@ -1633,8 +1918,6 @@ static int img_convert(int argc, char **argv) out_bs->bl.discard_alignment)) ); - buf =qemu_blockalign (out_bs, bufsectors *BDRV_SECTOR_SIZE ); -if (skip_create) {int64_t output_sectors =bdrv_nb_sectors (out_bs);if (output_sectors <0 ) {
@@ -1661,203 +1944,20 @@ static int img_convert(int argc, char **argv) cluster_sectors = bdi.cluster_size /BDRV_SECTOR_SIZE ; } -if (compress) { -if (cluster_sectors <=0 || cluster_sectors > bufsectors) { -error_report ("invalid cluster size" ); - ret =-1 ; - goto out; - } - sector_num =0 ; - - nb_sectors = total_sectors; - -for (;;) { -int64_t bs_num; -int remainder; -uint8_t *buf2; - - nb_sectors = total_sectors - sector_num; -if (nb_sectors <=0 ) -break ; -if (nb_sectors >= cluster_sectors) - n = cluster_sectors; -else - n = nb_sectors; - - bs_num = sector_num - bs_offset; -assert (bs_num >=0 ); - remainder = n; - buf2 = buf; -while (remainder >0 ) { -int nlow; -while (bs_num == bs_sectors[bs_i]) { - bs_offset += bs_sectors[bs_i]; - bs_i++; -assert (bs_i < bs_n); - bs_num =0 ; -/* printf("changing part: sector_num=%" PRId64 ", " -"bs_i=%d, bs_offset=%" PRId64 ", bs_sectors=%" PRId64 -"\n", sector_num, bs_i, bs_offset, bs_sectors[bs_i]); */ - } -assert (bs_num < bs_sectors[bs_i]); - - nlow = remainder > bs_sectors[bs_i] - bs_num - ? bs_sectors[bs_i] - bs_num : remainder; - - ret =bdrv_read (bs[bs_i], bs_num, buf2, nlow); -if (ret <0 ) { -error_report ("error while reading sector %" PRId64": %s" , - bs_num,strerror (-ret)); - goto out; - } - - buf2 += nlow *512 ; - bs_num += nlow; - - remainder -= nlow; - } -assert (remainder ==0 ); - -if (!buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) { - ret =bdrv_write_compressed (out_bs, sector_num, buf, n); -if (ret !=0 ) { -error_report ("error while compressing sector %" PRId64 -": %s" , sector_num,strerror (-ret)); - goto out; - } - } - sector_num += n; -qemu_progress_print (100.0 * sector_num / total_sectors,0 ); - } -/* signal EOF to align */ -bdrv_write_compressed (out_bs,0 ,NULL ,0 ); - }else { -int64_t sectors_to_read, sectors_read, sector_num_next_status; -bool count_allocated_sectors; -int has_zero_init = min_sparse ?bdrv_has_zero_init (out_bs) :0 ; - -if (!has_zero_init &&bdrv_can_write_zeroes_with_unmap (out_bs)) { - ret =bdrv_make_zero (out_bs,BDRV_REQ_MAY_UNMAP ); -if (ret <0 ) { - goto out; - } - has_zero_init =1 ; - } - - sectors_to_read = total_sectors; - count_allocated_sectors = progress && (out_baseimg || has_zero_init); -restart: - sector_num =0 ;// total number of sectors converted so far - sectors_read =0 ; - sector_num_next_status =0 ; - -for (;;) { - nb_sectors = total_sectors - sector_num; -if (nb_sectors <=0 ) { -if (count_allocated_sectors) { - sectors_to_read = sectors_read; - count_allocated_sectors = false; - goto restart; - } - ret =0 ; -break ; - } - -while (sector_num - bs_offset >= bs_sectors[bs_i]) { - bs_offset += bs_sectors[bs_i]; - bs_i ++; -assert (bs_i < bs_n); -/* printf("changing part: sector_num=%" PRId64 ", bs_i=%d, " -"bs_offset=%" PRId64 ", bs_sectors=%" PRId64 "\n", -sector_num, bs_i, bs_offset, bs_sectors[bs_i]); */ - } - -if ((out_baseimg || has_zero_init) && - sector_num >= sector_num_next_status) { - n = nb_sectors >INT_MAX ?INT_MAX : nb_sectors; - ret =bdrv_get_block_status (bs[bs_i], sector_num - bs_offset, - n, &n1); -if (ret <0 ) { -error_report ("error while reading block status of sector %" - PRId64": %s" , sector_num - bs_offset, -strerror (-ret)); - goto out; - } -/* If the output image is zero initialized, we are not working -* on a shared base and the input is zero we can skip the next -* n1 sectors */ -if (has_zero_init && !out_baseimg && (ret &BDRV_BLOCK_ZERO )) { - sector_num += n1; -continue ; - } -/* If the output image is being created as a copy on write -* image, assume that sectors which are unallocated in the -* input image are present in both the output's and input's -* base images (no need to copy them). */ -if (out_baseimg) { -if (!(ret &BDRV_BLOCK_DATA )) { - sector_num += n1; -continue ; - } -/* The next 'n1' sectors are allocated in the input image. -* Copy only those as they may be followed by unallocated -* sectors. */ - nb_sectors = n1; - } -/* avoid redundant callouts to get_block_status */ - sector_num_next_status = sector_num + n1; - } - - n =MIN (nb_sectors, bufsectors); - -/* round down request length to an aligned sector, but -* do not bother doing this on short requests. They happen -* when we found an all-zero area, and the next sector to -* write will not be sector_num + n. */ -if (cluster_sectors >0 && n >= cluster_sectors) { -int64_t next_aligned_sector = (sector_num + n); - next_aligned_sector -= next_aligned_sector % cluster_sectors; -if (sector_num + n > next_aligned_sector) { - n = next_aligned_sector - sector_num; - } - } - - n =MIN (n, bs_sectors[bs_i] - (sector_num - bs_offset)); - - sectors_read += n; -if (count_allocated_sectors) { - sector_num += n; -continue ; - } + state = (ImgConvertState ) { + .src = bs, + .src_sectors = bs_sectors, + .src_num = bs_n, + .total_sectors = total_sectors, + .target = out_bs, + .compressed = compress, + .out_backing = (bool ) out_baseimg, + .min_sparse = min_sparse, + .cluster_sectors = cluster_sectors, + .buf_sectors = bufsectors, + }; + ret =convert_do_copy (&state); - n1 = n; - ret =bdrv_read (bs[bs_i], sector_num - bs_offset, buf, n); -if (ret <0 ) { -error_report ("error while reading sector %" PRId64": %s" , - sector_num - bs_offset,strerror (-ret)); - goto out; - } -/* NOTE: at the same time we convert, we do not write zero -sectors to have a chance to compress the image. Ideally, we -should add a specific call to have the info to go faster */ - buf1 = buf; -while (n >0 ) { -if (!has_zero_init || -is_allocated_sectors_min (buf1, n, &n1, min_sparse)) { - ret =bdrv_write (out_bs, sector_num, buf1, n1); -if (ret <0 ) { -error_report ("error while writing sector %" PRId64 -": %s" , sector_num,strerror (-ret)); - goto out; - } - } - sector_num += n1; - n -= n1; - buf1 += n1 *512 ; - } -qemu_progress_print (100.0 * sectors_read / sectors_to_read,0 ); - } - } out:if (!ret) {qemu_progress_print (100 ,0 );
@@ -1865,7 +1965,6 @@ out:qemu_progress_end ();qemu_opts_del (opts);qemu_opts_free (create_opts); -qemu_vfree (buf);qemu_opts_del (sn_opts);blk_unref (out_blk);g_free (bs);
Signature · 1 line
1.8.3.1
Quoted 59 lines · Kevin Wolf
The implementation of qemu-img convert is (a) messy, (b) buggy, and
(c) less efficient than possible. The changes required to beat some sense into it are massive enough that incremental changes would only make my and the reviewers' life harder. So throw it away and reimplement it from scratch.Let me give some examples what I mean by messy, buggy and inefficient:
(a) The copying logic of qemu-img convert has two separate branches for compressed and normal target images, which roughly do the same - except for a little code that handles actual differences between compressed and uncompressed images, and much more code that implements just a different set of optimisations and bugs. This is unnecessary code duplication, and makes the code for compressed output (unsurprisingly) suffer from bitrot.
The code for uncompressed ouput is run twice to count the the total length for the progress bar. In the first run it just takes a shortcut and runs only half the loop, and when it's done, it toggles a boolean, jumps out of the loop with a backwards goto and starts over. Works, but pretty is something different.
(b) Converting while keeping a backing file (-B option) is broken in several ways. This includes not writing to the image file if the input has zero clusters or data filled with zeros (ignoring that the backing file will be visible instead).
It also doesn't correctly limit every iteration of the copy loop to sectors of the same status so that too many sectors may be copied to in the target image. For -B this gives an unexpected result, for other images it just does more work than necessary.
Conversion with a compressed target completely ignores any target backing file.
(c) qemu-img convert skips reading and writing an area if it knows from metadata that copying isn't needed (except for the bug mentioned above that ignores a status change in some cases). It does, however, read from the source even if it knows that it will read zeros, and then search for non-zero bytes in the read buffer, if it's possible that a write might be needed.
This reimplementation of the copying core reorganises the code to remove the duplication and have a much more obvious code flow, by essentially splitting the copy iteration loop into three parts:
1. Find the number of contiguous sectors of the same status at the current offset (This can also be called in a separate loop for the copying loop in order to determine the total sectors for the progress bar.)
2. Read sectors. If the status implies that there is no data there to read (zero or unallocated cluster), don't do anything.
3. Write sectors depending on the status. If it's data, write it. If we want the backing file to be visible (with -B), don't write it. If it's zeroed, skip it if you can, otherwise use
bdrv_write_zeroes()to optimise the write at least where possible.
Quoted 4 lines
Signed-off-by Kevin Wolf <[email protected]>--- qemu-img.c | 511 ++++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 305 insertions(+), 206 deletions(-)
You have been very careful not to write "Rewrite img_convert()" or something like that in the subject, so I can't really complain that there are still bugs in the function (which are not related to the copying logic), for instance:
$ ./qemu-img create -f qcow2 i1.qcow2 64M
Formatting 'i1.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off
$ ./qemu-img create -f qcow2 i2.qcow2 64M
Formatting 'i2.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off
$ ./qemu-img snapshot -c foo i1.qcow2
$ ./qemu-img snapshot -c foo i2.qcow2
$ ./qemu-io -c 'write 0 64M' i2.qcow2
wrote 67108864/67108864 bytes at offset 0
64 MiB, 1 ops; 0:00:01.32 (48.152 MiB/sec and 0.7524 ops/sec)
$ ./qemu-img convert -l snapshot.name=foo -O qcow2 i{1,2}.qcow2 o.qcow2
qemu-img: error while reading sector 131072: Input/output error("No support for concatenating multiple snapshot" should be enforced for sn_opts != NULL)
Quoted 17 lines
qemu-img.c +9
-0@@ -1306,20 +1306,307 @@ out3:return ret; } +enum ImgConvertBlockStatus { +BLK_DATA , +BLK_ZERO , +BLK_BACKING_FILE , +}; + +typedef struct ImgConvertState { + BlockDriverState **src; + int64_t *src_sectors;
qemu-img.c out3: · line 1327
Quoted 10 lines
qemu-img.c +10
-0+int src_cur, src_num; +int64_t src_cur_offset; +int64_t total_sectors; +int64_t allocated_sectors; +enum ImgConvertBlockStatus status; +int64_t sector_next_status; +BlockDriverState *target; +bool has_zero_init; +bool compressed; +bool out_backing;
Maybe "out_backed" (or "out_is_backed" or "out_has_backing") would be better; "out_backing" to me sounds like this should describe the backing file.
Quoted 13 lines
qemu-img.c +13
-0+int min_sparse; +size_t cluster_sectors; +size_t buf_sectors; +} ImgConvertState; + +static void convert_select_part (ImgConvertState *s,int64_t sector_num) +{ +while (sector_num - s->src_cur_offset >= s->src_sectors[s->src_cur]) { + s->src_cur_offset += s->src_sectors[s->src_cur]; + s->src_cur++; +assert (s->src_cur < s->src_num); + } +}
qemu-img.c out3: · line 1350
Quoted 10 lines
qemu-img.c +10
-0+ +static int convert_iteration_sectors (ImgConvertState *s,int64_t sector_num) +{ +int64_t ret; +int n; + +convert_select_part (s, sector_num); + +assert (s->total_sectors > sector_num); + n =MIN (s->total_sectors - sector_num,INT_MAX );
Maybe it would be better to use INT_MAX / BDRV_SECTOR_SIZE here (and in other places, too)... In practice, this would only be relevant to reads and writes, though, and those are limited by s->buf_sectors.
Quoted 61 lines
qemu-img.c +61
-0+ +if (s->sector_next_status <= sector_num) { + ret =bdrv_get_block_status (s->src[s->src_cur], + sector_num - s->src_cur_offset, + n, &n); +if (ret <0 ) { +return ret; + } + +if (ret &BDRV_BLOCK_ZERO ) { + s->status =BLK_ZERO ; + }else if (ret &BDRV_BLOCK_DATA ) { + s->status =BLK_DATA ; + }else if (!s->out_backing) { +/* Without a target backing file we must copy over the contents of +* the backing file as well. */ +/* TODO Check block status of the backing file chain to avoid +* needlessly reading zeroes and limiting the iteration to the +* buffer size */ + s->status =BLK_DATA ; + }else { + s->status =BLK_BACKING_FILE ; + } + + s->sector_next_status = sector_num + n; + } + + n =MIN (n, s->sector_next_status - sector_num); +if (s->status ==BLK_DATA ) { + n =MIN (n, s->buf_sectors); + } + +/* We need to write complete clusters for compressed images, so if an +* unallocated area is shorter than that, we must consider the whole +* cluster allocated. */ +if (s->compressed) { +if (n < s->cluster_sectors) { + n =MIN (s->cluster_sectors, s->total_sectors - sector_num); + s->status =BLK_DATA ; + }else { + n =QEMU_ALIGN_DOWN (n, s->cluster_sectors); + } + } + +return n; +} + +static int convert_read (ImgConvertState *s,int64_t sector_num,int nb_sectors, +uint8_t *buf) +{ +int n; +int ret; + +if (s->status ==BLK_ZERO || s->status ==BLK_BACKING_FILE ) { +return 0 ; + } + +assert (nb_sectors <= s->buf_sectors); +while (nb_sectors >0 ) { +BlockDriverState *bs; +int64_t bs_sectors;
qemu-img.c out3: · line 1421
Quoted 10 lines
qemu-img.c +10
-0+ +/* In the case of compression with multiple source files, we can get a +* nb_sectors that spreads into the next part. So we must be able to +* read across multiple BDSes for one convert_read() call. */ +convert_select_part (s, sector_num); + bs = s->src[s->src_cur]; + bs_sectors = s->src_sectors[s->src_cur]; + + n =MIN (nb_sectors, bs_sectors - (sector_num - s->src_cur_offset)); + ret =bdrv_read (bs, sector_num, buf, n);
Shouldn't this be sector_num - s->src_cur_offset?
Quoted 9 lines
qemu-img.c +9
-0+if (ret <0 ) { +return ret; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } +
qemu-img.c out3: · line 1440
Quoted 10 lines
qemu-img.c +10
-0+return 0 ; +} + +static int convert_write (ImgConvertState *s,int64_t sector_num,int nb_sectors, +const uint8_t *buf) +{ +int ret; + +while (nb_sectors >0 ) { +
This empty line looks a bit strange to me...
Quoted 4 lines
qemu-img.c +4
-0+int n = nb_sectors; + +switch (s->status) { +case BLK_BACKING_FILE :
qemu-img.c out3: · line 1454
Quoted 10 lines
qemu-img.c +10
-0+/* If we have a backing file, leave clusters unallocated that are +* unallocated in the source image, so that the backing file is +* visible at the respective offset. */ +assert (s->out_backing); +break ; + +case BLK_DATA : +/* We must always write compressed clusters as a whole, so don't +* try to find zeroed parts in the buffer. We can only save the +* write if the buffer is completely zeroed. */
But you shouldn't be doing that if n < s->min_sparse, I think (if people specify -S 0, they don't want you to zero anything, and that's guaranteed by the man page). Considering that is_allocated_sectors_min() basically has a min = MIN(min, n) part, too, I'd be fine with making -S 0 a special case here ("if (s->has_zero_init && s->min_sparse && buffer_is_zero(...))").
Actually, now that I'm looking at is_allocated_sectors_min(), if the first sectors is not allocated, it will return false and thus both the current qemu-img convert and the new one after this patch won't write data. That's a bug, I think (because it is guaranteed by the man page).
Quoted 8 lines
qemu-img.c +8
-0+if (s->compressed) { +if (s->has_zero_init && +buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) + { +assert (!s->out_backing); +break ; + } +
qemu-img.c out3: · line 1472
Quoted 10 lines
qemu-img.c +10
-0+ ret =bdrv_write_compressed (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } + +/* If there is real non-zero data, we must write it. Otherwise we +* can treat it as zero sectors. */ +if (is_allocated_sectors_min (buf, n, &n, s->min_sparse)) {
So I think this should be "if (!s->min_sparse || ...)".
qemu-img.c out3: · line 1482
Quoted 10 lines
qemu-img.c +10
-0+ ret =bdrv_write (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } +/* fall-through */ + +case BLK_ZERO : +if (s->has_zero_init) {
And I don't really know what to do about this. Should we write zeros if !s->min_sparse? Or is this a special case and the user can't expect qemu-img convert to really write zeros if the source image contained unallocated/zero clusters? That would make sense, but once again, the man page clearly states that "If sparse_size is 0, […] the destination image will always be fully allocated." Maybe we could argue that "fully" in this case means "as fully as the source image was allocated".
qemu-img.c out3: · line 1485
Quoted 3 lines
qemu-img.c +3
-0+break ; + } +/* TODO Should this use BDRV_REQ_MAY_UNMAP? */
Can't we remove this comment? If BDRV_REQ_MAP_UNMAP results in zeros read, bdrv_make_zero() below will have been invoked and thus s->has_zero_init = true (unless you take my suggestion and leave s->has_zero_init false if bdrv_make_zero() failed, but I wouldn't really mind that case here).
Quoted 23 lines
qemu-img.c +23
-0+ ret =bdrv_write_zeroes (s->target, sector_num, n,0 ); +if (ret <0 ) { +return ret; + } +break ; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_do_copy (ImgConvertState *s) +{ +uint8_t *buf =NULL ; +int64_t sector_num, allocated_done; +int ret; +int n; + +/* Check whether we have zero initialisation or can get it efficiently */
qemu-img.c out3: · line 1518
Quoted 10 lines
qemu-img.c +10
-0+ s->has_zero_init = s->min_sparse && !s->out_backing + ?bdrv_has_zero_init (s->target) + : false; + +if (!s->has_zero_init && !s->out_backing && +bdrv_can_write_zeroes_with_unmap (s->target)) + { + ret =bdrv_make_zero (s->target,BDRV_REQ_MAY_UNMAP ); +if (ret <0 ) { + goto fail;
I think just leaving s->has_zero_init false should be fine, too (instead of failing).
qemu-img.c out3: · line 1520
Quoted 2 lines
qemu-img.c +2
-0+ } + s->has_zero_init =1 ;
s/1/true/
Quoted 59 lines
qemu-img.c +59
-0+ } + +/* Allocate buffer for copied data. For compressed images, only one cluster +* can be copied at a time. */ +if (s->compressed) { +if (s->cluster_sectors <=0 || s->cluster_sectors > s->buf_sectors) { +error_report ("invalid cluster size" ); + ret = -EINVAL ; + goto fail; + } + s->buf_sectors = s->cluster_sectors; + } + buf =qemu_blockalign (s->target, s->buf_sectors *BDRV_SECTOR_SIZE ); + +/* Calculate allocated sectors for progress */ + s->allocated_sectors =0 ; + sector_num =0 ; +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + s->allocated_sectors += n; + } + sector_num += n; + } + +/* Do the copy */ + s->src_cur =0 ; + s->src_cur_offset =0 ; + s->sector_next_status =0 ; + + sector_num =0 ; + allocated_done =0 ; + +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + allocated_done += n; +qemu_progress_print (100.0 * allocated_done / s->allocated_sectors, +0 ); + } + + ret =convert_read (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while reading sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + ret =convert_write (s, sector_num, n, buf); +if (ret <0 ) { + error_report("error while writing sector %" PRId64
qemu-img.c out3: · line 1589
Quoted 10 lines
qemu-img.c +10
-0+ ": %s", sector_num, strerror(-ret)); + goto fail; + } + + sector_num += n; + } + +if (s->compressed) { +/* signal EOF to align */ +bdrv_write_compressed (s->target,0 ,NULL ,0 );
Is there a reason for ignoring the return value other than "That's how img_convert() used to do it"?
All in all, this patch looks pretty good to me and certainly makes the function much more clear. But as the bdrv_read() call probably needs to be fixed, and because I think the -S 0 behavior is buggy, I cannot give an R-b at this point (but I will if those are fixed, or if you tell me why they don't need to be).
Max
Quoted 279 lines
qemu-img.c +24
-206+ } + + ret =0 ; +fail: +qemu_vfree (buf); +return ret; +} +static int img_convert (int argc,char **argv) { -int c, n, n1, bs_n, bs_i, compress, cluster_sectors, skip_create; +int c, bs_n, bs_i, compress, cluster_sectors, skip_create;int64_t ret =0 ;int progress =0 , flags, src_flags;const char *fmt, *out_fmt, *cache, *src_cache, *out_baseimg, *out_filename;BlockDriver *drv, *proto_drv;BlockBackend **blk =NULL , *out_blk =NULL ;BlockDriverState **bs =NULL , *out_bs =NULL ; -int64_t total_sectors, nb_sectors, sector_num, bs_offset; +int64_t total_sectors;int64_t *bs_sectors =NULL ; -uint8_t * buf =NULL ;size_t bufsectors =IO_BUF_SIZE /BDRV_SECTOR_SIZE ; -const uint8_t *buf1;BlockDriverInfo bdi;QemuOpts *opts =NULL ;QemuOptsList *create_opts =NULL ;@@ -1330,6 +1617,7 @@ static int img_convert(int argc, char **argv)bool quiet = false;Error *local_err =NULL ;QemuOpts *sn_opts =NULL ; +ImgConvertState state; fmt =NULL ; out_fmt ="raw" ;@@ -1622,9 +1910,6 @@ static int img_convert(int argc, char **argv) } out_bs =blk_bs (out_blk); - bs_i =0 ; - bs_offset =0 ; -/* increase bufsectors from the default 4096 (2M) if opt_transfer_length * or discard_alignment of the out_bs is greater. Limit to 32768 (16MB) * as maximum. */ @@ -1633,8 +1918,6 @@ static int img_convert(int argc, char **argv) out_bs->bl.discard_alignment)) ); - buf =qemu_blockalign (out_bs, bufsectors *BDRV_SECTOR_SIZE ); -if (skip_create) {int64_t output_sectors =bdrv_nb_sectors (out_bs);if (output_sectors <0 ) {@@ -1661,203 +1944,20 @@ static int img_convert(int argc, char **argv) cluster_sectors = bdi.cluster_size /BDRV_SECTOR_SIZE ; } -if (compress) { -if (cluster_sectors <=0 || cluster_sectors > bufsectors) { -error_report ("invalid cluster size" ); - ret =-1 ; - goto out; - } - sector_num =0 ; - - nb_sectors = total_sectors; - -for (;;) { -int64_t bs_num; -int remainder; -uint8_t *buf2; - - nb_sectors = total_sectors - sector_num; -if (nb_sectors <=0 ) -break ; -if (nb_sectors >= cluster_sectors) - n = cluster_sectors; -else - n = nb_sectors; - - bs_num = sector_num - bs_offset; -assert (bs_num >=0 ); - remainder = n; - buf2 = buf; -while (remainder >0 ) { -int nlow; -while (bs_num == bs_sectors[bs_i]) { - bs_offset += bs_sectors[bs_i]; - bs_i++; -assert (bs_i < bs_n); - bs_num =0 ; -/* printf("changing part: sector_num=%" PRId64 ", " -"bs_i=%d, bs_offset=%" PRId64 ", bs_sectors=%" PRId64 -"\n", sector_num, bs_i, bs_offset, bs_sectors[bs_i]); */ - } -assert (bs_num < bs_sectors[bs_i]); - - nlow = remainder > bs_sectors[bs_i] - bs_num - ? bs_sectors[bs_i] - bs_num : remainder; - - ret =bdrv_read (bs[bs_i], bs_num, buf2, nlow); -if (ret <0 ) { -error_report ("error while reading sector %" PRId64": %s" , - bs_num,strerror (-ret)); - goto out; - } - - buf2 += nlow *512 ; - bs_num += nlow; - - remainder -= nlow; - } -assert (remainder ==0 ); - -if (!buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) { - ret =bdrv_write_compressed (out_bs, sector_num, buf, n); -if (ret !=0 ) { -error_report ("error while compressing sector %" PRId64 -": %s" , sector_num,strerror (-ret)); - goto out; - } - } - sector_num += n; -qemu_progress_print (100.0 * sector_num / total_sectors,0 ); - } -/* signal EOF to align */ -bdrv_write_compressed (out_bs,0 ,NULL ,0 ); - }else { -int64_t sectors_to_read, sectors_read, sector_num_next_status; -bool count_allocated_sectors; -int has_zero_init = min_sparse ?bdrv_has_zero_init (out_bs) :0 ; - -if (!has_zero_init &&bdrv_can_write_zeroes_with_unmap (out_bs)) { - ret =bdrv_make_zero (out_bs,BDRV_REQ_MAY_UNMAP ); -if (ret <0 ) { - goto out; - } - has_zero_init =1 ; - } - - sectors_to_read = total_sectors; - count_allocated_sectors = progress && (out_baseimg || has_zero_init); -restart: - sector_num =0 ;// total number of sectors converted so far - sectors_read =0 ; - sector_num_next_status =0 ; - -for (;;) { - nb_sectors = total_sectors - sector_num; -if (nb_sectors <=0 ) { -if (count_allocated_sectors) { - sectors_to_read = sectors_read; - count_allocated_sectors = false; - goto restart; - } - ret =0 ; -break ; - } - -while (sector_num - bs_offset >= bs_sectors[bs_i]) { - bs_offset += bs_sectors[bs_i]; - bs_i ++; -assert (bs_i < bs_n); -/* printf("changing part: sector_num=%" PRId64 ", bs_i=%d, " -"bs_offset=%" PRId64 ", bs_sectors=%" PRId64 "\n", -sector_num, bs_i, bs_offset, bs_sectors[bs_i]); */ - } - -if ((out_baseimg || has_zero_init) && - sector_num >= sector_num_next_status) { - n = nb_sectors >INT_MAX ?INT_MAX : nb_sectors; - ret =bdrv_get_block_status (bs[bs_i], sector_num - bs_offset, - n, &n1); -if (ret <0 ) { -error_report ("error while reading block status of sector %" - PRId64": %s" , sector_num - bs_offset, -strerror (-ret)); - goto out; - } -/* If the output image is zero initialized, we are not working -* on a shared base and the input is zero we can skip the next -* n1 sectors */ -if (has_zero_init && !out_baseimg && (ret &BDRV_BLOCK_ZERO )) { - sector_num += n1; -continue ; - } -/* If the output image is being created as a copy on write -* image, assume that sectors which are unallocated in the -* input image are present in both the output's and input's -* base images (no need to copy them). */ -if (out_baseimg) { -if (!(ret &BDRV_BLOCK_DATA )) { - sector_num += n1; -continue ; - } -/* The next 'n1' sectors are allocated in the input image. -* Copy only those as they may be followed by unallocated -* sectors. */ - nb_sectors = n1; - } -/* avoid redundant callouts to get_block_status */ - sector_num_next_status = sector_num + n1; - } - - n =MIN (nb_sectors, bufsectors); - -/* round down request length to an aligned sector, but -* do not bother doing this on short requests. They happen -* when we found an all-zero area, and the next sector to -* write will not be sector_num + n. */ -if (cluster_sectors >0 && n >= cluster_sectors) { -int64_t next_aligned_sector = (sector_num + n); - next_aligned_sector -= next_aligned_sector % cluster_sectors; -if (sector_num + n > next_aligned_sector) { - n = next_aligned_sector - sector_num; - } - } - - n =MIN (n, bs_sectors[bs_i] - (sector_num - bs_offset)); - - sectors_read += n; -if (count_allocated_sectors) { - sector_num += n; -continue ; - } + state = (ImgConvertState ) { + .src = bs, + .src_sectors = bs_sectors, + .src_num = bs_n, + .total_sectors = total_sectors, + .target = out_bs, + .compressed = compress, + .out_backing = (bool ) out_baseimg, + .min_sparse = min_sparse, + .cluster_sectors = cluster_sectors, + .buf_sectors = bufsectors, + }; + ret =convert_do_copy (&state); - n1 = n; - ret =bdrv_read (bs[bs_i], sector_num - bs_offset, buf, n); -if (ret <0 ) { -error_report ("error while reading sector %" PRId64": %s" , - sector_num - bs_offset,strerror (-ret)); - goto out; - } -/* NOTE: at the same time we convert, we do not write zero -sectors to have a chance to compress the image. Ideally, we -should add a specific call to have the info to go faster */ - buf1 = buf; -while (n >0 ) { -if (!has_zero_init || -is_allocated_sectors_min (buf1, n, &n1, min_sparse)) { - ret =bdrv_write (out_bs, sector_num, buf1, n1); -if (ret <0 ) { -error_report ("error while writing sector %" PRId64 -": %s" , sector_num,strerror (-ret)); - goto out; - } - } - sector_num += n1; - n -= n1; - buf1 += n1 *512 ; - } -qemu_progress_print (100.0 * sectors_read / sectors_to_read,0 ); - } - } out:if (!ret) {qemu_progress_print (100 ,0 );@@ -1865,7 +1965,6 @@ out:qemu_progress_end ();qemu_opts_del (opts);qemu_opts_free (create_opts); -qemu_vfree (buf);qemu_opts_del (sn_opts);blk_unref (out_blk);g_free (bs);
Am 18.02.2015 um 18:18 hat Max Reitz geschrieben:
Quoted 84 lines · Kevin Wolf
Quoted 59 lines
The implementation of qemu-img convert is (a) messy, (b) buggy, and
(c) less efficient than possible. The changes required to beat some sense into it are massive enough that incremental changes would only make my and the reviewers' life harder. So throw it away and reimplement it from scratch.Let me give some examples what I mean by messy, buggy and inefficient:
(a) The copying logic of qemu-img convert has two separate branches for compressed and normal target images, which roughly do the same - except for a little code that handles actual differences between compressed and uncompressed images, and much more code that implements just a different set of optimisations and bugs. This is unnecessary code duplication, and makes the code for compressed output (unsurprisingly) suffer from bitrot.
The code for uncompressed ouput is run twice to count the the total length for the progress bar. In the first run it just takes a shortcut and runs only half the loop, and when it's done, it toggles a boolean, jumps out of the loop with a backwards goto and starts over. Works, but pretty is something different.
(b) Converting while keeping a backing file (-B option) is broken in several ways. This includes not writing to the image file if the input has zero clusters or data filled with zeros (ignoring that the backing file will be visible instead).
It also doesn't correctly limit every iteration of the copy loop to sectors of the same status so that too many sectors may be copied to in the target image. For -B this gives an unexpected result, for other images it just does more work than necessary.
Conversion with a compressed target completely ignores any target backing file.
(c) qemu-img convert skips reading and writing an area if it knows from metadata that copying isn't needed (except for the bug mentioned above that ignores a status change in some cases). It does, however, read from the source even if it knows that it will read zeros, and then search for non-zero bytes in the read buffer, if it's possible that a write might be needed.
This reimplementation of the copying core reorganises the code to remove the duplication and have a much more obvious code flow, by essentially splitting the copy iteration loop into three parts:
1. Find the number of contiguous sectors of the same status at the current offset (This can also be called in a separate loop for the copying loop in order to determine the total sectors for the progress bar.)
2. Read sectors. If the status implies that there is no data there to read (zero or unallocated cluster), don't do anything.
3. Write sectors depending on the status. If it's data, write it. If we want the backing file to be visible (with -B), don't write it. If it's zeroed, skip it if you can, otherwise use
bdrv_write_zeroes()to optimise the write at least where possible.Quoted 4 lines
Signed-off-by Kevin Wolf <[email protected]>--- qemu-img.c | 511 ++++++++++++++++++++++++++++++++++++------------------------- 1 file changed, 305 insertions(+), 206 deletions(-)You have been very careful not to write "Rewrite
img_convert()" or something like that in the subject, so I can't really complain that there are still bugs in the function (which are not related to the copying logic), for instance:$ ./qemu-img create -f qcow2 i1.qcow2 64M Formatting 'i1.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off $ ./qemu-img create -f qcow2 i2.qcow2 64M Formatting 'i2.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off $ ./qemu-img snapshot -c foo i1.qcow2 $ ./qemu-img snapshot -c foo i2.qcow2 $ ./qemu-io -c 'write 0 64M' i2.qcow2 wrote 67108864/67108864 bytes at offset 0 64 MiB, 1 ops; 0:00:01.32 (48.152 MiB/sec and 0.7524 ops/sec) $ ./qemu-img convert -l snapshot.name=foo -O qcow2 i{1,2}.qcow2 o.qcow2 qemu-img: error while reading sector 131072: Input/output error
Quoted 2 lines
("No support for concatenating multiple snapshot" should be enforced for
sn_opts!= NULL)
Probably worth addressing, though not in this patch.
Quoted 22 lines
Quoted 22 lines
qemu-img.c +15
-0@@ -1306,20 +1306,307 @@ out3:return ret; } +enum ImgConvertBlockStatus { +BLK_DATA , +BLK_ZERO , +BLK_BACKING_FILE , +}; + +typedef struct ImgConvertState { + BlockDriverState **src; + int64_t *src_sectors; + int src_cur, src_num; + int64_t src_cur_offset; + int64_t total_sectors; + int64_t allocated_sectors; + enum ImgConvertBlockStatus status; + int64_t sector_next_status;
Quoted 8 lines
qemu-img.c out3: · line 1327
Quoted 4 lines
qemu-img.c +4
-0+BlockDriverState *target; +bool has_zero_init; +bool compressed; +bool out_backing;Maybe "
out_backed" (or "out_is_backed" or "out_has_backing") would be better; "out_backing" to me sounds like this should describe the backing file.
Okay, will rename as target_has_backing.
Quoted 19 lines
Quoted 19 lines
qemu-img.c +19
-0+int min_sparse; +size_t cluster_sectors; +size_t buf_sectors; +} ImgConvertState; + +static void convert_select_part (ImgConvertState *s,int64_t sector_num) +{ +while (sector_num - s->src_cur_offset >= s->src_sectors[s->src_cur]) { + s->src_cur_offset += s->src_sectors[s->src_cur]; + s->src_cur++; +assert (s->src_cur < s->src_num); + } +} + +static int convert_iteration_sectors (ImgConvertState *s,int64_t sector_num) +{ +int64_t ret; +int n; +
Quoted 8 lines
qemu-img.c out3: · line 1350
Quoted 4 lines
qemu-img.c +4
-0+convert_select_part (s, sector_num); + +assert (s->total_sectors > sector_num); + n =MIN (s->total_sectors - sector_num,INT_MAX );Maybe it would be better to use
INT_MAX/BDRV_SECTOR_SIZEhere (and in other places, too)... In practice, this would only be relevant to reads and writes, though, and those are limited bys->buf_sectors.
I'll make that BDRV_REQUEST_MAX_SECTORS.
Quoted 65 lines
Quoted 65 lines
qemu-img.c +65
-0+ +if (s->sector_next_status <= sector_num) { + ret =bdrv_get_block_status (s->src[s->src_cur], + sector_num - s->src_cur_offset, + n, &n); +if (ret <0 ) { +return ret; + } + +if (ret &BDRV_BLOCK_ZERO ) { + s->status =BLK_ZERO ; + }else if (ret &BDRV_BLOCK_DATA ) { + s->status =BLK_DATA ; + }else if (!s->out_backing) { +/* Without a target backing file we must copy over the contents of +* the backing file as well. */ +/* TODO Check block status of the backing file chain to avoid +* needlessly reading zeroes and limiting the iteration to the +* buffer size */ + s->status =BLK_DATA ; + }else { + s->status =BLK_BACKING_FILE ; + } + + s->sector_next_status = sector_num + n; + } + + n =MIN (n, s->sector_next_status - sector_num); +if (s->status ==BLK_DATA ) { + n =MIN (n, s->buf_sectors); + } + +/* We need to write complete clusters for compressed images, so if an +* unallocated area is shorter than that, we must consider the whole +* cluster allocated. */ +if (s->compressed) { +if (n < s->cluster_sectors) { + n =MIN (s->cluster_sectors, s->total_sectors - sector_num); + s->status =BLK_DATA ; + }else { + n =QEMU_ALIGN_DOWN (n, s->cluster_sectors); + } + } + +return n; +} + +static int convert_read (ImgConvertState *s,int64_t sector_num,int nb_sectors, +uint8_t *buf) +{ +int n; +int ret; + +if (s->status ==BLK_ZERO || s->status ==BLK_BACKING_FILE ) { +return 0 ; + } + +assert (nb_sectors <= s->buf_sectors); +while (nb_sectors >0 ) { +BlockDriverState *bs; +int64_t bs_sectors; + +/* In the case of compression with multiple source files, we can get a +* nb_sectors that spreads into the next part. So we must be able to +* read across multiple BDSes for one convert_read() call. */
Quoted 8 lines
qemu-img.c out3: · line 1421
Quoted 6 lines
qemu-img.c +6
-0+convert_select_part (s, sector_num); + bs = s->src[s->src_cur]; + bs_sectors = s->src_sectors[s->src_cur]; + + n =MIN (nb_sectors, bs_sectors - (sector_num - s->src_cur_offset)); + ret =bdrv_read (bs, sector_num, buf, n);Shouldn't this be
sector_num-s->src_cur_offset?
I suppose it should be, thanks for catching this. Not a good sign with respect to our qemu-iotests coverage.
Quoted 13 lines
Quoted 13 lines
qemu-img.c +13
-0+if (ret <0 ) { +return ret; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_write(ImgConvertState *s, int64_t sector_num, int nb_sectors,
Quoted 8 lines
qemu-img.c out3: · line 1440
Quoted 6 lines
qemu-img.c +6
-0+ const uint8_t *buf) +{ +int ret; + +while (nb_sectors >0 ) { +This empty line looks a bit strange to me...
Removed.
Quoted 13 lines
Quoted 13 lines
qemu-img.c +13
-0+int n = nb_sectors; + +switch (s->status) { +case BLK_BACKING_FILE : +/* If we have a backing file, leave clusters unallocated that are +* unallocated in the source image, so that the backing file is +* visible at the respective offset. */ +assert (s->out_backing); +break ; + +case BLK_DATA : +/* We must always write compressed clusters as a whole, so don't +* try to find zeroed parts in the buffer. We can only save the
Quoted 8 lines
qemu-img.c out3: · line 1454
Quoted 1 line
qemu-img.c +1
-0+* write if the buffer is completely zeroed. */ But you shouldn't be doing that if n <
s->min_sparse, I think (if people specify -S 0, they don't want you to zero anything, and that's guaranteed by the man page). Considering thatis_allocated_sectors_min()basically has a min =MIN(min, n)part, too, I'd be fine with making -S 0 a special case here ("if (s->has_zero_init&&s->min_sparse&&buffer_is_zero(...))").
Hm, okay. With a compressed target n isn't variable, so there's only always or never, depending on min_sparse < cluster_size. But I can do it.
I'm actually not really sure what the use case for -S with a compressed target would be.
Quoted 5 lines
Actually, now that I'm looking at
is_allocated_sectors_min(), if the first sectors is not allocated, it will return false and thus both the current qemu-img convert and the new one after this patch won't write data. That's a bug, I think (because it is guaranteed by the man page).
Or the description in the man page is wrong.
The intention with -S was that we avoid splitting up writes into too many small chunks because that costs a lot of time. If you look at it from that angle, it's doing exactly the right thing because skipping zeros at the start doesn't increase the number of write requests.
Quoted 7 lines
qemu-img.c out3: · line 1461
Quoted 7 lines
qemu-img.c +7
-0+if (s->compressed) { +if (s->has_zero_init && +buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) + { +assert (!s->out_backing); +break ; + }
I wonder whether !s->has_zero_init shouldn't actually treat it as BLK_ZERO. That should "compress" even better than writing a gzip compressed cluster of zeros.
Quoted 11 lines
qemu-img.c out3: · line 1472
Quoted 10 lines
qemu-img.c +10
-0+ ret =bdrv_write_compressed (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } + +/* If there is real non-zero data, we must write it. Otherwise we +* can treat it as zero sectors. */ +if (is_allocated_sectors_min (buf, n, &n, s->min_sparse)) {
Quoted 1 line
So I think this should be "if (!s->min_sparse || ...)".
I'd rather change is_allocated_sectors_min(). But only if there is a use case, otherwise we should fix the description in the man page.
Quoted 11 lines
qemu-img.c out3: · line 1482
Quoted 10 lines
qemu-img.c +10
-0+ ret =bdrv_write (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } +/* fall-through */ + +case BLK_ZERO : +if (s->has_zero_init) {
Quoted 8 lines
And I don't really know what to do about this. Should we write zeros if !s->min_sparse? Or is this a special case and the user can't expect qemu-img convert to really write zeros if the source image contained unallocated/zero clusters? That would make sense, but once again, the man page clearly states that "If
sparse_sizeis 0, […] the destination image will always be fully allocated." Maybe we could argue that "fully" in this case means "as fully as the source image was allocated".
FWIW, bdrv_is_allocated() considers zero clusters allocated. ;-)
I don't think that -S 0 should mean full preallocation. If you want that, I think you can get it with -o (at the cost of writing some parts of the image twice).
Quoted 9 lines
qemu-img.c out3: · line 1485
Quoted 3 lines
qemu-img.c +3
-0+break ; + } +/* TODO Should this use BDRV_REQ_MAY_UNMAP? */ Can't we remove this comment? If
BDRV_REQ_MAP_UNMAPresults in zeros read,bdrv_make_zero()below will have been invoked and thuss->has_zero_init= true (unless you take my suggestion and leaves->has_zero_initfalse ifbdrv_make_zero()failed, but I wouldn't really mind that case here).
Good point.
(I wanted to remove the comment either way, it was just there to get input during the review.)
Quoted 37 lines
Quoted 23 lines
qemu-img.c +23
-0+ ret =bdrv_write_zeroes (s->target, sector_num, n,0 ); +if (ret <0 ) { +return ret; + } +break ; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_do_copy (ImgConvertState *s) +{ +uint8_t *buf =NULL ; +int64_t sector_num, allocated_done; +int ret; +int n; + +/* Check whether we have zero initialisation or can get it efficiently */ qemu-img.c out3: · line 1518
Quoted 10 lines
qemu-img.c +10
-0+ s->has_zero_init = s->min_sparse && !s->out_backing + ?bdrv_has_zero_init (s->target) + : false; + +if (!s->has_zero_init && !s->out_backing && +bdrv_can_write_zeroes_with_unmap (s->target)) + { + ret =bdrv_make_zero (s->target,BDRV_REQ_MAY_UNMAP ); +if (ret <0 ) { + goto fail;I think just leaving
s->has_zero_initfalse should be fine, too (instead of failing).
Quoted 4 lines
qemu-img.c out3: · line 1518
Quoted 2 lines
qemu-img.c +2
-0+ } + s->has_zero_init =1 ;s/1/true/
Yes.
Quoted 64 lines
Quoted 64 lines
qemu-img.c +64
-0+ } + +/* Allocate buffer for copied data. For compressed images, only one cluster +* can be copied at a time. */ +if (s->compressed) { +if (s->cluster_sectors <=0 || s->cluster_sectors > s->buf_sectors) { +error_report ("invalid cluster size" ); + ret = -EINVAL ; + goto fail; + } + s->buf_sectors = s->cluster_sectors; + } + buf =qemu_blockalign (s->target, s->buf_sectors *BDRV_SECTOR_SIZE ); + +/* Calculate allocated sectors for progress */ + s->allocated_sectors =0 ; + sector_num =0 ; +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + s->allocated_sectors += n; + } + sector_num += n; + } + +/* Do the copy */ + s->src_cur =0 ; + s->src_cur_offset =0 ; + s->sector_next_status =0 ; + + sector_num =0 ; + allocated_done =0 ; + +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + allocated_done += n; +qemu_progress_print (100.0 * allocated_done / s->allocated_sectors, +0 ); + } + + ret =convert_read (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while reading sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + ret =convert_write (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while writing sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + sector_num += n;
Quoted 8 lines
qemu-img.c out3: · line 1589
Quoted 5 lines
qemu-img.c +5
-0+ } + +if (s->compressed) { +/* signal EOF to align */ +bdrv_write_compressed (s->target,0 ,NULL ,0 );Is there a reason for ignoring the return value other than "That's how
img_convert()used to do it"?
No. Isn't that one good enough? ;-)
So the code in qcow2 says this:
if (nb_sectors == 0) {
/* align end of file to a sector boundary to ease reading with
sector based I/Os */
cluster_offset = bdrv_getlength(bs->file);
return bdrv_truncate(bs->file, cluster_offset);
}I don't think we have any such restrictions any more, so it's mostly useless. Perhaps ancient qemu versions would fail to read such an image, but recent ones shouldn't.
In fact, our bdrv_pwrite() currently maps to sector-aligned functions in the protocol driver, so I think at the moment we already get the alignment automatically. This might change again once we convert block drivers to byte offsets.
Kevin
Quoted 1 line · Kevin Wolf
Am 18.02.2015 um 18:18 hat Max Reitz geschrieben:
[snip]
Quoted 11 lines
Quoted 5 lines
Actually, now that I'm looking at
is_allocated_sectors_min(), if the first sectors is not allocated, it will return false and thus both the current qemu-img convert and the new one after this patch won't write data. That's a bug, I think (because it is guaranteed by the man page).Or the description in the man page is wrong.
The intention with -S was that we avoid splitting up writes into too many small chunks because that costs a lot of time. If you look at it from that angle, it's doing exactly the right thing because skipping zeros at the start doesn't increase the number of write requests.
Feel free to fix it. But I remember someone recently asking about preallocation for qemu-img convert in the #qemu IRC channel, and "-S 0" was one of the valid answers.
The nice thing about -S 0 is that it works with all image formats; I know that qcow2 is always our main concern (especially when it comes to the target of qemu-img convert), but I think using -S 0 for preallocation is justified.
Considering that in this case the man page was lying (because the code did not allocate everything), I'd be fine with fixing up the man page and leaving the behavior as-is, though, too.
[snip]
Quoted 7 lines
Quoted 6 lines
qemu-img.c out3: · line 1589
Quoted 4 lines
qemu-img.c +4
-0+ +if (s->compressed) { +/* signal EOF to align */ +bdrv_write_compressed (s->target,0 ,NULL ,0 );Is there a reason for ignoring the return value other than "That's how
img_convert()used to do it"?No. Isn't that one good enough? ;-)
I don't know, your commit message states that the old implementation is buggy, so I don't think that's enough. :-)
Quoted 9 lines
So the code in qcow2 says this:
if (nb_sectors == 0) { /* align end of file to a sector boundary to ease reading with sector based I/Os */ cluster_offset = bdrv_getlength(bs->file); return bdrv_truncate(bs->file, cluster_offset); }
Quoted 8 lines
I don't think we have any such restrictions any more, so it's mostly useless. Perhaps ancient qemu versions would fail to read such an image, but recent ones shouldn't.
In fact, our
bdrv_pwrite()currently maps to sector-aligned functions in the protocol driver, so I think at the moment we already get the alignment automatically. This might change again once we convert block drivers to byte offsets.
So you're intending to drop the bdrv_write_compressed() completely? I'd be fine with that as well.
Max
PATCH 2/2 qemu-iotests: Some qemu-img convert tests
In reply to Kevin WolfThis adds a regression test for some problems that the qemu-img convert rewrite just fixed.
--- tests/qemu-iotests/122 | 83 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 30 +++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 114 insertions(+) create mode 100755 tests/qemu-iotests/122 create mode 100644 tests/qemu-iotests/122.out
tests/qemu-iotests/122 new file +83 -0
@@ -0,0 +1,83 @@ +#!/bin/bash +# +# Test some qemu-img convert cases +# +# Copyright (C) 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +here="$PWD" +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto generic +_supported_os Linux + + +TEST_IMG=$TEST_IMG.base _make_test_img 64M +$QEMU_IO -c "write -P 0x11 0 64M" $TEST_IMG.base 2>&1 | _filter_qemu_io | _filter_testdir + + +echo +echo "=== Check allocation status regression with -B ===" +echo + +_make_test_img -b $TEST_IMG.base +$QEMU_IO -c "write -P 0x22 0 3M" $TEST_IMG 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IMG map $TEST_IMG.orig | _filter_qemu_img_map + + +echo +echo "=== Check that zero clusters are kept in overlay ===" +echo + +_make_test_img -b $TEST_IMG.base + +$QEMU_IO -c "write -P 0 0 3M" $TEST_IMG 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -c -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir + +$QEMU_IO -c "write -z 0 3M" $TEST_IMG 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -c -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir + +# TODO -B + -S 0 -> full allocation with no use of the backing file + +# success, all done +echo '*** done' +rm -f $seq.full +status=0
tests/qemu-iotests/122.out new file +30 -0
@@ -0,0 +1,30 @@ +QA output created by 122 +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +wrote 67108864/67108864 bytes at offset 0 +64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Check allocation status regression with -B === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file='TEST_DIR/t.IMGFMT.base' +wrote 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Offset Length File +0 0x300000 TEST_DIR/t.IMGFMT.orig +0x300000 0x3d00000 TEST_DIR/t.IMGFMT.base + +=== Check that zero clusters are kept in overlay === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file='TEST_DIR/t.IMGFMT.base' +wrote 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** done
tests/qemu-iotests/group +1 -0
@@ -117,3 +117,4 @@ 113 rw auto quick 114 rw auto quick 116 rw auto quick +122 rw auto quick
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH 2/2] qemu-iotests: Some qemu-img convert tests
Quoted 86 lines · Kevin Wolf
This adds a regression test for some problems that the qemu-img convert rewrite just fixed.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 83 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 30 +++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 114 insertions(+)create mode 100755 tests/qemu-iotests/122
create mode 100644 tests/qemu-iotests/122.outtests/qemu-iotests/122 new file +68
-0@@ -0,0 +1,83 @@ +#!/bin/bash +# +# Test some qemu-img convert cases +# +# Copyright (C) 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +here="$PWD" +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto generic +_supported_os Linux + + +TEST_IMG=$TEST_IMG.base _make_test_img 64M +$QEMU_IO -c "write -P 0x11 0 64M" $TEST_IMG.base 2>&1 | _filter_qemu_io | _filter_testdir + + +echo +echo "=== Check allocation status regression with -B ===" +echo + +_make_test_img -b $TEST_IMG.base +$QEMU_IO -c "write -P 0x22 0 3M" $TEST_IMG 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IMG map $TEST_IMG.orig | _filter_qemu_img_map + + +echo +echo "=== Check that zero clusters are kept in overlay ===" +echo + +_make_test_img -b $TEST_IMG.base + +$QEMU_IO -c "write -P 0 0 3M" $TEST_IMG 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir
tests/qemu-iotests/122 line 78
Quoted 10 lines
tests/qemu-iotests/122 new file +10
-0+$QEMU_IMG convert -O $IMGFMT -c -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir + +$QEMU_IO -c "write -z 0 3M" $TEST_IMG 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -c -B $TEST_IMG.base $TEST_IMG $TEST_IMG.orig +$QEMU_IO -c "read -P 0 0 3M" $TEST_IMG.orig 2>&1 | _filter_qemu_io | _filter_testdir + +# TODO -B + -S 0 -> full allocation with no use of the backing file
Also, -S 0 with -B (that is, with has_zero_init) and concatenation, please.
But even without those:
Quoted 50 lines
tests/qemu-iotests/122 new file +5
-0+ +# success, all done +echo '*** done' +rm -f $seq.full +status=0tests/qemu-iotests/122.out new file +30
-0@@ -0,0 +1,30 @@ +QA output created by 122 +Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864 +wrote 67108864/67108864 bytes at offset 0 +64 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Check allocation status regression with -B === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file='TEST_DIR/t.IMGFMT.base' +wrote 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Offset Length File +0 0x300000 TEST_DIR/t.IMGFMT.orig +0x300000 0x3d00000 TEST_DIR/t.IMGFMT.base + +=== Check that zero clusters are kept in overlay === + +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file='TEST_DIR/t.IMGFMT.base' +wrote 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +read 3145728/3145728 bytes at offset 0 +3 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +*** donetests/qemu-iotests/group +1
-0@@ -117,3 +117,4 @@ 113 rw auto quick 114 rw auto quick 116 rw auto quick +122 rw auto quick
Re: [Qemu-devel] [PATCH 2/2] qemu-iotests: Some qemu-img convert tests
In reply to Kevin WolfQuoted 11 lines · Kevin Wolf
This adds a regression test for some problems that the qemu-img convert rewrite just fixed.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 83 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 30 +++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 114 insertions(+)create mode 100755 tests/qemu-iotests/122
create mode 100644 tests/qemu-iotests/122.out
[snip]
tests/qemu-iotests/group line 120
Quoted 9 lines
tests/qemu-iotests/group +1
-0@@ -117,3 +117,4 @@ 113 rw auto quick 114 rw auto quick 116 rw auto quick +122 rw auto quick
This test takes 17 s on my HDD, and 7 s in tmpfs. Are you sure it should be added to the "quick" group?
Other than that, looks good.
Max
Re: [Qemu-devel] [PATCH 2/2] qemu-iotests: Some qemu-img convert tests
Quoted 28 lines · Max Reitz
Quoted 12 lines · Kevin Wolf
This adds a regression test for some problems that the qemu-img convert rewrite just fixed.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 83 ++++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 30 +++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 114 insertions(+)create mode 100755 tests/qemu-iotests/122
create mode 100644 tests/qemu-iotests/122.out[snip]
tests/qemu-iotests/group line 120
Quoted 9 lines
tests/qemu-iotests/group +1
-0@@ -117,3 +117,4 @@ 113 rw auto quick 114 rw auto quick 116 rw auto quick +122 rw auto quickThis test takes 17 s on my HDD, and 7 s in tmpfs. Are you sure it should be added to the "quick" group?
Quoted 3 lines
Other than that, looks good.
Max
Argh, disregard that, I had the wrong version open. Sorry.
Max