qemu-img convert: Rewrite copying logic
v2 cover qemu-img convert: Rewrite copying logic
See the commit message of patch 2 for the background.
v2:
- Rebased. Commit f1d3cd79 changed many bdrv_* calls to blk_* calls. This version tries to keep those conversions and use BlockBackend where qemu.git master does or where it makes sense. Added patch 1 to get blk_write_zeroes().
- Renamed s->out_backing as s->target_has_backing [Max]
- convert_iteration_sectors(): Limit n to BDRV_REQUEST_MAX_SECTORS instead of INT_MAX [Max]
- convert_read(): Fixed sector number for image concatenation [Max]
- Fixed -S 0 for full allocation [Max, Peter]
- Leave has_zero_init = false instead of failing the conversion if bdrv_make_zero() returns an error [Max]
- Set has_zero_init = true instead of 1 [Max]
- Don't ignore error returns in a final blk_write_compressed() [Max]
- Removed TODO comment after deciding that no change is needed
- Many added tests
Kevin Wolf (3):
block-backend: Expose bdrv_write_zeroes()
qemu-img convert: Rewrite copying logic
qemu-iotests: Some qemu-img convert tests
block/block-backend.c | 11 + include/sysemu/block-backend.h | 2 + qemu-img.c | 516 +++++++++++++++++++++++++---------------- tests/qemu-iotests/122 | 223 ++++++++++++++++++ tests/qemu-iotests/122.out | 209 +++++++++++++++++ tests/qemu-iotests/group | 1 + 6 files changed, 756 insertions(+), 206 deletions(-) create mode 100755 tests/qemu-iotests/122 create mode 100644 tests/qemu-iotests/122.out
Signature · 1 line
1.8.3.1
PATCH v2 1/3 block-backend: Expose bdrv_write_zeroes()
--- block/block-backend.c | 11 +++++++++++ include/sysemu/block-backend.h | 2 ++ 2 files changed, 13 insertions(+)
block/block-backend.c +11 -0
@@ -515,6 +515,17 @@ int blk_write(BlockBackend *blk, int64_t sector_num, const uint8_t *buf,return bdrv_write (blk->bs, sector_num, buf, nb_sectors); } +int blk_write_zeroes (BlockBackend *blk,int64_t sector_num, +int nb_sectors,BdrvRequestFlags flags) +{ +int ret =blk_check_request (blk, sector_num, nb_sectors); +if (ret <0 ) { +return ret; + } + +return bdrv_write_zeroes (blk->bs, sector_num, nb_sectors, flags); +} +static void error_callback_bh (void *opaque) {struct BlockBackendAIOCB *acb = opaque;
include/sysemu/block-backend.h +2 -0
@@ -87,6 +87,8 @@ int blk_read_unthrottled(BlockBackend *blk, int64_t sector_num, uint8_t *buf, int nb_sectors);int blk_write (BlockBackend *blk,int64_t sector_num,const uint8_t *buf,int nb_sectors); +int blk_write_zeroes (BlockBackend *blk,int64_t sector_num, +int nb_sectors,BdrvRequestFlags flags);BlockAIOCB *blk_aio_write_zeroes (BlockBackend *blk,int64_t sector_num,int nb_sectors,BdrvRequestFlags flags,BlockCompletionFunc *cb,void *opaque);
Signature · 1 line
1.8.3.1
PATCH v2 2/3 qemu-img convert: Rewrite copying logic
In reply to Kevin WolfThe implementation of qemu-img convert is (a) messy, (b) buggy, and
(c) less efficient than possible. The changes required to beat some sense into it are massive enough that incremental changes would only make my and the reviewers' life harder. So throw it away and reimplement it from scratch.
Let me give some examples what I mean by messy, buggy and inefficient:
(a) The copying logic of qemu-img convert has two separate branches for compressed and normal target images, which roughly do the same - except for a little code that handles actual differences between compressed and uncompressed images, and much more code that implements just a different set of optimisations and bugs. This is unnecessary code duplication, and makes the code for compressed output (unsurprisingly) suffer from bitrot.
The code for uncompressed ouput is run twice to count the the total length for the progress bar. In the first run it just takes a shortcut and runs only half the loop, and when it's done, it toggles a boolean, jumps out of the loop with a backwards goto and starts over. Works, but pretty is something different.
(b) Converting while keeping a backing file (-B option) is broken in several ways. This includes not writing to the image file if the input has zero clusters or data filled with zeros (ignoring that the backing file will be visible instead).
It also doesn't correctly limit every iteration of the copy loop to sectors of the same status so that too many sectors may be copied to in the target image. For -B this gives an unexpected result, for other images it just does more work than necessary.
Conversion with a compressed target completely ignores any target backing file.
(c) qemu-img convert skips reading and writing an area if it knows from metadata that copying isn't needed (except for the bug mentioned above that ignores a status change in some cases). It does, however, read from the source even if it knows that it will read zeros, and then search for non-zero bytes in the read buffer, if it's possible that a write might be needed.
This reimplementation of the copying core reorganises the code to remove the duplication and have a much more obvious code flow, by essentially splitting the copy iteration loop into three parts:
1. Find the number of contiguous sectors of the same status at the current offset (This can also be called in a separate loop before the copying loop in order to determine the total sectors for the progress bar.)
2. Read sectors. If the status implies that there is no data there to read (zero or unallocated cluster), don't do anything.
3. Write sectors depending on the status. If it's data, write it. If we want the backing file to be visible (with -B), don't write it. If it's zeroed, skip it if you can, otherwise use bdrv_write_zeroes() to optimise the write at least where possible.
--- qemu-img.c | 516 +++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 310 insertions(+), 206 deletions(-)
qemu-img.c +310 -206
@@ -1305,20 +1305,312 @@ out3:return ret; } +enum ImgConvertBlockStatus { +BLK_DATA , +BLK_ZERO , +BLK_BACKING_FILE , +}; + +typedef struct ImgConvertState { +BlockBackend **src; +int64_t *src_sectors; +int src_cur, src_num; +int64_t src_cur_offset; +int64_t total_sectors; +int64_t allocated_sectors; +enum ImgConvertBlockStatus status; +int64_t sector_next_status; +BlockBackend *target; +bool has_zero_init; +bool compressed; +bool target_has_backing; +int min_sparse; +size_t cluster_sectors; +size_t buf_sectors; +}ImgConvertState ; + +static void convert_select_part (ImgConvertState *s,int64_t sector_num) +{ +assert (sector_num >= s->src_cur_offset); +while (sector_num - s->src_cur_offset >= s->src_sectors[s->src_cur]) { + s->src_cur_offset += s->src_sectors[s->src_cur]; + s->src_cur++; +assert (s->src_cur < s->src_num); + } +} + +static int convert_iteration_sectors (ImgConvertState *s,int64_t sector_num) +{ +int64_t ret; +int n; + +convert_select_part (s, sector_num); + +assert (s->total_sectors > sector_num); + n =MIN (s->total_sectors - sector_num,BDRV_REQUEST_MAX_SECTORS ); + +if (s->sector_next_status <= sector_num) { + ret =bdrv_get_block_status (blk_bs (s->src[s->src_cur]), + sector_num - s->src_cur_offset, + n, &n); +if (ret <0 ) { +return ret; + } + +if (ret &BDRV_BLOCK_ZERO ) { + s->status =BLK_ZERO ; + }else if (ret &BDRV_BLOCK_DATA ) { + s->status =BLK_DATA ; + }else if (!s->target_has_backing) { +/* Without a target backing file we must copy over the contents of +* the backing file as well. */ +/* TODO Check block status of the backing file chain to avoid +* needlessly reading zeroes and limiting the iteration to the +* buffer size */ + s->status =BLK_DATA ; + }else { + s->status =BLK_BACKING_FILE ; + } + + s->sector_next_status = sector_num + n; + } + + n =MIN (n, s->sector_next_status - sector_num); +if (s->status ==BLK_DATA ) { + n =MIN (n, s->buf_sectors); + } + +/* We need to write complete clusters for compressed images, so if an +* unallocated area is shorter than that, we must consider the whole +* cluster allocated. */ +if (s->compressed) { +if (n < s->cluster_sectors) { + n =MIN (s->cluster_sectors, s->total_sectors - sector_num); + s->status =BLK_DATA ; + }else { + n =QEMU_ALIGN_DOWN (n, s->cluster_sectors); + } + } + +return n; +} + +static int convert_read (ImgConvertState *s,int64_t sector_num,int nb_sectors, +uint8_t *buf) +{ +int n; +int ret; + +if (s->status ==BLK_ZERO || s->status ==BLK_BACKING_FILE ) { +return 0 ; + } + +assert (nb_sectors <= s->buf_sectors); +while (nb_sectors >0 ) { +BlockBackend *blk; +int64_t bs_sectors; + +/* In the case of compression with multiple source files, we can get a +* nb_sectors that spreads into the next part. So we must be able to +* read across multiple BDSes for one convert_read() call. */ +convert_select_part (s, sector_num); + blk = s->src[s->src_cur]; + bs_sectors = s->src_sectors[s->src_cur]; + + n =MIN (nb_sectors, bs_sectors - (sector_num - s->src_cur_offset)); + ret =blk_read (blk, sector_num - s->src_cur_offset, buf, n); +if (ret <0 ) { +return ret; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_write (ImgConvertState *s,int64_t sector_num,int nb_sectors, +const uint8_t *buf) +{ +int ret; + +while (nb_sectors >0 ) { +int n = nb_sectors; + +switch (s->status) { +case BLK_BACKING_FILE : +/* If we have a backing file, leave clusters unallocated that are +* unallocated in the source image, so that the backing file is +* visible at the respective offset. */ +assert (s->target_has_backing); +break ; + +case BLK_DATA : +/* We must always write compressed clusters as a whole, so don't +* try to find zeroed parts in the buffer. We can only save the +* write if the buffer is completely zeroed and we're allowed to +* keep the target sparse. */ +if (s->compressed) { +if (s->has_zero_init && s->min_sparse && +buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) + { +assert (!s->target_has_backing); +break ; + } + + ret =blk_write_compressed (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } + +/* If there is real non-zero data or we're told to keep the target +* fully allocated (-S 0), we must write it. Otherwise we can treat +* it as zero sectors. */ +if (!s->min_sparse || +is_allocated_sectors_min (buf, n, &n, s->min_sparse)) + { + ret =blk_write (s->target, sector_num, buf, n); +if (ret <0 ) { +return ret; + } +break ; + } +/* fall-through */ + +case BLK_ZERO : +if (s->has_zero_init) { +break ; + } + ret =blk_write_zeroes (s->target, sector_num, n,0 ); +if (ret <0 ) { +return ret; + } +break ; + } + + sector_num += n; + nb_sectors -= n; + buf += n *BDRV_SECTOR_SIZE ; + } + +return 0 ; +} + +static int convert_do_copy (ImgConvertState *s) +{ +uint8_t *buf =NULL ; +int64_t sector_num, allocated_done; +int ret; +int n; + +/* Check whether we have zero initialisation or can get it efficiently */ + s->has_zero_init = s->min_sparse && !s->target_has_backing + ?bdrv_has_zero_init (blk_bs (s->target)) + : false; + +if (!s->has_zero_init && !s->target_has_backing && +bdrv_can_write_zeroes_with_unmap (blk_bs (s->target))) + { + ret =bdrv_make_zero (blk_bs (s->target),BDRV_REQ_MAY_UNMAP ); +if (ret ==0 ) { + s->has_zero_init = true; + } + } + +/* Allocate buffer for copied data. For compressed images, only one cluster +* can be copied at a time. */ +if (s->compressed) { +if (s->cluster_sectors <=0 || s->cluster_sectors > s->buf_sectors) { +error_report ("invalid cluster size" ); + ret = -EINVAL ; + goto fail; + } + s->buf_sectors = s->cluster_sectors; + } + buf =blk_blockalign (s->target, s->buf_sectors *BDRV_SECTOR_SIZE ); + +/* Calculate allocated sectors for progress */ + s->allocated_sectors =0 ; + sector_num =0 ; +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + s->allocated_sectors += n; + } + sector_num += n; + } + +/* Do the copy */ + s->src_cur =0 ; + s->src_cur_offset =0 ; + s->sector_next_status =0 ; + + sector_num =0 ; + allocated_done =0 ; + +while (sector_num < s->total_sectors) { + n =convert_iteration_sectors (s, sector_num); +if (n <0 ) { + ret = n; + goto fail; + } +if (s->status ==BLK_DATA ) { + allocated_done += n; +qemu_progress_print (100.0 * allocated_done / s->allocated_sectors, +0 ); + } + + ret =convert_read (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while reading sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + ret =convert_write (s, sector_num, n, buf); +if (ret <0 ) { +error_report ("error while writing sector %" PRId64 +": %s" , sector_num,strerror (-ret)); + goto fail; + } + + sector_num += n; + } + +if (s->compressed) { +/* signal EOF to align */ + ret =blk_write_compressed (s->target,0 ,NULL ,0 ); +if (ret <0 ) { + goto fail; + } + } + + ret =0 ; +fail: +qemu_vfree (buf); +return ret; +} +static int img_convert (int argc,char **argv) { -int c, n, n1, bs_n, bs_i, compress, cluster_sectors, skip_create; +int c, bs_n, bs_i, compress, cluster_sectors, skip_create;int64_t ret =0 ;int progress =0 , flags, src_flags;const char *fmt, *out_fmt, *cache, *src_cache, *out_baseimg, *out_filename;BlockDriver *drv, *proto_drv;BlockBackend **blk =NULL , *out_blk =NULL ;BlockDriverState **bs =NULL , *out_bs =NULL ; -int64_t total_sectors, nb_sectors, sector_num, bs_offset; +int64_t total_sectors;int64_t *bs_sectors =NULL ; -uint8_t * buf =NULL ;size_t bufsectors =IO_BUF_SIZE /BDRV_SECTOR_SIZE ; -const uint8_t *buf1;BlockDriverInfo bdi;QemuOpts *opts =NULL ;QemuOptsList *create_opts =NULL ;
@@ -1329,6 +1621,7 @@ static int img_convert(int argc, char **argv)bool quiet = false;Error *local_err =NULL ;QemuOpts *sn_opts =NULL ; +ImgConvertState state; fmt =NULL ; out_fmt ="raw" ;
@@ -1627,9 +1920,6 @@ static int img_convert(int argc, char **argv) } out_bs =blk_bs (out_blk); - bs_i =0 ; - bs_offset =0 ; -/* increase bufsectors from the default 4096 (2M) if opt_transfer_length * or discard_alignment of the out_bs is greater. Limit to 32768 (16MB) * as maximum. */
@@ -1638,8 +1928,6 @@ static int img_convert(int argc, char **argv) out_bs->bl.discard_alignment)) ); - buf =blk_blockalign (out_blk, bufsectors *BDRV_SECTOR_SIZE ); -if (skip_create) {int64_t output_sectors =blk_nb_sectors (out_blk);if (output_sectors <0 ) {
@@ -1666,203 +1954,20 @@ static int img_convert(int argc, char **argv) cluster_sectors = bdi.cluster_size /BDRV_SECTOR_SIZE ; } -if (compress) { -if (cluster_sectors <=0 || cluster_sectors > bufsectors) { -error_report ("invalid cluster size" ); - ret =-1 ; - goto out; - } - sector_num =0 ; - - nb_sectors = total_sectors; - -for (;;) { -int64_t bs_num; -int remainder; -uint8_t *buf2; - - nb_sectors = total_sectors - sector_num; -if (nb_sectors <=0 ) -break ; -if (nb_sectors >= cluster_sectors) - n = cluster_sectors; -else - n = nb_sectors; - - bs_num = sector_num - bs_offset; -assert (bs_num >=0 ); - remainder = n; - buf2 = buf; -while (remainder >0 ) { -int nlow; -while (bs_num == bs_sectors[bs_i]) { - bs_offset += bs_sectors[bs_i]; - bs_i++; -assert (bs_i < bs_n); - bs_num =0 ; -/* printf("changing part: sector_num=%" PRId64 ", " -"bs_i=%d, bs_offset=%" PRId64 ", bs_sectors=%" PRId64 -"\n", sector_num, bs_i, bs_offset, bs_sectors[bs_i]); */ - } -assert (bs_num < bs_sectors[bs_i]); - - nlow = remainder > bs_sectors[bs_i] - bs_num - ? bs_sectors[bs_i] - bs_num : remainder; - - ret =blk_read (blk[bs_i], bs_num, buf2, nlow); -if (ret <0 ) { -error_report ("error while reading sector %" PRId64": %s" , - bs_num,strerror (-ret)); - goto out; - } - - buf2 += nlow *512 ; - bs_num += nlow; - - remainder -= nlow; - } -assert (remainder ==0 ); - -if (!buffer_is_zero (buf, n *BDRV_SECTOR_SIZE )) { - ret =blk_write_compressed (out_blk, sector_num, buf, n); -if (ret !=0 ) { -error_report ("error while compressing sector %" PRId64 -": %s" , sector_num,strerror (-ret)); - goto out; - } - } - sector_num += n; -qemu_progress_print (100.0 * sector_num / total_sectors,0 ); - } -/* signal EOF to align */ -blk_write_compressed (out_blk,0 ,NULL ,0 ); - }else { -int64_t sectors_to_read, sectors_read, sector_num_next_status; -bool count_allocated_sectors; -int has_zero_init = min_sparse ?bdrv_has_zero_init (out_bs) :0 ; - -if (!has_zero_init &&bdrv_can_write_zeroes_with_unmap (out_bs)) { - ret =bdrv_make_zero (out_bs,BDRV_REQ_MAY_UNMAP ); -if (ret <0 ) { - goto out; - } - has_zero_init =1 ; - } - - sectors_to_read = total_sectors; - count_allocated_sectors = progress && (out_baseimg || has_zero_init); -restart: - sector_num =0 ;// total number of sectors converted so far - sectors_read =0 ; - sector_num_next_status =0 ; - -for (;;) { - nb_sectors = total_sectors - sector_num; -if (nb_sectors <=0 ) { -if (count_allocated_sectors) { - sectors_to_read = sectors_read; - count_allocated_sectors = false; - goto restart; - } - ret =0 ; -break ; - } - -while (sector_num - bs_offset >= bs_sectors[bs_i]) { - bs_offset += bs_sectors[bs_i]; - bs_i ++; -assert (bs_i < bs_n); -/* printf("changing part: sector_num=%" PRId64 ", bs_i=%d, " -"bs_offset=%" PRId64 ", bs_sectors=%" PRId64 "\n", -sector_num, bs_i, bs_offset, bs_sectors[bs_i]); */ - } - -if ((out_baseimg || has_zero_init) && - sector_num >= sector_num_next_status) { - n = nb_sectors >INT_MAX ?INT_MAX : nb_sectors; - ret =bdrv_get_block_status (bs[bs_i], sector_num - bs_offset, - n, &n1); -if (ret <0 ) { -error_report ("error while reading block status of sector %" - PRId64": %s" , sector_num - bs_offset, -strerror (-ret)); - goto out; - } -/* If the output image is zero initialized, we are not working -* on a shared base and the input is zero we can skip the next -* n1 sectors */ -if (has_zero_init && !out_baseimg && (ret &BDRV_BLOCK_ZERO )) { - sector_num += n1; -continue ; - } -/* If the output image is being created as a copy on write -* image, assume that sectors which are unallocated in the -* input image are present in both the output's and input's -* base images (no need to copy them). */ -if (out_baseimg) { -if (!(ret &BDRV_BLOCK_DATA )) { - sector_num += n1; -continue ; - } -/* The next 'n1' sectors are allocated in the input image. -* Copy only those as they may be followed by unallocated -* sectors. */ - nb_sectors = n1; - } -/* avoid redundant callouts to get_block_status */ - sector_num_next_status = sector_num + n1; - } - - n =MIN (nb_sectors, bufsectors); - -/* round down request length to an aligned sector, but -* do not bother doing this on short requests. They happen -* when we found an all-zero area, and the next sector to -* write will not be sector_num + n. */ -if (cluster_sectors >0 && n >= cluster_sectors) { -int64_t next_aligned_sector = (sector_num + n); - next_aligned_sector -= next_aligned_sector % cluster_sectors; -if (sector_num + n > next_aligned_sector) { - n = next_aligned_sector - sector_num; - } - } - - n =MIN (n, bs_sectors[bs_i] - (sector_num - bs_offset)); - - sectors_read += n; -if (count_allocated_sectors) { - sector_num += n; -continue ; - } + state = (ImgConvertState ) { + .src = blk, + .src_sectors = bs_sectors, + .src_num = bs_n, + .total_sectors = total_sectors, + .target = out_blk, + .compressed = compress, + .target_has_backing = (bool ) out_baseimg, + .min_sparse = min_sparse, + .cluster_sectors = cluster_sectors, + .buf_sectors = bufsectors, + }; + ret =convert_do_copy (&state); - n1 = n; - ret =blk_read (blk[bs_i], sector_num - bs_offset, buf, n); -if (ret <0 ) { -error_report ("error while reading sector %" PRId64": %s" , - sector_num - bs_offset,strerror (-ret)); - goto out; - } -/* NOTE: at the same time we convert, we do not write zero -sectors to have a chance to compress the image. Ideally, we -should add a specific call to have the info to go faster */ - buf1 = buf; -while (n >0 ) { -if (!has_zero_init || -is_allocated_sectors_min (buf1, n, &n1, min_sparse)) { - ret =blk_write (out_blk, sector_num, buf1, n1); -if (ret <0 ) { -error_report ("error while writing sector %" PRId64 -": %s" , sector_num,strerror (-ret)); - goto out; - } - } - sector_num += n1; - n -= n1; - buf1 += n1 *512 ; - } -qemu_progress_print (100.0 * sectors_read / sectors_to_read,0 ); - } - } out:if (!ret) {qemu_progress_print (100 ,0 );
@@ -1870,7 +1975,6 @@ out:qemu_progress_end ();qemu_opts_del (opts);qemu_opts_free (create_opts); -qemu_vfree (buf);qemu_opts_del (sn_opts);blk_unref (out_blk);g_free (bs);
Signature · 1 line
1.8.3.1
PATCH v2 3/3 qemu-iotests: Some qemu-img convert tests
In reply to Kevin WolfThis adds a regression test for some problems that the qemu-img convert rewrite just fixed.
--- tests/qemu-iotests/122 | 223 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 209 ++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 433 insertions(+) create mode 100755 tests/qemu-iotests/122 create mode 100644 tests/qemu-iotests/122.out
tests/qemu-iotests/122 new file +223 -0
@@ -0,0 +1,223 @@ +#!/bin/bash +# +# Test some qemu-img convert cases +# +# Copyright (C) 2015 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq="$(basename $0)" +echo "QA output created by $seq" + +here="$PWD" +tmp=/tmp/$$ +status=1 # failure is the default! + +_cleanup() +{ + rm -f "$TEST_IMG".[123] + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + + +TEST_IMG="$TEST_IMG".base _make_test_img 64M +$QEMU_IO -c "write -P 0x11 0 64M" "$TEST_IMG".base 2>&1 | _filter_qemu_io | _filter_testdir + + +echo +echo "=== Check allocation status regression with -B ===" +echo + +_make_test_img -b "$TEST_IMG".base +$QEMU_IO -c "write -P 0x22 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IMG map "$TEST_IMG".orig | _filter_qemu_img_map + + +echo +echo "=== Check that zero clusters are kept in overlay ===" +echo + +_make_test_img -b "$TEST_IMG".base + +$QEMU_IO -c "write -P 0 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -c -B "$TEST_IMG".base "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir + +$QEMU_IO -c "write -z 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG convert -O $IMGFMT -c -B "$TEST_IMG".base "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir + + +echo +echo "=== Concatenate multiple source images ===" +echo + +TEST_IMG="$TEST_IMG".1 _make_test_img 4M +TEST_IMG="$TEST_IMG".2 _make_test_img 4M +TEST_IMG="$TEST_IMG".3 _make_test_img 4M + +$QEMU_IO -c "write -P 0x11 0 64k" "$TEST_IMG".1 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write -P 0x22 0 64k" "$TEST_IMG".2 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write -P 0x33 0 64k" "$TEST_IMG".3 2>&1 | _filter_qemu_io | _filter_testdir + +$QEMU_IMG convert -O $IMGFMT "$TEST_IMG".[123] "$TEST_IMG" +$QEMU_IMG map "$TEST_IMG" | _filter_qemu_img_map +$QEMU_IO -c "read -P 0x11 0 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x22 4M 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x33 8M 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir + +$QEMU_IMG convert -c -O $IMGFMT "$TEST_IMG".[123] "$TEST_IMG" +$QEMU_IMG map --output=json "$TEST_IMG" | _filter_qemu_img_map +$QEMU_IO -c "read -P 0x11 0 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x22 4M 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x33 8M 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir + +# -B can't be combined with concatenation +$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base "$TEST_IMG".[123] "$TEST_IMG" +$QEMU_IMG convert -O $IMGFMT -c -B "$TEST_IMG".base "$TEST_IMG".[123] "$TEST_IMG" + + +echo +echo "=== Compression with misaligned allocations and image sizes ===" +echo + +TEST_IMG="$TEST_IMG".1 _make_test_img 1023k -o cluster_size=1024 +TEST_IMG="$TEST_IMG".2 _make_test_img 1023k -o cluster_size=1024 + +$QEMU_IO -c "write -P 0x11 16k 16k" "$TEST_IMG".1 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write -P 0x22 130k 130k" "$TEST_IMG".1 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write -P 0x33 1022k 1k" "$TEST_IMG".1 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write -P 0x44 0k 1k" "$TEST_IMG".2 2>&1 | _filter_qemu_io | _filter_testdir + +$QEMU_IMG convert -c -O $IMGFMT "$TEST_IMG".[12] "$TEST_IMG" +$QEMU_IMG map --output=json "$TEST_IMG" | _filter_qemu_img_map +$QEMU_IO -c "read -P 0 0k 16k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x11 16k 16k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 32k 98k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x22 130k 130k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 260k 762k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x33 1022k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x44 1023k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 1024k 1022k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir + + +echo +echo "=== Full allocation with -S 0 ===" +echo + +# Standalone image +_make_test_img 64M +$QEMU_IO -c "write -P 0x22 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write -P 0 3M 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir + +echo +echo convert -S 0: +$QEMU_IMG convert -O $IMGFMT -S 0 "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0x22 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 3M 61M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + +echo +echo convert -c -S 0: +$QEMU_IMG convert -O $IMGFMT -c -S 0 "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0x22 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 3M 61M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + +# With backing file +TEST_IMG="$TEST_IMG".base _make_test_img 64M +$QEMU_IO -c "write -P 0x11 0 32M" "$TEST_IMG".base 2>&1 | _filter_qemu_io | _filter_testdir + +_make_test_img -b "$TEST_IMG".base 64M +$QEMU_IO -c "write -P 0x22 0 3M" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir + +echo +echo convert -S 0 with source backing file: +$QEMU_IMG convert -O $IMGFMT -S 0 "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0x22 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x11 3M 29M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 32M 32M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + +echo +echo convert -c -S 0 with source backing file: +$QEMU_IMG convert -O $IMGFMT -c -S 0 "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0x22 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x11 3M 29M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 32M 32M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + +# With keeping the backing file +echo +echo convert -S 0 -B ... +$QEMU_IMG convert -O $IMGFMT -S 0 "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0x22 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x11 3M 29M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 32M 32M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + +echo +echo convert -c -S 0 -B ... +$QEMU_IMG convert -O $IMGFMT -c -S 0 "$TEST_IMG" "$TEST_IMG".orig +$QEMU_IO -c "read -P 0x22 0 3M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0x11 3M 29M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "read -P 0 32M 32M" "$TEST_IMG".orig 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + + +echo +echo "=== Non-zero -S ===" +echo + +_make_test_img 64M -o cluster_size=1k +$QEMU_IO -c "write -P 0 0 64k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write 0 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write 8k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir +$QEMU_IO -c "write 17k 1k" "$TEST_IMG" 2>&1 | _filter_qemu_io | _filter_testdir + +for min_sparse in 4k 8k; do + echo + echo convert -S $min_sparse + $QEMU_IMG convert -O $IMGFMT -o cluster_size=1k -S $min_sparse "$TEST_IMG" "$TEST_IMG".orig + $QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map + + echo + echo convert -c -S $min_sparse + # For compressed images, -S values other than 0 are ignored + $QEMU_IMG convert -O $IMGFMT -o cluster_size=1k -c -S $min_sparse "$TEST_IMG" "$TEST_IMG".orig + $QEMU_IMG map --output=json "$TEST_IMG".orig | _filter_qemu_img_map +done + +# success, all done +echo '*** done' +rm -f $seq.full +status=0
tests/qemu-iotests/122.out new file +209 -0
209 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -122,5 +122,6 @@ 115 rw auto 116 rw auto quick 121 rw auto +122 rw auto quick 123 rw auto quick 128 rw auto quick
Signature · 1 line
1.8.3.1
Re: [Qemu-devel] [PATCH v2 1/3] block-backend: Expose bdrv_write_zeroes()
In reply to Kevin WolfQuoted 5 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- block/block-backend.c | 11 +++++++++++ include/sysemu/block-backend.h | 2 ++ 2 files changed, 13 insertions(+)
Quoted 59 lines · Kevin Wolf
The implementation of qemu-img convert is (a) messy, (b) buggy, and
(c) less efficient than possible. The changes required to beat some sense into it are massive enough that incremental changes would only make my and the reviewers' life harder. So throw it away and reimplement it from scratch.Let me give some examples what I mean by messy, buggy and inefficient:
(a) The copying logic of qemu-img convert has two separate branches for compressed and normal target images, which roughly do the same - except for a little code that handles actual differences between compressed and uncompressed images, and much more code that implements just a different set of optimisations and bugs. This is unnecessary code duplication, and makes the code for compressed output (unsurprisingly) suffer from bitrot.
The code for uncompressed ouput is run twice to count the the total length for the progress bar. In the first run it just takes a shortcut and runs only half the loop, and when it's done, it toggles a boolean, jumps out of the loop with a backwards goto and starts over. Works, but pretty is something different.
(b) Converting while keeping a backing file (-B option) is broken in several ways. This includes not writing to the image file if the input has zero clusters or data filled with zeros (ignoring that the backing file will be visible instead).
It also doesn't correctly limit every iteration of the copy loop to sectors of the same status so that too many sectors may be copied to in the target image. For -B this gives an unexpected result, for other images it just does more work than necessary.
Conversion with a compressed target completely ignores any target backing file.
(c) qemu-img convert skips reading and writing an area if it knows from metadata that copying isn't needed (except for the bug mentioned above that ignores a status change in some cases). It does, however, read from the source even if it knows that it will read zeros, and then search for non-zero bytes in the read buffer, if it's possible that a write might be needed.
This reimplementation of the copying core reorganises the code to remove the duplication and have a much more obvious code flow, by essentially splitting the copy iteration loop into three parts:
1. Find the number of contiguous sectors of the same status at the current offset (This can also be called in a separate loop before the copying loop in order to determine the total sectors for the progress bar.)
2. Read sectors. If the status implies that there is no data there to read (zero or unallocated cluster), don't do anything.
3. Write sectors depending on the status. If it's data, write it. If we want the backing file to be visible (with -B), don't write it. If it's zeroed, skip it if you can, otherwise use
bdrv_write_zeroes()to optimise the write at least where possible.
Quoted 4 lines
Signed-off-by Kevin Wolf <[email protected]>--- qemu-img.c | 516 +++++++++++++++++++++++++++++++++++++------------------------ 1 file changed, 310 insertions(+), 206 deletions(-)
Re: [Qemu-devel] [PATCH v2 3/3] qemu-iotests: Some qemu-img convert tests
In reply to Kevin WolfQuoted 11 lines · Kevin Wolf
This adds a regression test for some problems that the qemu-img convert rewrite just fixed.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 223 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 209 ++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 433 insertions(+)create mode 100755 tests/qemu-iotests/122
create mode 100644 tests/qemu-iotests/122.out
[snip]
tests/qemu-iotests/group line 127
Quoted 11 lines
tests/qemu-iotests/group +1
-0@@ -122,5 +122,6 @@ 115 rw auto 116 rw auto quick 121 rw auto +122 rw auto quick 123 rw auto quick 128 rw auto quick
As I mistakenly replied to v2 just now, this test takes 17 s on my HDD, and 7 s in tmpfs. So are you sure it should be added to the "quick" group?
The test itself looks good, though.
Max
Re: [Qemu-devel] [PATCH v2 3/3] qemu-iotests: Some qemu-img convert tests
Quoted 17 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
This adds a regression test for some problems that the qemu-img convert rewrite just fixed.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/122 | 223 +++++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/122.out | 209 ++++++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 433 insertions(+)create mode 100755 tests/qemu-iotests/122
create mode 100644 tests/qemu-iotests/122.out[snip]
Quoted 13 lines
tests/qemu-iotests/group line 127
Quoted 11 lines
tests/qemu-iotests/group +1
-0@@ -122,5 +122,6 @@ 115 rw auto 116 rw auto quick 121 rw auto +122 rw auto quick 123 rw auto quick 128 rw auto quickAs I mistakenly replied to v2 just now, this test takes 17 s on my HDD, and 7 s in tmpfs. So are you sure it should be added to the "quick" group?
*v1... I somehow don't seem too awake today.
Quoted 3 lines
The test itself looks good, though.
Max
Am 19.03.2015 um 13:33 hat Kevin Wolf geschrieben:
Quoted 19 lines
See the commit message of patch 2 for the background.
v2:
- Rebased. Commitf1d3cd79changed many bdrv_* calls to blk_* calls. This version tries to keep those conversions and use BlockBackend where qemu.git master does or where it makes sense. Added patch 1 to getblk_write_zeroes().
- Renameds->out_backingass->target_has_backing[Max]
-convert_iteration_sectors(): Limit n toBDRV_REQUEST_MAX_SECTORSinstead ofINT_MAX[Max]
-convert_read(): Fixed sector number for image concatenation [Max]
- Fixed -S 0 for full allocation [Max, Peter]
- Leavehas_zero_init= false instead of failing the conversion ifbdrv_make_zero()returns an error [Max]
- Sethas_zero_init= true instead of 1 [Max]
- Don't ignore error returns in a finalblk_write_compressed()[Max]
- Removed TODO comment after deciding that no change is needed
- Many added tests
Quoted 4 lines
Kevin Wolf (3):
block-backend: Exposebdrv_write_zeroes()
qemu-img convert: Rewrite copying logic
qemu-iotests: Some qemu-img convert tests
Removed the test case from the "quick" group as suggested by Max, and applied to block-next.
Kevin