block: Fix resize (extending) of short overlays
cover block: Fix resize (extending) of short overlays
Consider the following scenario where the overlay is shorter than its backing file:
base.qcow2: AAAAAAAA
overlay.qcow2: BBBB
When resizing (extending) overlay.qcow2, the new blocks should not stay unallocated and make the additional As from base.qcow2 visible like before this series, but zeros should be read.
A similar case happens with the various variants of a commit job when an intermediate file is short (- for unallocated):
base.qcow2: A-A-AAAA
mid.qcow2: BB-B
top.qcow2: C--C--C-After commit top.qcow2 to mid.qcow2, the following happens:
mid.qcow2: CB-C00C0 (correct result)
mid.qcow2: CB-C--C- (before this fix)
Without the fix, blocks that previously read as zeros on top.qcow2 suddenly turn into A.
Kevin Wolf (6):
block: bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter
block: truncate: Don't make backing file data visible
iotests: Add qemu_io_log()
iotests: Fix timeout in run_job()
iotests: Support job-complete in run_job()
iotests: Test committing to short backing file
block/io.c | 31 ++++++- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 11 ++- 5 files changed, 319 insertions(+), 5 deletions(-) create mode 100755 tests/qemu-iotests/274 create mode 100644 tests/qemu-iotests/274.out
Signature · 1 line
2.20.1
PATCH 3/6 iotests: Add qemu_io_log()
Add a function that runs qemu-io and logs the output with the appropriate filters applied.
--- tests/qemu-iotests/iotests.py | 5 +++++ 1 file changed, 5 insertions(+)
tests/qemu-iotests/iotests.py +5 -0
@@ -157,6 +157,11 @@ def qemu_io(*args): sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args))) return subp.communicate()[0] +def qemu_io_log(*args): + result = qemu_io(*args) + log(result, filters=[filter_testfiles, filter_qemu_io]) + return result + def qemu_io_silent(*args): '''Run qemu-io and return the exit code, suppressing stdout''' args = qemu_io_args + list(args)
Signature · 1 line
2.20.1
PATCH 4/6 iotests: Fix timeout in run_job()
In reply to Kevin Wolfrun_job() accepts a wait parameter for a timeout, but it doesn't actually use it. The only thing that is missing is passing it to events_wait(), so do that now.
--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/iotests.py +1 -1
@@ -604,7 +604,7 @@ class VM(qtest.QEMUQtestMachine): ] error = None while True: - ev = filter_qmp_event(self.events_wait(events)) + ev = filter_qmp_event(self.events_wait(events, timeout=wait)) if ev['event'] != 'JOB_STATUS_CHANGE': if use_log: log(ev)
Signature · 1 line
2.20.1
PATCH 1/6 block: bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter
In reply to Kevin Wolfbdrv_co_do_pwrite_zeroes() can already cope with maximum request sizes by calling the driver in a loop until everything is done. Make the small remaining change that is necessary to let it accept a 64 bit byte count.
--- block/io.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
block/io.c +3 -3
@@ -42,7 +42,7 @@static void bdrv_parent_cb_resize (BlockDriverState *bs); static int coroutine_fn bdrv_co_do_pwrite_zeroes(BlockDriverState *bs, -int64_t offset,int bytes,BdrvRequestFlags flags); +int64_t offset,int64_t bytes,BdrvRequestFlags flags);static void bdrv_parent_drained_begin (BlockDriverState *bs,BdrvChild *ignore,bool ignore_bds_parents)
@@ -1730,7 +1730,7 @@ int coroutine_fn bdrv_co_preadv_part(BdrvChild *child, } static int coroutine_fn bdrv_co_do_pwrite_zeroes(BlockDriverState *bs, -int64_t offset,int bytes,BdrvRequestFlags flags) +int64_t offset,int64_t bytes,BdrvRequestFlags flags) {BlockDriver *drv = bs->drv;QEMUIOVector qiov;
@@ -1760,7 +1760,7 @@ static int coroutine_fn bdrv_co_do_pwrite_zeroes(BlockDriverState *bs,assert (max_write_zeroes >= bs->bl.request_alignment);while (bytes >0 && !ret) { -int num = bytes; +int num =MIN (bytes,BDRV_REQUEST_MAX_BYTES );/* Align request. Block drivers can expect the "bulk" of the request * to be aligned, and that unaligned requests do not cross cluster
Signature · 1 line
2.20.1
PATCH 2/6 block: truncate: Don't make backing file data visible
In reply to Kevin WolfWhen extending the size of an image that has a backing file larger than its old size, make sure that the backing file data doesn't become visible in the guest, but the added area is properly zeroed out.
The old behaviour made a difference in 'block_resize' (where showing the backing file data from an old snapshot rather than zeros is questionable) as well as in commit block jobs (both from active and intermediate nodes) and HMP 'commit', where committing to a short backing file would incorrectly omit writing zeroes for unallocated blocks on the top layer after the EOF of the short backing file.
--- block/io.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
block/io.c +25 -0
@@ -3382,6 +3382,31 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } +/* +* If the image has a backing file that is large enough that it would +* provide data for the new area, we cannot leave it unallocated because +* then the backing file content would become visible. Instead, zero-fill +* the area where backing file and new area overlap. +*/ +if (new_bytes && bs->backing && prealloc ==PREALLOC_MODE_OFF ) { +int64_t backing_len; + + backing_len =bdrv_getlength (backing_bs (bs)); +if (backing_len <0 ) { + ret = backing_len; + goto out; + } + +if (backing_len > old_size) { + ret =bdrv_co_do_pwrite_zeroes (bs, old_size, +MIN (new_bytes, backing_len - old_size), +BDRV_REQ_ZERO_WRITE |BDRV_REQ_MAY_UNMAP ); +if (ret <0 ) { + goto out; + } + } + } + ret =refresh_total_sectors (bs, offset >>BDRV_SECTOR_BITS );if (ret <0 ) {error_setg_errno (errp, -ret,"Could not refresh total sector count" );
Signature · 1 line
2.20.1
PATCH 5/6 iotests: Support job-complete in run_job()
In reply to Kevin WolfAutomatically complete jobs that have a 'ready' state and need an explicit job-complete. Without this, run_job() would hang for such jobs.
--- tests/qemu-iotests/iotests.py | 2 ++ 1 file changed, 2 insertions(+)
tests/qemu-iotests/iotests.py +2 -0
@@ -617,6 +617,8 @@ class VM(qtest.QEMUQtestMachine): error = j['error'] if use_log: log('Job failed: %s' % (j['error'])) + elif status == 'ready': + self.qmp_log('job-complete', id=job) elif status == 'pending' and not auto_finalize: if pre_finalize: pre_finalize()
Signature · 1 line
2.20.1
PATCH 6/6 iotests: Test committing to short backing file
In reply to Kevin Wolf--- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 2 +- 4 files changed, 283 insertions(+), 1 deletion(-) create mode 100755 tests/qemu-iotests/274 create mode 100644 tests/qemu-iotests/274.out
tests/qemu-iotests/274 new file +131 -0
@@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Creator/Owner: Kevin Wolf <[email protected]> +# +# Some tests for short backing files and short overlays + +import iotests +import os + +iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.verify_platform(['linux']) + +size_short = 1 * 1024 * 1024 +size_long = 2 * 1024 * 1024 +size_diff = size_long - size_short + +def create_chain(): + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, + str(size_long)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, mid, + str(size_short)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', mid, top, + str(size_long)) + + iotests.qemu_io_log('-c', 'write -P 1 0 %d' % size_long, base) + +def create_vm(): + vm = iotests.VM() + vm.add_blockdev('file,filename=%s,node-name=base-file' % (base)) + vm.add_blockdev('%s,file=base-file,node-name=base' % (iotests.imgfmt)) + vm.add_blockdev('file,filename=%s,node-name=mid-file' % (mid)) + vm.add_blockdev('%s,file=mid-file,node-name=mid,backing=base' % (iotests.imgfmt)) + vm.add_drive(top, 'backing=mid,node-name=top') + return vm + +with iotests.FilePath('base') as base, \ + iotests.FilePath('mid') as mid, \ + iotests.FilePath('top') as top: + + iotests.log('== Commit tests ==') + + create_chain() + + iotests.log('=== Check visible data ===') + + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, top) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), top) + + iotests.log('=== Checking allocation status ===') + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + base) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + mid) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + top) + + iotests.log('=== Checking map ===') + + iotests.qemu_img_log('map', '--output=json', base) + iotests.qemu_img_log('map', '--output=human', base) + iotests.qemu_img_log('map', '--output=json', mid) + iotests.qemu_img_log('map', '--output=human', mid) + iotests.qemu_img_log('map', '--output=json', top) + iotests.qemu_img_log('map', '--output=human', top) + + iotests.log('=== Testing qemu-img commit (top -> mid) ===') + + iotests.qemu_img_log('commit', top) + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing HMP commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('human-monitor-command', command_line='commit drive0') + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing QMP active commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('block-commit', device='top', base_node='mid', + job_id='job0', auto_dismiss=False) + vm.run_job('job0', wait=5) + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + + iotests.log('== Resize tests ==') + + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, '6G') + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, top, '1G') + iotests.qemu_io_log('-c', 'write -P 1 3G 64k', base) + iotests.qemu_io_log('-c', 'write -P 2 5G 64k', base) + + # After this, 0 to 6 GB should be allocated/zeroed + # 6 GB to 8 BG should be unallocated + iotests.qemu_img_log('resize', '-f', iotests.imgfmt, top, '8G') + iotests.qemu_io_log('-c', 'read -P 0 3G 64k', top) + iotests.qemu_io_log('-c', 'read -P 0 5G 64k', top) + iotests.qemu_img_log('map', '--output=json', top)
tests/qemu-iotests/274.out new file +150 -0
150 lines not on this page · raw message
tests/qemu-iotests/group +1 -0
@@ -285,3 +285,4 @@ 270 rw backing quick 272 rw 273 backing quick +274 rw backing quick
tests/qemu-iotests/iotests.py +1 -1
@@ -144,7 +144,7 @@ def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): output = qemu_img_pipe(*args) if not filter_path: filter_path = filename - log(filter_img_info(output, filter_path)) + log(filter_img_info(output, filter_path), filters=[filter_testfiles]) def qemu_io(*args): '''Run qemu-io and return the stdout data'''
Signature · 1 line
2.20.1
Re: [PATCH 1/6] block: bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter
In reply to Kevin WolfQuoted 8 lines · Kevin Wolf
bdrv_co_do_pwrite_zeroes()can already cope with maximum request sizes by calling the driver in a loop until everything is done. Make the small remaining change that is necessary to let it accept a 64 bit byte count.Signed-off-by Kevin Wolf <[email protected]>--- block/io.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/6] block: truncate: Don't make backing file data visible
In reply to Kevin WolfQuoted 11 lines · Kevin Wolf
When extending the size of an image that has a backing file larger than its old size, make sure that the backing file data doesn't become visible in the guest, but the added area is properly zeroed out.
The old behaviour made a difference in '
block_resize' (where showing the backing file data from an old snapshot rather than zeros is questionable) as well as in commit block jobs (both from active and intermediate nodes) and HMP 'commit', where committing to a short backing file would incorrectly omit writing zeroes for unallocated blocks on the top layer after the EOF of the short backing file.
Quoted 4 lines
Signed-off-by Kevin Wolf <[email protected]>--- block/io.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 3/6] iotests: Add qemu_io_log()
In reply to Kevin WolfQuoted 7 lines · Kevin Wolf
Add a function that runs qemu-io and logs the output with the appropriate filters applied.
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/iotests.py | 5 +++++ 1 file changed, 5 insertions(+)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 1/6] block: bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter
In reply to Kevin WolfQuoted 5 lines · Kevin Wolf
bdrv_co_do_pwrite_zeroes()can already cope with maximum request sizes by calling the driver in a loop until everything is done. Make the small remaining change that is necessary to let it accept a 64 bit byte count.Signed-off-by Kevin Wolf <[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 4/6] iotests: Fix timeout in run_job()
In reply to Kevin WolfQuoted 8 lines · Kevin Wolf
run_job()accepts a wait parameter for a timeout, but it doesn't actually use it. The only thing that is missing is passing it toevents_wait(), so do that now.Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/iotests.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 15 lines
tests/qemu-iotests/iotests.py +1
-1@@ -604,7 +604,7 @@ class VM(qtest.QEMUQtestMachine): ] error = None while True: - ev = filter_qmp_event(self.events_wait(events)) + ev = filter_qmp_event(self.events_wait(events, timeout=wait)) if ev['event'] != 'JOB_STATUS_CHANGE': if use_log: log(ev)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 5/6] iotests: Support job-complete in run_job()
In reply to Kevin WolfQuoted 8 lines · Kevin Wolf
Automatically complete jobs that have a 'ready' state and need an explicit job-complete. Without this,
run_job()would hang for such jobs.Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/iotests.py | 2 ++ 1 file changed, 2 insertions(+)
Quoted 15 lines
tests/qemu-iotests/iotests.py +2
-0@@ -617,6 +617,8 @@ class VM(qtest.QEMUQtestMachine): error = j['error'] if use_log: log('Job failed: %s' % (j['error'])) + elif status == 'ready': + self.qmp_log('job-complete', id=job) elif status == 'pending' and not auto_finalize: if pre_finalize: pre_finalize()
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/6] block: truncate: Don't make backing file data visible
In reply to Kevin WolfQuoted 21 lines · Kevin Wolf
When extending the size of an image that has a backing file larger than its old size, make sure that the backing file data doesn't become visible in the guest, but the added area is properly zeroed out.
The old behaviour made a difference in '
block_resize' (where showing the backing file data from an old snapshot rather than zeros is questionable) as well as in commit block jobs (both from active and intermediate nodes) and HMP 'commit', where committing to a short backing file would incorrectly omit writing zeroes for unallocated blocks on the top layer after the EOF of the short backing file.Signed-off-by Kevin Wolf <[email protected]>--- block/io.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)block/io.c +0
-0
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3390
Quoted 9 lines
block/io.c +6
-0@@ -3382,6 +3382,31 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } +/* +* If the image has a backing file that is large enough that it would +* provide data for the new area, we cannot leave it unallocated because +* then the backing file content would become visible. Instead, zero-fill +* the area where backing file and new area overlap. +*/
Should we mention that, still, we don't care if user for some reason will change backing file in future?
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3403
Quoted 13 lines
block/io.c +13
-0+if (new_bytes && bs->backing && prealloc ==PREALLOC_MODE_OFF ) { +int64_t backing_len; + + backing_len =bdrv_getlength (backing_bs (bs)); +if (backing_len <0 ) { + ret = backing_len; + goto out; + } + +if (backing_len > old_size) { + ret =bdrv_co_do_pwrite_zeroes (bs, old_size, +MIN (new_bytes, backing_len - old_size), +BDRV_REQ_ZERO_WRITE |BDRV_REQ_MAY_UNMAP );
two over-80 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3408
Quoted 5 lines
block/io.c +5
-0+if (ret <0 ) { + goto out; + } + } + }
should we improve "off" mode specification in qapi?
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3412
Quoted 5 lines
block/io.c +1
-0+ ret =refresh_total_sectors (bs, offset >>BDRV_SECTOR_BITS );if (ret <0 ) {error_setg_errno (errp, -ret,"Could not refresh total sector count" );
Hmm. is it correct to call write_zeroes before refresh_total_sectors? Note that qcow2_co_pwrite_zeroes rely on bs->total_sectors...
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 3/6] iotests: Add qemu_io_log()
In reply to Kevin WolfQuoted 4 lines · Kevin Wolf
Add a function that runs qemu-io and logs the output with the appropriate filters applied.
Signed-off-by Kevin Wolf<[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 4/6] iotests: Fix timeout in run_job()
In reply to Kevin WolfQuoted 5 lines · Kevin Wolf
run_job()accepts a wait parameter for a timeout, but it doesn't actually use it. The only thing that is missing is passing it toevents_wait(), so do that now.Signed-off-by Kevin Wolf<[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 6/6] iotests: Test committing to short backing file
In reply to Kevin WolfQuoted 11 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 2 +- 4 files changed, 283 insertions(+), 1 deletion(-)create mode 100755 tests/qemu-iotests/274
create mode 100644 tests/qemu-iotests/274.out
tests/qemu-iotests/274 line 40
Quoted 9 lines
tests/qemu-iotests/274 new file +9
-0+def create_chain(): + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, + str(size_long)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, mid, + str(size_short)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', mid, top, + str(size_long)) + + iotests.qemu_io_log('-c', 'write -P 1 0 %d' % size_long, base)
Matches the setup in the cover letter, and definitely exposes the issues at hand.
As leaving this bug open represents a data leak/corruption to guests, I think this series counts as worthy of -rc3.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH 5/6] iotests: Support job-complete in run_job()
In reply to Kevin WolfQuoted 5 lines · Kevin Wolf
Automatically complete jobs that have a 'ready' state and need an explicit job-complete. Without this,
run_job()would hang for such jobs.Signed-off-by Kevin Wolf <[email protected]>
Quoted 18 lines
--- tests/qemu-iotests/iotests.py | 2 ++ 1 file changed, 2 insertions(+)tests/qemu-iotests/iotests.py +2
-0@@ -617,6 +617,8 @@ class VM(qtest.QEMUQtestMachine): error = j['error'] if use_log: log('Job failed: %s' % (j['error'])) + elif status == 'ready': + self.qmp_log('job-complete', id=job) elif status == 'pending' and not auto_finalize: if pre_finalize: pre_finalize()
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 2/6] block: truncate: Don't make backing file data visible
In reply to Vladimir Sementsov-OgievskiyAm 20.11.2019 um 15:47 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 25 lines
Quoted 24 lines · Kevin Wolf
When extending the size of an image that has a backing file larger than its old size, make sure that the backing file data doesn't become visible in the guest, but the added area is properly zeroed out.
The old behaviour made a difference in '
block_resize' (where showing the backing file data from an old snapshot rather than zeros is questionable) as well as in commit block jobs (both from active and intermediate nodes) and HMP 'commit', where committing to a short backing file would incorrectly omit writing zeroes for unallocated blocks on the top layer after the EOF of the short backing file.Signed-off-by Kevin Wolf <[email protected]>--- block/io.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)block/io.c +0
-0@@ -3382,6 +3382,31 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; }
Quoted 9 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3390
Quoted 6 lines
block/io.c +6
-0+/* +* If the image has a backing file that is large enough that it would +* provide data for the new area, we cannot leave it unallocated because +* then the backing file content would become visible. Instead, zero-fill +* the area where backing file and new area overlap. +*/ Should we mention that, still, we don't care if user for some reason will change backing file in future?
This should be obvious, but I can add something.
Quoted 7 lines
Quoted 7 lines
block/io.c +7
-0+if (new_bytes && bs->backing && prealloc ==PREALLOC_MODE_OFF ) { +int64_t backing_len; + + backing_len =bdrv_getlength (backing_bs (bs)); +if (backing_len <0 ) { + ret = backing_len; + goto out;
Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3403
Quoted 6 lines
block/io.c +6
-0+ } + +if (backing_len > old_size) { + ret =bdrv_co_do_pwrite_zeroes (bs, old_size, +MIN (new_bytes, backing_len - old_size), +BDRV_REQ_ZERO_WRITE |BDRV_REQ_MAY_UNMAP );two over-80 lines
Will fix.
Quoted 7 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3408
Quoted 5 lines
block/io.c +5
-0+if (ret <0 ) { + goto out; + } + } + }should we improve "off" mode specification in qapi?
I don't think we're changing the semantics of "off". We're merely fixing a bug that happens not to exist with preallocation.
Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3412
Quoted 5 lines
block/io.c +1
-0+ ret =refresh_total_sectors (bs, offset >>BDRV_SECTOR_BITS );if (ret <0 ) {error_setg_errno (errp, -ret,"Could not refresh total sector count" );Hmm. is it correct to call
write_zeroesbeforerefresh_total_sectors? Note thatqcow2_co_pwrite_zeroesrely onbs->total_sectors...
Hm... I placed the code where I did because I didn't want to make the new area valid before it isn't zeroed. But you're probably right that we shouldn't run requests with inconsistent bs->total_sectors, so I'll switch the order.
Kevin
Re: [PATCH 6/6] iotests: Test committing to short backing file
In reply to Kevin WolfQuoted 133 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 2 +- 4 files changed, 283 insertions(+), 1 deletion(-)create mode 100755 tests/qemu-iotests/274
create mode 100644 tests/qemu-iotests/274.outtests/qemu-iotests/274 new file +117
-0@@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Creator/Owner: Kevin Wolf <[email protected]> +# +# Some tests for short backing files and short overlays + +import iotests +import os + +iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.verify_platform(['linux']) + +size_short = 1 * 1024 * 1024 +size_long = 2 * 1024 * 1024 +size_diff = size_long - size_short + +def create_chain(): + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, + str(size_long)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, mid, + str(size_short)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', mid, top, + str(size_long)) + + iotests.qemu_io_log('-c', 'write -P 1 0 %d' % size_long, base) + +def create_vm(): + vm = iotests.VM() + vm.add_blockdev('file,filename=%s,node-name=base-file' % (base)) + vm.add_blockdev('%s,file=base-file,node-name=base' % (iotests.imgfmt)) + vm.add_blockdev('file,filename=%s,node-name=mid-file' % (mid)) + vm.add_blockdev('%s,file=mid-file,node-name=mid,backing=base' % (iotests.imgfmt)) + vm.add_drive(top, 'backing=mid,node-name=top') + return vm + +with iotests.FilePath('base') as base, \ + iotests.FilePath('mid') as mid, \ + iotests.FilePath('top') as top: + + iotests.log('== Commit tests ==') + + create_chain() + + iotests.log('=== Check visible data ===') + + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, top) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), top) + + iotests.log('=== Checking allocation status ===') + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + base) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + mid) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + top) + + iotests.log('=== Checking map ===') + + iotests.qemu_img_log('map', '--output=json', base) + iotests.qemu_img_log('map', '--output=human', base) + iotests.qemu_img_log('map', '--output=json', mid) + iotests.qemu_img_log('map', '--output=human', mid) + iotests.qemu_img_log('map', '--output=json', top) + iotests.qemu_img_log('map', '--output=human', top) + + iotests.log('=== Testing qemu-img commit (top -> mid) ===') + + iotests.qemu_img_log('commit', top) + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing HMP commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('human-monitor-command', command_line='commit drive0') + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing QMP active commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('block-commit', device='top', base_node='mid', + job_id='job0', auto_dismiss=False) + vm.run_job('job0', wait=5) + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) +
tests/qemu-iotests/274 line 127
Quoted 10 lines
tests/qemu-iotests/274 new file +10
-0+ + iotests.log('== Resize tests ==') + + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, '6G') + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, top, '1G') + iotests.qemu_io_log('-c', 'write -P 1 3G 64k', base) + iotests.qemu_io_log('-c', 'write -P 2 5G 64k', base) + + # After this, 0 to 6 GB should be allocated/zeroed + # 6 GB to 8 BG should be unallocated
Hmm, the problem is that the following qemu-img map can't show it, as it merges 1G..6G and 6G..8G into one chunk..
Quoted 40 lines
tests/qemu-iotests/274 new file +4
-0+ iotests.qemu_img_log('resize', '-f', iotests.imgfmt, top, '8G') + iotests.qemu_io_log('-c', 'read -P 0 3G 64k', top) + iotests.qemu_io_log('-c', 'read -P 0 5G 64k', top) + iotests.qemu_img_log('map', '--output=json', top)tests/qemu-iotests/274.out new file +30
-0@@ -0,0 +1,150 @@ +== Commit tests == +Formatting 'TEST_DIR/PID-base', fmt=qcow2 size=2097152 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-mid', fmt=qcow2 size=1048576 backing_file=TEST_DIR/PID-base cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-top', fmt=qcow2 size=2097152 backing_file=TEST_DIR/PID-mid cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +wrote 2097152/2097152 bytes at offset 0 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Check visible data === +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Checking allocation status === +1048576/1048576 bytes allocated at offset 0 bytes +1048576/1048576 bytes allocated at offset 1 MiB + +0/1048576 bytes allocated at offset 0 bytes +0/0 bytes allocated at offset 1 MiB + +0/1048576 bytes allocated at offset 0 bytes +0/1048576 bytes allocated at offset 1 MiB + +=== Checking map === +[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}] +
tests/qemu-iotests/274.out line 40
Quoted 10 lines
tests/qemu-iotests/274.out new file +10
-0+Offset Length Mapped to File +0 0x200000 0x50000 TEST_DIR/PID-base + +[{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}] + +Offset Length Mapped to File +0 0x100000 0x50000 TEST_DIR/PID-base + +[{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": true, "data": false}]
I think depth of second chunk should be 1, not 0.. But this is for another fixing series.
Quoted 125 lines
tests/qemu-iotests/274.out new file +110
-0+ +Offset Length Mapped to File +0 0x100000 0x50000 TEST_DIR/PID-base + +=== Testing qemu-img commit (top -> mid) === +Image committed. + +image: TEST_IMG +file format: IMGFMT +virtual size: 2 MiB (2097152 bytes) +cluster_size: 65536 +backing file: TEST_DIR/PID-base +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Testing HMP commit (top -> mid) === +Formatting 'TEST_DIR/PID-base', fmt=qcow2 size=2097152 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-mid', fmt=qcow2 size=1048576 backing_file=TEST_DIR/PID-base cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-top', fmt=qcow2 size=2097152 backing_file=TEST_DIR/PID-mid cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +wrote 2097152/2097152 bytes at offset 0 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +{"execute": "human-monitor-command", "arguments": {"command-line": "commit drive0"}} +{"return": ""} +image: TEST_IMG +file format: IMGFMT +virtual size: 2 MiB (2097152 bytes) +cluster_size: 65536 +backing file: TEST_DIR/PID-base +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Testing QMP active commit (top -> mid) === +Formatting 'TEST_DIR/PID-base', fmt=qcow2 size=2097152 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-mid', fmt=qcow2 size=1048576 backing_file=TEST_DIR/PID-base cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-top', fmt=qcow2 size=2097152 backing_file=TEST_DIR/PID-mid cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +wrote 2097152/2097152 bytes at offset 0 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +{"execute": "block-commit", "arguments": {"auto-dismiss": false, "base-node": "mid", "device": "top", "job-id": "job0"}} +{"return": {}} +{"execute": "job-complete", "arguments": {"id": "job0"}} +{"return": {}} +{"data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}} +{"execute": "job-dismiss", "arguments": {"id": "job0"}} +{"return": {}} +image: TEST_IMG +file format: IMGFMT +virtual size: 2 MiB (2097152 bytes) +cluster_size: 65536 +backing file: TEST_DIR/PID-base +Format specific information: + compat: 1.1 + lazy refcounts: false + refcount bits: 16 + corrupt: false + +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +== Resize tests == +Formatting 'TEST_DIR/PID-base', fmt=qcow2 size=6442450944 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-top', fmt=qcow2 size=1073741824 backing_file=TEST_DIR/PID-base cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +wrote 65536/65536 bytes at offset 3221225472 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +wrote 65536/65536 bytes at offset 5368709120 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +Image resized. + +read 65536/65536 bytes at offset 3221225472 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 65536/65536 bytes at offset 5368709120 +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +[{ "start": 0, "length": 1073741824, "depth": 1, "zero": true, "data": false}, +{ "start": 1073741824, "length": 7516192768, "depth": 0, "zero": true, "data": false}] +tests/qemu-iotests/group +1
-0@@ -285,3 +285,4 @@ 270 rw backing quick 272 rw 273 backing quick +274 rw backing quicktests/qemu-iotests/iotests.py +0
-0
tests/qemu-iotests/iotests.py def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): · line 150
Quoted 8 lines
tests/qemu-iotests/iotests.py +1
-1@@ -144,7 +144,7 @@ def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]): output = qemu_img_pipe(*args) if not filter_path: filter_path = filename - log(filter_img_info(output, filter_path)) + log(filter_img_info(output, filter_path), filters=[filter_testfiles]) def qemu_io(*args): '''Run qemu-io and return the stdout data'''
anyway,
Signature · 2 lines
Best regards, Vladimir
Quoted 17 lines · Kevin Wolf
Consider the following scenario where the overlay is shorter than its backing file:
base.qcow2: AAAAAAAA
overlay.qcow2: BBBBWhen resizing (extending) overlay.qcow2, the new blocks should not stay unallocated and make the additional As from base.qcow2 visible like before this series, but zeros should be read.
A similar case happens with the various variants of a commit job when an intermediate file is short (- for unallocated):
base.qcow2: A-A-AAAA mid.qcow2: BB-B top.qcow2: C--C--C-
Quoted 7 lines
After commit top.qcow2 to mid.qcow2, the following happens:
mid.qcow2: CB-C00C0 (correct result)
mid.qcow2: CB-C--C- (before this fix)Without the fix, blocks that previously read as zeros on top.qcow2 suddenly turn into A.
Note, that committing to base should still be broken after these series, and is fixed in my "[PATCH 0/4] fix & merge block_status_above and is_allocated_above".
I think, I'll post a 5/4 patch, to add this case to your new test.
Quoted 18 lines
Kevin Wolf (6):
block:bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter
block: truncate: Don't make backing file data visible
iotests: Addqemu_io_log()
iotests: Fix timeout inrun_job()
iotests: Support job-complete inrun_job()
iotests: Test committing to short backing fileblock/io.c | 31 ++++++- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 11 ++- 5 files changed, 319 insertions(+), 5 deletions(-)create mode 100755 tests/qemu-iotests/274
create mode 100644 tests/qemu-iotests/274.out
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 6/6] iotests: Test committing to short backing file
In reply to Kevin WolfQuoted 1 line · Kevin Wolf
Signed-off-by Kevin Wolf<[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 6/6] iotests: Test committing to short backing file
In reply to Vladimir Sementsov-OgievskiyAm 20.11.2019 um 16:41 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 139 lines
Quoted 138 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 2 +- 4 files changed, 283 insertions(+), 1 deletion(-)create mode 100755 tests/qemu-iotests/274
create mode 100644 tests/qemu-iotests/274.outtests/qemu-iotests/274 new file +122
-0@@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Creator/Owner: Kevin Wolf <[email protected]> +# +# Some tests for short backing files and short overlays + +import iotests +import os + +iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.verify_platform(['linux']) + +size_short = 1 * 1024 * 1024 +size_long = 2 * 1024 * 1024 +size_diff = size_long - size_short + +def create_chain(): + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, + str(size_long)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, mid, + str(size_short)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', mid, top, + str(size_long)) + + iotests.qemu_io_log('-c', 'write -P 1 0 %d' % size_long, base) + +def create_vm(): + vm = iotests.VM() + vm.add_blockdev('file,filename=%s,node-name=base-file' % (base)) + vm.add_blockdev('%s,file=base-file,node-name=base' % (iotests.imgfmt)) + vm.add_blockdev('file,filename=%s,node-name=mid-file' % (mid)) + vm.add_blockdev('%s,file=mid-file,node-name=mid,backing=base' % (iotests.imgfmt)) + vm.add_drive(top, 'backing=mid,node-name=top') + return vm + +with iotests.FilePath('base') as base, \ + iotests.FilePath('mid') as mid, \ + iotests.FilePath('top') as top: + + iotests.log('== Commit tests ==') + + create_chain() + + iotests.log('=== Check visible data ===') + + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, top) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), top) + + iotests.log('=== Checking allocation status ===') + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + base) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + mid) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + top) + + iotests.log('=== Checking map ===') + + iotests.qemu_img_log('map', '--output=json', base) + iotests.qemu_img_log('map', '--output=human', base) + iotests.qemu_img_log('map', '--output=json', mid) + iotests.qemu_img_log('map', '--output=human', mid) + iotests.qemu_img_log('map', '--output=json', top) + iotests.qemu_img_log('map', '--output=human', top) + + iotests.log('=== Testing qemu-img commit (top -> mid) ===') + + iotests.qemu_img_log('commit', top) + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing HMP commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('human-monitor-command', command_line='commit drive0') + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing QMP active commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('block-commit', device='top', base_node='mid', + job_id='job0', auto_dismiss=False) + vm.run_job('job0', wait=5) + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + + iotests.log('== Resize tests ==') + + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, '6G') + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, top, '1G')
Quoted 8 lines
tests/qemu-iotests/274 line 127
Quoted 5 lines
tests/qemu-iotests/274 new file +5
-0+ iotests.qemu_io_log('-c', 'write -P 1 3G 64k', base) + iotests.qemu_io_log('-c', 'write -P 2 5G 64k', base) + + # After this, 0 to 6 GB should be allocated/zeroed + # 6 GB to 8 BG should be unallocatedHmm, the problem is that the following qemu-img map can't show it, as it merges 1G..6G and 6G..8G into one chunk..
Hm, true, because it's more about the content of the blocks than about the allocation status. I'll add a qemu-io 'map' call, which display the actual allocation status:
1 GiB (0x40000000) bytes not allocated at offset 0 bytes (0x0)
5 GiB (0x140000000) bytes allocated at offset 1 GiB (0x40000000)
2 GiB (0x80000000) bytes not allocated at offset 6 GiB (0x180000000)
Quoted 45 lines
Quoted 45 lines
tests/qemu-iotests/274 new file +4
-0+ iotests.qemu_img_log('resize', '-f', iotests.imgfmt, top, '8G') + iotests.qemu_io_log('-c', 'read -P 0 3G 64k', top) + iotests.qemu_io_log('-c', 'read -P 0 5G 64k', top) + iotests.qemu_img_log('map', '--output=json', top)tests/qemu-iotests/274.out new file +35
-0@@ -0,0 +1,150 @@ +== Commit tests == +Formatting 'TEST_DIR/PID-base', fmt=qcow2 size=2097152 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-mid', fmt=qcow2 size=1048576 backing_file=TEST_DIR/PID-base cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-top', fmt=qcow2 size=2097152 backing_file=TEST_DIR/PID-mid cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +wrote 2097152/2097152 bytes at offset 0 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Check visible data === +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Checking allocation status === +1048576/1048576 bytes allocated at offset 0 bytes +1048576/1048576 bytes allocated at offset 1 MiB + +0/1048576 bytes allocated at offset 0 bytes +0/0 bytes allocated at offset 1 MiB + +0/1048576 bytes allocated at offset 0 bytes +0/1048576 bytes allocated at offset 1 MiB + +=== Checking map === +[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}] + +Offset Length Mapped to File +0 0x200000 0x50000 TEST_DIR/PID-base + +[{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}] +
Quoted 8 lines
tests/qemu-iotests/274.out line 40
Quoted 5 lines
tests/qemu-iotests/274.out new file +5
-0+Offset Length Mapped to File +0 0x100000 0x50000 TEST_DIR/PID-base + +[{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": true, "data": false}]I think depth of second chunk should be 1, not 0.. But this is for another fixing series.
The part from 1 GB to 6 GB should be 0 without any question, this is where we wrote zeros into the overlay.
The part from 7 GB to 8 GB is a bit more open to interpretation because this is unallocated in the overlay and reads zeros because the backing file is shorter. I think 0 makes sense, but it's debatable.
Kevin
Re: [PATCH 1/6] block: bdrv_co_do_pwrite_zeroes: 64 bit 'bytes' parameter
In reply to Kevin WolfQuoted 5 lines · Kevin Wolf
bdrv_co_do_pwrite_zeroes()can already cope with maximum request sizes by calling the driver in a loop until everything is done. Make the small remaining change that is necessary to let it accept a 64 bit byte count.Signed-off-by Kevin Wolf <[email protected]>
Berto
Re: [PATCH 6/6] iotests: Test committing to short backing file
In reply to Kevin WolfQuoted 211 lines · Kevin Wolf
Am 20.11.2019 um 16:41 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 139 lines
Quoted 138 lines · Kevin Wolf
Signed-off-by Kevin Wolf <[email protected]>--- tests/qemu-iotests/274 | 131 +++++++++++++++++++++++++++++ tests/qemu-iotests/274.out | 150 ++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + tests/qemu-iotests/iotests.py | 2 +- 4 files changed, 283 insertions(+), 1 deletion(-)create mode 100755 tests/qemu-iotests/274
create mode 100644 tests/qemu-iotests/274.outtests/qemu-iotests/274 new file +122
-0@@ -0,0 +1,131 @@ +#!/usr/bin/env python +# +# Copyright (C) 2019 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# Creator/Owner: Kevin Wolf <[email protected]> +# +# Some tests for short backing files and short overlays + +import iotests +import os + +iotests.verify_image_format(supported_fmts=['qcow2']) +iotests.verify_platform(['linux']) + +size_short = 1 * 1024 * 1024 +size_long = 2 * 1024 * 1024 +size_diff = size_long - size_short + +def create_chain(): + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, + str(size_long)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, mid, + str(size_short)) + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', mid, top, + str(size_long)) + + iotests.qemu_io_log('-c', 'write -P 1 0 %d' % size_long, base) + +def create_vm(): + vm = iotests.VM() + vm.add_blockdev('file,filename=%s,node-name=base-file' % (base)) + vm.add_blockdev('%s,file=base-file,node-name=base' % (iotests.imgfmt)) + vm.add_blockdev('file,filename=%s,node-name=mid-file' % (mid)) + vm.add_blockdev('%s,file=mid-file,node-name=mid,backing=base' % (iotests.imgfmt)) + vm.add_drive(top, 'backing=mid,node-name=top') + return vm + +with iotests.FilePath('base') as base, \ + iotests.FilePath('mid') as mid, \ + iotests.FilePath('top') as top: + + iotests.log('== Commit tests ==') + + create_chain() + + iotests.log('=== Check visible data ===') + + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, top) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), top) + + iotests.log('=== Checking allocation status ===') + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + base) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + mid) + + iotests.qemu_io_log('-c', 'alloc 0 %d' % size_short, + '-c', 'alloc %d %d' % (size_short, size_diff), + top) + + iotests.log('=== Checking map ===') + + iotests.qemu_img_log('map', '--output=json', base) + iotests.qemu_img_log('map', '--output=human', base) + iotests.qemu_img_log('map', '--output=json', mid) + iotests.qemu_img_log('map', '--output=human', mid) + iotests.qemu_img_log('map', '--output=json', top) + iotests.qemu_img_log('map', '--output=human', top) + + iotests.log('=== Testing qemu-img commit (top -> mid) ===') + + iotests.qemu_img_log('commit', top) + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing HMP commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('human-monitor-command', command_line='commit drive0') + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + iotests.log('=== Testing QMP active commit (top -> mid) ===') + + create_chain() + with create_vm() as vm: + vm.launch() + vm.qmp_log('block-commit', device='top', base_node='mid', + job_id='job0', auto_dismiss=False) + vm.run_job('job0', wait=5) + + iotests.img_info_log(mid) + iotests.qemu_io_log('-c', 'read -P 1 0 %d' % size_short, mid) + iotests.qemu_io_log('-c', 'read -P 0 %d %d' % (size_short, size_diff), mid) + + + iotests.log('== Resize tests ==') + + iotests.qemu_img_log('create', '-f', iotests.imgfmt, base, '6G') + iotests.qemu_img_log('create', '-f', iotests.imgfmt, '-b', base, top, '1G')Quoted 8 lines
tests/qemu-iotests/274 line 127
Quoted 5 lines
tests/qemu-iotests/274 new file +5
-0+ iotests.qemu_io_log('-c', 'write -P 1 3G 64k', base) + iotests.qemu_io_log('-c', 'write -P 2 5G 64k', base) + + # After this, 0 to 6 GB should be allocated/zeroed + # 6 GB to 8 BG should be unallocatedHmm, the problem is that the following qemu-img map can't show it, as it merges 1G..6G and 6G..8G into one chunk..
Hm, true, because it's more about the content of the blocks than about the allocation status. I'll add a qemu-io 'map' call, which display the actual allocation status:
1 GiB (
0x40000000) bytes not allocated at offset 0 bytes (0x0)
5 GiB (0x140000000) bytes allocated at offset 1 GiB (0x40000000)
2 GiB (0x80000000) bytes not allocated at offset 6 GiB (0x180000000)Quoted 45 lines
Quoted 45 lines
tests/qemu-iotests/274 new file +4
-0+ iotests.qemu_img_log('resize', '-f', iotests.imgfmt, top, '8G') + iotests.qemu_io_log('-c', 'read -P 0 3G 64k', top) + iotests.qemu_io_log('-c', 'read -P 0 5G 64k', top) + iotests.qemu_img_log('map', '--output=json', top)tests/qemu-iotests/274.out new file +35
-0@@ -0,0 +1,150 @@ +== Commit tests == +Formatting 'TEST_DIR/PID-base', fmt=qcow2 size=2097152 cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-mid', fmt=qcow2 size=1048576 backing_file=TEST_DIR/PID-base cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +Formatting 'TEST_DIR/PID-top', fmt=qcow2 size=2097152 backing_file=TEST_DIR/PID-mid cluster_size=65536 lazy_refcounts=off refcount_bits=16 + +wrote 2097152/2097152 bytes at offset 0 +2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Check visible data === +read 1048576/1048576 bytes at offset 0 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +read 1048576/1048576 bytes at offset 1048576 +1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) + +=== Checking allocation status === +1048576/1048576 bytes allocated at offset 0 bytes +1048576/1048576 bytes allocated at offset 1 MiB + +0/1048576 bytes allocated at offset 0 bytes +0/0 bytes allocated at offset 1 MiB + +0/1048576 bytes allocated at offset 0 bytes +0/1048576 bytes allocated at offset 1 MiB + +=== Checking map === +[{ "start": 0, "length": 2097152, "depth": 0, "zero": false, "data": true, "offset": 327680}] + +Offset Length Mapped to File +0 0x200000 0x50000 TEST_DIR/PID-base + +[{ "start": 0, "length": 1048576, "depth": 1, "zero": false, "data": true, "offset": 327680}] +Quoted 8 lines
tests/qemu-iotests/274.out line 40
Quoted 5 lines
tests/qemu-iotests/274.out new file +5
-0+Offset Length Mapped to File +0 0x100000 0x50000 TEST_DIR/PID-base + +[{ "start": 0, "length": 1048576, "depth": 2, "zero": false, "data": true, "offset": 327680}, +{ "start": 1048576, "length": 1048576, "depth": 0, "zero": true, "data": false}]I think depth of second chunk should be 1, not 0.. But this is for another fixing series.
Quoted 9 lines
The part from 1 GB to 6 GB should be 0 without any question, this is where we wrote zeros into the overlay.
The part from 7 GB to 8 GB is a bit more open to interpretation because this is unallocated in the overlay and reads zeros because the backing file is shorter. I think 0 makes sense, but it's debatable.
Kevin
Here I'm not about last test-case but about "depth" in second element of map, which covers [1M,2M) region. Zeroes comes from second layer (after EOF), not from first. On first this are is unallocated in all possible meanings. Still, I doubt that "depth" in qemu-img map output is something well-documented, so it's not significant.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 2/6] block: truncate: Don't make backing file data visible
In reply to Kevin WolfQuoted 43 lines · Kevin Wolf
When extending the size of an image that has a backing file larger than its old size, make sure that the backing file data doesn't become visible in the guest, but the added area is properly zeroed out.
The old behaviour made a difference in '
block_resize' (where showing the backing file data from an old snapshot rather than zeros is questionable) as well as in commit block jobs (both from active and intermediate nodes) and HMP 'commit', where committing to a short backing file would incorrectly omit writing zeroes for unallocated blocks on the top layer after the EOF of the short backing file.Signed-off-by Kevin Wolf <[email protected]>--- block/io.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)block/io.c +18
-0@@ -3382,6 +3382,31 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } +/* +* If the image has a backing file that is large enough that it would +* provide data for the new area, we cannot leave it unallocated because +* then the backing file content would become visible. Instead, zero-fill +* the area where backing file and new area overlap. +*/ +if (new_bytes && bs->backing && prealloc ==PREALLOC_MODE_OFF ) { +int64_t backing_len; + + backing_len =bdrv_getlength (backing_bs (bs)); +if (backing_len <0 ) { + ret = backing_len; + goto out; + } + +if (backing_len > old_size) { + ret = bdrv_co_do_pwrite_zeroes(bs, old_size, + MIN(new_bytes, backing_len - old_size),
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3412
Quoted 10 lines
block/io.c +7
-0+ BDRV_REQ_ZERO_WRITE | BDRV_REQ_MAY_UNMAP); +if (ret <0 ) { + goto out; + } + } + } + ret =refresh_total_sectors (bs, offset >>BDRV_SECTOR_BITS );if (ret <0 ) {error_setg_errno (errp, -ret,"Could not refresh total sector count" );
Hmmmm. I'm think that for commit, we also should zero truncated area if !bdrv_has_zero_init_truncate(bs). But we should not do it here, as it should not be done if we just resizing disk..
What formats are that bad?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH 2/6] block: truncate: Don't make backing file data visible
Am 20.11.2019 um 19:01 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 49 lines
Quoted 48 lines · Kevin Wolf
When extending the size of an image that has a backing file larger than its old size, make sure that the backing file data doesn't become visible in the guest, but the added area is properly zeroed out.
The old behaviour made a difference in '
block_resize' (where showing the backing file data from an old snapshot rather than zeros is questionable) as well as in commit block jobs (both from active and intermediate nodes) and HMP 'commit', where committing to a short backing file would incorrectly omit writing zeroes for unallocated blocks on the top layer after the EOF of the short backing file.Signed-off-by Kevin Wolf <[email protected]>--- block/io.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)block/io.c +24
-0@@ -3382,6 +3382,31 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } +/* +* If the image has a backing file that is large enough that it would +* provide data for the new area, we cannot leave it unallocated because +* then the backing file content would become visible. Instead, zero-fill +* the area where backing file and new area overlap. +*/ +if (new_bytes && bs->backing && prealloc ==PREALLOC_MODE_OFF ) { +int64_t backing_len; + + backing_len =bdrv_getlength (backing_bs (bs)); +if (backing_len <0 ) { + ret = backing_len; + goto out; + } + +if (backing_len > old_size) { + ret =bdrv_co_do_pwrite_zeroes (bs, old_size, +MIN (new_bytes, backing_len - old_size), +BDRV_REQ_ZERO_WRITE |BDRV_REQ_MAY_UNMAP ); +if (ret <0 ) { + goto out; + } + } + }
Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3412
Quoted 4 lines
block/io.c +1
-0+ ret =refresh_total_sectors (bs, offset >>BDRV_SECTOR_BITS );if (ret <0 ) {error_setg_errno (errp, -ret,"Could not refresh total sector count" );Hmmmm. I'm think that for commit, we also should zero truncated area if !bdrv_has_zero_init_truncate(bs). But we should not do it here, as it should not be done if we just resizing disk..
Hm, yes, we need to do this for METADATA and FALLOC preallocation at least. I think we already guarantee zeros for FULL, do we?
Resize needs zero init in the opposite case: When you are resizing a short backing file, the longer overlay still needs to read the same zeros it read before after EOF of the backing file. This one sounds actually even nastier to fix than what this series does. :-/
Anyway, maybe instead of the no_fallback parameter I introduced in v3, what we really want is a need_zero_init parameter that only commit jobs set for now? Or actually add a new preallocation mode like you suggested that would add a zero write and pass OFF to the driver implementations. Then we wouldn't have to add a new parameter everywhere.
We'd still unconditionally write zeros where it's necessary to allocate blocks to cover the backing file (and to provide correct data to the overlay if we ever figure out how to check this condition). I think I've come to the conclusion that blocking on block_resize is better than failing.
Quoted 1 line
What formats are that bad?
You mean that they don't have zero init? The usual suspect for bad image formats is raw, but fortunately that doesn't support backing files. So maybe it's not a problem we would see in practice.
Kevin