Fix overflow bug in qcow2 discard
v3 cover Fix overflow bug in qcow2 discard
v3: don't filter mapping info from qemu-img map output, otherwise
it don't show what I try to check [sorry for extra noise in list]
v2: [mostly by Eric's review]
01: new
02: point to bug introducing commit in cover letter [Eric]
[but I failed to compile it, to check]
drop s/INT_MAX/BDRV_REQUEST_MAX_BYTES/ chunk
03: - improve wording
- cheating with preallocation=metadata and discards to make test quick and not eating disk space
- use new trace-point
- move it to be 250 iotest
- filter out extra qemu-img info output
Vladimir Sementsov-Ogievskiy (3):
block/qcow2-refcount: add trace-point to qcow2_process_discards block/io: bdrv_pdiscard: support int64_t bytes parameter
iotests: test big qcow2 shrink
include/block/block.h | 4 +-- block/io.c | 16 ++++----- block/qcow2-refcount.c | 7 +++- block/trace-events | 3 ++ tests/qemu-iotests/250 | 72 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/250.out | 21 +++++++++++ tests/qemu-iotests/group | 1 + 7 files changed, 113 insertions(+), 11 deletions(-) create mode 100755 tests/qemu-iotests/250 create mode 100644 tests/qemu-iotests/250.out
Signature · 1 line
2.18.0
PATCH v3 1/3 block/qcow2-refcount: add trace-point to qcow2_process_discards
Let's at least trace ignored failure.
--- block/qcow2-refcount.c | 7 ++++++- block/trace-events | 3 +++ 2 files changed, 9 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +6 -1
@@ -30,6 +30,7 @@#include "qemu/range.h" #include "qemu/bswap.h" #include "qemu/cutils.h" +#include "trace.h" static int64_t alloc_clusters_noref (BlockDriverState *bs,uint64_t size,uint64_t max);
@@ -738,7 +739,11 @@ void qcow2_process_discards(BlockDriverState *bs, int ret)/* Discard is optional, ignore the return value */ if (ret >=0 ) { -bdrv_pdiscard (bs->file, d->offset, d->bytes); +int r2 =bdrv_pdiscard (bs->file, d->offset, d->bytes); +if (r2 <0 ) { +trace_qcow2_process_discards_failed_region (d->offset, d->bytes, + r2); + } }g_free (d);
block/trace-events +3 -0
@@ -91,6 +91,9 @@ qcow2_cache_get_done(void *co, int c, int i) "co %p is_l2_cache %d index %d" qcow2_cache_flush(void *co, int c) "co %p is_l2_cache %d" qcow2_cache_entry_flush(void *co, int c, int i) "co %p is_l2_cache %d index %d" +# qcow2-refcount.c +qcow2_process_discards_failed_region(uint64_t offset, uint64_t bytes, int ret) "offset 0x%" PRIx64 " bytes 0x%" PRIx64 " ret %d" + # qed-l2-cache.c qed_alloc_l2_cache_entry(void *l2_cache, void *entry) "l2_cache %p entry %p" qed_unref_l2_cache_entry(void *entry, int ref) "entry %p ref %d"
Signature · 1 line
2.18.0
Re: [Qemu-devel] [PATCH v3 1/3] block/qcow2-refcount: add trace-point to qcow2_process_discards
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
Let's at least trace ignored failure.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2-refcount.c | 7 ++++++- block/trace-events | 3 +++ 2 files changed, 9 insertions(+), 1 deletion(-)
Quoted 41 lines
block/qcow2-refcount.c +6
-1@@ -30,6 +30,7 @@#include "qemu/range.h" #include "qemu/bswap.h" #include "qemu/cutils.h" +#include "trace.h" static int64_t alloc_clusters_noref (BlockDriverState *bs,uint64_t size,uint64_t max);@@ -738,7 +739,11 @@ void qcow2_process_discards(BlockDriverState *bs, int ret)/* Discard is optional, ignore the return value */ if (ret >=0 ) { -bdrv_pdiscard (bs->file, d->offset, d->bytes); +int r2 =bdrv_pdiscard (bs->file, d->offset, d->bytes); +if (r2 <0 ) { +trace_qcow2_process_discards_failed_region (d->offset, d->bytes, + r2); + } }g_free (d);block/trace-events +3
-0@@ -91,6 +91,9 @@ qcow2_cache_get_done(void *co, int c, int i) "co %p is_l2_cache %d index %d" qcow2_cache_flush(void *co, int c) "co %p is_l2_cache %d" qcow2_cache_entry_flush(void *co, int c, int i) "co %p is_l2_cache %d index %d" +# qcow2-refcount.c +qcow2_process_discards_failed_region(uint64_t offset, uint64_t bytes, int ret) "offset 0x%" PRIx64 " bytes 0x%" PRIx64 " ret %d" + # qed-l2-cache.c qed_alloc_l2_cache_entry(void *l2_cache, void *entry) "l2_cache %p entry %p" qed_unref_l2_cache_entry(void *entry, int ref) "entry %p ref %d"
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v3 2/3 block/io: bdrv_pdiscard: support int64_t bytes parameter
In reply to Vladimir Sementsov-OgievskiyThis fixes at least one overflow in qcow2_process_discards, which passes 64bit region length to bdrv_pdiscard which bytes (or sectors in the past) parameter is int since it's introduction in 0b919fae.
--- include/block/block.h | 4 ++-- block/io.c | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-)
include/block/block.h +2 -2
@@ -432,8 +432,8 @@ void bdrv_drain_all(void);AIO_WAIT_WHILE (bdrv_get_aio_context (bs_), \ cond); }) -int bdrv_pdiscard (BdrvChild *child,int64_t offset,int bytes); -int bdrv_co_pdiscard (BdrvChild *child,int64_t offset,int bytes); +int bdrv_pdiscard (BdrvChild *child,int64_t offset,int64_t bytes); +int bdrv_co_pdiscard (BdrvChild *child,int64_t offset,int64_t bytes);int bdrv_has_zero_init_1 (BlockDriverState *bs);int bdrv_has_zero_init (BlockDriverState *bs);bool bdrv_unallocated_blocks_are_zero (BlockDriverState *bs);
block/io.c +8 -8
@@ -2653,7 +2653,7 @@ int bdrv_flush(BlockDriverState *bs)typedef struct DiscardCo {BdrvChild *child;int64_t offset; -int bytes; +int64_t bytes;int ret; }DiscardCo ; static void coroutine_fn bdrv_pdiscard_co_entry(void *opaque)
@@ -2664,14 +2664,15 @@ static void coroutine_fn bdrv_pdiscard_co_entry(void *opaque)aio_wait_kick (); } -int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes) +int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, +int64_t bytes) {BdrvTrackedRequest req;int max_pdiscard, ret;int head, tail, align;BlockDriverState *bs = child->bs; -if (!bs || !bs->drv) { +if (!bs || !bs->drv || !bdrv_is_inserted (bs)) {return -ENOMEDIUM ; }
@@ -2679,9 +2680,8 @@ int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes)return -EPERM ; } - ret =bdrv_check_byte_request (bs, offset, bytes); -if (ret <0 ) { -return ret; +if (offset <0 ) { +return -EIO ; }/* Do nothing if disabled. */
@@ -2716,7 +2716,7 @@ int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes)assert (max_pdiscard >= bs->bl.request_alignment);while (bytes >0 ) { -int num = bytes; +int64_t num = bytes;if (head) {/* Make small requests to get to alignment boundaries. */
@@ -2778,7 +2778,7 @@ out:return ret; } -int bdrv_pdiscard (BdrvChild *child,int64_t offset,int bytes) +int bdrv_pdiscard (BdrvChild *child,int64_t offset,int64_t bytes) {Coroutine *co;DiscardCo rwco = {
Signature · 1 line
2.18.0
Re: [Qemu-devel] [PATCH v3 2/3] block/io: bdrv_pdiscard: support int64_t bytes parameter
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
This fixes at least one overflow in
qcow2_process_discards, which passes 64bit region length tobdrv_pdiscardwhich bytes (or sectors in
s/which/where/
Quoted 1 line
the past) parameter is int since it's introduction in
0b919fae.
s/it's/its/
Quoted 7 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block.h | 4 ++-- block/io.c | 16 ++++++++-------- 2 files changed, 10 insertions(+), 10 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v3 3/3 iotests: test big qcow2 shrink
In reply to Vladimir Sementsov-OgievskiyThis test checks bug in qcow2_process_discards, fixed by previous commit.
--- tests/qemu-iotests/250 | 72 ++++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/250.out | 21 +++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 94 insertions(+) create mode 100755 tests/qemu-iotests/250 create mode 100644 tests/qemu-iotests/250.out
tests/qemu-iotests/250 new file +72 -0
@@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# +# Test big discard in qcow2 shrink +# +# Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +# creator [email protected] + +seq=`basename $0` +echo "QA output created by $seq" + +status=1 # failure is the default! + +_cleanup() +{ + _cleanup_test_img +} +trap "_cleanup; exit \$status" 0 1 2 3 15 + +# get standard environment, filters and checks +. ./common.rc +. ./common.filter + +_supported_fmt qcow2 +_supported_proto file +_supported_os Linux + +# This test checks that qcow2_process_discards does not truncate a discard +# request > 2G. +# To reproduce bug we need to overflow int by one sequential discard, so we +# need size > 2G, bigger cluster size (as with default 64k we may have maximum +# of 512M sequential data, corresponding to one L1 entry), and we need some +# data of the beginning of the disk mapped to the end of file to prevent +# bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which may success +# anyway. + +size=2100M +IMGOPTS="cluster_size=1M,preallocation=metadata" + +_make_test_img $size +$QEMU_IO -c 'discard 0 10M' -c 'discard 2090M 10M' \ + -c 'write 2090M 10M' -c 'write 0 10M' "$TEST_IMG" | _filter_qemu_io + +# Check that our trick with swapping first and last 10M chunks succeeded. +# Otherwise test will may pass even if bdrv_pdiscard() fails in +# qcow2_process_discards() +$QEMU_IMG map "$TEST_IMG" | _filter_testdir +$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdir + +$QEMU_IMG -T 'qcow2_process_discards_failed*' resize --shrink "$TEST_IMG" 5M + +$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdir + +# success, all done +echo "*** done" +rm -f $seq.full +status=0
tests/qemu-iotests/250.out new file +21 -0
@@ -0,0 +1,21 @@ +QA output created by 250 +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2202009600 preallocation=metadata +discard 10485760/10485760 bytes at offset 0 +10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +discard 10485760/10485760 bytes at offset 2191523840 +10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 10485760/10485760 bytes at offset 2191523840 +10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +wrote 10485760/10485760 bytes at offset 0 +10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec) +Offset Length Mapped to File +0 0xa00000 0x82f00000 TEST_DIR/t.qcow2 +0x82a00000 0xa00000 0x500000 TEST_DIR/t.qcow2 +virtual size: 2.1G (2202009600 bytes) +disk size: 24M +cluster_size: 1048576 +Image resized. +virtual size: 5.0M (5242880 bytes) +disk size: 9.0M +cluster_size: 1048576 +*** done
tests/qemu-iotests/group +1 -0
@@ -248,3 +248,4 @@ 246 rw auto quick 247 rw auto quick 248 rw auto quick +250 rw auto quick
Signature · 1 line
2.18.0
Re: [Qemu-devel] [PATCH v3 3/3] iotests: test big qcow2 shrink
Review
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
This test checks bug in
qcow2_process_discards, fixed by previous commit.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
(quoted diff) +7
-0+# This test checks that qcow2_process_discards does not truncate a discard +# request > 2G. +# To reproduce bug we need to overflow int by one sequential discard, so we +# need size > 2G, bigger cluster size (as with default 64k we may have maximum +# of 512M sequential data, corresponding to one L1 entry), and we need some +# data of the beginning of the disk mapped to the end of file to prevent +# bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which may success
s/may success/might succeed/
Quoted 4 lines
(quoted diff) +4
-0+# anyway. + +size=2100M +IMGOPTS="cluster_size=1M,preallocation=metadata"
Review
Quoted 10 lines
(quoted diff) +10
-0+ +_make_test_img $size +$QEMU_IO -c 'discard 0 10M' -c 'discard 2090M 10M' \ + -c 'write 2090M 10M' -c 'write 0 10M' "$TEST_IMG" | _filter_qemu_io + +# Check that our trick with swapping first and last 10M chunks succeeded. +# Otherwise test will may pass even if bdrv_pdiscard() fails in +# qcow2_process_discards() +$QEMU_IMG map "$TEST_IMG" | _filter_testdir +$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdir
Nice - that's a lot faster than v1! And makes the test fit in the quick group after all.
Quoted 2 lines
+
+$QEMU_IMG -T 'qcow2_process_discards_failed*' resize --shrink "$TEST_IMG" 5M
However, I'm quite certain that trace output is not reliable in iotests. Depending on configure options, traces might not be enabled at all, or might not go to stderr. Drop the -T '...'. Even without the trace line,
Quoted 2 lines
+
+$QEMU_IMG info "$TEST_IMG" | grep size |_filter_testdir
this second image info is sufficient to prove whether the resize had an effect (post-patch) or exposes the bug (without patch 2/3). That is, applying this patch but not 2/3, I see this (with my cleanups to qemu-img info in place, from Kevin's block-next branch):
+++ /home/eblake/qemu/tests/qemu-iotests/250.out.bad 2019-04-22
08:52:26.072968731 -0500
@@ -14,8 +14,9 @@
virtual size: 2.05 GiB (2202009600 bytes)
disk size: 24 MiB
cluster_size: 1048576
[email protected]:qcow2_process_discards_failed_region offset 0x500000 bytes 0x82a00000 ret -5
Image resized.
virtual size: 5 MiB (5242880 bytes)
-disk size: 9.01 MiB
+disk size: 19 MiB
where the trace did indeed show show that we had a bug, but where even without the trace, the difference in size between 19M with incomplete discards vs. 9M with patch 2/3 is enough to show that patch 2/3 fixes a bug.
Review
Quoted 7 lines
(quoted diff) +7
-0+virtual size: 2.1G (2202009600 bytes) +disk size: 24M +cluster_size: 1048576 +Image resized. +virtual size: 5.0M (5242880 bytes) +disk size: 9.0M +cluster_size: 1048576
When Kevin's block-next branch is applied, you'll need this squashed in:
tests/qemu-iotests/250.out +4 -4
@@ -11,11 +11,11 @@ wrote 10485760/10485760 bytes at offset 0 Offset Length Mapped to File 0 0xa00000 0x82f00000 TEST_DIR/t.qcow2 0x82a00000 0xa00000 0x500000 TEST_DIR/t.qcow2 -virtual size: 2.1G (2202009600 bytes) -disk size: 24M +virtual size: 2.05 GiB (2202009600 bytes) +disk size: 24 MiB cluster_size: 1048576 Image resized. -virtual size: 5.0M (5242880 bytes) -disk size: 9.0M +virtual size: 5 MiB (5242880 bytes) +disk size: 9.01 MiB cluster_size: 1048576 *** done
With the updated output to match changes to qemu-img output, the grammar fixes, and with the -T option removed,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v3 3/3] iotests: test big qcow2 shrink
Quoted 98 lines · Eric Blake
Review
Quoted 13 lines · Vladimir Sementsov-Ogievskiy
This test checks bug in
qcow2_process_discards, fixed by previous commit.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
(quoted diff) +7
-0+# This test checks that qcow2_process_discards does not truncate a discard +# request > 2G. +# To reproduce bug we need to overflow int by one sequential discard, so we +# need size > 2G, bigger cluster size (as with default 64k we may have maximum +# of 512M sequential data, corresponding to one L1 entry), and we need some +# data of the beginning of the disk mapped to the end of file to prevent +# bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which may successs/may success/might succeed/
Quoted 4 lines
(quoted diff) +4
-0+# anyway. + +size=2100M +IMGOPTS="cluster_size=1M,preallocation=metadata"Review
Quoted 10 lines
(quoted diff) +10
-0+ +_make_test_img $size +$QEMU_IO -c 'discard 0 10M' -c 'discard 2090M 10M' \ + -c 'write 2090M 10M' -c 'write 0 10M' "$TEST_IMG" | _filter_qemu_io + +# Check that our trick with swapping first and last 10M chunks succeeded. +# Otherwise test will may pass even if bdrv_pdiscard() fails in +# qcow2_process_discards() +$QEMU_IMG map "$TEST_IMG" | _filter_testdir +$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdirNice - that's a lot faster than v1! And makes the test fit in the quick group after all.
Quoted 2 lines
+
+$QEMU_IMG -T 'qcow2_process_discards_failed*' resize --shrink "$TEST_IMG" 5MHowever, I'm quite certain that trace output is not reliable in iotests. Depending on configure options, traces might not be enabled at all, or might not go to stderr. Drop the -T '...'. Even without the trace line,
Quoted 2 lines
+
+$QEMU_IMG info "$TEST_IMG" | grep size |_filter_testdirthis second image info is sufficient to prove whether the resize had an effect (post-patch) or exposes the bug (without patch 2/3). That is, applying this patch but not 2/3, I see this (with my cleanups to qemu-img info in place, from Kevin's block-next branch):
+++
/home/eblake/qemu/tests/qemu-iotests/250.out.bad2019-04-22
08:52:26.072968731 -0500(quoted diff) +1
-0@@ -14,8 +14,9 @@ virtual size: 2.05 GiB (2202009600 bytes) disk size: 24 MiB cluster_size: 1048576 [email protected]:qcow2_process_discards_failed_region offset
0x500000bytes0x82a00000ret -5
Image resized.
virtual size: 5 MiB (5242880 bytes)(quoted diff) +1
-1-disk size: 9.01 MiB +disk size: 19 MiBwhere the trace did indeed show show that we had a bug, but where even without the trace, the difference in size between 19M with incomplete discards vs. 9M with patch 2/3 is enough to show that patch 2/3 fixes a bug.
Review
Quoted 7 lines
(quoted diff) +7
-0+virtual size: 2.1G (2202009600 bytes) +disk size: 24M +cluster_size: 1048576 +Image resized. +virtual size: 5.0M (5242880 bytes) +disk size: 9.0M +cluster_size: 1048576When Kevin's block-next branch is applied, you'll need this squashed in:
tests/qemu-iotests/250.out +4
-4@@ -11,11 +11,11 @@ wrote 10485760/10485760 bytes at offset 0 Offset Length Mapped to File 0 0xa00000 0x82f00000 TEST_DIR/t.qcow2 0x82a00000 0xa00000 0x500000 TEST_DIR/t.qcow2 -virtual size: 2.1G (2202009600 bytes) -disk size: 24M +virtual size: 2.05 GiB (2202009600 bytes) +disk size: 24 MiB cluster_size: 1048576 Image resized. -virtual size: 5.0M (5242880 bytes) -disk size: 9.0M +virtual size: 5 MiB (5242880 bytes) +disk size: 9.01 MiB cluster_size: 1048576 *** done
Quoted 6 lines
With the updated output to match changes to qemu-img output, the grammar fixes, and with the -T option removed,
Reviewed-by Eric Blake <[email protected]>Tested-by Eric Blake <[email protected]>
Thank you!
I'll resend soon.
Signature · 2 lines
Best regards, Vladimir