Skip to contentSkip to conversation

Fix overflow bug in qcow2 discard

PATCH Superseded Vladimir Sementsov-Ogievskiy 2 participants 8 messages Apr 19, 2019 – Apr 22, 2019 Block layer

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Fix overflow bug in qcow2 discard

v3: don't filter mapping info from qemu-img map output, otherwise
it don't show what I try to check [sorry for extra noise in list]

v2: [mostly by Eric's review]
01: new
02: point to bug introducing commit in cover letter [Eric]
[but I failed to compile it, to check]
drop s/INT_MAX/BDRV_REQUEST_MAX_BYTES/ chunk
03: - improve wording
- cheating with preallocation=metadata and discards to make test quick and not eating disk space
- use new trace-point
- move it to be 250 iotest
- filter out extra qemu-img info output

Vladimir Sementsov-Ogievskiy (3):
block/qcow2-refcount: add trace-point to qcow2_process_discards block/io: bdrv_pdiscard: support int64_t bytes parameter
iotests: test big qcow2 shrink

 include/block/block.h      |  4 +--
 block/io.c                 | 16 ++++-----
 block/qcow2-refcount.c     |  7 +++-
 block/trace-events         |  3 ++
 tests/qemu-iotests/250     | 72 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/250.out | 21 +++++++++++
 tests/qemu-iotests/group   |  1 +
 7 files changed, 113 insertions(+), 11 deletions(-)
 create mode 100755 tests/qemu-iotests/250
 create mode 100644 tests/qemu-iotests/250.out
Signature · 1 line
2.18.0
#1 lore raw

PATCH v3 1/3 block/qcow2-refcount: add trace-point to qcow2_process_discards

Let's at least trace ignored failure.

---
 block/qcow2-refcount.c | 7 ++++++-
 block/trace-events     | 3 +++
 2 files changed, 9 insertions(+), 1 deletion(-)
block/qcow2-refcount.c +6 -1
index e0fe322500..60284bcaac 100644
@@ -30,6 +30,7 @@
 #include "qemu/range.h"
 #include "qemu/bswap.h"
 #include "qemu/cutils.h"
+#include "trace.h"
 
 static int64_t alloc_clusters_noref(BlockDriverState *bs, uint64_t size,
                                     uint64_t max);
@@ -738,7 +739,11 @@ void qcow2_process_discards(BlockDriverState *bs, int ret)
 
         /* Discard is optional, ignore the return value */
         if (ret >= 0) {
-            bdrv_pdiscard(bs->file, d->offset, d->bytes);
+            int r2 = bdrv_pdiscard(bs->file, d->offset, d->bytes);
+            if (r2 < 0) {
+                trace_qcow2_process_discards_failed_region(d->offset, d->bytes,
+                                                           r2);
+            }
         }
 
         g_free(d);
block/trace-events +3 -0
index 7335a42540..ea508f637e 100644
@@ -91,6 +91,9 @@ qcow2_cache_get_done(void *co, int c, int i) "co %p is_l2_cache %d index %d"
 qcow2_cache_flush(void *co, int c) "co %p is_l2_cache %d"
 qcow2_cache_entry_flush(void *co, int c, int i) "co %p is_l2_cache %d index %d"
 
+# qcow2-refcount.c
+qcow2_process_discards_failed_region(uint64_t offset, uint64_t bytes, int ret) "offset 0x%" PRIx64 " bytes 0x%" PRIx64 " ret %d"
+
 # qed-l2-cache.c
 qed_alloc_l2_cache_entry(void *l2_cache, void *entry) "l2_cache %p entry %p"
 qed_unref_l2_cache_entry(void *entry, int ref) "entry %p ref %d"
Signature · 1 line
2.18.0
Reviewed-by on v3 1/3
#2 lore raw

Re: [Qemu-devel] [PATCH v3 1/3] block/qcow2-refcount: add trace-point to qcow2_process_discards

Quoted 7 lines · Vladimir Sementsov-Ogievskiy

Let's at least trace ignored failure.

---
 block/qcow2-refcount.c | 7 ++++++-
 block/trace-events     | 3 +++
 2 files changed, 9 insertions(+), 1 deletion(-)
Quoted 41 lines
block/qcow2-refcount.c +6 -1
index e0fe322500..60284bcaac 100644
@@ -30,6 +30,7 @@
 #include "qemu/range.h"
 #include "qemu/bswap.h"
 #include "qemu/cutils.h"
+#include "trace.h"
 
 static int64_t alloc_clusters_noref(BlockDriverState *bs, uint64_t size,
                                     uint64_t max);
@@ -738,7 +739,11 @@ void qcow2_process_discards(BlockDriverState *bs, int ret)
 
         /* Discard is optional, ignore the return value */
         if (ret >= 0) {
-            bdrv_pdiscard(bs->file, d->offset, d->bytes);
+            int r2 = bdrv_pdiscard(bs->file, d->offset, d->bytes);
+            if (r2 < 0) {
+                trace_qcow2_process_discards_failed_region(d->offset, d->bytes,
+                                                           r2);
+            }
         }
 
         g_free(d);
block/trace-events +3 -0
index 7335a42540..ea508f637e 100644
@@ -91,6 +91,9 @@ qcow2_cache_get_done(void *co, int c, int i) "co %p is_l2_cache %d index %d"
 qcow2_cache_flush(void *co, int c) "co %p is_l2_cache %d"
 qcow2_cache_entry_flush(void *co, int c, int i) "co %p is_l2_cache %d index %d"
 
+# qcow2-refcount.c
+qcow2_process_discards_failed_region(uint64_t offset, uint64_t bytes, int ret) "offset 0x%" PRIx64 " bytes 0x%" PRIx64 " ret %d"
+
 # qed-l2-cache.c
 qed_alloc_l2_cache_entry(void *l2_cache, void *entry) "l2_cache %p entry %p"
 qed_unref_l2_cache_entry(void *entry, int ref) "entry %p ref %d"
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#3 lore raw

PATCH v3 2/3 block/io: bdrv_pdiscard: support int64_t bytes parameter

In reply to Vladimir Sementsov-Ogievskiy

This fixes at least one overflow in qcow2_process_discards, which passes 64bit region length to bdrv_pdiscard which bytes (or sectors in the past) parameter is int since it's introduction in 0b919fae.

---
 include/block/block.h |  4 ++--
 block/io.c            | 16 ++++++++--------
 2 files changed, 10 insertions(+), 10 deletions(-)
include/block/block.h +2 -2
index c7a26199aa..69fa18867e 100644
@@ -432,8 +432,8 @@ void bdrv_drain_all(void);
     AIO_WAIT_WHILE(bdrv_get_aio_context(bs_),              \
                    cond); })
 
-int bdrv_pdiscard(BdrvChild *child, int64_t offset, int bytes);
-int bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes);
+int bdrv_pdiscard(BdrvChild *child, int64_t offset, int64_t bytes);
+int bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int64_t bytes);
 int bdrv_has_zero_init_1(BlockDriverState *bs);
 int bdrv_has_zero_init(BlockDriverState *bs);
 bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs);
block/io.c +8 -8
index dfc153b8d8..35c4157669 100644
@@ -2653,7 +2653,7 @@ int bdrv_flush(BlockDriverState *bs)
 typedef struct DiscardCo {
     BdrvChild *child;
     int64_t offset;
-    int bytes;
+    int64_t bytes;
     int ret;
 } DiscardCo;
 static void coroutine_fn bdrv_pdiscard_co_entry(void *opaque)
@@ -2664,14 +2664,15 @@ static void coroutine_fn bdrv_pdiscard_co_entry(void *opaque)
     aio_wait_kick();
 }
 
-int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes)
+int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset,
+                                  int64_t bytes)
 {
     BdrvTrackedRequest req;
     int max_pdiscard, ret;
     int head, tail, align;
     BlockDriverState *bs = child->bs;
 
-    if (!bs || !bs->drv) {
+    if (!bs || !bs->drv || !bdrv_is_inserted(bs)) {
         return -ENOMEDIUM;
     }
 
@@ -2679,9 +2680,8 @@ int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes)
         return -EPERM;
     }
 
-    ret = bdrv_check_byte_request(bs, offset, bytes);
-    if (ret < 0) {
-        return ret;
+    if (offset < 0) {
+        return -EIO;
     }
 
     /* Do nothing if disabled.  */
@@ -2716,7 +2716,7 @@ int coroutine_fn bdrv_co_pdiscard(BdrvChild *child, int64_t offset, int bytes)
     assert(max_pdiscard >= bs->bl.request_alignment);
 
     while (bytes > 0) {
-        int num = bytes;
+        int64_t num = bytes;
 
         if (head) {
             /* Make small requests to get to alignment boundaries. */
@@ -2778,7 +2778,7 @@ out:
     return ret;
 }
 
-int bdrv_pdiscard(BdrvChild *child, int64_t offset, int bytes)
+int bdrv_pdiscard(BdrvChild *child, int64_t offset, int64_t bytes)
 {
     Coroutine *co;
     DiscardCo rwco = {
Signature · 1 line
2.18.0
Reviewed-by on v3 2/3
#4 lore raw

Re: [Qemu-devel] [PATCH v3 2/3] block/io: bdrv_pdiscard: support int64_t bytes parameter

Quoted 2 lines · Vladimir Sementsov-Ogievskiy

This fixes at least one overflow in qcow2_process_discards, which passes 64bit region length to bdrv_pdiscard which bytes (or sectors in

s/which/where/

Quoted 1 line

the past) parameter is int since it's introduction in 0b919fae.

s/it's/its/

Quoted 7 lines
---
 include/block/block.h |  4 ++--
 block/io.c            | 16 ++++++++--------
 2 files changed, 10 insertions(+), 10 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#5 lore raw

PATCH v3 3/3 iotests: test big qcow2 shrink

In reply to Vladimir Sementsov-Ogievskiy

This test checks bug in qcow2_process_discards, fixed by previous commit.

---
 tests/qemu-iotests/250     | 72 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/250.out | 21 +++++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 94 insertions(+)
 create mode 100755 tests/qemu-iotests/250
 create mode 100644 tests/qemu-iotests/250.out
tests/qemu-iotests/250 new file +72 -0
new file mode 100755
index 0000000000..618ddef5e4
@@ -0,0 +1,72 @@
+#!/usr/bin/env bash
+#
+# Test big discard in qcow2 shrink
+#
+# Copyright (c) 2019 Virtuozzo International GmbH. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
[email protected]
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+# This test checks that qcow2_process_discards does not truncate a discard
+# request > 2G.
+# To reproduce bug we need to overflow int by one sequential discard, so we
+# need size > 2G, bigger cluster size (as with default 64k we may have maximum
+# of 512M sequential data, corresponding to one L1 entry), and we need some
+# data of the beginning of the disk mapped to the end of file to prevent
+# bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which may success
+# anyway.
+
+size=2100M
+IMGOPTS="cluster_size=1M,preallocation=metadata"
+
+_make_test_img $size
+$QEMU_IO -c 'discard 0 10M' -c 'discard 2090M 10M' \
+         -c 'write 2090M 10M' -c 'write 0 10M' "$TEST_IMG" | _filter_qemu_io
+
+# Check that our trick with swapping first and last 10M chunks succeeded.
+# Otherwise test will may pass even if bdrv_pdiscard() fails in
+# qcow2_process_discards()
+$QEMU_IMG map "$TEST_IMG" | _filter_testdir
+$QEMU_IMG info "$TEST_IMG" | grep size |  _filter_testdir
+
+$QEMU_IMG -T 'qcow2_process_discards_failed*' resize --shrink "$TEST_IMG" 5M
+
+$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdir
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/250.out new file +21 -0
new file mode 100644
index 0000000000..37e37f0c9e
@@ -0,0 +1,21 @@
+QA output created by 250
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=2202009600 preallocation=metadata
+discard 10485760/10485760 bytes at offset 0
+10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+discard 10485760/10485760 bytes at offset 2191523840
+10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 10485760/10485760 bytes at offset 2191523840
+10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 10485760/10485760 bytes at offset 0
+10 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Offset          Length          Mapped to       File
+0               0xa00000        0x82f00000      TEST_DIR/t.qcow2
+0x82a00000      0xa00000        0x500000        TEST_DIR/t.qcow2
+virtual size: 2.1G (2202009600 bytes)
+disk size: 24M
+cluster_size: 1048576
+Image resized.
+virtual size: 5.0M (5242880 bytes)
+disk size: 9.0M
+cluster_size: 1048576
+*** done
tests/qemu-iotests/group +1 -0
index bae7718380..588ae8b8b1 100644
@@ -248,3 +248,4 @@
 246 rw auto quick
 247 rw auto quick
 248 rw auto quick
+250 rw auto quick
Signature · 1 line
2.18.0
Reviewed-by Tested-by on v3 3/3
#6 lore raw

Re: [Qemu-devel] [PATCH v3 3/3] iotests: test big qcow2 shrink

Review

Quoted 13 lines · Vladimir Sementsov-Ogievskiy

This test checks bug in qcow2_process_discards, fixed by previous commit.

---

(quoted diff) +7 -0
+# This test checks that qcow2_process_discards does not truncate a discard
+# request > 2G.
+# To reproduce bug we need to overflow int by one sequential discard, so we
+# need size > 2G, bigger cluster size (as with default 64k we may have maximum
+# of 512M sequential data, corresponding to one L1 entry), and we need some
+# data of the beginning of the disk mapped to the end of file to prevent
+# bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which may success

s/may success/might succeed/

Quoted 4 lines
(quoted diff) +4 -0
+# anyway.
+
+size=2100M
+IMGOPTS="cluster_size=1M,preallocation=metadata"

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+_make_test_img $size
+$QEMU_IO -c 'discard 0 10M' -c 'discard 2090M 10M' \
+         -c 'write 2090M 10M' -c 'write 0 10M' "$TEST_IMG" | _filter_qemu_io
+
+# Check that our trick with swapping first and last 10M chunks succeeded.
+# Otherwise test will may pass even if bdrv_pdiscard() fails in
+# qcow2_process_discards()
+$QEMU_IMG map "$TEST_IMG" | _filter_testdir
+$QEMU_IMG info "$TEST_IMG" | grep size |  _filter_testdir

Nice - that's a lot faster than v1! And makes the test fit in the quick group after all.

Quoted 2 lines

+
+$QEMU_IMG -T 'qcow2_process_discards_failed*' resize --shrink "$TEST_IMG" 5M

However, I'm quite certain that trace output is not reliable in iotests. Depending on configure options, traces might not be enabled at all, or might not go to stderr. Drop the -T '...'. Even without the trace line,

Quoted 2 lines

+
+$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdir

this second image info is sufficient to prove whether the resize had an effect (post-patch) or exposes the bug (without patch 2/3). That is, applying this patch but not 2/3, I see this (with my cleanups to qemu-img info in place, from Kevin's block-next branch):

+++ /home/eblake/qemu/tests/qemu-iotests/250.out.bad 2019-04-22
08:52:26.072968731 -0500
@@ -14,8 +14,9 @@
virtual size: 2.05 GiB (2202009600 bytes)
disk size: 24 MiB
cluster_size: 1048576
[email protected]:qcow2_process_discards_failed_region offset 0x500000 bytes 0x82a00000 ret -5
Image resized.
virtual size: 5 MiB (5242880 bytes)
-disk size: 9.01 MiB
+disk size: 19 MiB

where the trace did indeed show show that we had a bug, but where even without the trace, the difference in size between 19M with incomplete discards vs. 9M with patch 2/3 is enough to show that patch 2/3 fixes a bug.

Review

Quoted 7 lines
(quoted diff) +7 -0
+virtual size: 2.1G (2202009600 bytes)
+disk size: 24M
+cluster_size: 1048576
+Image resized.
+virtual size: 5.0M (5242880 bytes)
+disk size: 9.0M
+cluster_size: 1048576

When Kevin's block-next branch is applied, you'll need this squashed in:

tests/qemu-iotests/250.out +4 -4
index 37e37f0c9e7..d1c1c7180b5 100644
@@ -11,11 +11,11 @@ wrote 10485760/10485760 bytes at offset 0
 Offset          Length          Mapped to       File
 0               0xa00000        0x82f00000      TEST_DIR/t.qcow2
 0x82a00000      0xa00000        0x500000        TEST_DIR/t.qcow2
-virtual size: 2.1G (2202009600 bytes)
-disk size: 24M
+virtual size: 2.05 GiB (2202009600 bytes)
+disk size: 24 MiB
 cluster_size: 1048576
 Image resized.
-virtual size: 5.0M (5242880 bytes)
-disk size: 9.0M
+virtual size: 5 MiB (5242880 bytes)
+disk size: 9.01 MiB
 cluster_size: 1048576
 *** done

With the updated output to match changes to qemu-img output, the grammar fixes, and with the -T option removed,

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#7 lore raw

Re: [Qemu-devel] [PATCH v3 3/3] iotests: test big qcow2 shrink

Quoted 98 lines · Eric Blake

Review

Quoted 13 lines · Vladimir Sementsov-Ogievskiy

This test checks bug in qcow2_process_discards, fixed by previous commit.

---

(quoted diff) +7 -0
+# This test checks that qcow2_process_discards does not truncate a discard
+# request > 2G.
+# To reproduce bug we need to overflow int by one sequential discard, so we
+# need size > 2G, bigger cluster size (as with default 64k we may have maximum
+# of 512M sequential data, corresponding to one L1 entry), and we need some
+# data of the beginning of the disk mapped to the end of file to prevent
+# bdrv_co_truncate(bs->file) call in qcow2_co_truncate(), which may success

s/may success/might succeed/

Quoted 4 lines
(quoted diff) +4 -0
+# anyway.
+
+size=2100M
+IMGOPTS="cluster_size=1M,preallocation=metadata"

Review

Quoted 10 lines
(quoted diff) +10 -0
+
+_make_test_img $size
+$QEMU_IO -c 'discard 0 10M' -c 'discard 2090M 10M' \
+         -c 'write 2090M 10M' -c 'write 0 10M' "$TEST_IMG" | _filter_qemu_io
+
+# Check that our trick with swapping first and last 10M chunks succeeded.
+# Otherwise test will may pass even if bdrv_pdiscard() fails in
+# qcow2_process_discards()
+$QEMU_IMG map "$TEST_IMG" | _filter_testdir
+$QEMU_IMG info "$TEST_IMG" | grep size |  _filter_testdir

Nice - that's a lot faster than v1! And makes the test fit in the quick group after all.

Quoted 2 lines

+
+$QEMU_IMG -T 'qcow2_process_discards_failed*' resize --shrink "$TEST_IMG" 5M

However, I'm quite certain that trace output is not reliable in iotests. Depending on configure options, traces might not be enabled at all, or might not go to stderr. Drop the -T '...'. Even without the trace line,

Quoted 2 lines

+
+$QEMU_IMG info "$TEST_IMG" | grep size | _filter_testdir

this second image info is sufficient to prove whether the resize had an effect (post-patch) or exposes the bug (without patch 2/3). That is, applying this patch but not 2/3, I see this (with my cleanups to qemu-img info in place, from Kevin's block-next branch):

+++ /home/eblake/qemu/tests/qemu-iotests/250.out.bad 2019-04-22
08:52:26.072968731 -0500

(quoted diff) +1 -0
@@ -14,8 +14,9 @@
  virtual size: 2.05 GiB (2202009600 bytes)
  disk size: 24 MiB
  cluster_size: 1048576
[email protected]:qcow2_process_discards_failed_region offset

0x500000 bytes 0x82a00000 ret -5
Image resized.
virtual size: 5 MiB (5242880 bytes)

(quoted diff) +1 -1
-disk size: 9.01 MiB
+disk size: 19 MiB

where the trace did indeed show show that we had a bug, but where even without the trace, the difference in size between 19M with incomplete discards vs. 9M with patch 2/3 is enough to show that patch 2/3 fixes a bug.

Review

Quoted 7 lines
(quoted diff) +7 -0
+virtual size: 2.1G (2202009600 bytes)
+disk size: 24M
+cluster_size: 1048576
+Image resized.
+virtual size: 5.0M (5242880 bytes)
+disk size: 9.0M
+cluster_size: 1048576

When Kevin's block-next branch is applied, you'll need this squashed in:

tests/qemu-iotests/250.out +4 -4
index 37e37f0c9e7..d1c1c7180b5 100644
@@ -11,11 +11,11 @@ wrote 10485760/10485760 bytes at offset 0
  Offset          Length          Mapped to       File
  0               0xa00000        0x82f00000      TEST_DIR/t.qcow2
  0x82a00000      0xa00000        0x500000        TEST_DIR/t.qcow2
-virtual size: 2.1G (2202009600 bytes)
-disk size: 24M
+virtual size: 2.05 GiB (2202009600 bytes)
+disk size: 24 MiB
  cluster_size: 1048576
  Image resized.
-virtual size: 5.0M (5242880 bytes)
-disk size: 9.0M
+virtual size: 5 MiB (5242880 bytes)
+disk size: 9.01 MiB
  cluster_size: 1048576
  *** done
Quoted 6 lines

With the updated output to match changes to qemu-img output, the grammar fixes, and with the -T option removed,

Thank you!

I'll resend soon.

Signature · 2 lines
Best regards,
Vladimir