Skip to contentSkip to conversation

block: Fix multiple blockdev-snapshot calls

PATCH In review Kevin Wolf 4 participants 12 messages Nov 8, 2019 – Nov 18, 2019

lore
Share & tools
lore

Kevin Wolf

author
#0 lore raw

cover block: Fix multiple blockdev-snapshot calls

Kevin Wolf (2):
block: Remove 'backing': null from bs->{explicit_,}options
iotests: Test multiple blockdev-snapshot calls

 block.c                    |   2 +
 tests/qemu-iotests/273     |  76 +++++++++
 tests/qemu-iotests/273.out | 337 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 4 files changed, 416 insertions(+)
 create mode 100755 tests/qemu-iotests/273
 create mode 100644 tests/qemu-iotests/273.out
Signature · 1 line
2.20.1
author
#1 lore raw

PATCH 1/2 block: Remove 'backing': null from bs->{explicit_, }options

bs->options and bs->explicit_options shouldn't contain any options for child nodes. bdrv_open_inherited() takes care to remove any options that match a child name after opening the image and the same is done when reopening.

However, we miss the case of 'backing': null, which is a child option, but results in no child being created. This means that a 'backing': null remains in bs->options and bs->explicit_options.

A typical use for 'backing': null is in live snapshots: blockdev-add for the qcow2 overlay makes sure not to open the backing file (because it is already opened and blockdev-snapshot will attach it). After doing a blockdev-snapshot, bs->options and bs->explicit_options become inconsistent with the actual state (bs has a backing file now, but the options still say null). On the next occasion that the image is reopened, e.g. switching it from read-write to read-only when another snapshot is taken, the option will take effect again and the node incorrectly loses its backing file.

Fix bdrv_open_inherited() to remove the 'backing' option from bs->options and bs->explicit_options even for the case where it specifies that no backing file is wanted.

---
 block.c | 2 ++
 1 file changed, 2 insertions(+)
block.c +2 -0
index dad5a3d8e0..74ba9acb08 100644
@@ -3019,6 +3019,8 @@ static BlockDriverState *bdrv_open_inherit(const char *filename,
                         "use \"backing\": null instead");
         }
         flags |= BDRV_O_NO_BACKING;
+        qdict_del(bs->explicit_options, "backing");
+        qdict_del(bs->options, "backing");
         qdict_del(options, "backing");
     }
 
Signature · 1 line
2.20.1
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] block: Remove 'backing': null from bs->{explicit_, }options

Quoted 19 lines · Kevin Wolf

bs->options and bs->explicit_options shouldn't contain any options for child nodes. bdrv_open_inherited() takes care to remove any options that match a child name after opening the image and the same is done when reopening.

However, we miss the case of 'backing': null, which is a child option, but results in no child being created. This means that a 'backing': null remains in bs->options and bs->explicit_options.

A typical use for 'backing': null is in live snapshots: blockdev-add for the qcow2 overlay makes sure not to open the backing file (because it is already opened and blockdev-snapshot will attach it). After doing a blockdev-snapshot, bs->options and bs->explicit_options become inconsistent with the actual state (bs has a backing file now, but the options still say null). On the next occasion that the image is reopened, e.g. switching it from read-write to read-only when another snapshot is taken, the option will take effect again and the node incorrectly loses its backing file.

Quoted 6 lines

Fix bdrv_open_inherited() to remove the 'backing' option from bs->options and bs->explicit_options even for the case where it specifies that no backing file is wanted.

Berto

Tested-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] block: Remove 'backing': null from bs->{explicit_,}options

In reply to Kevin Wolf
Quoted 11 lines · Kevin Wolf

bs->options and bs->explicit_options shouldn't contain any options for child nodes. bdrv_open_inherited() takes care to remove any options that match a child name after opening the image and the same is done when reopening.

However, we miss the case of 'backing': null, which is a child option, but results in no child being created. This means that a 'backing': null remains in bs->options and bs->explicit_options.

A typical use for 'backing': null is in live snapshots: blockdev-add for the qcow2 overlay makes sure not to open the backing file (because it is

Note that we also use '"backing": null' as a terminator for the last image in the chain if the user configures the chain manually.

This is kind-of a protection from opening the backing file from the header if it was misconfigured somehow. I think this functionality should be kept despite probably not making practical sense.

In my testing this scenario worked properly.

Quoted 8 lines

already opened and blockdev-snapshot will attach it). After doing a blockdev-snapshot, bs->options and bs->explicit_options become inconsistent with the actual state (bs has a backing file now, but the options still say null). On the next occasion that the image is reopened, e.g. switching it from read-write to read-only when another snapshot is taken, the option will take effect again and the node incorrectly loses its backing file.

Quoted 6 lines

Fix bdrv_open_inherited() to remove the 'backing' option from bs->options and bs->explicit_options even for the case where it specifies that no backing file is wanted.

The fix looks sane-enough to me and works as expected, but since I'm not familiar enough with this code I'm comfortable only with a:

Quoted 3 lines
---
 block.c | 2 ++
 1 file changed, 2 insertions(+)
author
#4 lore raw

PATCH 2/2 iotests: Test multiple blockdev-snapshot calls

In reply to Kevin Wolf

Test that doing a second blockdev-snapshot doesn't make the first overlay's backing file go away.

---
 tests/qemu-iotests/273     |  76 +++++++++
 tests/qemu-iotests/273.out | 337 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 414 insertions(+)
 create mode 100755 tests/qemu-iotests/273
 create mode 100644 tests/qemu-iotests/273.out
tests/qemu-iotests/273 new file +76 -0
new file mode 100755
index 0000000000..60076de7ce
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+#
+# Test large write to a qcow2 image
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+seq=$(basename "$0")
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+# This is a qcow2 regression test
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+do_run_qemu()
+{
+    echo Testing: "$@"
+    $QEMU -nographic -qmp-pretty stdio -nodefaults "$@"
+    echo
+}
+
+run_qemu()
+{
+    do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qemu | _filter_qmp |
+        _filter_generated_node_ids | _filter_imgfmt | _filter_actual_image_size
+}
+
+TEST_IMG="$TEST_IMG.base" _make_test_img 64M
+TEST_IMG="$TEST_IMG.mid" _make_test_img -b "$TEST_IMG.base"
+_make_test_img -b "$TEST_IMG.mid"
+
+run_qemu \
+    -blockdev file,node-name=base,filename="$TEST_IMG.base" \
+     -blockdev file,node-name=midf,filename="$TEST_IMG.mid" \
+     -blockdev '{"driver":"qcow2","node-name":"mid","file":"midf","backing":null}' \
+     -blockdev file,node-name=topf,filename="$TEST_IMG" \
+     -blockdev '{"driver":"qcow2","file":"topf","node-name":"top","backing":null}' \
+<<EOF
+{"execute":"qmp_capabilities"}
+{"execute":"blockdev-snapshot","arguments":{"node":"base","overlay":"mid"}}
+{"execute":"blockdev-snapshot","arguments":{"node":"mid","overlay":"top"}}
+{"execute":"query-named-block-nodes"}
+{"execute":"x-debug-query-block-graph"}
+{"execute":"quit"}
+EOF
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
tests/qemu-iotests/273.out new file +337 -0

337 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index af322af756..5aabdfefde 100644
@@ -282,3 +282,4 @@
 267 rw auto quick snapshot
 268 rw auto quick
 270 rw backing quick
+273 backing quick
Signature · 1 line
2.20.1
#5 lore raw

Re: [PATCH 2/2] iotests: Test multiple blockdev-snapshot calls

Quoted 1 line · Kevin Wolf

+# Test large write to a qcow2 image

This doesn't belong here I guess :)

I wonder if this test could go in 245 instead.

Berto

author
#6 lore raw

Re: [PATCH 2/2] iotests: Test multiple blockdev-snapshot calls

Am 08.11.2019 um 15:32 hat Alberto Garcia geschrieben:

Quoted 4 lines · Kevin Wolf
Quoted 1 line

+# Test large write to a qcow2 image

This doesn't belong here I guess :)

Yes, fixed.

Quoted 1 line

I wonder if this test could go in 245 instead.

The headline for 245 is "Test cases for the QMP 'x-blockdev-reopen' command", so I don't think so?

We end up testing the reopen code, but only as a side effect of snapshotting, not through x-blockdev-reopen.

Kevin

Reviewed-by on v1 2/2
#7 lore raw

Re: [PATCH 2/2] iotests: Test multiple blockdev-snapshot calls

In reply to Kevin Wolf
Quoted 11 lines · Kevin Wolf

Test that doing a second blockdev-snapshot doesn't make the first overlay's backing file go away.

---
 tests/qemu-iotests/273     |  76 +++++++++
 tests/qemu-iotests/273.out | 337 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 414 insertions(+)
 create mode 100755 tests/qemu-iotests/273
 create mode 100644 tests/qemu-iotests/273.out

Didn't apply cleanly for me.

tests/qemu-iotests/273 line 3

Quoted 10 lines
tests/qemu-iotests/273 new file +3 -0
new file mode 100755
index 0000000000..60076de7ce
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+#
+# Test large write to a qcow2 image

Cut&paste?

Rest looks good

Quoted 31 lines
tests/qemu-iotests/273 new file +31 -0
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+seq=$(basename "$0")
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter

tests/qemu-iotests/273 line 44

Quoted 10 lines
tests/qemu-iotests/273 new file +10 -0
+
+# This is a qcow2 regression test
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+do_run_qemu()
+{
+    echo Testing: "$@"
+    $QEMU -nographic -qmp-pretty stdio -nodefaults "$@"

-qmp-pretty, that's useful

Quoted 15 lines
tests/qemu-iotests/273 new file +15 -0
+    echo
+}
+
+run_qemu()
+{
+    do_run_qemu "$@" 2>&1 | _filter_testdir | _filter_qemu | _filter_qmp |
+        _filter_generated_node_ids | _filter_imgfmt | _filter_actual_image_size
+}
+
+TEST_IMG="$TEST_IMG.base" _make_test_img 64M
+TEST_IMG="$TEST_IMG.mid" _make_test_img -b "$TEST_IMG.base"
+_make_test_img -b "$TEST_IMG.mid"
+
+run_qemu \
+    -blockdev file,node-name=base,filename="$TEST_IMG.base" \

tests/qemu-iotests/273 line 69

Quoted 10 lines
tests/qemu-iotests/273 new file +10 -0
+     -blockdev file,node-name=midf,filename="$TEST_IMG.mid" \
+     -blockdev '{"driver":"qcow2","node-name":"mid","file":"midf","backing":null}' \
+     -blockdev file,node-name=topf,filename="$TEST_IMG" \
+     -blockdev '{"driver":"qcow2","file":"topf","node-name":"top","backing":null}' \
+<<EOF
+{"execute":"qmp_capabilities"}
+{"execute":"blockdev-snapshot","arguments":{"node":"base","overlay":"mid"}}
+{"execute":"blockdev-snapshot","arguments":{"node":"mid","overlay":"top"}}
+{"execute":"query-named-block-nodes"}
+{"execute":"x-debug-query-block-graph"}

Oh, this too!

Quoted 7 lines
tests/qemu-iotests/273 new file +7 -0
+{"execute":"quit"}
+EOF
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
author
#8 lore raw

Re: [PATCH 2/2] iotests: Test multiple blockdev-snapshot calls

Am 12.11.2019 um 17:07 hat Peter Krempa geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 11 lines

Test that doing a second blockdev-snapshot doesn't make the first overlay's backing file go away.

---
 tests/qemu-iotests/273     |  76 +++++++++
 tests/qemu-iotests/273.out | 337 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 414 insertions(+)
 create mode 100755 tests/qemu-iotests/273
 create mode 100644 tests/qemu-iotests/273.out

Didn't apply cleanly for me.

Quoted 1 line
Quoted 11 lines

tests/qemu-iotests/273 line 3

Quoted 9 lines
tests/qemu-iotests/273 new file +3 -0
new file mode 100755
index 0000000000..60076de7ce
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+#
+# Test large write to a qcow2 image

Cut&paste?

Indeed. I'll change it like this:

# Test multiple blockdev-snapshot calls with 'backing': null

Quoted 3 lines

Rest looks good

Thanks!

Kevin

#9 lore raw

Re: [PATCH 2/2] iotests: Test multiple blockdev-snapshot calls

In reply to Kevin Wolf
Quoted 11 lines · Kevin Wolf

Test that doing a second blockdev-snapshot doesn't make the first overlay's backing file go away.

---
 tests/qemu-iotests/273     |  76 +++++++++
 tests/qemu-iotests/273.out | 337 +++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |   1 +
 3 files changed, 414 insertions(+)
 create mode 100755 tests/qemu-iotests/273
 create mode 100644 tests/qemu-iotests/273.out

[...]

tests/qemu-iotests/273.out line 48

Quoted 8 lines
tests/qemu-iotests/273.out new file +8 -0
+                "format-specific": {
+                    "type": "IMGFMT",
+                    "data": {
+                        "compat": "1.1",
+                        "lazy-refcounts": false,
+                        "refcount-bits": 16,
+                        "corrupt": false
+                    }

I get a mismatch here with -o compat=0.10 or -o refcount_bits=1.

Max

#10 lore raw
In reply to Kevin Wolf
Quoted 3 lines · Kevin Wolf

Kevin Wolf (2):
block: Remove 'backing': null from bs->{explicit_,}options
iotests: Test multiple blockdev-snapshot calls

Hi,

I'm not sure how the freeze rules for qemu are at this point thus:

Will this patchset make it into 4.2. I argue it's a pretty important fix as it ends up in image corruption.

If it won't make into we will need to add a QMP feature flag to detect presence of the fix so that I can gate libvirt's support of blockdev with it. (This can also be done if we make it into 4.2 but I'll be okay infering it from other fixes present in 4.2).

Thanks

Peter

author
#11 lore raw

Am 18.11.2019 um 17:26 hat Peter Krempa geschrieben:

Quoted 9 lines · Kevin Wolf
Quoted 3 lines

Kevin Wolf (2):
block: Remove 'backing': null from bs->{explicit_,}options
iotests: Test multiple blockdev-snapshot calls

Hi,

I'm not sure how the freeze rules for qemu are at this point thus:

Quoted 7 lines

Will this patchset make it into 4.2. I argue it's a pretty important fix as it ends up in image corruption.

If it won't make into we will need to add a QMP feature flag to detect presence of the fix so that I can gate libvirt's support of blockdev with it. (This can also be done if we make it into 4.2 but I'll be okay infering it from other fixes present in 4.2).

Yes, this is planned for 4.2. I'm in the process of preparing a pull request right now.

Kevin