qemu-img: Support bitmap --merge into backing image
PATCH v2 1/1 qemu-img: Support bitmap --merge into backing image
If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a bitmap from top into base, qemu-img was failing with:
qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock Is another process using the image [base.qcow2]?
The easiest fix is to not open the entire backing chain of either image (source or destination); after all, the point of 'qemu-img bitmap' is solely to manipulate bitmaps directly within a single qcow2 image, and this is made more precise if we don't pay attention to other images in the chain that may happen to have a bitmap by the same name.
However, note that during normal usage, it is a feature that qemu will allow a bitmap from a backing image to be exposed by an overlay BDS; doing so makes it easier to perform incremental backup, where we have:
Base <- Active <- temporrary
\--block job ->/
with temporary being fed by a block-copy 'sync' job; when exposing temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and, when NOT using 'qemu-img bitmap', that bitmaps are indeed visible through a backing chain.
[1] Full disclosure: prior to the recent commit 374eedd1c4 and friends, we were NOT able to see bitmaps through filters, which meant that we actually did not have nice clean semantics for uniformly being able to pick up bitmaps from anywhere in the backing chain (seen as a change in behavior between qemu 4.1 and 4.2 at commit 00e30f05de, when block-copy swapped from a one-off to a filter). Which means libvirt was already coded to copy bitmaps around for the sake of older qemu, even though modern qemu no longer needs it. Oh well.
---
In v2:
- also use BDRV_O_NO_BACKING on source [Max]
- improved commit message [Max]
qemu-img.c | 11 ++++++-- tests/qemu-iotests/291 | 12 ++++++++ tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 3 deletions(-)
qemu-img.c +8 -3
@@ -4757,14 +4757,19 @@ static int img_bitmap(int argc, char **argv) filename = argv[optind]; bitmap = argv[optind +1 ]; - blk =img_open (image_opts, filename, fmt,BDRV_O_RDWR , false, false, - false); +/* +* No need to open backing chains; we will be manipulating bitmaps +* directly in this image without reference to image contents. +*/ + blk =img_open (image_opts, filename, fmt,BDRV_O_RDWR |BDRV_O_NO_BACKING , + false, false, false);if (!blk) { goto out; } bs =blk_bs (blk);if (src_filename) { - src =img_open (false, src_filename, src_fmt,0 , false, false, false); + src =img_open (false, src_filename, src_fmt,BDRV_O_NO_BACKING , + false, false, false);if (!src) { goto out; }
tests/qemu-iotests/291 +12 -0
@@ -91,6 +91,15 @@ $QEMU_IMG bitmap --remove --image-opts \ driver=$IMGFMT,file.driver=file,file.filename="$TEST_IMG" tmp _img_info --format-specific +echo +echo "=== Merge from top layer into backing image ===" +echo + +$QEMU_IMG rebase -u -F qcow2 -b "$TEST_IMG.base" "$TEST_IMG" +$QEMU_IMG bitmap --add --merge b2 -b "$TEST_IMG" -F $IMGFMT \ + -f $IMGFMT "$TEST_IMG.base" b3 +_img_info --format-specific --backing-chain + echo echo "=== Check bitmap contents ===" echo
@@ -107,6 +116,9 @@ $QEMU_IMG map --output=json --image-opts \ nbd_server_start_unix_socket -r -f qcow2 -B b2 "$TEST_IMG" $QEMU_IMG map --output=json --image-opts \ "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b2" | _filter_qemu_img_map +nbd_server_start_unix_socket -r -f qcow2 -B b3 "$TEST_IMG" +$QEMU_IMG map --output=json --image-opts \ + "$IMG,x-dirty-bitmap=qemu:dirty-bitmap:b3" | _filter_qemu_img_map # success, all done echo '*** done'
tests/qemu-iotests/291.out +56 -0
66 lines not on this page · raw message
Signature · 1 line
2.28.0
Quoted 13 lines · Eric Blake
If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a bitmap from top into base, qemu-img was failing with:
qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock Is another process using the image [base.qcow2]?
The easiest fix is to not open the entire backing chain of either image (source or destination); after all, the point of 'qemu-img bitmap' is solely to manipulate bitmaps directly within a single qcow2 image, and this is made more precise if we don't pay attention to other images in the chain that may happen to have a bitmap by the same name.
Quoted 5 lines
However, note that during normal usage, it is a feature that qemu will allow a bitmap from a backing image to be exposed by an overlay BDS; doing so makes it easier to perform incremental backup, where we have:
Base <- Active <- temporrary
*temporary
(Also it’s a bit strange that “Base” and “Active” are capitalized, but “temporary” isn’t)
Quoted 3 lines
\--block job ->/with temporary being fed by a block-copy 'sync' job; when exposing
s/block-copy 'sync'/backup 'sync=none'/?
Quoted 5 lines
temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and, when NOT using 'qemu-img bitmap', that bitmaps are indeed visible through a backing chain.
Well. It is useful over NBD but I would doubt that it isn’t useful in general. For example, the QMP commands that refer to bitmaps always do so through a node-name + bitmap-name combination, and they require that the given bitmap is exactly on the given node.
So I think this is a very much a case-by-case question. (And in practice, NBD seems to be the outlier, not qemu-img bitmap.)
Quoted 14 lines
[1] Full disclosure: prior to the recent commit
374eedd1c4and friends, we were NOT able to see bitmaps through filters, which meant that we actually did not have nice clean semantics for uniformly being able to pick up bitmaps from anywhere in the backing chain (seen as a change in behavior between qemu 4.1 and 4.2 at commit00e30f05de, when block-copy swapped from a one-off to a filter). Which means libvirt was already coded to copy bitmaps around for the sake of older qemu, even though modern qemu no longer needs it. Oh well.Reported-by Eyal Shenitzky <[email protected]>Signed-off-by Eric Blake <[email protected]>---
Quoted 8 lines
In v2:
- also useBDRV_O_NO_BACKINGon source [Max]
- improved commit message [Max]qemu-img.c | 11 ++++++-- tests/qemu-iotests/291 | 12 ++++++++ tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 3 deletions(-)
The code looks good to me, but I wonder whether in the commit message it should be noted that we don’t want to let bitmaps from deeper nodes shine through by default everywhere, but just in specific cases where that’s useful (i.e. only NBD so far AFAIA).
Max
Quoted 22 lines · Max Reitz
Quoted 13 lines · Eric Blake
If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a bitmap from top into base, qemu-img was failing with:
qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock Is another process using the image [base.qcow2]?
The easiest fix is to not open the entire backing chain of either image (source or destination); after all, the point of 'qemu-img bitmap' is solely to manipulate bitmaps directly within a single qcow2 image, and this is made more precise if we don't pay attention to other images in the chain that may happen to have a bitmap by the same name.
Quoted 5 lines
However, note that during normal usage, it is a feature that qemu will allow a bitmap from a backing image to be exposed by an overlay BDS; doing so makes it easier to perform incremental backup, where we have:
Base <- Active <- temporrary
*temporary
Quoted 8 lines
(Also it’s a bit strange that “Base” and “Active” are capitalized, but “temporary” isn’t)
Quoted 3 lines
\--block job ->/with temporary being fed by a block-copy 'sync' job; when exposing
s/block-copy 'sync'/backup 'sync=none'/?
Will fix both.
Quoted 7 lines
Quoted 6 lines
temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and, when NOT using 'qemu-img bitmap', that bitmaps are indeed visible through a backing chain.
Quoted 8 lines
Well. It is useful over NBD but I would doubt that it isn’t useful in general. For example, the QMP commands that refer to bitmaps always do so through a node-name + bitmap-name combination, and they require that the given bitmap is exactly on the given node.
So I think this is a very much a case-by-case question. (And in practice, NBD seems to be the outlier, not qemu-img bitmap.)
I'm happy to reword slightly to give that caveat.
Quoted 23 lines
Quoted 14 lines
[1] Full disclosure: prior to the recent commit
374eedd1c4and friends, we were NOT able to see bitmaps through filters, which meant that we actually did not have nice clean semantics for uniformly being able to pick up bitmaps from anywhere in the backing chain (seen as a change in behavior between qemu 4.1 and 4.2 at commit00e30f05de, when block-copy swapped from a one-off to a filter). Which means libvirt was already coded to copy bitmaps around for the sake of older qemu, even though modern qemu no longer needs it. Oh well.Reported-by Eyal Shenitzky <[email protected]>Signed-off-by Eric Blake <[email protected]>---
Quoted 8 lines
In v2:
- also useBDRV_O_NO_BACKINGon source [Max]
- improved commit message [Max]qemu-img.c | 11 ++++++-- tests/qemu-iotests/291 | 12 ++++++++ tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 76 insertions(+), 3 deletions(-)
Quoted 4 lines
The code looks good to me, but I wonder whether in the commit message it should be noted that we don’t want to let bitmaps from deeper nodes shine through by default everywhere, but just in specific cases where that’s useful (i.e. only NBD so far AFAIA).
So is this a Reviewed-by? I'm happy to queue it through my bitmaps tree, if so.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Max Reitz
Reviewed-byQuoted 76 lines · Eric Blake
Quoted 23 lines · Max Reitz
Quoted 14 lines · Eric Blake
If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a bitmap from top into base, qemu-img was failing with:
qemu-img: Could not open 'top.qcow2': Could not open backing file:
Failed to get shared "write" lock
Is another process using the image [base.qcow2]?The easiest fix is to not open the entire backing chain of either image (source or destination); after all, the point of 'qemu-img bitmap' is solely to manipulate bitmaps directly within a single qcow2 image, and this is made more precise if we don't pay attention to other images in the chain that may happen to have a bitmap by the same name.
Quoted 5 lines
However, note that during normal usage, it is a feature that qemu will allow a bitmap from a backing image to be exposed by an overlay BDS; doing so makes it easier to perform incremental backup, where we have:
Base <- Active <- temporrary
*temporary
Quoted 8 lines
(Also it’s a bit strange that “Base” and “Active” are capitalized, but “temporary” isn’t)
Quoted 3 lines
\--block job ->/
with temporary being fed by a block-copy 'sync' job; when exposing
s/block-copy 'sync'/backup 'sync=none'/?
Will fix both.
Quoted 7 lines
Quoted 6 lines
temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and, when NOT using 'qemu-img bitmap', that bitmaps are indeed visible through a backing chain.
Quoted 8 lines
Well. It is useful over NBD but I would doubt that it isn’t useful in general. For example, the QMP commands that refer to bitmaps always do so through a node-name + bitmap-name combination, and they require that the given bitmap is exactly on the given node.
So I think this is a very much a case-by-case question. (And in practice, NBD seems to be the outlier, not qemu-img bitmap.)
I'm happy to reword slightly to give that caveat.
Quoted 23 lines
Quoted 22 lines
[1] Full disclosure: prior to the recent commit
374eedd1c4and friends, we were NOT able to see bitmaps through filters, which meant that we actually did not have nice clean semantics for uniformly being able to pick up bitmaps from anywhere in the backing chain (seen as a change in behavior between qemu 4.1 and 4.2 at commit00e30f05de, when block-copy swapped from a one-off to a filter). Which means libvirt was already coded to copy bitmaps around for the sake of older qemu, even though modern qemu no longer needs it. Oh well.Reported-by Eyal Shenitzky <[email protected]>Signed-off-by Eric Blake <[email protected]>---
In v2:
- also useBDRV_O_NO_BACKINGon source [Max]
- improved commit message [Max]qemu-img.c | 11 ++++++-- tests/qemu-iotests/291 | 12 ++++++++ tests/qemu-iotests/291.out | 56 ++++++++++++++++++++++++++++++++++++++
3 files changed, 76 insertions(+), 3 deletions(-)
Quoted 7 lines
Quoted 4 lines
The code looks good to me, but I wonder whether in the commit message it should be noted that we don’t want to let bitmaps from deeper nodes shine through by default everywhere, but just in specific cases where that’s useful (i.e. only NBD so far AFAIA).
So is this a Reviewed-by? I'm happy to queue it through my bitmaps tree, if so.
It wasn’t meant as an R-b, because my R-b depends on how the commit message addresses the question of when exactly bitmaps from the backing chain should be visible on the top image. Whether qemu-img bitmap is an exception, or whether this is really a case-by-case question.
(I wanted to know whether you agree on including it. Normally, I’m happy to give an R-b on the basis of “with that done”, but in this case I wasn’t entirely sure whether my request was reasonable, but I also felt that in case it was, it wasn’t optional, given that you do have an entire paragraph in the commit message dedicated to why the backing image’s bitmap is visible on an image exported over NBD.)
I have to say I would like to see how you do phrase it in the end, but given that you do agree on including it, I can give a
Now.
Quoted 31 lines · Max Reitz
Quoted 18 lines
Quoted 6 lines
Quoted 5 lines
temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and, when NOT using 'qemu-img bitmap', that bitmaps are indeed visible through a backing chain.
Quoted 8 lines
Well. It is useful over NBD but I would doubt that it isn’t useful in general. For example, the QMP commands that refer to bitmaps always do so through a node-name + bitmap-name combination, and they require that the given bitmap is exactly on the given node.
So I think this is a very much a case-by-case question. (And in practice, NBD seems to be the outlier, not qemu-img bitmap.)
I'm happy to reword slightly to give that caveat.
Quoted 7 lines
Quoted 4 lines
The code looks good to me, but I wonder whether in the commit message it should be noted that we don’t want to let bitmaps from deeper nodes shine through by default everywhere, but just in specific cases where that’s useful (i.e. only NBD so far AFAIA).
So is this a Reviewed-by? I'm happy to queue it through my bitmaps tree, if so.
It wasn’t meant as an R-b, because my R-b depends on how the commit message addresses the question of when exactly bitmaps from the backing chain should be visible on the top image. Whether qemu-img bitmap is an exception, or whether this is really a case-by-case question.
Quoted 6 lines
(I wanted to know whether you agree on including it. Normally, I’m happy to give an R-b on the basis of “with that done”, but in this case I wasn’t entirely sure whether my request was reasonable, but I also felt that in case it was, it wasn’t optional, given that you do have an entire paragraph in the commit message dedicated to why the backing image’s bitmap is visible on an image exported over NBD.)
Here's my rewording:
However, note that on a case-by-case analysis, there _are_ times where we treat it as a feature that we can access a bitmap from a backing layer in association with an overlay BDS. A demonstration of this is using NBD to expose both an overlay BDS (for constant contents) and a bitmap (for learning which blocks are interesting) during an incremental backup:
Base <- Active <- Temporary
\--block job ->/
where Temporary is being fed by a backup 'sync=none' job. When exposing Temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into Temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and that qemu-nbd is indeed able to access a bitmap inherited from the backing chain since it is a different use case than 'qemu-img bitmap'.
Quoted 7 lines
I have to say I would like to see how you do phrase it in the end, but given that you do agree on including it, I can give a
Reviewed-by Max Reitz <[email protected]>Now.
Okay, I think I've met your request, so I'll go ahead and send the pull request today.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Quoted 36 lines · Eric Blake
If you have the chain 'base.qcow2 <- top.qcow2' and want to merge a bitmap from top into base, qemu-img was failing with:
qemu-img: Could not open 'top.qcow2': Could not open backing file: Failed to get shared "write" lock Is another process using the image [base.qcow2]?
The easiest fix is to not open the entire backing chain of either image (source or destination); after all, the point of 'qemu-img bitmap' is solely to manipulate bitmaps directly within a single qcow2 image, and this is made more precise if we don't pay attention to other images in the chain that may happen to have a bitmap by the same name.
However, note that during normal usage, it is a feature that qemu will allow a bitmap from a backing image to be exposed by an overlay BDS; doing so makes it easier to perform incremental backup, where we have:
Base <- Active <- temporrary
\--block job ->/with temporary being fed by a block-copy 'sync' job; when exposing temporary over NBD, referring to a bitmap that lives only in Active is less effort than having to copy a bitmap into temporary [1]. So the testsuite additions in this patch check both where bitmaps get allocated (the qemu-img info output), and, when NOT using 'qemu-img bitmap', that bitmaps are indeed visible through a backing chain.
[1] Full disclosure: prior to the recent commit
374eedd1c4and friends, we were NOT able to see bitmaps through filters, which meant that we actually did not have nice clean semantics for uniformly being able to pick up bitmaps from anywhere in the backing chain (seen as a change in behavior between qemu 4.1 and 4.2 at commit00e30f05de, when block-copy swapped from a one-off to a filter). Which means libvirt was already coded to copy bitmaps around for the sake of older qemu, even though modern qemu no longer needs it. Oh well.
Quoted 4 lines
Reported-by Eyal Shenitzky <[email protected]>Signed-off-by Eric Blake <[email protected]>---
Honestly I don't want to bother with carefully checking new test output, at least I see that it doesn't produce errors:) Code change seems obvious.
Signature · 2 lines
Best regards, Vladimir