Skip to contentSkip to conversation

block/monitor: force read-only on live-added persistent bitmap

PATCH In review Denis V. Lunev 4 participants 4 messages Jul 6 – Jul 13 Block layer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 block/monitor: force read-only on live-added persistent bitmap

qmp_block_dirty_bitmap_add() marks a new bitmap persistent without checking write access to its node. bdrv_create_dirty_bitmap() always creates bitmaps writable, so a persistent bitmap added live to an already read-only node stays writable in memory on a node that can never store it. The next global inactivation (migration completion, save, or any other bdrv_inactivate_all() caller) then fails trying to flush it back:

Lost persistent bitmaps during inactivation of node '<node>': No write access migration_block_inactivate: bdrv_inactivate_all() failed: -22

qcow2_load_dirty_bitmaps() and qcow2_reopen_bitmaps_ro() already force a persistent bitmap read-only when its node is read-only; do the same right after the bitmap is marked persistent, whether it arrived through a plain block-dirty-bitmap-add or the transaction action that calls the same function. qcow2_reopen_bitmaps_rw() already promotes any persistent, read-only bitmap back to writable once its node is reopened read-write, so the reverse direction needs no changes.

Also check BDRV_O_INACTIVE, matching qcow2's private can_write() helper, to cover adding a bitmap to a node that is already inactive. Document the new behavior on BlockDirtyBitmapAdd.persistent.

Extend remove-bitmap-from-backing to cover this: add a persistent bitmap live to the already read-only "base" node, then inactivate all nodes the way migration completion, virsh save and qemu-update do, and check the bitmap ends up as protected as one already on disk.

CC Eric Blake <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
CC John Snow <[email protected]>
---
 block/monitor/bitmap-qmp-cmds.c                  |  9 +++++++++
 qapi/block-core.json                             |  4 +++-
 .../tests/remove-bitmap-from-backing             | 16 ++++++++++++++++
 .../tests/remove-bitmap-from-backing.out         | 11 +++++++++++
 4 files changed, 39 insertions(+), 1 deletion(-)
block/monitor/bitmap-qmp-cmds.c +9 -0
index a738e7bbf7..1f2c0c9cbc 100644
@@ -141,6 +141,15 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
     }
 
     bdrv_dirty_bitmap_set_persistence(bitmap, persistent);
+
+    /*
+     * A persistent bitmap on a node that can't be written must not be
+     * writable either, same as one loaded from or reopened onto it.
+     */
+    if (persistent &&
+        (bdrv_is_read_only(bs) || (bdrv_get_flags(bs) & BDRV_O_INACTIVE))) {
+        bdrv_dirty_bitmap_set_readonly(bitmap, true);
+    }
 }
 
 BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name,
qapi/block-core.json +3 -1
index 1f87b07850..a52397922b 100644
@@ -2353,7 +2353,9 @@
 # @persistent: the bitmap is persistent, i.e. it will be saved to the
 #     corresponding block device image file on its close.  For now
 #     only Qcow2 disks support persistent bitmaps.  Default is false
-#     for `block-dirty-bitmap-add`.  (Since: 2.10)
+#     for `block-dirty-bitmap-add`.  If the node is read-only, the
+#     bitmap is created read-only instead, the same as a persistent
+#     bitmap already stored on such a node.  (Since: 2.10)
 #
 # @disabled: the bitmap is created in the disabled state, which means
 #     that it will not track drive changes.  The bitmap may be enabled
tests/qemu-iotests/tests/remove-bitmap-from-backing +16 -0
index 15be32dcb9..61e1f3f10f 100755
@@ -66,6 +66,22 @@ result = vm.qmp('blockdev-reopen', **new_base_opts)
 if result != {'return': {}}:
     log('Failed to reopen: ' + str(result))
 
+# A bitmap added live (not loaded from disk) must be protected the same
+# way as bitmap0 above once its node is read-only.
+log('Adding a persistent bitmap live to the r-o base node:')
+vm.qmp_log('block-dirty-bitmap-add', node='base', name='bitmap1',
+          persistent=True)
+
+# Inactivate globally, like migration completion/save/qemu-update do;
+# inactivating "base" directly would hit an unrelated guard, since it is
+# a backing child of the still-active "top".
+log('Inactivating all nodes must not fail flushing bitmap1:')
+vm.qmp_log('blockdev-set-active', active=False)
+vm.qmp_log('blockdev-set-active', active=True)
+
+log('Trying to remove the live-added bitmap, should fail same as bitmap0:')
+vm.qmp_log('block-dirty-bitmap-remove', node='base', name='bitmap1')
+
 vm.shutdown()
 
 if 'bitmaps' in qemu_img_info(base)['format-specific']['data']:
tests/qemu-iotests/tests/remove-bitmap-from-backing.out +11 -0
index c28af82c75..ea492d167f 100644
@@ -4,3 +4,14 @@ Trying to remove persistent bitmap from r-o base node, should fail:
 Remove persistent bitmap from base node reopened to RW:
 {"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "base"}}
 {"return": {}}
+Adding a persistent bitmap live to the r-o base node:
+{"execute": "block-dirty-bitmap-add", "arguments": {"name": "bitmap1", "node": "base", "persistent": true}}
+{"return": {}}
+Inactivating all nodes must not fail flushing bitmap1:
+{"execute": "blockdev-set-active", "arguments": {"active": false}}
+{"return": {}}
+{"execute": "blockdev-set-active", "arguments": {"active": true}}
+{"return": {}}
+Trying to remove the live-added bitmap, should fail same as bitmap0:
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap1", "node": "base"}}
+{"error": {"class": "GenericError", "desc": "Bitmap 'bitmap1' is readonly and cannot be modified"}}
Signature · 1 line
2.53.0
#1 lore raw
Quoted 8 lines · Denis V. Lunev

qmp_block_dirty_bitmap_add() marks a new bitmap persistent without checking write access to its node. bdrv_create_dirty_bitmap() always creates bitmaps writable, so a persistent bitmap added live to an already read-only node stays writable in memory on a node that can never store it. The next global inactivation (migration completion, save, or any other bdrv_inactivate_all() caller) then fails trying to flush it back:

Quoted 4 lines

Lost persistent bitmaps during inactivation of node '<node>': No write access migration_block_inactivate: bdrv_inactivate_all() failed: -22

qcow2_load_dirty_bitmaps()

Hm, but qcow2_load_dirty_bitmaps() and qmp_block_dirty_bitmap_add() are different:

In qcow2_load_dirty_bitmaps(), we load bitmap from the image, so we are sure, that the bitmap is already stored. And if image is readonly, we mark bitmap readonly as well. So, on exit, we assume, that bitmap is already stored, and don't produce "Lost bitmaps" error.

In qmp_block_dirty_bitmap_add(), we create a new bitmap, that not yet stored. So, even if we don't modify it, the error "Lost bitmap" would be correct on exit, as bitmap is not stored at all.

---

Also, this patch breaks the scenario of create and than merge some bitmap into created one. That's a breaking change, and would need a deprecation period.

---

Still. Maybe better be more strict, and simply forbid creating persistent bitmaps in read-only images at all? (this will need a deprecation period anyway).

What is your scenario?

Quoted 109 lines

and qcow2_reopen_bitmaps_ro() already force
a persistent bitmap read-only when its node is read-only; do the same right after the bitmap is marked persistent, whether it arrived through a plain block-dirty-bitmap-add or the transaction action that calls the same function. qcow2_reopen_bitmaps_rw() already promotes any persistent, read-only bitmap back to writable once its node is reopened read-write, so the reverse direction needs no changes.

Also check BDRV_O_INACTIVE, matching qcow2's private can_write() helper, to cover adding a bitmap to a node that is already inactive. Document the new behavior on BlockDirtyBitmapAdd.persistent.

Extend remove-bitmap-from-backing to cover this: add a persistent bitmap live to the already read-only "base" node, then inactivate all nodes the way migration completion, virsh save and qemu-update do, and check the bitmap ends up as protected as one already on disk.

CC Eric Blake <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
CC John Snow <[email protected]>
---
  block/monitor/bitmap-qmp-cmds.c                  |  9 +++++++++
  qapi/block-core.json                             |  4 +++-
  .../tests/remove-bitmap-from-backing             | 16 ++++++++++++++++
  .../tests/remove-bitmap-from-backing.out         | 11 +++++++++++
  4 files changed, 39 insertions(+), 1 deletion(-)
block/monitor/bitmap-qmp-cmds.c +9 -0
index a738e7bbf7..1f2c0c9cbc 100644
@@ -141,6 +141,15 @@ void qmp_block_dirty_bitmap_add(const char *node, const char *name,
      }
 
      bdrv_dirty_bitmap_set_persistence(bitmap, persistent);
+
+    /*
+     * A persistent bitmap on a node that can't be written must not be
+     * writable either, same as one loaded from or reopened onto it.
+     */
+    if (persistent &&
+        (bdrv_is_read_only(bs) || (bdrv_get_flags(bs) & BDRV_O_INACTIVE))) {
+        bdrv_dirty_bitmap_set_readonly(bitmap, true);
+    }
  }
 
  BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name,
qapi/block-core.json +3 -1
index 1f87b07850..a52397922b 100644
@@ -2353,7 +2353,9 @@
  # @persistent: the bitmap is persistent, i.e. it will be saved to the
  #     corresponding block device image file on its close.  For now
  #     only Qcow2 disks support persistent bitmaps.  Default is false
-#     for `block-dirty-bitmap-add`.  (Since: 2.10)
+#     for `block-dirty-bitmap-add`.  If the node is read-only, the
+#     bitmap is created read-only instead, the same as a persistent
+#     bitmap already stored on such a node.  (Since: 2.10)
  #
  # @disabled: the bitmap is created in the disabled state, which means
  #     that it will not track drive changes.  The bitmap may be enabled
tests/qemu-iotests/tests/remove-bitmap-from-backing +16 -0
index 15be32dcb9..61e1f3f10f 100755
@@ -66,6 +66,22 @@ result = vm.qmp('blockdev-reopen', **new_base_opts)
  if result != {'return': {}}:
      log('Failed to reopen: ' + str(result))
 
+# A bitmap added live (not loaded from disk) must be protected the same
+# way as bitmap0 above once its node is read-only.
+log('Adding a persistent bitmap live to the r-o base node:')
+vm.qmp_log('block-dirty-bitmap-add', node='base', name='bitmap1',
+          persistent=True)
+
+# Inactivate globally, like migration completion/save/qemu-update do;
+# inactivating "base" directly would hit an unrelated guard, since it is
+# a backing child of the still-active "top".
+log('Inactivating all nodes must not fail flushing bitmap1:')
+vm.qmp_log('blockdev-set-active', active=False)
+vm.qmp_log('blockdev-set-active', active=True)
+
+log('Trying to remove the live-added bitmap, should fail same as bitmap0:')
+vm.qmp_log('block-dirty-bitmap-remove', node='base', name='bitmap1')
+
  vm.shutdown()
 
  if 'bitmaps' in qemu_img_info(base)['format-specific']['data']:
tests/qemu-iotests/tests/remove-bitmap-from-backing.out +11 -0
index c28af82c75..ea492d167f 100644
@@ -4,3 +4,14 @@ Trying to remove persistent bitmap from r-o base node, should fail:
  Remove persistent bitmap from base node reopened to RW:
  {"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap0", "node": "base"}}
  {"return": {}}
+Adding a persistent bitmap live to the r-o base node:
+{"execute": "block-dirty-bitmap-add", "arguments": {"name": "bitmap1", "node": "base", "persistent": true}}
+{"return": {}}
+Inactivating all nodes must not fail flushing bitmap1:
+{"execute": "blockdev-set-active", "arguments": {"active": false}}
+{"return": {}}
+{"execute": "blockdev-set-active", "arguments": {"active": true}}
+{"return": {}}
+Trying to remove the live-added bitmap, should fail same as bitmap0:
+{"execute": "block-dirty-bitmap-remove", "arguments": {"name": "bitmap1", "node": "base"}}
+{"error": {"class": "GenericError", "desc": "Bitmap 'bitmap1' is readonly and cannot be modified"}}
Signature · 2 lines
Best regards,
Vladimir
#2 lore raw
Quoted 31 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines · Denis V. Lunev

qmp_block_dirty_bitmap_add() marks a new bitmap persistent without checking write access to its node. bdrv_create_dirty_bitmap() always creates bitmaps writable, so a persistent bitmap added live to an already read-only node stays writable in memory on a node that can never store it. The next global inactivation (migration completion, save, or any other bdrv_inactivate_all() caller) then fails trying to flush it back:

Quoted 4 lines

Lost persistent bitmaps during inactivation of node '<node>': No write access migration_block_inactivate: bdrv_inactivate_all() failed: -22

qcow2_load_dirty_bitmaps()

Hm, but qcow2_load_dirty_bitmaps() and qmp_block_dirty_bitmap_add() are different:

In qcow2_load_dirty_bitmaps(), we load bitmap from the image, so we are sure, that the bitmap is already stored. And if image is readonly, we mark bitmap readonly as well. So, on exit, we assume, that bitmap is already stored, and don't produce "Lost bitmaps" error.

In qmp_block_dirty_bitmap_add(), we create a new bitmap, that not yet stored. So, even if we don't modify it, the error "Lost bitmap" would be correct on exit, as bitmap is not stored at all.

---

Also, this patch breaks the scenario of create and than merge some bitmap into created one. That's a breaking change, and would need a deprecation period.

Quoted 5 lines

---

Still. Maybe better be more strict, and simply forbid creating persistent bitmaps in read-only images at all? (this will need a deprecation period anyway).

this was also an option. We will need a couple of additional tweaks but the approach is ok.

I'll followup with new series.

Den

#3 lore raw
In reply to Denis V. Lunev
Quoted 11 lines · Denis V. Lunev

qmp_block_dirty_bitmap_add() marks a new bitmap persistent without checking write access to its node. bdrv_create_dirty_bitmap() always creates bitmaps writable, so a persistent bitmap added live to an already read-only node stays writable in memory on a node that can never store it. The next global inactivation (migration completion, save, or any other bdrv_inactivate_all() caller) then fails trying to flush it back:

Lost persistent bitmaps during inactivation of node '<node>': No write access migration_block_inactivate: bdrv_inactivate_all() failed: -22

Quoted 8 lines

qcow2_load_dirty_bitmaps() and qcow2_reopen_bitmaps_ro() already force a persistent bitmap read-only when its node is read-only; do the same right after the bitmap is marked persistent, whether it arrived through a plain block-dirty-bitmap-add or the transaction action that calls the same function. qcow2_reopen_bitmaps_rw() already promotes any persistent, read-only bitmap back to writable once its node is reopened read-write, so the reverse direction needs no changes.

As Vladimir noticed, we don't really get our non-persistent bitmap automatically converted to RW on reopen, as qcow2_reopen_bitmaps_rw() -> bitmap_list_load() only loads bitmaps actually stored in the image. Before that patch, after RW reopen we have both the node and the bitmap in RW. But with this patch the bitmap stays RO, and we can trigger an assert crash by issuing a backup. Here's a script which reproduces it:

Quoted 28 lines

#!/bin/bash
set -u

SRCDIR=$(dirname "$(readlink -f "$0")")
BUILD=$SRCDIR/build
QSD=$BUILD/storage-daemon/qemu-storage-daemon
TEST=$(mktemp -d)
QMPSHELL="$BUILD/run qmp-shell"
QMPSOCK=$TEST/qmp.sock
QSD_PID=

cleanup() {
    [ -n "$QSD_PID" ] && kill "$QSD_PID" 2>/dev/null
    rm -rf "$TEST"
}

trap cleanup EXIT

qemu-img create -f qcow2 "$TEST/tgt.qcow2" 1M >/dev/null qemu-img create -f qcow2 "$TEST/src.qcow2" 1M >/dev/null qemu-io -f qcow2 -c 'write 0 1M' "$TEST/src.qcow2" >/dev/null

"$QSD" \
    --blockdev file,node-name=tgtfile,filename="$TEST/tgt.qcow2" \
    --blockdev qcow2,node-name=tgt,file=tgtfile,read-only=on \
    --blockdev file,node-name=srcfile,filename="$TEST/src.qcow2" \
    --blockdev qcow2,node-name=src,file=srcfile \
    --chardev socket,id=qmp0,path=$QMPSOCK,server=on,wait=off \
    --monitor chardev=qmp0 &
Quoted 7 lines

QSD_PID=$!

$QMPSHELL -p $QMPSOCK <<EOF

    block-dirty-bitmap-add node=tgt name=bmp0 persistent=true
    blockdev-reopen options=[{"driver":"qcow2","node-name":"tgt","file":"tgtfile","read-only":false}]
    blockdev-backup job-id=job0 device=src target=tgt sync=full x-perf={"use-copy-range":true}

EOF

qemu-storage-daemon: ../block/dirty-bitmap.c:670: void bdrv_set_dirty(BlockDriverState *, int64_t, int64_t): Assertion `!bdrv_dirty_bitmap_readonly(bitmap)' failed.

Quoted 15 lines

Also check BDRV_O_INACTIVE, matching qcow2's private can_write() helper, to cover adding a bitmap to a node that is already inactive. Document the new behavior on BlockDirtyBitmapAdd.persistent.

Extend remove-bitmap-from-backing to cover this: add a persistent bitmap live to the already read-only "base" node, then inactivate all nodes the way migration completion, virsh save and qemu-update do, and check the bitmap ends up as protected as one already on disk.

CC Eric Blake <[email protected]>
CC Vladimir Sementsov-Ogievskiy <[email protected]>
CC John Snow <[email protected]>

---
[...]