Skip to contentSkip to conversation

bitmaps: allow bitmaps to be used with full and top

PATCH Superseded John Snow 2 participants 35 messages Jul 10, 2019 – Jul 11, 2019 1 message not in the archive Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover bitmaps: allow bitmaps to be used with full and top

Requires: <[email protected]> [PATCH v4 00/18] bitmaps: introduce 'bitmap' sync mode

This follows the previous series which adds the 'bitmap' sync mode and uses it to add interactions with bitmaps to the 'full' and 'top' modes to blockdev-backup and drive-backup.

Why?
on-success: Can conveniently synchronize a bitmap to a full backup.
Allows for transactionless anchor backups. Allows us to attempt an anchor backup without damaging
our bitmap until the backup is successful.
Allows for transactional, ungrouped anchor backups.
always: Allows us to resume full/top style backups with a later
invocation to sync=bitmap. Neat!

Summary:
1-3: Refactor iotest 257 to accommodate this;
4-5: Augment 257 to test trivial failure cases
6-7: Implement feature
8: Test new feature

John Snow (8):
iotests/257: add Pattern class
iotests/257: add EmulatedBitmap class
iotests/257: Refactor backup helpers
block/backup: hoist bitmap check into QMP interface iotests/257: test API failures
block/backup: issue progress updates for skipped regions block/backup: support bitmap sync modes for non-bitmap backups iotests/257: test traditional sync modes

 block/backup.c             |   22 +-
 blockdev.c                 |   32 +
 qapi/block-core.json       |    6 +-
 tests/qemu-iotests/257     |  329 +++-
 tests/qemu-iotests/257.out | 3366 +++++++++++++++++++++++++++++++++++-
 5 files changed, 3548 insertions(+), 207 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH 1/8 iotests/257: add Pattern class

Just kidding, this is easier to manage with a full class instead of a namedtuple.

---
 tests/qemu-iotests/257 | 58 +++++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 26 deletions(-)
tests/qemu-iotests/257 +32 -26
index 75a651c7c3..f576a35a5c 100755
@@ -19,7 +19,6 @@
 #
 # [email protected]
 
-from collections import namedtuple
 import math
 import os
 
@@ -29,10 +28,18 @@ from iotests import log, qemu_img
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
-Pattern = namedtuple('Pattern', ['byte', 'offset', 'size'])
-def mkpattern(byte, offset, size=GRANULARITY):
-    """Constructor for Pattern() with default size"""
-    return Pattern(byte, offset, size)
+
+class Pattern:
+    def __init__(self, byte, offset, size=GRANULARITY):
+        self.byte = byte
+        self.offset = offset
+        self.size = size
+
+    def bits(self, granularity):
+        lower = math.floor(self.offset / granularity)
+        upper = math.floor((self.offset + self.size - 1) / granularity)
+        return set(range(lower, upper + 1))
+
 
 class PatternGroup:
     """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
@@ -43,40 +50,39 @@ class PatternGroup:
         """Calculate the unique bits dirtied by this pattern grouping"""
         res = set()
         for pattern in self.patterns:
-            lower = math.floor(pattern.offset / granularity)
-            upper = math.floor((pattern.offset + pattern.size - 1) / granularity)
-            res = res | set(range(lower, upper + 1))
+            res = res | pattern.bits(granularity)
         return res
 
+
 GROUPS = [
     PatternGroup([
         # Batch 0: 4 clusters
-        mkpattern('0x49', 0x0000000),
-        mkpattern('0x6c', 0x0100000),   # 1M
-        mkpattern('0x6f', 0x2000000),   # 32M
-        mkpattern('0x76', 0x3ff0000)]), # 64M - 64K
+        Pattern('0x49', 0x0000000),
+        Pattern('0x6c', 0x0100000),   # 1M
+        Pattern('0x6f', 0x2000000),   # 32M
+        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
     PatternGroup([
         # Batch 1: 6 clusters (3 new)
-        mkpattern('0x65', 0x0000000),   # Full overwrite
-        mkpattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
-        mkpattern('0x72', 0x2008000),   # Partial-right (32M+32K)
-        mkpattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
+        Pattern('0x65', 0x0000000),   # Full overwrite
+        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
+        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
+        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
     PatternGroup([
         # Batch 2: 7 clusters (3 new)
-        mkpattern('0x74', 0x0010000),   # Adjacent-right
-        mkpattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
-        mkpattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
-        mkpattern('0x67', 0x3fe0000,
-                  2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
+        Pattern('0x74', 0x0010000),   # Adjacent-right
+        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
+        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
+        Pattern('0x67', 0x3fe0000,
+                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
     PatternGroup([
         # Batch 3: 8 clusters (5 new)
         # Carefully chosen such that nothing re-dirties the one cluster
         # that copies out successfully before failure in Group #1.
-        mkpattern('0xaa', 0x0010000,
-                  3*GRANULARITY),       # Overwrite and 2x Adjacent-right
-        mkpattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
-        mkpattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
-        mkpattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
+        Pattern('0xaa', 0x0010000,
+                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
+        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
+        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
+        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
 ]
 
 class Drive:
Signature · 1 line
2.21.0
Reviewed-by on v1 1/8
#2 lore raw

Re: [Qemu-devel] [PATCH 1/8] iotests/257: add Pattern class

Quoted 7 lines · John Snow

Just kidding, this is easier to manage with a full class instead of a namedtuple.

---
 tests/qemu-iotests/257 | 58 +++++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 26 deletions(-)
#3 lore raw

Re: [Qemu-devel] [PATCH 1/8] iotests/257: add Pattern class

In reply to John Snow
Quoted 29 lines · John Snow

Just kidding, this is easier to manage with a full class instead of a namedtuple.

---
 tests/qemu-iotests/257 | 58 +++++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 26 deletions(-)
tests/qemu-iotests/257 +1 -5
index 75a651c7c3..f576a35a5c 100755
@@ -19,7 +19,6 @@
 #
 # [email protected]
 
-from collections import namedtuple
 import math
 import os
 
@@ -29,10 +28,18 @@ from iotests import log, qemu_img
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
-Pattern = namedtuple('Pattern', ['byte', 'offset', 'size'])
-def mkpattern(byte, offset, size=GRANULARITY):
-    """Constructor for Pattern() with default size"""
-    return Pattern(byte, offset, size)
+

tests/qemu-iotests/257 from iotests import log, qemu_img · line 41

Quoted 10 lines
tests/qemu-iotests/257 +10 -0
+class Pattern:
+    def __init__(self, byte, offset, size=GRANULARITY):
+        self.byte = byte
+        self.offset = offset
+        self.size = size
+
+    def bits(self, granularity):
+        lower = math.floor(self.offset / granularity)
+        upper = math.floor((self.offset + self.size - 1) / granularity)
+        return set(range(lower, upper + 1))

By the way, this doesn’t work with Python2 (pre-existing in your other series). It complains that these are floats.

Now I don’t know whether you care but there is the fact that the expressions would be shorter if they were of the form x // y instead of math.floor(x / y).

Max

author
#4 lore raw

Re: [Qemu-devel] [PATCH 1/8] iotests/257: add Pattern class

Quoted 44 lines · Max Reitz
Quoted 29 lines · John Snow

Just kidding, this is easier to manage with a full class instead of a namedtuple.

---
 tests/qemu-iotests/257 | 58 +++++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 26 deletions(-)
tests/qemu-iotests/257 +1 -5
index 75a651c7c3..f576a35a5c 100755
@@ -19,7 +19,6 @@
 #
 # [email protected]
 
-from collections import namedtuple
 import math
 import os
 
@@ -29,10 +28,18 @@ from iotests import log, qemu_img
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
-Pattern = namedtuple('Pattern', ['byte', 'offset', 'size'])
-def mkpattern(byte, offset, size=GRANULARITY):
-    """Constructor for Pattern() with default size"""
-    return Pattern(byte, offset, size)
+

tests/qemu-iotests/257 from iotests import log, qemu_img · line 41

Quoted 10 lines
tests/qemu-iotests/257 +10 -0
+class Pattern:
+    def __init__(self, byte, offset, size=GRANULARITY):
+        self.byte = byte
+        self.offset = offset
+        self.size = size
+
+    def bits(self, granularity):
+        lower = math.floor(self.offset / granularity)
+        upper = math.floor((self.offset + self.size - 1) / granularity)
+        return set(range(lower, upper + 1))

By the way, this doesn’t work with Python2 (pre-existing in your other series). It complains that these are floats.

Quoted 6 lines

Now I don’t know whether you care but there is the fact that the expressions would be shorter if they were of the form x // y instead of math.floor(x / y).

Max

Ah, crud; OK -- I'll play nice with python2 for a while longer. Thanks for pointing this out.

author
#5 lore raw

PATCH 2/8 iotests/257: add EmulatedBitmap class

In reply to John Snow

Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.

---
 tests/qemu-iotests/257 | 125 +++++++++++++++++++++++++----------------
 1 file changed, 76 insertions(+), 49 deletions(-)
tests/qemu-iotests/257 +76 -49
index f576a35a5c..2ff4aa8695 100755
@@ -85,6 +85,60 @@ GROUPS = [
         Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
 ]
 
+
+class EmulatedBitmap:
+    def __init__(self, granularity=GRANULARITY):
+        self._bits = set()
+        self.groups = set()
+        self.granularity = granularity
+
+    def dirty_bits(self, bits):
+        self._bits |= set(bits)
+
+    def dirty_group(self, n):
+        self.dirty_bits(GROUPS[n].bits(self.granularity))
+
+    def clear(self):
+        self._bits = set()
+
+    def clear_bits(self, bits):
+        self._bits = self._bits - set(bits)
+
+    def clear_bit(self, bit):
+        self.clear_bits({bit})
+
+    def clear_group(self, n):
+        self.clear_bits(GROUPS[n].bits(self.granularity))
+
+    @property
+    def first_bit(self):
+        return sorted(self.bits)[0]
+
+    @property
+    def bits(self):
+        return self._bits
+
+    @property
+    def count(self):
+        return len(self.bits)
+
+    def compare(self, qmp_bitmap):
+        """
+        Print a nice human-readable message checking that a bitmap as reported
+        by the QMP interface has as many bits set as we expect it to.
+        """
+
+        name = qmp_bitmap.get('name', '(anonymous)')
+        log("= Checking Bitmap {:s} =".format(name))
+
+        want = self.count
+        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
+
+        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
+            want, have, "OK!" if want == have else "ERROR!"))
+        log('')
+
+
 class Drive:
     """Represents, vaguely, a drive attached to a VM.
     Includes format, graph, and device information."""
@@ -195,27 +249,6 @@ def perform_writes(drive, n):
     log('')
     return bitmaps
 
-def calculate_bits(groups=None):
-    """Calculate how many bits we expect to see dirtied."""
-    if groups:
-        bits = set.union(*(GROUPS[group].bits(GRANULARITY) for group in groups))
-        return len(bits)
-    return 0
-
-def bitmap_comparison(bitmap, groups=None, want=0):
-    """
-    Print a nice human-readable message checking that this bitmap has as
-    many bits set as we expect it to.
-    """
-    log("= Checking Bitmap {:s} =".format(bitmap.get('name', '(anonymous)')))
-
-    if groups:
-        want = calculate_bits(groups)
-    have = bitmap['count'] // bitmap['granularity']
-
-    log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
-        want, have, "OK!" if want == have else "ERROR!"))
-    log('')
 
 def compare_images(image, reference, baseimg=None, expected_match=True):
     """
@@ -321,12 +354,13 @@ def test_bitmap_sync(bsync_mode, failure=None):
         vm.qmp_log("block-dirty-bitmap-add", node=drive0.name,
                    name="bitmap0", granularity=GRANULARITY)
         log('')
+        ebitmap = EmulatedBitmap()
 
         # 1 - Writes and Reference Backup
         bitmaps = perform_writes(drive0, 1)
-        dirty_groups = {1}
+        ebitmap.dirty_group(1)
         bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
-        bitmap_comparison(bitmap, groups=dirty_groups)
+        ebitmap.compare(bitmap)
         reference_backup(drive0, 1, fbackup1)
 
         # 1 - Bitmap Backup (Optional induced failure)
@@ -342,54 +376,47 @@ def test_bitmap_sync(bsync_mode, failure=None):
             log('')
             bitmaps = perform_writes(drive0, 2)
             # Named bitmap (static, should be unchanged)
-            bitmap_comparison(get_bitmap(bitmaps, drive0.device, 'bitmap0'),
-                              groups=dirty_groups)
+            ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
             # Anonymous bitmap (dynamic, shows new writes)
-            bitmap_comparison(get_bitmap(bitmaps, drive0.device, '',
-                                         recording=True), groups={2})
-            dirty_groups.add(2)
+            anonymous = EmulatedBitmap()
+            anonymous.dirty_group(2)
+            anonymous.compare(get_bitmap(bitmaps, drive0.device, '',
+                                         recording=True))
+
+            # Simulate the order in which this will happen:
+            # group 1 gets cleared first, then group two gets written.
+            if ((bsync_mode == 'on-success' and not failure) or
+                (bsync_mode == 'always')):
+                ebitmap.clear_group(1)
+            ebitmap.dirty_group(2)
 
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
                    pre_finalize=_callback,
                    cancel=(failure == 'simulated'))
         bitmaps = query_bitmaps(vm)
-        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
         log(bitmaps, indent=2)
         log('')
 
-        if ((bsync_mode == 'on-success' and not failure) or
-                (bsync_mode == 'always' and failure != 'intermediate')):
-            dirty_groups.remove(1)
-
         if bsync_mode == 'always' and failure == 'intermediate':
             # We manage to copy one sector (one bit) before the error.
-            bitmap_comparison(bitmap,
-                              want=calculate_bits(groups=dirty_groups) - 1)
-        else:
-            bitmap_comparison(bitmap, groups=dirty_groups)
+            ebitmap.clear_bit(ebitmap.first_bit)
+        ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
         bitmaps = perform_writes(drive0, 3)
-        dirty_groups.add(3)
-        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
-        if bsync_mode == 'always' and failure == 'intermediate':
-            # We're one bit short, still.
-            bitmap_comparison(bitmap,
-                              want=calculate_bits(groups=dirty_groups) - 1)
-        else:
-            bitmap_comparison(bitmap, groups=dirty_groups)
+        ebitmap.dirty_group(3)
+        ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
         reference_backup(drive0, 2, fbackup2)
 
         # 2 - Bitmap Backup (In failure modes, this is a recovery.)
         job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode)
         vm.run_job(job, auto_dismiss=True, auto_finalize=False)
         bitmaps = query_bitmaps(vm)
-        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
         log(bitmaps, indent=2)
         log('')
-        bitmap_comparison(bitmap, groups={}
-                          if bsync_mode != 'never'
-                          else dirty_groups)
+        if bsync_mode != 'never':
+            ebitmap.clear()
+        ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         log('--- Cleanup ---\n')
         vm.qmp_log("block-dirty-bitmap-remove",
Signature · 1 line
2.21.0
Reviewed-by on v1 2/8
#6 lore raw

Re: [Qemu-devel] [PATCH 2/8] iotests/257: add EmulatedBitmap class

Quoted 25 lines · John Snow

Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.

---
 tests/qemu-iotests/257 | 125 +++++++++++++++++++++++++----------------
 1 file changed, 76 insertions(+), 49 deletions(-)
tests/qemu-iotests/257 +8 -0
index f576a35a5c..2ff4aa8695 100755
@@ -85,6 +85,60 @@ GROUPS = [
         Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
 ]
 
+
+class EmulatedBitmap:
+    def __init__(self, granularity=GRANULARITY):
+        self._bits = set()
+        self.groups = set()
+        self.granularity = granularity
+
+    def dirty_bits(self, bits):

tests/qemu-iotests/257 GROUPS = [ · line 105

Quoted 10 lines
tests/qemu-iotests/257 +10 -0
+        self._bits |= set(bits)
+
+    def dirty_group(self, n):
+        self.dirty_bits(GROUPS[n].bits(self.granularity))
+
+    def clear(self):
+        self._bits = set()
+
+    def clear_bits(self, bits):
+        self._bits = self._bits - set(bits)

Does -= not work here?

No real complaints. Sorry.

author
#7 lore raw

Re: [Qemu-devel] [PATCH 2/8] iotests/257: add EmulatedBitmap class

Quoted 39 lines · Max Reitz
Quoted 25 lines · John Snow

Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.

---
 tests/qemu-iotests/257 | 125 +++++++++++++++++++++++++----------------
 1 file changed, 76 insertions(+), 49 deletions(-)
tests/qemu-iotests/257 +8 -0
index f576a35a5c..2ff4aa8695 100755
@@ -85,6 +85,60 @@ GROUPS = [
         Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
 ]
 
+
+class EmulatedBitmap:
+    def __init__(self, granularity=GRANULARITY):
+        self._bits = set()
+        self.groups = set()
+        self.granularity = granularity
+
+    def dirty_bits(self, bits):

tests/qemu-iotests/257 GROUPS = [ · line 105

Quoted 10 lines
tests/qemu-iotests/257 +10 -0
+        self._bits |= set(bits)
+
+    def dirty_group(self, n):
+        self.dirty_bits(GROUPS[n].bits(self.granularity))
+
+    def clear(self):
+        self._bits = set()
+
+    def clear_bits(self, bits):
+        self._bits = self._bits - set(bits)

Does -= not work here?

Quoted 4 lines

No real complaints. Sorry.

It's okay. I forget which shorthand operators Python has for which types sometimes.

Quoted 3 lines
Quoted 3 lines
tests/qemu-iotests/257 +0 -0
 > a = {'a', 'b', 'c'}
 > a -= {'c'}
Quoted 1 line

a

{'b', 'a'}

Well, apparently it does. I'll change it.

--js

author
#8 lore raw

PATCH 3/8 iotests/257: Refactor backup helpers

In reply to John Snow

This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.

---
 tests/qemu-iotests/257     |  46 +++++----
 tests/qemu-iotests/257.out | 192 ++++++++++++++++++-------------------
 2 files changed, 124 insertions(+), 114 deletions(-)
tests/qemu-iotests/257 +28 -18
index 2ff4aa8695..2eb4f26c28 100755
@@ -208,6 +208,14 @@ def get_bitmap(bitmaps, drivename, name, recording=None):
                 return bitmap
     return None
 
+def blockdev_backup(vm, device, target, sync, **kwargs):
+    result = vm.qmp_log('blockdev-backup',
+                        device=device,
+                        target=target,
+                        sync=sync,
+                        **kwargs)
+    return result
+
 def reference_backup(drive, n, filepath):
     log("--- Reference Backup #{:d} ---\n".format(n))
     target_id = "ref_target_{:d}".format(n)
@@ -215,24 +223,26 @@ def reference_backup(drive, n, filepath):
     target_drive = Drive(filepath, vm=drive.vm)
 
     target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup",
-                     job_id=job_id, device=drive.name,
-                     target=target_id, sync="full")
+    blockdev_backup(drive.vm, drive.name, target_id, "full", job_id=job_id)
     drive.vm.run_job(job_id, auto_dismiss=True)
     log('')
 
-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode):
-    log("--- Bitmap Backup #{:d} ---\n".format(n))
-    target_id = "bitmap_target_{:d}".format(n)
-    job_id = "bitmap_backup_{:d}".format(n)
+def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'):
+    log("--- Test Backup #{:d} ---\n".format(n))
+    target_id = "backup_target_{:d}".format(n)
+    job_id = "backup_{:d}".format(n)
     target_drive = Drive(filepath, vm=drive.vm)
 
     target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name,
-                     target=target_id, sync="bitmap",
-                     bitmap_mode=bitmap_mode,
-                     bitmap=bitmap,
-                     auto_finalize=False)
+
+    kwargs = {
+        'job_id': job_id,
+        'auto_finalize': False,
+        'bitmap': bitmap,
+        'bitmap_mode': bitmap_mode,
+    }
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}
+    blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs)
     return job_id
 
 def perform_writes(drive, n):
@@ -264,7 +274,7 @@ def compare_images(image, reference, baseimg=None, expected_match=True):
         "OK!" if ret == expected_ret else "ERROR!"),
         filters=[iotests.filter_testfiles])
 
-def test_bitmap_sync(bsync_mode, failure=None):
+def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
     """
     Test bitmap backup routines.
 
@@ -292,7 +302,7 @@ def test_bitmap_sync(bsync_mode, failure=None):
                              fbackup0, fbackup1, fbackup2), \
          iotests.VM() as vm:
 
-        mode = "Bitmap Sync Mode {:s}".format(bsync_mode)
+        mode = "Mode {:s}; Bitmap Sync {:s}".format(msync_mode, bsync_mode)
         preposition = "with" if failure else "without"
         cond = "{:s} {:s}".format(preposition,
                                   "{:s} failure".format(failure) if failure
@@ -363,12 +373,12 @@ def test_bitmap_sync(bsync_mode, failure=None):
         ebitmap.compare(bitmap)
         reference_backup(drive0, 1, fbackup1)
 
-        # 1 - Bitmap Backup (Optional induced failure)
+        # 1 - Test Backup (w/ Optional induced failure)
         if failure == 'intermediate':
             # Activate blkdebug induced failure for second-to-next read
             log(vm.hmp_qemu_io(drive0.name, 'flush'))
             log('')
-        job = bitmap_backup(drive0, 1, bsync1, "bitmap0", bsync_mode)
+        job = backup(drive0, 1, bsync1, "bitmap0", bsync_mode, sync=msync_mode)
 
         def _callback():
             """Issue writes while the job is open to test bitmap divergence."""
@@ -409,7 +419,7 @@ def test_bitmap_sync(bsync_mode, failure=None):
         reference_backup(drive0, 2, fbackup2)
 
         # 2 - Bitmap Backup (In failure modes, this is a recovery.)
-        job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode)
+        job = backup(drive0, 2, bsync2, "bitmap0", bsync_mode)
         vm.run_job(job, auto_dismiss=True, auto_finalize=False)
         bitmaps = query_bitmaps(vm)
         log(bitmaps, indent=2)
@@ -443,7 +453,7 @@ def test_bitmap_sync(bsync_mode, failure=None):
 def main():
     for bsync_mode in ("never", "on-success", "always"):
         for failure in ("simulated", "intermediate", None):
-            test_bitmap_sync(bsync_mode, failure)
+            test_bitmap_sync(bsync_mode, "bitmap", failure)
 
 if __name__ == '__main__':
     iotests.script_main(main, supported_fmts=['qcow2'])
tests/qemu-iotests/257.out +96 -96

523 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v1 3/8
#9 lore raw

Re: [Qemu-devel] [PATCH 3/8] iotests/257: Refactor backup helpers

Quoted 14 lines · John Snow

This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.

---
 tests/qemu-iotests/257     |  46 +++++----
 tests/qemu-iotests/257.out | 192 ++++++++++++++++++-------------------
 2 files changed, 124 insertions(+), 114 deletions(-)
tests/qemu-iotests/257 +0 -0
index 2ff4aa8695..2eb4f26c28 100755

[...]

Quoted 14 lines
tests/qemu-iotests/257 +4 -7
-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode):
-    log("--- Bitmap Backup #{:d} ---\n".format(n))
-    target_id = "bitmap_target_{:d}".format(n)
-    job_id = "bitmap_backup_{:d}".format(n)
+def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'):
+    log("--- Test Backup #{:d} ---\n".format(n))
+    target_id = "backup_target_{:d}".format(n)
+    job_id = "backup_{:d}".format(n)
     target_drive = Drive(filepath, vm=drive.vm)
 
     target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name,
-                     target=target_id, sync="bitmap",
-                     bitmap_mode=bitmap_mode,

tests/qemu-iotests/257 def reference_backup(drive, n, filepath): · line 244

Quoted 10 lines
tests/qemu-iotests/257 +8 -2
-                     bitmap=bitmap,
-                     auto_finalize=False)
+
+    kwargs = {
+        'job_id': job_id,
+        'auto_finalize': False,
+        'bitmap': bitmap,
+        'bitmap_mode': bitmap_mode,
+    }
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}

I suppose this is to remove items that are None?

Very cute, but why not just

  kwargs = {
    'job_id': job_id,
    'auto_finalize': False,
  }
  if bitmap is not None:
    kwargs['bitmap'] = bitmap
    kwargs['bitmap_mode'] = bitmap_mode

Exactly the same number of lines, but immediately makes it clear what’s going on. Not as cute, I admit.

(Yes, I am indeed actively trying to train you not to write cute code.)

The rest looks good to me:

author
#10 lore raw

Re: [Qemu-devel] [PATCH 3/8] iotests/257: Refactor backup helpers

Quoted 55 lines · Max Reitz
Quoted 14 lines · John Snow

This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.

---
 tests/qemu-iotests/257     |  46 +++++----
 tests/qemu-iotests/257.out | 192 ++++++++++++++++++-------------------
 2 files changed, 124 insertions(+), 114 deletions(-)
tests/qemu-iotests/257 +0 -0
index 2ff4aa8695..2eb4f26c28 100755

[...]

Quoted 14 lines
tests/qemu-iotests/257 +4 -7
-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode):
-    log("--- Bitmap Backup #{:d} ---\n".format(n))
-    target_id = "bitmap_target_{:d}".format(n)
-    job_id = "bitmap_backup_{:d}".format(n)
+def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'):
+    log("--- Test Backup #{:d} ---\n".format(n))
+    target_id = "backup_target_{:d}".format(n)
+    job_id = "backup_{:d}".format(n)
     target_drive = Drive(filepath, vm=drive.vm)
 
     target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name,
-                     target=target_id, sync="bitmap",
-                     bitmap_mode=bitmap_mode,

tests/qemu-iotests/257 def reference_backup(drive, n, filepath): · line 244

Quoted 10 lines
tests/qemu-iotests/257 +8 -2
-                     bitmap=bitmap,
-                     auto_finalize=False)
+
+    kwargs = {
+        'job_id': job_id,
+        'auto_finalize': False,
+        'bitmap': bitmap,
+        'bitmap_mode': bitmap_mode,
+    }
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}

I suppose this is to remove items that are None?

Very cute, but why not just

  kwargs = {
    'job_id': job_id,
    'auto_finalize': False,
  }
  if bitmap is not None:
    kwargs['bitmap'] = bitmap
    kwargs['bitmap_mode'] = bitmap_mode
Quoted 5 lines

Exactly the same number of lines, but immediately makes it clear what’s going on. Not as cute, I admit.

(Yes, I am indeed actively trying to train you not to write cute code.)

It sneaks in. I genuinely struggle with understanding what other people will find readable; I have an authentically hard time reviewing other people's patches too. I'm earnestly not sure how I can help improve this, but I would like to.

I wasn't sure what the easiest way to avoid sending the "None" over the wire was, so I went with a general thing, but yes: it's because bitmap and bitmap_mode are set to None sometimes and I need to omit such keys.

In this case, though, I do test bitmap and bitmap_mode separately, so for the purposes of testing intentionally bad combinations you do need:

if bitmap is not None:
kwargs['bitmap'] = bitmap
if bitmap_mode is not None:
kwargs['bitmap_mode'] = bitmap_mode

And I just looked at this and it did not spark joy; so I went with a generic filter to remove nulled keys. I admit it's /slightly/ cute and not immediately obvious why it needs to be done.

This is even cuter, so maybe I am traveling in the wrong direction:

def backup(drive, n, filepath, sync, **kwargs):
log("--- Test Backup #{:d} ---\n".format(n))
target_id = "backup_target_{:d}".format(n)
job_id = "backup_{:d}".format(n)
target_drive = Drive(filepath, vm=drive.vm)

target_drive.create_target(target_id, drive.fmt, drive.size) kwargs.setdefault('auto_finalize', False)
# Strip any arguments explicitly nulled by the caller:
kwargs = {key: val for key, val in kwargs.items()
if val is not None}
blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs) return job_id

It's quite a bit shorter and also makes backup() more flexible by omitting the bitmap and bitmap_mode arguments entirely, allowing the caller to override the auto_finalize default, etc. In this permutation, we don't know the full extent of kwargs so it makes sense to generically filter it.

Manually conditionally setting arguments is probably also fine. Do you still have a preference for the more static approach?

Quoted 4 lines

The rest looks good to me:

Thanks for reviewing, as always!

#11 lore raw

Re: [Qemu-devel] [PATCH 3/8] iotests/257: Refactor backup helpers

Quoted 102 lines · John Snow
Quoted 55 lines · Max Reitz
Quoted 14 lines · John Snow

This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.

---
 tests/qemu-iotests/257     |  46 +++++----
 tests/qemu-iotests/257.out | 192 ++++++++++++++++++-------------------
 2 files changed, 124 insertions(+), 114 deletions(-)
tests/qemu-iotests/257 +0 -0
index 2ff4aa8695..2eb4f26c28 100755

[...]

Quoted 14 lines
tests/qemu-iotests/257 +4 -7
-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode):
-    log("--- Bitmap Backup #{:d} ---\n".format(n))
-    target_id = "bitmap_target_{:d}".format(n)
-    job_id = "bitmap_backup_{:d}".format(n)
+def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'):
+    log("--- Test Backup #{:d} ---\n".format(n))
+    target_id = "backup_target_{:d}".format(n)
+    job_id = "backup_{:d}".format(n)
     target_drive = Drive(filepath, vm=drive.vm)
 
     target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name,
-                     target=target_id, sync="bitmap",
-                     bitmap_mode=bitmap_mode,

tests/qemu-iotests/257 def reference_backup(drive, n, filepath): · line 244

Quoted 10 lines
tests/qemu-iotests/257 +8 -2
-                     bitmap=bitmap,
-                     auto_finalize=False)
+
+    kwargs = {
+        'job_id': job_id,
+        'auto_finalize': False,
+        'bitmap': bitmap,
+        'bitmap_mode': bitmap_mode,
+    }
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}

I suppose this is to remove items that are None?

Very cute, but why not just

  kwargs = {
    'job_id': job_id,
    'auto_finalize': False,
  }
  if bitmap is not None:
    kwargs['bitmap'] = bitmap
    kwargs['bitmap_mode'] = bitmap_mode
Quoted 5 lines

Exactly the same number of lines, but immediately makes it clear what’s going on. Not as cute, I admit.

(Yes, I am indeed actively trying to train you not to write cute code.)

It sneaks in. I genuinely struggle with understanding what other people will find readable; I have an authentically hard time reviewing other people's patches too. I'm earnestly not sure how I can help improve this, but I would like to.

I wasn't sure what the easiest way to avoid sending the "None" over the wire was, so I went with a general thing, but yes: it's because bitmap and bitmap_mode are set to None sometimes and I need to omit such keys.

In this case, though, I do test bitmap and bitmap_mode separately, so for the purposes of testing intentionally bad combinations you do need:

if bitmap is not None:
kwargs['bitmap'] = bitmap
if bitmap_mode is not None:
kwargs['bitmap_mode'] = bitmap_mode

And I just looked at this and it did not spark joy; so I went with a generic filter to remove nulled keys. I admit it's /slightly/ cute and not immediately obvious why it needs to be done.

This is even cuter, so maybe I am traveling in the wrong direction:

def backup(drive, n, filepath, sync, **kwargs):
log("--- Test Backup #{:d} ---\n".format(n))
target_id = "backup_target_{:d}".format(n)
job_id = "backup_{:d}".format(n)
target_drive = Drive(filepath, vm=drive.vm)

target_drive.create_target(target_id, drive.fmt, drive.size) kwargs.setdefault('auto_finalize', False)
# Strip any arguments explicitly nulled by the caller:
kwargs = {key: val for key, val in kwargs.items()
if val is not None}
blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs) return job_id

Quoted 8 lines

It's quite a bit shorter and also makes backup() more flexible by omitting the bitmap and bitmap_mode arguments entirely, allowing the caller to override the auto_finalize default, etc. In this permutation, we don't know the full extent of kwargs so it makes sense to generically filter it.

Manually conditionally setting arguments is probably also fine. Do you still have a preference for the more static approach?

It’s OK with the comment.
Although it needs a kwargs.setdefault('job_id', job_id), too.

(Hm. Shouldn’t 'auto-finalize' and 'job-id' work just fine? It’s not like the QMP scripts swap - and _. They just replace _ by -.)

Max

Quoted 7 lines
Quoted 4 lines

The rest looks good to me:

Thanks for reviewing, as always!

author
#12 lore raw

PATCH 4/8 block/backup: hoist bitmap check into QMP interface

In reply to John Snow

This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.

---
 block/backup.c | 13 ++++---------
 blockdev.c     | 10 ++++++++++
 2 files changed, 14 insertions(+), 9 deletions(-)
block/backup.c +4 -9
index e2729cf6fa..a64b768e24 100644
@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     assert(bs);
     assert(target);
 
+    /* QMP interface protects us from these cases */
+    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
+    assert(sync_bitmap || sync_mode != MIRROR_SYNC_MODE_BITMAP);
+
     if (bs == target) {
         error_setg(errp, "Source and target cannot be the same");
         return NULL;
@@ -597,16 +601,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    /* QMP interface should have handled translating this to bitmap mode */
-    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
-
     if (sync_mode == MIRROR_SYNC_MODE_BITMAP) {
-        if (!sync_bitmap) {
-            error_setg(errp, "must provide a valid bitmap name for "
-                       "'%s' sync mode", MirrorSyncMode_str(sync_mode));
-            return NULL;
-        }
-
         /* If we need to write to this bitmap, check that we can: */
         if (bitmap_mode != BITMAP_SYNC_MODE_NEVER &&
             bdrv_dirty_bitmap_check(sync_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
blockdev.c +10 -0
index 3e30bc2ca7..f0b7da53b0 100644
@@ -3464,6 +3464,16 @@ static BlockJob *do_backup_common(BackupCommon *backup,
         return NULL;
     }
 
+    if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) ||
+        (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL)) {
+        /* done before desugaring 'incremental' to print the right message */
+        if (!backup->has_bitmap) {
+            error_setg(errp, "must provide a valid bitmap name for "
+                       "'%s' sync mode", MirrorSyncMode_str(backup->sync));
+            return NULL;
+        }
+    }
+
     if (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL) {
         if (backup->has_bitmap_mode &&
             backup->bitmap_mode != BITMAP_SYNC_MODE_ON_SUCCESS) {
Signature · 1 line
2.21.0
Reviewed-by on v1 4/8
#13 lore raw

Re: [Qemu-devel] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface

Quoted 14 lines · John Snow

This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.

---
 block/backup.c | 13 ++++---------
 blockdev.c     | 10 ++++++++++
 2 files changed, 14 insertions(+), 9 deletions(-)
block/backup.c +0 -0
index e2729cf6fa..a64b768e24 100644

block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 571

Quoted 6 lines
block/backup.c +3 -0
@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     assert(bs);
     assert(target);
 
+    /* QMP interface protects us from these cases */
+    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
+    assert(sync_bitmap || sync_mode != MIRROR_SYNC_MODE_BITMAP);

Implication would be a nice operator sometimes.

("assert(sync_mode == MIRROR_SYNC_MODE_BITMAP -> sync_bitmap)")

(Can you do that in C++? No, you can’t overload bool’s operators, right?)

author
#14 lore raw

Re: [Qemu-devel] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface

Quoted 24 lines · Max Reitz
Quoted 14 lines · John Snow

This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.

---
 block/backup.c | 13 ++++---------
 blockdev.c     | 10 ++++++++++
 2 files changed, 14 insertions(+), 9 deletions(-)
block/backup.c +0 -0
index e2729cf6fa..a64b768e24 100644

block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 571

Quoted 6 lines
block/backup.c +3 -0
@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     assert(bs);
     assert(target);
 
+    /* QMP interface protects us from these cases */
+    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
+    assert(sync_bitmap || sync_mode != MIRROR_SYNC_MODE_BITMAP);

Implication would be a nice operator sometimes.

Quoted 6 lines

("assert(sync_mode == MIRROR_SYNC_MODE_BITMAP -> sync_bitmap)")

(Can you do that in C++? No, you can’t overload bool’s operators, right?)

Yes, I also find this assertion kind of hard to read personally, but it feels somewhat clunky to write:

if (antecedent) {
    assert(condition);
}

I suppose we can also phrase this as:

assert(sync_mode == MIRROR_SYNC_MODE_BITMAP ? sync_bitmap : true);

Which might honestly be pretty good. Mind if I change it to this?

--js

#15 lore raw

Re: [Qemu-devel] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface

Quoted 41 lines · John Snow
Quoted 24 lines · Max Reitz
Quoted 14 lines · John Snow

This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.

---
 block/backup.c | 13 ++++---------
 blockdev.c     | 10 ++++++++++
 2 files changed, 14 insertions(+), 9 deletions(-)
block/backup.c +0 -0
index e2729cf6fa..a64b768e24 100644

block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 571

Quoted 6 lines
block/backup.c +3 -0
@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     assert(bs);
     assert(target);
 
+    /* QMP interface protects us from these cases */
+    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
+    assert(sync_bitmap || sync_mode != MIRROR_SYNC_MODE_BITMAP);

Implication would be a nice operator sometimes.

Quoted 6 lines

("assert(sync_mode == MIRROR_SYNC_MODE_BITMAP -> sync_bitmap)")

(Can you do that in C++? No, you can’t overload bool’s operators, right?)

Yes, I also find this assertion kind of hard to read personally, but it feels somewhat clunky to write:

if (antecedent) {
    assert(condition);
}
Quoted 5 lines

I suppose we can also phrase this as:

assert(sync_mode == MIRROR_SYNC_MODE_BITMAP ? sync_bitmap : true);

Which might honestly be pretty good. Mind if I change it to this?

Looks weird (mostly unfamiliar), but I do not.

Max

author
#16 lore raw

PATCH 5/8 iotests/257: test API failures

In reply to John Snow
---
 tests/qemu-iotests/257     | 69 +++++++++++++++++++++++++++++++
 tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 154 insertions(+)
tests/qemu-iotests/257 +69 -0
index 2eb4f26c28..de8707cb19 100755
@@ -450,10 +450,79 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         compare_images(img_path, fbackup2)
         log('')
 
+def test_backup_api():
+    """
+    """
+    with iotests.FilePaths(['img', 'bsync1']) as \
+         (img_path, backup_path), \
+         iotests.VM() as vm:
+
+        log("\n=== API failure tests ===\n")
+        log('--- Preparing image & VM ---\n')
+        drive0 = Drive(img_path, vm=vm)
+        drive0.img_create(iotests.imgfmt, SIZE)
+        vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device()))
+        vm.launch()
+
+        file_config = {
+            'driver': 'file',
+            'filename': drive0.path
+        }
+
+        vm.qmp_log('blockdev-add',
+                   filters=[iotests.filter_qmp_testfiles],
+                   node_name="drive0",
+                   driver=drive0.fmt,
+                   file=file_config)
+        drive0.node = 'drive0'
+        drive0.device = 'device0'
+        vm.qmp_log("device_add", id=drive0.device,
+                   drive=drive0.name, driver="scsi-hd")
+        log('')
+
+        target0 = Drive(backup_path, vm=vm)
+        target0.create_target("backup_target", drive0.fmt, drive0.size)
+        log('')
+
+        vm.qmp_log("block-dirty-bitmap-add", node=drive0.name,
+                   name="bitmap0", granularity=GRANULARITY)
+        log('')
+
+        log('-- Testing invalid QMP commands --\n')
+
+        error_cases = {
+            'incremental': {
+                None:        ['on-success', 'always', 'never', None],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['always', 'never']
+            },
+            'bitmap': {
+                None:        ['on-success', 'always', 'never', None],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   [None],
+            },
+        }
+
+        for sync_mode, config in error_cases.items():
+            log("-- Sync mode {:s} tests --\n".format(sync_mode))
+            for bitmap, policies in config.items():
+                for policy in policies:
+                    kwargs = { 'job_id': 'api_job' }
+                    if bitmap is not None:
+                        kwargs['bitmap'] = bitmap
+                    if policy is not None:
+                        kwargs['bitmap-mode'] = policy
+                    blockdev_backup(drive0.vm, drive0.name, "backup_target",
+                                    sync_mode, **kwargs)
+                    log('')
+
+
 def main():
     for bsync_mode in ("never", "on-success", "always"):
         for failure in ("simulated", "intermediate", None):
             test_bitmap_sync(bsync_mode, "bitmap", failure)
 
+    test_backup_api()
+
 if __name__ == '__main__':
     iotests.script_main(main, supported_fmts=['qcow2'])
tests/qemu-iotests/257.out +85 -0

88 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v1 5/8
#17 lore raw

Re: [Qemu-devel] [PATCH 5/8] iotests/257: test API failures

Quoted 11 lines · John Snow
---
 tests/qemu-iotests/257     | 69 +++++++++++++++++++++++++++++++
 tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 154 insertions(+)
tests/qemu-iotests/257 +0 -0
index 2eb4f26c28..de8707cb19 100755

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 455

Quoted 6 lines
tests/qemu-iotests/257 +3 -0
@@ -450,10 +450,79 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         compare_images(img_path, fbackup2)
         log('')
 
+def test_backup_api():
+    """
+    """

Er, OK?

[...]

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 508

Quoted 3 lines
tests/qemu-iotests/257 +3 -0
+        for sync_mode, config in error_cases.items():
+            log("-- Sync mode {:s} tests --\n".format(sync_mode))
+            for bitmap, policies in config.items():

You might be interested in the fact that the iteration order is different for Python2. Or maybe you aren’t.

author
#18 lore raw

Re: [Qemu-devel] [PATCH 5/8] iotests/257: test API failures

Quoted 15 lines · Max Reitz
Quoted 14 lines · John Snow
---
 tests/qemu-iotests/257     | 69 +++++++++++++++++++++++++++++++
 tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 154 insertions(+)
tests/qemu-iotests/257 +0 -0
index 2eb4f26c28..de8707cb19 100755
@@ -450,10 +450,79 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         compare_images(img_path, fbackup2)
         log('')
 
Quoted 8 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 455

Quoted 3 lines
tests/qemu-iotests/257 +3 -0
+def test_backup_api():
+    """
+    """

Er, OK?

[...]

Whooooooooops.

Quoted 7 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 508

Quoted 3 lines
tests/qemu-iotests/257 +3 -0
+        for sync_mode, config in error_cases.items():
+            log("-- Sync mode {:s} tests --\n".format(sync_mode))
+            for bitmap, policies in config.items():

You might be interested in the fact that the iteration order is different for Python2. Or maybe you aren’t.

asdf. This is undoubtedly the worst thing about Python. I'll have to fix this, because the ordering isn't guaranteed until 3.5 or some such and we're only EOLing Python2.

--js

author
#19 lore raw

PATCH 6/8 block/backup: issue progress updates for skipped regions

In reply to John Snow

The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.

The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.

Update the progress when we skip regions.

---
 block/backup.c | 1 +
 1 file changed, 1 insertion(+)
block/backup.c +1 -0
index a64b768e24..38c4a688c6 100644
@@ -417,6 +417,7 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)
         if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
             bdrv_is_unallocated_range(bs, offset, job->cluster_size))
         {
+            job_progress_update(&job->common.job, job->cluster_size);
             bdrv_reset_dirty_bitmap(job->copy_bitmap, offset,
                                     job->cluster_size);
             continue;
Signature · 1 line
2.21.0
#20 lore raw

Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions

Quoted 2 lines · John Snow

The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.

Although there is this comment:

Quoted 1 line

/* TODO job_progress_set_remaining() would make more sense */

So...

Quoted 5 lines

The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.

Update the progress when we skip regions.

Wouldn’t it be more correct to decrease the job length?

Max

Quoted 18 lines
---
 block/backup.c | 1 +
 1 file changed, 1 insertion(+)
block/backup.c +1 -0
index a64b768e24..38c4a688c6 100644
@@ -417,6 +417,7 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)
         if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
             bdrv_is_unallocated_range(bs, offset, job->cluster_size))
         {
+            job_progress_update(&job->common.job, job->cluster_size);
             bdrv_reset_dirty_bitmap(job->copy_bitmap, offset,
                                     job->cluster_size);
             continue;
author
#21 lore raw

Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions

Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow

The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.

Although there is this comment:

Quoted 1 line

/* TODO job_progress_set_remaining() would make more sense */

So...

Quoted 4 lines

The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.

Quoted 6 lines
Quoted 1 line

Update the progress when we skip regions.

Wouldn’t it be more correct to decrease the job length?

Max

Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:

(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]

This means we can do stuff like:

- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particular

Bitmap sync works by artificially increasing progress for NOP regions, seen in init_copy_bitmap.

Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.

Top sync is the odd one out, which just omits progress for regions it skips.

My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?

¯\_(ツ)_/¯

Quoted 20 lines
Quoted 18 lines
---
 block/backup.c | 1 +
 1 file changed, 1 insertion(+)
block/backup.c +1 -0
index a64b768e24..38c4a688c6 100644
@@ -417,6 +417,7 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)
         if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
             bdrv_is_unallocated_range(bs, offset, job->cluster_size))
         {
+            job_progress_update(&job->common.job, job->cluster_size);
             bdrv_reset_dirty_bitmap(job->copy_bitmap, offset,
                                     job->cluster_size);
             continue;
Reviewed-by
#22 lore raw

Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions

Quoted 31 lines · John Snow
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow

The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.

Although there is this comment:

Quoted 1 line

/* TODO job_progress_set_remaining() would make more sense */

So...

Quoted 4 lines

The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.

Quoted 6 lines
Quoted 1 line

Update the progress when we skip regions.

Wouldn’t it be more correct to decrease the job length?

Max

Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:

(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]

Quoted 8 lines

This means we can do stuff like:

- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particular

Bitmap sync works by artificially increasing progress for NOP regions, seen in init_copy_bitmap.

Yes, and it has a TODO comment that says it should be done differently.

Quoted 11 lines

Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.

Top sync is the odd one out, which just omits progress for regions it skips.

My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?

If you look at the commit (05df8a6a2b4), I suppose it was because that commit simply did not intend to change behavior. It just touched that piece of code and noted that maybe there should be a follow-up commit to change it.

But yeah, whatever.

author
#23 lore raw

Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions

Quoted 55 lines · Max Reitz
Quoted 31 lines · John Snow
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow

The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.

Although there is this comment:

Quoted 1 line

/* TODO job_progress_set_remaining() would make more sense */

So...

Quoted 4 lines

The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.

Quoted 6 lines
Quoted 1 line

Update the progress when we skip regions.

Wouldn’t it be more correct to decrease the job length?

Max

Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:

(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]

Quoted 8 lines

This means we can do stuff like:

- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particular

Bitmap sync works by artificially increasing progress for NOP regions, seen in init_copy_bitmap.

Yes, and it has a TODO comment that says it should be done differently.

Quoted 11 lines

Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.

Top sync is the odd one out, which just omits progress for regions it skips.

My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?

Quoted 9 lines

If you look at the commit (05df8a6a2b4), I suppose it was because that commit simply did not intend to change behavior. It just touched that piece of code and noted that maybe there should be a follow-up commit to change it.

But yeah, whatever.

I mean. I'll make it consistent either way, but I actually don't know which way I should make it go. I just think that all the modes should work the same if we can help it.

Flip a coin?

--js

#24 lore raw

Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions

Quoted 65 lines · John Snow
Quoted 62 lines · Max Reitz
Quoted 31 lines · John Snow
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow

The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.

Although there is this comment:

Quoted 1 line

/* TODO job_progress_set_remaining() would make more sense */

So...

Quoted 4 lines

The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.

Quoted 6 lines
Quoted 1 line

Update the progress when we skip regions.

Wouldn’t it be more correct to decrease the job length?

Max

Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:

(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]

Quoted 8 lines

This means we can do stuff like:

- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particular

Bitmap sync works by artificially increasing progress for NOP regions, seen in init_copy_bitmap.

Yes, and it has a TODO comment that says it should be done differently.

Quoted 11 lines

Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.

Top sync is the odd one out, which just omits progress for regions it skips.

My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?

If you look at the commit (05df8a6a2b4), I suppose it was because that commit simply did not intend to change behavior. It just touched that piece of code and noted that maybe there should be a follow-up commit to change it.

But yeah, whatever.

Quoted 8 lines
Quoted 2 lines

I mean. I'll make it consistent either way, but I actually don't know which way I should make it go. I just think that all the modes should work the same if we can help it.

Flip a coin?

If you’d flip a coin, I can say that I’d find it a bit more meaningful to reduce the length. Especially if you change sync=top to calculate beforehand how much we need to copy, so the length isn’t even reduced while the job is running.

And it’s easier to remove the TODO comment this way. If we decide against it, you’d have to remove the TODO comment aso (because we decided against it), but you’d need to justify it in the commit message.
And we all know that writing explanations and documentation is the
hardest thing of all.

Max

author
#25 lore raw

PATCH 7/8 block/backup: support bitmap sync modes for non-bitmap backups

In reply to John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---
 block/backup.c       |  8 +-------
 blockdev.c           | 22 ++++++++++++++++++++++
 qapi/block-core.json |  6 ++++--
 3 files changed, 27 insertions(+), 9 deletions(-)
block/backup.c +1 -7
index 38c4a688c6..b94ed595ba 100644
@@ -602,7 +602,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    if (sync_mode == MIRROR_SYNC_MODE_BITMAP) {
+    if (sync_bitmap) {
         /* If we need to write to this bitmap, check that we can: */
         if (bitmap_mode != BITMAP_SYNC_MODE_NEVER &&
             bdrv_dirty_bitmap_check(sync_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
@@ -613,12 +613,6 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         if (bdrv_dirty_bitmap_create_successor(bs, sync_bitmap, errp) < 0) {
             return NULL;
         }
-    } else if (sync_bitmap) {
-        error_setg(errp,
-                   "a bitmap was given to backup_job_create, "
-                   "but it received an incompatible sync_mode (%s)",
-                   MirrorSyncMode_str(sync_mode));
-        return NULL;
     }
 
     len = bdrv_getlength(bs);
blockdev.c +22 -0
index f0b7da53b0..bc152f8e0d 100644
@@ -3502,6 +3502,28 @@ static BlockJob *do_backup_common(BackupCommon *backup,
         if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_ALLOW_RO, errp)) {
             return NULL;
         }
+
+        /* This does not produce a useful bitmap artifact: */
+        if (backup->sync == MIRROR_SYNC_MODE_NONE) {
+            error_setg(errp, "sync mode '%s' does not produce meaningful bitmap"
+                       " outputs", MirrorSyncMode_str(backup->sync));
+            return NULL;
+        }
+
+        /* If the bitmap isn't used for input or output, this is useless: */
+        if (backup->bitmap_mode == BITMAP_SYNC_MODE_NEVER &&
+            backup->sync != MIRROR_SYNC_MODE_BITMAP) {
+            error_setg(errp, "Bitmap sync mode '%s' has no meaningful effect"
+                       " when combined with sync mode '%s'",
+                       BitmapSyncMode_str(backup->bitmap_mode),
+                       MirrorSyncMode_str(backup->sync));
+            return NULL;
+        }
+    }
+
+    if (!backup->has_bitmap && backup->has_bitmap_mode) {
+        error_setg(errp, "Cannot specify Bitmap sync mode without a bitmap");
+        return NULL;
     }
 
     if (!backup->auto_finalize) {
qapi/block-core.json +4 -2
index 5a578806c5..099e4f37b2 100644
@@ -1352,13 +1352,15 @@
 # @speed: the maximum speed, in bytes per second. The default is 0,
 #         for unlimited.
 #
-# @bitmap: the name of a dirty bitmap if sync is "bitmap" or "incremental".
+# @bitmap: The name of a dirty bitmap to use.
 #          Must be present if sync is "bitmap" or "incremental".
+#          Can be present if sync is "full" or "top".
 #          Must not be present otherwise.
 #          (Since 2.4 (drive-backup), 3.1 (blockdev-backup))
 #
 # @bitmap-mode: Specifies the type of data the bitmap should contain after
-#               the operation concludes. Must be present if sync is "bitmap".
+#               the operation concludes.
+#               Must be present if a bitmap was provided,
 #               Must NOT be present otherwise. (Since 4.2)
 #
 # @compress: true to compress data, if the target format supports it.
Signature · 1 line
2.21.0
Reviewed-by on v1 7/8
#26 lore raw

Re: [Qemu-devel] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups

Quoted 26 lines · John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---
 block/backup.c       |  8 +-------
 blockdev.c           | 22 ++++++++++++++++++++++
 qapi/block-core.json |  6 ++++--
 3 files changed, 27 insertions(+), 9 deletions(-)

[...]

Quoted 4 lines
blockdev.c +0 -0
index f0b7da53b0..bc152f8e0d 100644

[...]

blockdev.c static BlockJob *do_backup_common(BackupCommon *backup, · line 3525

Quoted 2 lines
blockdev.c +2 -0
+    if (!backup->has_bitmap && backup->has_bitmap_mode) {
+        error_setg(errp, "Cannot specify Bitmap sync mode without a bitmap");

Any reason for capitalizing the first “Bitmap”?

With a reason or it fixed:

author
#27 lore raw

Re: [Qemu-devel] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups

Quoted 37 lines · Max Reitz
Quoted 26 lines · John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---
 block/backup.c       |  8 +-------
 blockdev.c           | 22 ++++++++++++++++++++++
 qapi/block-core.json |  6 ++++--
 3 files changed, 27 insertions(+), 9 deletions(-)

[...]

Quoted 4 lines
blockdev.c +0 -0
index f0b7da53b0..bc152f8e0d 100644

[...]

Quoted 9 lines

blockdev.c static BlockJob *do_backup_common(BackupCommon *backup, · line 3525

Quoted 2 lines
blockdev.c +2 -0
+    if (!backup->has_bitmap && backup->has_bitmap_mode) {
+        error_setg(errp, "Cannot specify Bitmap sync mode without a bitmap");

Any reason for capitalizing the first “Bitmap”?

With a reason or it fixed:

Hanging around Germans too much?

Actually, I can explain why: because a "bitmap" is a generic term, but whenever I capitalize it as "Bitmap" I am referring to a Block Dirty Bitmap which is a specific sort of thing. I do this unconsciously.

But in that case, I should actually be consistent in the interface (and docstrings and docs and error strings) and always call it that specific thing, which I don't.

"bitmap" is probably more correct for now, but I ought to go through all the interface and make it consistent in some way or another.

(Actually: I'd like to see if I can't rename the "BdrvDirtyBitmap" object to something more generic and shorter so I can rename many of the functions we have something shorter.

Because the structure is "BdrvDirtyBitmap", there's some confusion when we name functions like bdrv_dirty_bitmap_{verb} because it's not clear if this is a bdrv function that does something with dirty bitmaps, or if it's a "BdrvDirtyBitmap" function that does something with that object.

I guess that seems like a subtle point, but it's why the naming conventions in dirty-bitmap.c are all over the place. I think at one point, the idea was that:

bdrv_{verb}_dirty_bitmap was an action applied to a BDS that happened to do something with dirty bitmaps. bdrv_dirty_bitmap_{verb} was an action that applied to a "BdrvDirtyBitmap". Crystal clear and not confusing at all, right?

It'd be nice to have functions that operate on a node be named bdrv_dbitmap_foo() and functions that operate on the bitmap structure itself named just dbitmap_bar().

Would it be okay if I named them such a thing, I wonder?

we have "bitmap" and "hbitmap" already, I could do something like "dbitmap" for "dirty bitmap" or some such. Kind of an arbitrary change I admit, but I'm itching to do a big spring cleaning in dirty-bitmap.c right after this series is done.)

#28 lore raw

Re: [Qemu-devel] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups

Quoted 43 lines · John Snow
Quoted 40 lines · Max Reitz
Quoted 26 lines · John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---
 block/backup.c       |  8 +-------
 blockdev.c           | 22 ++++++++++++++++++++++
 qapi/block-core.json |  6 ++++--
 3 files changed, 27 insertions(+), 9 deletions(-)

[...]

Quoted 4 lines
blockdev.c +0 -0
index f0b7da53b0..bc152f8e0d 100644

[...]

Quoted 2 lines
blockdev.c +2 -0
+    if (!backup->has_bitmap && backup->has_bitmap_mode) {
+        error_setg(errp, "Cannot specify Bitmap sync mode without a bitmap");
Quoted 8 lines
Quoted 6 lines

Any reason for capitalizing the first “Bitmap”?

With a reason or it fixed:

Hanging around Germans too much?

You should know then that the korrekt way to write it would be:

„Specifying Binarydigitmapsynchronizationmode without a Binarydigitmap is absolutely verboten!“

Quoted 8 lines

Actually, I can explain why: because a "bitmap" is a generic term, but whenever I capitalize it as "Bitmap" I am referring to a Block Dirty Bitmap which is a specific sort of thing. I do this unconsciously.

But in that case, I should actually be consistent in the interface (and docstrings and docs and error strings) and always call it that specific thing, which I don't.

Quoted 7 lines

"bitmap" is probably more correct for now, but I ought to go through all the interface and make it consistent in some way or another.

(Actually: I'd like to see if I can't rename the "BdrvDirtyBitmap" object to something more generic and shorter so I can rename many of the functions we have something shorter.

Well, BDB is free. That path has worked fine for BlockDriverStates.

Or what I said on IRC, but you know.

Quoted 5 lines

Because the structure is "BdrvDirtyBitmap", there's some confusion when we name functions like bdrv_dirty_bitmap_{verb} because it's not clear if this is a bdrv function that does something with dirty bitmaps, or if it's a "BdrvDirtyBitmap" function that does something with that object.

Quoted 8 lines

I guess that seems like a subtle point, but it's why the naming conventions in dirty-bitmap.c are all over the place. I think at one point, the idea was that:

bdrv_{verb}_dirty_bitmap was an action applied to a BDS that happened to do something with dirty bitmaps. bdrv_dirty_bitmap_{verb} was an action that applied to a "BdrvDirtyBitmap". Crystal clear and not confusing at all, right?

I just thought people named their functions whatever they felt like at the time.

Quoted 10 lines

It'd be nice to have functions that operate on a node be named bdrv_dbitmap_foo() and functions that operate on the bitmap structure itself named just dbitmap_bar().

Would it be okay if I named them such a thing, I wonder?

we have "bitmap" and "hbitmap" already, I could do something like "dbitmap" for "dirty bitmap" or some such. Kind of an arbitrary change I admit, but I'm itching to do a big spring cleaning in dirty-bitmap.c right after this series is done.)

HBitmaps are generally used to track dirty areas, so I’d find this a misnomer. BDBitmap would be OK. The “block” part should be in there somewhere.

Max

author
#29 lore raw

PATCH 8/8 iotests/257: test traditional sync modes

In reply to John Snow
---
 tests/qemu-iotests/257     |   31 +
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3120 insertions(+)
tests/qemu-iotests/257 +31 -0
index de8707cb19..8de1c4da19 100755
@@ -288,6 +288,12 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
                       Bitmaps are always synchronized, regardless of failure.
                       (Partial images must be kept.)
 
+    :param msync_mode: The mirror sync mode to use for the first backup.
+                       Can be any one of:
+        - bitmap: Backups based on bitmap manifest.
+        - full:   Full backups.
+        - top:    Full backups of the top layer only.
+
     :param failure: Is the (optional) failure mode, and can be any of:
         - None:         No failure. Test the normative path. Default.
         - simulated:    Cancel the job right before it completes.
@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         if bsync_mode == 'always' and failure == 'intermediate':
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)
+            if msync_mode in ('full', 'top'):
+                # These modes return all bits set except what was copied/skipped
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
@@ -501,6 +512,21 @@ def test_backup_api():
                 'bitmap404': ['on-success', 'always', 'never', None],
                 'bitmap0':   [None],
             },
+            'full': {
+                None:        ['on-success', 'always', 'never'],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['never', None],
+            },
+            'top': {
+                None:        ['on-success', 'always', 'never'],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['never', None],
+            },
+            'none': {
+                None:        ['on-success', 'always', 'never'],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['on-success', 'always', 'never', None],
+            }
         }
 
         for sync_mode, config in error_cases.items():
@@ -522,6 +548,11 @@ def main():
         for failure in ("simulated", "intermediate", None):
             test_bitmap_sync(bsync_mode, "bitmap", failure)
 
+    for sync_mode in ('full', 'top'):
+        for bsync_mode in ('on-success', 'always'):
+            for failure in ('simulated', 'intermediate', None):
+                test_bitmap_sync(bsync_mode, sync_mode, failure)
+
     test_backup_api()
 
 if __name__ == '__main__':
tests/qemu-iotests/257.out +3089 -0

2,945 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.21.0

Part of the diff is not shown here. View the raw message.

#30 lore raw

Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes

Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   31 +
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3120 insertions(+)

Oof.

Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index de8707cb19..8de1c4da19 100755

[...]

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 420

Quoted 6 lines
tests/qemu-iotests/257 +2 -0
@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         if bsync_mode == 'always' and failure == 'intermediate':
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)
+            if msync_mode in ('full', 'top'):
+                # These modes return all bits set except what was copied/skipped

Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.

Max

Quoted 6 lines
tests/qemu-iotests/257 +3 -0
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup

[...]

author
#31 lore raw

Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes

Quoted 9 lines · Max Reitz
Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   31 +
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3120 insertions(+)

Oof.

Yeah, it's... a lot of test output. We probably shouldn't count the reference test output against any kind of SLOC metrics.

Quoted 18 lines
Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index de8707cb19..8de1c4da19 100755

[...]

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 420

Quoted 6 lines
tests/qemu-iotests/257 +2 -0
@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         if bsync_mode == 'always' and failure == 'intermediate':
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)
+            if msync_mode in ('full', 'top'):
+                # These modes return all bits set except what was copied/skipped

Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.

Quoted 2 lines

Max

Good point!

I think this can be fixed by doing an initialization pass of the copy_bitmap when sync=top to set only the allocated regions in the bitmap.

This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.

So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.

This doesn't touch sync=none at all, which needs more invasive fixes if we wanted it to have useful bitmap return values (it needs to differentiate the idea between must-copy and can-copy, and I still don't know if this is worthwhile to do, so until I hear otherwise, I'm not gonna.)

Quoted 8 lines
Quoted 6 lines
tests/qemu-iotests/257 +3 -0
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup

[...]

#32 lore raw

Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes

Quoted 39 lines · John Snow
Quoted 9 lines · Max Reitz
Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   31 +
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3120 insertions(+)

Oof.

Yeah, it's... a lot of test output. We probably shouldn't count the reference test output against any kind of SLOC metrics.

Quoted 18 lines
Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index de8707cb19..8de1c4da19 100755

[...]

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 420

Quoted 6 lines
tests/qemu-iotests/257 +2 -0
@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         if bsync_mode == 'always' and failure == 'intermediate':
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)
+            if msync_mode in ('full', 'top'):
+                # These modes return all bits set except what was copied/skipped

Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.

Quoted 2 lines

Max

Good point!

Quoted 7 lines

I think this can be fixed by doing an initialization pass of the copy_bitmap when sync=top to set only the allocated regions in the bitmap.

This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.

Now that you mention it... I didn’t realize that. Yes, you’re right.

Quoted 9 lines

So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.

This doesn't touch sync=none at all, which needs more invasive fixes if we wanted it to have useful bitmap return values (it needs to differentiate the idea between must-copy and can-copy, and I still don't know if this is worthwhile to do, so until I hear otherwise, I'm not gonna.)

No, I’m with you on that one.

Max

Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
tests/qemu-iotests/257 +3 -0
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup

[...]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#34 lore raw

Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes

Quoted 31 lines · John Snow
Quoted 16 lines · Max Reitz
Quoted 15 lines · John Snow
Quoted 8 lines · Max Reitz

On 10.07.19 03:05, John Snow wrote:

Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.

Max

Good point!

I think this can be fixed by doing an initialization pass of the copy_bitmap when sync=top to set only the allocated regions in the bitmap.

Quoted 12 lines
Quoted 4 lines

This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.

Now that you mention it... I didn’t realize that. Yes, you’re right.

Quoted 5 lines

So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.

Quoted 7 lines

This might be harder than I first thought.

initializing the copy_bitmap generally happens before we install the write notifier, which means that it occurs before the first yield.

However, checking the allocation status can potentially be very slow, can't it? I can't just hog the thread while I check.

I was thinking about that myself. It isn’t that bad, because you aren’t doing the full block_status dance but just checking allocation status, which is reasonably quick (it just needs to look at the image format metadata, it doesn’t go down to the protocol layer).

But it’s probably not so good to halt the monitor for this, yes.

Quoted 7 lines

There are ways to cooperatively process write notifier interrupts and continue to check allocated status once we enter the main loop, but the problem there becomes: if we fail early, how can we tell if the backup is worth resuming?

We might not have reached a convergence point for the copy_bitmap before we failed, and still have a lot of extra bits set.

Is that so bad?

Quoted 4 lines

I suppose at least in the case where we aren't trying to save the copy_bitmap and need it to mean something specific, this is a reasonable approach to fixing sync=TOP.

Quoted 8 lines

As far as resume is concerned, I don't think I have good ideas. I could emit an event or something if you're using sync=top with a bitmap for output, but that feels *so* specialized for a niche(?) command that I don't know if it's worth pursuing.

(Plus, even then, what do you do if it fails before you see that event? You just have to give up on what we copied out? That seems like a waste and not the point of this exercise.)

Before that event, the bitmap can still be usable, as long as all “unknown” areas are set to dirty. Sure, your resumed backup will then copy too much data. But who cares.

So I don’t think you even need an event.

Quoted 3 lines

The only way I can think of at all to get a retry on sync=top is to take an always policy, and to allow a special invocation with something like mode=bitmap+top:

Yes, that was my first idea, too. But I didn’t even write about it, because of...

Quoted 5 lines

"Assume we need to copy anything set in the bitmap, unless it's not in the top layer, and then skip it."

Which seems awful, because it would be a specialty mode for the exclusive purpose of re-trying sync=top backups.

...exactly this.

Quoted 1 line

Meh.

I don’t think it’s all that bad.

Max

author
#35 lore raw

Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes

Quoted 86 lines · Max Reitz
Quoted 31 lines · John Snow
Quoted 16 lines · Max Reitz
Quoted 15 lines · John Snow
Quoted 8 lines · Max Reitz

On 10.07.19 03:05, John Snow wrote:

Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.

Max

Good point!

I think this can be fixed by doing an initialization pass of the copy_bitmap when sync=top to set only the allocated regions in the bitmap.

Quoted 12 lines
Quoted 4 lines

This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.

Now that you mention it... I didn’t realize that. Yes, you’re right.

Quoted 5 lines

So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.

Quoted 7 lines

This might be harder than I first thought.

initializing the copy_bitmap generally happens before we install the write notifier, which means that it occurs before the first yield.

However, checking the allocation status can potentially be very slow, can't it? I can't just hog the thread while I check.

I was thinking about that myself. It isn’t that bad, because you aren’t doing the full block_status dance but just checking allocation status, which is reasonably quick (it just needs to look at the image format metadata, it doesn’t go down to the protocol layer).

But it’s probably not so good to halt the monitor for this, yes.

Quoted 7 lines

There are ways to cooperatively process write notifier interrupts and continue to check allocated status once we enter the main loop, but the problem there becomes: if we fail early, how can we tell if the backup is worth resuming?

We might not have reached a convergence point for the copy_bitmap before we failed, and still have a lot of extra bits set.

Is that so bad?

Quoted 4 lines

I suppose at least in the case where we aren't trying to save the copy_bitmap and need it to mean something specific, this is a reasonable approach to fixing sync=TOP.

Quoted 8 lines

As far as resume is concerned, I don't think I have good ideas. I could emit an event or something if you're using sync=top with a bitmap for output, but that feels *so* specialized for a niche(?) command that I don't know if it's worth pursuing.

(Plus, even then, what do you do if it fails before you see that event? You just have to give up on what we copied out? That seems like a waste and not the point of this exercise.)

Before that event, the bitmap can still be usable, as long as all “unknown” areas are set to dirty. Sure, your resumed backup will then copy too much data. But who cares.

So I don’t think you even need an event.

Quoted 3 lines

The only way I can think of at all to get a retry on sync=top is to take an always policy, and to allow a special invocation with something like mode=bitmap+top:

Yes, that was my first idea, too. But I didn’t even write about it, because of...

Quoted 3 lines

"Assume we need to copy anything set in the bitmap, unless it's not in the top layer, and then skip it."

Quoted 9 lines
Quoted 2 lines

Which seems awful, because it would be a specialty mode for the exclusive purpose of re-trying sync=top backups.

...exactly this.

Quoted 1 line

Meh.

I don’t think it’s all that bad.

No, it's just not ideal and it's something I'd have to defend in a patch. It's a caveat that would need documenting.

"Hey, depending on how far the job got before it failed, you might not want to resume it because it may not have finished determining which segments held allocated data. There's no way to tell if this happened or not."

It makes the decision making process by e.g. libvirt harder, though there are still some heuristics you could use, like:

- Is the bitmap count less than the size of the top image?
- Is it bigger?

And that might be good enough when deciding how to proceed. I suppose if we want to give more precise mechanisms for this we'd always be within our right to continue refining it and just document that it MIGHT have extra bits set.

--js