Skip to contentSkip to conversation

bitmaps: allow bitmaps to be used with full and top

PATCH Applied John Snow 3 participants 31 messages Jul 16, 2019 – Jul 17, 2019 Block layer

lore
Share & tools
lore

John Snow

author
#0 lore raw

v2 cover bitmaps: allow bitmaps to be used with full and top

Based-on: https://github.com/jnsnow/qemu/tree/bitmaps

This follows the previous series which adds the 'bitmap' sync mode and uses it to add interactions with bitmaps to the 'full' and 'top' modes to blockdev-backup and drive-backup.

Why?
on-success: Can conveniently synchronize a bitmap to a full backup.
Allows for transactionless anchor backups. Allows us to attempt an anchor backup without damaging
our bitmap until the backup is successful.
Allows for transactional, ungrouped anchor backups.
always: Allows us to resume full/top style backups with a later
invocation to sync=bitmap. Neat!

Summary:
1-3: Refactor iotest 257 to accommodate this;
4-5: Augment 257 to test trivial failure cases
6-9: Refactor sync=top for block/backup
10: Implement feature
11: Test feature

===
V2:
===

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/11:[0010] [FC] 'iotests/257: add Pattern class'
002/11:[0003] [FC] 'iotests/257: add EmulatedBitmap class' 003/11:[0042] [FC] 'iotests/257: Refactor backup helpers' 004/11:[----] [--] 'block/backup: hoist bitmap check into QMP interface' 005/11:[0016] [FC] 'iotests/257: test API failures'
006/11:[down] 'block/backup: improve sync=bitmap work estimates' 007/11:[down] 'block/backup: centralize copy_bitmap initialization' 008/11:[down] 'block/backup: add backup_is_cluster_allocated' 009/11:[down] 'block/backup: teach TOP to never copy unallocated regions' 010/11:[0002] [FC] 'block/backup: support bitmap sync modes for non-bitmap backups' 011/11:[0058] [FC] 'iotests/257: test traditional sync modes'

001: Fallout from changing floor(x / y) to x // y.
Fallout from changing x = x | y to x |= y.
(Decided not to keep RB.)
002: Removed dead variable
Change x = x - y to x -= y in clear_bits()
003: Substantially reorganize patch. Hopefully 10% more clever and 10% less cute. 004: (Added RB.)
005: Added docstring to test_backup_api
Fixed test matrix enumeration to have consistent order in python2/3 (Declined RB.)
---: What was patch 006 was dropped, and became patches 6-9.
6-9: New!

010: Was 007;
     Formatting fix.
     (Added RB.)

011: Was 008;
Adjust bitmap clearing/expected code as a consequence of #9. Fallout from changes to 005.

John Snow (11):
iotests/257: add Pattern class
iotests/257: add EmulatedBitmap class
iotests/257: Refactor backup helpers
block/backup: hoist bitmap check into QMP interface iotests/257: test API failures
block/backup: improve sync=bitmap work estimates block/backup: centralize copy_bitmap initialization block/backup: add backup_is_cluster_allocated block/backup: teach TOP to never copy unallocated regions block/backup: support bitmap sync modes for non-bitmap backups iotests/257: test traditional sync modes

 block/backup.c             |  188 +-
 block/trace-events         |    1 +
 blockdev.c                 |   32 +
 qapi/block-core.json       |    6 +-
 tests/qemu-iotests/256.out |    4 +-
 tests/qemu-iotests/257     |  342 ++--
 tests/qemu-iotests/257.out | 3366 +++++++++++++++++++++++++++++++++++-
 7 files changed, 3683 insertions(+), 256 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v2 1/11 iotests/257: add Pattern class

Just kidding, this is easier to manage with a full class instead of a namedtuple.

---
 tests/qemu-iotests/257 | 58 +++++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 26 deletions(-)
tests/qemu-iotests/257 +32 -26
index 3952683749..02f9ae0649 100755
@@ -19,7 +19,6 @@
 #
 # [email protected]
 
-from collections import namedtuple
 import math
 import os
 
@@ -29,10 +28,18 @@ from iotests import log, qemu_img
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
-Pattern = namedtuple('Pattern', ['byte', 'offset', 'size'])
-def mkpattern(byte, offset, size=GRANULARITY):
-    """Constructor for Pattern() with default size"""
-    return Pattern(byte, offset, size)
+
+class Pattern:
+    def __init__(self, byte, offset, size=GRANULARITY):
+        self.byte = byte
+        self.offset = offset
+        self.size = size
+
+    def bits(self, granularity):
+        lower = self.offset // granularity
+        upper = (self.offset + self.size - 1) // granularity
+        return set(range(lower, upper + 1))
+
 
 class PatternGroup:
     """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
@@ -43,40 +50,39 @@ class PatternGroup:
         """Calculate the unique bits dirtied by this pattern grouping"""
         res = set()
         for pattern in self.patterns:
-            lower = pattern.offset // granularity
-            upper = (pattern.offset + pattern.size - 1) // granularity
-            res = res | set(range(lower, upper + 1))
+            res |= pattern.bits(granularity)
         return res
 
+
 GROUPS = [
     PatternGroup([
         # Batch 0: 4 clusters
-        mkpattern('0x49', 0x0000000),
-        mkpattern('0x6c', 0x0100000),   # 1M
-        mkpattern('0x6f', 0x2000000),   # 32M
-        mkpattern('0x76', 0x3ff0000)]), # 64M - 64K
+        Pattern('0x49', 0x0000000),
+        Pattern('0x6c', 0x0100000),   # 1M
+        Pattern('0x6f', 0x2000000),   # 32M
+        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
     PatternGroup([
         # Batch 1: 6 clusters (3 new)
-        mkpattern('0x65', 0x0000000),   # Full overwrite
-        mkpattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
-        mkpattern('0x72', 0x2008000),   # Partial-right (32M+32K)
-        mkpattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
+        Pattern('0x65', 0x0000000),   # Full overwrite
+        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
+        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
+        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
     PatternGroup([
         # Batch 2: 7 clusters (3 new)
-        mkpattern('0x74', 0x0010000),   # Adjacent-right
-        mkpattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
-        mkpattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
-        mkpattern('0x67', 0x3fe0000,
-                  2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
+        Pattern('0x74', 0x0010000),   # Adjacent-right
+        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
+        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
+        Pattern('0x67', 0x3fe0000,
+                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
     PatternGroup([
         # Batch 3: 8 clusters (5 new)
         # Carefully chosen such that nothing re-dirties the one cluster
         # that copies out successfully before failure in Group #1.
-        mkpattern('0xaa', 0x0010000,
-                  3*GRANULARITY),       # Overwrite and 2x Adjacent-right
-        mkpattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
-        mkpattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
-        mkpattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
+        Pattern('0xaa', 0x0010000,
+                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
+        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
+        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
+        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
 ]
 
 class Drive:
Signature · 1 line
2.21.0
Reviewed-by on v2 1/11
#2 lore raw

Re: [Qemu-devel] [PATCH v2 01/11] iotests/257: add Pattern class

Quoted 7 lines · John Snow

Just kidding, this is easier to manage with a full class instead of a namedtuple.

---
 tests/qemu-iotests/257 | 58 +++++++++++++++++++++++-------------------
 1 file changed, 32 insertions(+), 26 deletions(-)
author
#3 lore raw

PATCH v2 2/11 iotests/257: add EmulatedBitmap class

In reply to John Snow

Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.

---
 tests/qemu-iotests/257 | 124 +++++++++++++++++++++++++----------------
 1 file changed, 75 insertions(+), 49 deletions(-)
tests/qemu-iotests/257 +75 -49
index 02f9ae0649..bc66ea03b2 100755
@@ -85,6 +85,59 @@ GROUPS = [
         Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
 ]
 
+
+class EmulatedBitmap:
+    def __init__(self, granularity=GRANULARITY):
+        self._bits = set()
+        self.granularity = granularity
+
+    def dirty_bits(self, bits):
+        self._bits |= set(bits)
+
+    def dirty_group(self, n):
+        self.dirty_bits(GROUPS[n].bits(self.granularity))
+
+    def clear(self):
+        self._bits = set()
+
+    def clear_bits(self, bits):
+        self._bits -= set(bits)
+
+    def clear_bit(self, bit):
+        self.clear_bits({bit})
+
+    def clear_group(self, n):
+        self.clear_bits(GROUPS[n].bits(self.granularity))
+
+    @property
+    def first_bit(self):
+        return sorted(self.bits)[0]
+
+    @property
+    def bits(self):
+        return self._bits
+
+    @property
+    def count(self):
+        return len(self.bits)
+
+    def compare(self, qmp_bitmap):
+        """
+        Print a nice human-readable message checking that a bitmap as reported
+        by the QMP interface has as many bits set as we expect it to.
+        """
+
+        name = qmp_bitmap.get('name', '(anonymous)')
+        log("= Checking Bitmap {:s} =".format(name))
+
+        want = self.count
+        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
+
+        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
+            want, have, "OK!" if want == have else "ERROR!"))
+        log('')
+
+
 class Drive:
     """Represents, vaguely, a drive attached to a VM.
     Includes format, graph, and device information."""
@@ -195,27 +248,6 @@ def perform_writes(drive, n):
     log('')
     return bitmaps
 
-def calculate_bits(groups=None):
-    """Calculate how many bits we expect to see dirtied."""
-    if groups:
-        bits = set.union(*(GROUPS[group].bits(GRANULARITY) for group in groups))
-        return len(bits)
-    return 0
-
-def bitmap_comparison(bitmap, groups=None, want=0):
-    """
-    Print a nice human-readable message checking that this bitmap has as
-    many bits set as we expect it to.
-    """
-    log("= Checking Bitmap {:s} =".format(bitmap.get('name', '(anonymous)')))
-
-    if groups:
-        want = calculate_bits(groups)
-    have = bitmap['count'] // bitmap['granularity']
-
-    log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
-        want, have, "OK!" if want == have else "ERROR!"))
-    log('')
 
 def compare_images(image, reference, baseimg=None, expected_match=True):
     """
@@ -321,12 +353,13 @@ def test_bitmap_sync(bsync_mode, failure=None):
         vm.qmp_log("block-dirty-bitmap-add", node=drive0.name,
                    name="bitmap0", granularity=GRANULARITY)
         log('')
+        ebitmap = EmulatedBitmap()
 
         # 1 - Writes and Reference Backup
         bitmaps = perform_writes(drive0, 1)
-        dirty_groups = {1}
+        ebitmap.dirty_group(1)
         bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
-        bitmap_comparison(bitmap, groups=dirty_groups)
+        ebitmap.compare(bitmap)
         reference_backup(drive0, 1, fbackup1)
 
         # 1 - Bitmap Backup (Optional induced failure)
@@ -342,54 +375,47 @@ def test_bitmap_sync(bsync_mode, failure=None):
             log('')
             bitmaps = perform_writes(drive0, 2)
             # Named bitmap (static, should be unchanged)
-            bitmap_comparison(get_bitmap(bitmaps, drive0.device, 'bitmap0'),
-                              groups=dirty_groups)
+            ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
             # Anonymous bitmap (dynamic, shows new writes)
-            bitmap_comparison(get_bitmap(bitmaps, drive0.device, '',
-                                         recording=True), groups={2})
-            dirty_groups.add(2)
+            anonymous = EmulatedBitmap()
+            anonymous.dirty_group(2)
+            anonymous.compare(get_bitmap(bitmaps, drive0.device, '',
+                                         recording=True))
+
+            # Simulate the order in which this will happen:
+            # group 1 gets cleared first, then group two gets written.
+            if ((bsync_mode == 'on-success' and not failure) or
+                (bsync_mode == 'always')):
+                ebitmap.clear_group(1)
+            ebitmap.dirty_group(2)
 
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
                    pre_finalize=_callback,
                    cancel=(failure == 'simulated'))
         bitmaps = query_bitmaps(vm)
-        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
         log(bitmaps, indent=2)
         log('')
 
-        if ((bsync_mode == 'on-success' and not failure) or
-                (bsync_mode == 'always' and failure != 'intermediate')):
-            dirty_groups.remove(1)
-
         if bsync_mode == 'always' and failure == 'intermediate':
             # We manage to copy one sector (one bit) before the error.
-            bitmap_comparison(bitmap,
-                              want=calculate_bits(groups=dirty_groups) - 1)
-        else:
-            bitmap_comparison(bitmap, groups=dirty_groups)
+            ebitmap.clear_bit(ebitmap.first_bit)
+        ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
         bitmaps = perform_writes(drive0, 3)
-        dirty_groups.add(3)
-        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
-        if bsync_mode == 'always' and failure == 'intermediate':
-            # We're one bit short, still.
-            bitmap_comparison(bitmap,
-                              want=calculate_bits(groups=dirty_groups) - 1)
-        else:
-            bitmap_comparison(bitmap, groups=dirty_groups)
+        ebitmap.dirty_group(3)
+        ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
         reference_backup(drive0, 2, fbackup2)
 
         # 2 - Bitmap Backup (In failure modes, this is a recovery.)
         job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode)
         vm.run_job(job, auto_dismiss=True, auto_finalize=False)
         bitmaps = query_bitmaps(vm)
-        bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0')
         log(bitmaps, indent=2)
         log('')
-        bitmap_comparison(bitmap, groups={}
-                          if bsync_mode != 'never'
-                          else dirty_groups)
+        if bsync_mode != 'never':
+            ebitmap.clear()
+        ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         log('--- Cleanup ---\n')
         vm.qmp_log("block-dirty-bitmap-remove",
Signature · 1 line
2.21.0
Reviewed-by on v2 2/11
#4 lore raw

Re: [Qemu-devel] [PATCH v2 02/11] iotests/257: add EmulatedBitmap class

Quoted 8 lines · John Snow

Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.

---
 tests/qemu-iotests/257 | 124 +++++++++++++++++++++++++----------------
 1 file changed, 75 insertions(+), 49 deletions(-)
author
#5 lore raw

PATCH v2 3/11 iotests/257: Refactor backup helpers

In reply to John Snow

This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.

---
 tests/qemu-iotests/257     |  56 ++++++-----
 tests/qemu-iotests/257.out | 192 ++++++++++++++++++-------------------
 2 files changed, 128 insertions(+), 120 deletions(-)
tests/qemu-iotests/257 +32 -24
index bc66ea03b2..aaa8f59504 100755
@@ -207,31 +207,37 @@ def get_bitmap(bitmaps, drivename, name, recording=None):
                 return bitmap
     return None
 
+def blockdev_backup(vm, device, target, sync, **kwargs):
+    # Strip any arguments explicitly nulled by the caller:
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}
+    result = vm.qmp_log('blockdev-backup',
+                        device=device,
+                        target=target,
+                        sync=sync,
+                        **kwargs)
+    return result
+
+def blockdev_backup_mktarget(drive, target_id, filepath, sync, **kwargs):
+    target_drive = Drive(filepath, vm=drive.vm)
+    target_drive.create_target(target_id, drive.fmt, drive.size)
+    blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs)
+
 def reference_backup(drive, n, filepath):
     log("--- Reference Backup #{:d} ---\n".format(n))
     target_id = "ref_target_{:d}".format(n)
     job_id = "ref_backup_{:d}".format(n)
-    target_drive = Drive(filepath, vm=drive.vm)
-
-    target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup",
-                     job_id=job_id, device=drive.name,
-                     target=target_id, sync="full")
+    blockdev_backup_mktarget(drive, target_id, filepath, "full",
+                             job_id=job_id)
     drive.vm.run_job(job_id, auto_dismiss=True)
     log('')
 
-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode):
-    log("--- Bitmap Backup #{:d} ---\n".format(n))
-    target_id = "bitmap_target_{:d}".format(n)
-    job_id = "bitmap_backup_{:d}".format(n)
-    target_drive = Drive(filepath, vm=drive.vm)
-
-    target_drive.create_target(target_id, drive.fmt, drive.size)
-    drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name,
-                     target=target_id, sync="bitmap",
-                     bitmap_mode=bitmap_mode,
-                     bitmap=bitmap,
-                     auto_finalize=False)
+def backup(drive, n, filepath, sync, **kwargs):
+    log("--- Test Backup #{:d} ---\n".format(n))
+    target_id = "backup_target_{:d}".format(n)
+    job_id = "backup_{:d}".format(n)
+    kwargs.setdefault('auto-finalize', False)
+    blockdev_backup_mktarget(drive, target_id, filepath, sync,
+                             job_id=job_id, **kwargs)
     return job_id
 
 def perform_writes(drive, n):
@@ -263,7 +269,7 @@ def compare_images(image, reference, baseimg=None, expected_match=True):
         "OK!" if ret == expected_ret else "ERROR!"),
         filters=[iotests.filter_testfiles])
 
-def test_bitmap_sync(bsync_mode, failure=None):
+def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
     """
     Test bitmap backup routines.
 
@@ -291,7 +297,7 @@ def test_bitmap_sync(bsync_mode, failure=None):
                              fbackup0, fbackup1, fbackup2), \
          iotests.VM() as vm:
 
-        mode = "Bitmap Sync Mode {:s}".format(bsync_mode)
+        mode = "Mode {:s}; Bitmap Sync {:s}".format(msync_mode, bsync_mode)
         preposition = "with" if failure else "without"
         cond = "{:s} {:s}".format(preposition,
                                   "{:s} failure".format(failure) if failure
@@ -362,12 +368,13 @@ def test_bitmap_sync(bsync_mode, failure=None):
         ebitmap.compare(bitmap)
         reference_backup(drive0, 1, fbackup1)
 
-        # 1 - Bitmap Backup (Optional induced failure)
+        # 1 - Test Backup (w/ Optional induced failure)
         if failure == 'intermediate':
             # Activate blkdebug induced failure for second-to-next read
             log(vm.hmp_qemu_io(drive0.name, 'flush'))
             log('')
-        job = bitmap_backup(drive0, 1, bsync1, "bitmap0", bsync_mode)
+        job = backup(drive0, 1, bsync1, msync_mode,
+                     bitmap="bitmap0", bitmap_mode=bsync_mode)
 
         def _callback():
             """Issue writes while the job is open to test bitmap divergence."""
@@ -408,7 +415,8 @@ def test_bitmap_sync(bsync_mode, failure=None):
         reference_backup(drive0, 2, fbackup2)
 
         # 2 - Bitmap Backup (In failure modes, this is a recovery.)
-        job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode)
+        job = backup(drive0, 2, bsync2, "bitmap",
+                     bitmap="bitmap0", bitmap_mode=bsync_mode)
         vm.run_job(job, auto_dismiss=True, auto_finalize=False)
         bitmaps = query_bitmaps(vm)
         log(bitmaps, indent=2)
@@ -442,7 +450,7 @@ def test_bitmap_sync(bsync_mode, failure=None):
 def main():
     for bsync_mode in ("never", "on-success", "always"):
         for failure in ("simulated", "intermediate", None):
-            test_bitmap_sync(bsync_mode, failure)
+            test_bitmap_sync(bsync_mode, "bitmap", failure)
 
 if __name__ == '__main__':
     iotests.script_main(main, supported_fmts=['qcow2'])
tests/qemu-iotests/257.out +96 -96

523 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v2 3/11
#6 lore raw

Re: [Qemu-devel] [PATCH v2 03/11] iotests/257: Refactor backup helpers

Quoted 9 lines · John Snow

This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.

---
 tests/qemu-iotests/257     |  56 ++++++-----
 tests/qemu-iotests/257.out | 192 ++++++++++++++++++-------------------
 2 files changed, 128 insertions(+), 120 deletions(-)
author
#7 lore raw

PATCH v2 4/11 block/backup: hoist bitmap check into QMP interface

In reply to John Snow

This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.

---
 block/backup.c | 13 ++++---------
 blockdev.c     | 10 ++++++++++
 2 files changed, 14 insertions(+), 9 deletions(-)
block/backup.c +4 -9
index e2729cf6fa..a64b768e24 100644
@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
     assert(bs);
     assert(target);
 
+    /* QMP interface protects us from these cases */
+    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
+    assert(sync_bitmap || sync_mode != MIRROR_SYNC_MODE_BITMAP);
+
     if (bs == target) {
         error_setg(errp, "Source and target cannot be the same");
         return NULL;
@@ -597,16 +601,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    /* QMP interface should have handled translating this to bitmap mode */
-    assert(sync_mode != MIRROR_SYNC_MODE_INCREMENTAL);
-
     if (sync_mode == MIRROR_SYNC_MODE_BITMAP) {
-        if (!sync_bitmap) {
-            error_setg(errp, "must provide a valid bitmap name for "
-                       "'%s' sync mode", MirrorSyncMode_str(sync_mode));
-            return NULL;
-        }
-
         /* If we need to write to this bitmap, check that we can: */
         if (bitmap_mode != BITMAP_SYNC_MODE_NEVER &&
             bdrv_dirty_bitmap_check(sync_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
blockdev.c +10 -0
index 020d566c1b..3c76c85cb5 100644
@@ -3527,6 +3527,16 @@ static BlockJob *do_backup_common(BackupCommon *backup,
         return NULL;
     }
 
+    if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) ||
+        (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL)) {
+        /* done before desugaring 'incremental' to print the right message */
+        if (!backup->has_bitmap) {
+            error_setg(errp, "must provide a valid bitmap name for "
+                       "'%s' sync mode", MirrorSyncMode_str(backup->sync));
+            return NULL;
+        }
+    }
+
     if (backup->sync == MIRROR_SYNC_MODE_INCREMENTAL) {
         if (backup->has_bitmap_mode &&
             backup->bitmap_mode != BITMAP_SYNC_MODE_ON_SUCCESS) {
Signature · 1 line
2.21.0
author
#8 lore raw

PATCH v2 5/11 iotests/257: test API failures

In reply to John Snow
---
 tests/qemu-iotests/257     | 67 ++++++++++++++++++++++++++++++
 tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)
tests/qemu-iotests/257 +67 -0
index aaa8f59504..53ab31c92e 100755
@@ -447,10 +447,77 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         compare_images(img_path, fbackup2)
         log('')
 
+def test_backup_api():
+    """
+    Test malformed and prohibited invocations of the backup API.
+    """
+    with iotests.FilePaths(['img', 'bsync1']) as \
+         (img_path, backup_path), \
+         iotests.VM() as vm:
+
+        log("\n=== API failure tests ===\n")
+        log('--- Preparing image & VM ---\n')
+        drive0 = Drive(img_path, vm=vm)
+        drive0.img_create(iotests.imgfmt, SIZE)
+        vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device()))
+        vm.launch()
+
+        file_config = {
+            'driver': 'file',
+            'filename': drive0.path
+        }
+
+        vm.qmp_log('blockdev-add',
+                   filters=[iotests.filter_qmp_testfiles],
+                   node_name="drive0",
+                   driver=drive0.fmt,
+                   file=file_config)
+        drive0.node = 'drive0'
+        drive0.device = 'device0'
+        vm.qmp_log("device_add", id=drive0.device,
+                   drive=drive0.name, driver="scsi-hd")
+        log('')
+
+        target0 = Drive(backup_path, vm=vm)
+        target0.create_target("backup_target", drive0.fmt, drive0.size)
+        log('')
+
+        vm.qmp_log("block-dirty-bitmap-add", node=drive0.name,
+                   name="bitmap0", granularity=GRANULARITY)
+        log('')
+
+        log('-- Testing invalid QMP commands --\n')
+
+        error_cases = {
+            'incremental': {
+                None:        ['on-success', 'always', 'never', None],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['always', 'never']
+            },
+            'bitmap': {
+                None:        ['on-success', 'always', 'never', None],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   [None],
+            },
+        }
+
+        # Dicts, as always, are not stably-ordered prior to 3.7, so use tuples:
+        for sync_mode in ('incremental', 'bitmap'):
+            log("-- Sync mode {:s} tests --\n".format(sync_mode))
+            for bitmap in (None, 'bitmap404', 'bitmap0'):
+                for policy in error_cases[sync_mode][bitmap]:
+                    blockdev_backup(drive0.vm, drive0.name, "backup_target",
+                                    sync_mode, job_id='api_job',
+                                    bitmap=bitmap, bitmap_mode=policy)
+                    log('')
+
+
 def main():
     for bsync_mode in ("never", "on-success", "always"):
         for failure in ("simulated", "intermediate", None):
             test_bitmap_sync(bsync_mode, "bitmap", failure)
 
+    test_backup_api()
+
 if __name__ == '__main__':
     iotests.script_main(main, supported_fmts=['qcow2'])
tests/qemu-iotests/257.out +85 -0

88 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v2 5/11
#9 lore raw

Re: [Qemu-devel] [PATCH v2 05/11] iotests/257: test API failures

Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     | 67 ++++++++++++++++++++++++++++++
 tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 152 insertions(+)
author
#10 lore raw

PATCH v2 6/11 block/backup: improve sync=bitmap work estimates

In reply to John Snow

When making backups based on bitmaps, the work estimate can be more accurate. Update iotests to reflect the new strategy.

TOP work estimates are broken, but do not get worse with this commit. That issue is addressed in the following commits instead.

---
 block/backup.c             |  8 +++-----
 tests/qemu-iotests/256.out |  4 ++--
 tests/qemu-iotests/257.out | 36 ++++++++++++++++++------------------
 3 files changed, 23 insertions(+), 25 deletions(-)
block/backup.c +3 -5
index a64b768e24..22fafbb80f 100644
@@ -449,9 +449,8 @@ static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
                                                 NULL, true);
     assert(ret);
 
-    /* TODO job_progress_set_remaining() would make more sense */
-    job_progress_update(&job->common.job,
-                        job->len - bdrv_get_dirty_count(job->copy_bitmap));
+    job_progress_set_remaining(&job->common.job,
+                               bdrv_get_dirty_count(job->copy_bitmap));
 }
 
 static int coroutine_fn backup_run(Job *job, Error **errp)
@@ -463,12 +462,11 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
     QLIST_INIT(&s->inflight_reqs);
     qemu_co_rwlock_init(&s->flush_rwlock);
 
-    job_progress_set_remaining(job, s->len);
-
     if (s->sync_mode == MIRROR_SYNC_MODE_BITMAP) {
         backup_incremental_init_copy_bitmap(s);
     } else {
         bdrv_set_dirty_bitmap(s->copy_bitmap, 0, s->len);
+        job_progress_set_remaining(job, s->len);
     }
 
     s->before_write.notify = backup_before_write_notify;
tests/qemu-iotests/256.out +2 -2
index eec38614ec..f18ecb0f91 100644
@@ -113,7 +113,7 @@
 {
   "return": {}
 }
-{"data": {"device": "j2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "j3", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j2", "len": 0, "offset": 0, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "j3", "len": 0, "offset": 0, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 --- Done ---
tests/qemu-iotests/257.out +18 -18
index 43f2e0f9c9..811b1b11f1 100644
@@ -150,7 +150,7 @@ expecting 7 dirty sectors; have 7. OK!
 {"execute": "job-cancel", "arguments": {"id": "backup_1"}}
 {"return": {}}
 {"data": {"id": "backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "len": 393216, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -228,7 +228,7 @@ expecting 15 dirty sectors; have 15. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 983040, "offset": 983040, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -367,7 +367,7 @@ expecting 6 dirty sectors; have 6. OK!
 {"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "never", "device": "drive0", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}}
 {"return": {}}
 {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "error": "Input/output error", "len": 67108864, "offset": 66781184, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "error": "Input/output error", "len": 393216, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -445,7 +445,7 @@ expecting 14 dirty sectors; have 14. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 917504, "offset": 917504, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -633,7 +633,7 @@ expecting 7 dirty sectors; have 7. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_1"}}
 {"return": {}}
 {"data": {"id": "backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "len": 393216, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -711,7 +711,7 @@ expecting 15 dirty sectors; have 15. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 983040, "offset": 983040, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -899,7 +899,7 @@ expecting 7 dirty sectors; have 7. OK!
 {"execute": "job-cancel", "arguments": {"id": "backup_1"}}
 {"return": {}}
 {"data": {"id": "backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "len": 393216, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -977,7 +977,7 @@ expecting 15 dirty sectors; have 15. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 983040, "offset": 983040, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1116,7 +1116,7 @@ expecting 6 dirty sectors; have 6. OK!
 {"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "on-success", "device": "drive0", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}}
 {"return": {}}
 {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "error": "Input/output error", "len": 67108864, "offset": 66781184, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "error": "Input/output error", "len": 393216, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1194,7 +1194,7 @@ expecting 14 dirty sectors; have 14. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 917504, "offset": 917504, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1382,7 +1382,7 @@ expecting 7 dirty sectors; have 7. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_1"}}
 {"return": {}}
 {"data": {"id": "backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "len": 393216, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1460,7 +1460,7 @@ expecting 12 dirty sectors; have 12. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 786432, "offset": 786432, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1648,7 +1648,7 @@ expecting 7 dirty sectors; have 7. OK!
 {"execute": "job-cancel", "arguments": {"id": "backup_1"}}
 {"return": {}}
 {"data": {"id": "backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "len": 393216, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1726,7 +1726,7 @@ expecting 12 dirty sectors; have 12. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 786432, "offset": 786432, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1865,7 +1865,7 @@ expecting 6 dirty sectors; have 6. OK!
 {"execute": "blockdev-backup", "arguments": {"auto-finalize": false, "bitmap": "bitmap0", "bitmap-mode": "always", "device": "drive0", "job-id": "backup_1", "sync": "bitmap", "target": "backup_target_1"}}
 {"return": {}}
 {"data": {"action": "report", "device": "backup_1", "operation": "read"}, "event": "BLOCK_JOB_ERROR", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "error": "Input/output error", "len": 67108864, "offset": 66781184, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "error": "Input/output error", "len": 393216, "offset": 65536, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -1943,7 +1943,7 @@ expecting 13 dirty sectors; have 13. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 851968, "offset": 851968, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -2131,7 +2131,7 @@ expecting 7 dirty sectors; have 7. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_1"}}
 {"return": {}}
 {"data": {"id": "backup_1", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_1", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_1", "len": 393216, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
@@ -2209,7 +2209,7 @@ expecting 12 dirty sectors; have 12. OK!
 {"execute": "job-finalize", "arguments": {"id": "backup_2"}}
 {"return": {}}
 {"data": {"id": "backup_2", "type": "backup"}, "event": "BLOCK_JOB_PENDING", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
-{"data": {"device": "backup_2", "len": 67108864, "offset": 67108864, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"device": "backup_2", "len": 786432, "offset": 786432, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 {
   "bitmaps": {
     "device0": [
Signature · 1 line
2.21.0
Reviewed-by on v2 6/11
#11 lore raw

Re: [Qemu-devel] [PATCH v2 06/11] block/backup: improve sync=bitmap work estimates

Quoted 6 lines · John Snow

When making backups based on bitmaps, the work estimate can be more accurate. Update iotests to reflect the new strategy.

TOP work estimates are broken, but do not get worse with this commit. That issue is addressed in the following commits instead.

Quoted 6 lines
---
 block/backup.c             |  8 +++-----
 tests/qemu-iotests/256.out |  4 ++--
 tests/qemu-iotests/257.out | 36 ++++++++++++++++++------------------
 3 files changed, 23 insertions(+), 25 deletions(-)
author
#12 lore raw

PATCH v2 7/11 block/backup: centralize copy_bitmap initialization

In reply to John Snow

Just a few housekeeping changes that keeps the following commit easier to read; perform the initial copy_bitmap initialization in one place.

---
 block/backup.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
block/backup.c +15 -14
index 22fafbb80f..c88a70fe10 100644
@@ -441,16 +441,22 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)
     return ret;
 }
 
-/* init copy_bitmap from sync_bitmap */
-static void backup_incremental_init_copy_bitmap(BackupBlockJob *job)
+static void backup_init_copy_bitmap(BackupBlockJob *job)
 {
-    bool ret = bdrv_dirty_bitmap_merge_internal(job->copy_bitmap,
-                                                job->sync_bitmap,
-                                                NULL, true);
-    assert(ret);
+    bool ret;
+    uint64_t estimate;
 
-    job_progress_set_remaining(&job->common.job,
-                               bdrv_get_dirty_count(job->copy_bitmap));
+    if (job->sync_mode == MIRROR_SYNC_MODE_BITMAP) {
+        ret = bdrv_dirty_bitmap_merge_internal(job->copy_bitmap,
+                                               job->sync_bitmap,
+                                               NULL, true);
+        assert(ret);
+    } else {
+        bdrv_set_dirty_bitmap(job->copy_bitmap, 0, job->len);
+    }
+
+    estimate = bdrv_get_dirty_count(job->copy_bitmap);
+    job_progress_set_remaining(&job->common.job, estimate);
 }
 
 static int coroutine_fn backup_run(Job *job, Error **errp)
@@ -462,12 +468,7 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
     QLIST_INIT(&s->inflight_reqs);
     qemu_co_rwlock_init(&s->flush_rwlock);
 
-    if (s->sync_mode == MIRROR_SYNC_MODE_BITMAP) {
-        backup_incremental_init_copy_bitmap(s);
-    } else {
-        bdrv_set_dirty_bitmap(s->copy_bitmap, 0, s->len);
-        job_progress_set_remaining(job, s->len);
-    }
+    backup_init_copy_bitmap(s);
 
     s->before_write.notify = backup_before_write_notify;
     bdrv_add_before_write_notifier(bs, &s->before_write);
Signature · 1 line
2.21.0
Reviewed-by on v2 7/11
#13 lore raw

Re: [Qemu-devel] [PATCH v2 07/11] block/backup: centralize copy_bitmap initialization

Quoted 7 lines · John Snow

Just a few housekeeping changes that keeps the following commit easier to read; perform the initial copy_bitmap initialization in one place.

---
 block/backup.c | 29 +++++++++++++++--------------
 1 file changed, 15 insertions(+), 14 deletions(-)
author
#14 lore raw

PATCH v2 8/11 block/backup: add backup_is_cluster_allocated

In reply to John Snow

Modify the existing bdrv_is_unallocated_range to utilize the pnum return from bdrv_is_allocated; optionally returning a full number of clusters that share the same allocation status.

This will be used to carefully toggle bits in the bitmap for sync=top initialization in the following commits.

---
 block/backup.c | 62 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 44 insertions(+), 18 deletions(-)
block/backup.c +44 -18
index c88a70fe10..b407d57954 100644
@@ -185,6 +185,48 @@ static int coroutine_fn backup_cow_with_offload(BackupBlockJob *job,
     return nbytes;
 }
 
+/*
+ * Check if the cluster starting at offset is allocated or not.
+ * return via pnum the number of contiguous clusters sharing this allocation.
+ */
+static int backup_is_cluster_allocated(BackupBlockJob *s, int64_t offset,
+                                       int64_t *pnum)
+{
+    BlockDriverState *bs = blk_bs(s->common.blk);
+    int64_t count, total_count = 0;
+    int64_t bytes = s->len - offset;
+    int ret;
+
+    assert(QEMU_IS_ALIGNED(offset, s->cluster_size));
+
+    while (true) {
+        ret = bdrv_is_allocated(bs, offset, bytes, &count);
+        if (ret < 0) {
+            return ret;
+        }
+
+        total_count += count;
+
+        if (ret || count == 0) {
+            /*
+             * ret: partial segment(s) are considered allocated.
+             * otherwise: unallocated tail is treated as an entire segment.
+             */
+            *pnum = DIV_ROUND_UP(total_count, s->cluster_size);
+            return ret;
+        }
+
+        /* Unallocated segment(s) with uncertain following segment(s) */
+        if (total_count >= s->cluster_size) {
+            *pnum = total_count / s->cluster_size;
+            return 0;
+        }
+
+        offset += count;
+        bytes -= count;
+    }
+}
+
 static int coroutine_fn backup_do_cow(BackupBlockJob *job,
                                       int64_t offset, uint64_t bytes,
                                       bool *error_is_read,
@@ -388,34 +430,18 @@ static bool coroutine_fn yield_and_check(BackupBlockJob *job)
     return false;
 }
 
-static bool bdrv_is_unallocated_range(BlockDriverState *bs,
-                                      int64_t offset, int64_t bytes)
-{
-    int64_t end = offset + bytes;
-
-    while (offset < end && !bdrv_is_allocated(bs, offset, bytes, &bytes)) {
-        if (bytes == 0) {
-            return true;
-        }
-        offset += bytes;
-        bytes = end - offset;
-    }
-
-    return offset >= end;
-}
-
 static int coroutine_fn backup_loop(BackupBlockJob *job)
 {
     bool error_is_read;
     int64_t offset;
     BdrvDirtyBitmapIter *bdbi;
-    BlockDriverState *bs = blk_bs(job->common.blk);
     int ret = 0;
+    int64_t dummy;
 
     bdbi = bdrv_dirty_iter_new(job->copy_bitmap);
     while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) {
         if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
-            bdrv_is_unallocated_range(bs, offset, job->cluster_size))
+            !backup_is_cluster_allocated(job, offset, &dummy))
         {
             bdrv_reset_dirty_bitmap(job->copy_bitmap, offset,
                                     job->cluster_size);
Signature · 1 line
2.21.0
Reviewed-by on v2 8/11
#15 lore raw

Re: [Qemu-devel] [PATCH v2 08/11] block/backup: add backup_is_cluster_allocated

Quoted 11 lines · John Snow

Modify the existing bdrv_is_unallocated_range to utilize the pnum return from bdrv_is_allocated; optionally returning a full number of clusters that share the same allocation status.

This will be used to carefully toggle bits in the bitmap for sync=top initialization in the following commits.

---
 block/backup.c | 62 +++++++++++++++++++++++++++++++++++---------------
 1 file changed, 44 insertions(+), 18 deletions(-)
author
#16 lore raw

PATCH v2 9/11 block/backup: teach TOP to never copy unallocated regions

In reply to John Snow

Presently, If sync=TOP is selected, we mark the entire bitmap as dirty. In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Taken together, the floor and ceiling move to meet each other.

---
 block/backup.c     | 78 ++++++++++++++++++++++++++++++++++++++++------
 block/trace-events |  1 +
 2 files changed, 70 insertions(+), 9 deletions(-)
block/backup.c +69 -9
index b407d57954..e28fd23f6a 100644
@@ -58,6 +58,7 @@ typedef struct BackupBlockJob {
     int64_t copy_range_size;
 
     bool serialize_target_writes;
+    bool initializing_bitmap;
 } BackupBlockJob;
 
 static const BlockJobDriver backup_job_driver;
@@ -227,6 +228,35 @@ static int backup_is_cluster_allocated(BackupBlockJob *s, int64_t offset,
     }
 }
 
+/**
+ * Reset bits in copy_bitmap starting at offset if they represent unallocated
+ * data in the image. May reset subsequent contiguous bits.
+ * @return 0 when the cluster at @offset was unallocated,
+ *         1 otherwise, and -ret on error.
+ */
+static int64_t backup_bitmap_reset_unallocated(BackupBlockJob *s,
+                                               int64_t offset, int64_t *count)
+{
+    int ret;
+    int64_t clusters, bytes, estimate;
+
+    ret = backup_is_cluster_allocated(s, offset, &clusters);
+    if (ret < 0) {
+        return ret;
+    }
+
+    bytes = clusters * s->cluster_size;
+
+    if (!ret) {
+        bdrv_reset_dirty_bitmap(s->copy_bitmap, offset, bytes);
+        estimate = bdrv_get_dirty_count(s->copy_bitmap);
+        job_progress_set_remaining(&s->common.job, estimate);
+    }
+
+    *count = bytes;
+    return ret;
+}
+
 static int coroutine_fn backup_do_cow(BackupBlockJob *job,
                                       int64_t offset, uint64_t bytes,
                                       bool *error_is_read,
@@ -236,6 +266,7 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
     int ret = 0;
     int64_t start, end; /* bytes */
     void *bounce_buffer = NULL;
+    int64_t skip_bytes;
 
     qemu_co_rwlock_rdlock(&job->flush_rwlock);
 
@@ -254,6 +285,15 @@ static int coroutine_fn backup_do_cow(BackupBlockJob *job,
             continue; /* already copied */
         }
 
+        if (job->initializing_bitmap) {
+            ret = backup_bitmap_reset_unallocated(job, start, &skip_bytes);
+            if (ret == 0) {
+                trace_backup_do_cow_skip_range(job, start, skip_bytes);
+                start += skip_bytes;
+                continue;
+            }
+        }
+
         trace_backup_do_cow_process(job, start);
 
         if (job->use_copy_range) {
@@ -436,18 +476,9 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)
     int64_t offset;
     BdrvDirtyBitmapIter *bdbi;
     int ret = 0;
-    int64_t dummy;
 
     bdbi = bdrv_dirty_iter_new(job->copy_bitmap);
     while ((offset = bdrv_dirty_iter_next(bdbi)) != -1) {
-        if (job->sync_mode == MIRROR_SYNC_MODE_TOP &&
-            !backup_is_cluster_allocated(job, offset, &dummy))
-        {
-            bdrv_reset_dirty_bitmap(job->copy_bitmap, offset,
-                                    job->cluster_size);
-            continue;
-        }
-
         do {
             if (yield_and_check(job)) {
                 goto out;
@@ -478,6 +509,13 @@ static void backup_init_copy_bitmap(BackupBlockJob *job)
                                                NULL, true);
         assert(ret);
     } else {
+        if (job->sync_mode == MIRROR_SYNC_MODE_TOP) {
+            /*
+             * We can't hog the coroutine to initialize this thoroughly.
+             * Set a flag and resume work when we are able to yield safely.
+             */
+            job->initializing_bitmap = true;
+        }
         bdrv_set_dirty_bitmap(job->copy_bitmap, 0, job->len);
     }
 
@@ -499,6 +537,26 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
     s->before_write.notify = backup_before_write_notify;
     bdrv_add_before_write_notifier(bs, &s->before_write);
 
+    if (s->sync_mode == MIRROR_SYNC_MODE_TOP) {
+        int64_t offset = 0;
+        int64_t count;
+
+        for (offset = 0; offset < s->len; ) {
+            if (yield_and_check(s)) {
+                ret = -ECANCELED;
+                goto out;
+            }
+
+            ret = backup_bitmap_reset_unallocated(s, offset, &count);
+            if (ret < 0) {
+                goto out;
+            }
+
+            offset += count;
+        }
+        s->initializing_bitmap = false;
+    }
+
     if (s->sync_mode == MIRROR_SYNC_MODE_NONE) {
         /* All bits are set in copy_bitmap to allow any cluster to be copied.
          * This does not actually require them to be copied. */
@@ -507,10 +565,12 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
              * notify callback service CoW requests. */
             job_yield(job);
         }
+        ret = -ECANCELED;
     } else {
         ret = backup_loop(s);
     }
 
+ out:
     notifier_with_return_remove(&s->before_write);
 
     /* wait until pending backup_do_cow() calls have completed */
block/trace-events +1 -0
index d724df0117..04209f058d 100644
@@ -41,6 +41,7 @@ mirror_yield_in_flight(void *s, int64_t offset, int in_flight) "s %p offset %" P
 backup_do_cow_enter(void *job, int64_t start, int64_t offset, uint64_t bytes) "job %p start %" PRId64 " offset %" PRId64 " bytes %" PRIu64
 backup_do_cow_return(void *job, int64_t offset, uint64_t bytes, int ret) "job %p offset %" PRId64 " bytes %" PRIu64 " ret %d"
 backup_do_cow_skip(void *job, int64_t start) "job %p start %"PRId64
+backup_do_cow_skip_range(void *job, int64_t start, uint64_t bytes) "job %p start %"PRId64" bytes %"PRId64
 backup_do_cow_process(void *job, int64_t start) "job %p start %"PRId64
 backup_do_cow_read_fail(void *job, int64_t start, int ret) "job %p start %"PRId64" ret %d"
 backup_do_cow_write_fail(void *job, int64_t start, int ret) "job %p start %"PRId64" ret %d"
Signature · 1 line
2.21.0
#17 lore raw

Re: [Qemu-devel] [PATCH v2 09/11] block/backup: teach TOP to never copy unallocated regions

Quoted 18 lines · John Snow

Presently, If sync=TOP is selected, we mark the entire bitmap as dirty. In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Quoted 7 lines

Taken together, the floor and ceiling move to meet each other.

---
 block/backup.c     | 78 ++++++++++++++++++++++++++++++++++++++++------
 block/trace-events |  1 +
 2 files changed, 70 insertions(+), 9 deletions(-)

Looks good to me but for a seemingly unrelated change:

Quoted 4 lines
block/backup.c +0 -0
index b407d57954..e28fd23f6a 100644

[...]

block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 568

Quoted 5 lines
block/backup.c +1 -0
@@ -507,10 +565,12 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
              * notify callback service CoW requests. */
             job_yield(job);
         }
+        ret = -ECANCELED;

This one. This doesn’t look like it belongs in this patch, and I’m not even sure it’s correct. Being cancelled is the normal state for sync=none, so I suppose it is correct to just return 0 then.

Max

Quoted 8 lines
block/backup.c +1 -0
     } else {
         ret = backup_loop(s);
     }
 
+ out:
     notifier_with_return_remove(&s->before_write);
 
     /* wait until pending backup_do_cow() calls have completed */
author
#18 lore raw

Re: [Qemu-devel] [PATCH v2 09/11] block/backup: teach TOP to never copy unallocated regions

Quoted 46 lines · Max Reitz
Quoted 18 lines · John Snow

Presently, If sync=TOP is selected, we mark the entire bitmap as dirty. In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Quoted 7 lines

Taken together, the floor and ceiling move to meet each other.

---
 block/backup.c     | 78 ++++++++++++++++++++++++++++++++++++++++------
 block/trace-events |  1 +
 2 files changed, 70 insertions(+), 9 deletions(-)

Looks good to me but for a seemingly unrelated change:

Quoted 4 lines
block/backup.c +0 -0
index b407d57954..e28fd23f6a 100644

[...]

block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 568

Quoted 5 lines
block/backup.c +1 -0
@@ -507,10 +565,12 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
              * notify callback service CoW requests. */
             job_yield(job);
         }
+        ret = -ECANCELED;

This one. This doesn’t look like it belongs in this patch, and I’m not even sure it’s correct. Being cancelled is the normal state for sync=none, so I suppose it is correct to just return 0 then.

Quoted 2 lines

Max

Yeah, this is wiggly, so... yes, we can return 0 here. The job infrastructure machinery is going to change it to an ECANCELED for us anyway:

job_completed
job_update_rc

    if (!job->ret && job_is_cancelled(job)) {
        job->ret = -ECANCELED;
    }

So in this case I just figured that I might as well make it explicit; this is an error exit.

(I guess just leaving it at 0 means "whatever the job machinery thinks" too, which is probably also fine. The job machinery does not distinguish between "canceled and 0" or "canceled and < 0".)

Since we're here, though... I was wondering if it shouldn't be the case that "canceling" a sync=none job should actually result in success, unless you force-cancel. OR, allow sync=none jobs to receive "COMPLETE" verbs to finish successfully, or "CANCEL" verbs to terminate with error.

(I don't like what mirror does and don't wish to mimic it. I continue to dislike the idea that canceling a ready mirror job allows it to complete with a successful error code.)

Quoted 9 lines
Quoted 8 lines
block/backup.c +1 -0
     } else {
         ret = backup_loop(s);
     }
 
+ out:
     notifier_with_return_remove(&s->before_write);
 
     /* wait until pending backup_do_cow() calls have completed */
#19 lore raw

Re: [Qemu-devel] [PATCH v2 09/11] block/backup: teach TOP to never copy unallocated regions

Quoted 61 lines · John Snow
Quoted 46 lines · Max Reitz
Quoted 18 lines · John Snow

Presently, If sync=TOP is selected, we mark the entire bitmap as dirty. In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Quoted 7 lines

Taken together, the floor and ceiling move to meet each other.

---
 block/backup.c     | 78 ++++++++++++++++++++++++++++++++++++++++------
 block/trace-events |  1 +
 2 files changed, 70 insertions(+), 9 deletions(-)

Looks good to me but for a seemingly unrelated change:

Quoted 4 lines
block/backup.c +0 -0
index b407d57954..e28fd23f6a 100644

[...]

block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 568

Quoted 5 lines
block/backup.c +1 -0
@@ -507,10 +565,12 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
              * notify callback service CoW requests. */
             job_yield(job);
         }
+        ret = -ECANCELED;

This one. This doesn’t look like it belongs in this patch, and I’m not even sure it’s correct. Being cancelled is the normal state for sync=none, so I suppose it is correct to just return 0 then.

Quoted 2 lines

Max

Yeah, this is wiggly, so... yes, we can return 0 here. The job infrastructure machinery is going to change it to an ECANCELED for us anyway:

job_completed
job_update_rc

    if (!job->ret && job_is_cancelled(job)) {
        job->ret = -ECANCELED;
    }
Quoted 6 lines

So in this case I just figured that I might as well make it explicit; this is an error exit.

(I guess just leaving it at 0 means "whatever the job machinery thinks" too, which is probably also fine. The job machinery does not distinguish between "canceled and 0" or "canceled and < 0".)

Hm, OK. I think it should be an own patch, though.

Quoted 4 lines

Since we're here, though... I was wondering if it shouldn't be the case that "canceling" a sync=none job should actually result in success, unless you force-cancel. OR, allow sync=none jobs to receive "COMPLETE" verbs to finish successfully, or "CANCEL" verbs to terminate with error.

That’s what I had thought. (That canceling would be a success.)

As for COMPLETE, you want it to emit a READY event right when it’s started? :-)

Quoted 3 lines

(I don't like what mirror does and don't wish to mimic it. I continue to dislike the idea that canceling a ready mirror job allows it to complete with a successful error code.)

Hm. Actually, I don’t even care that much. If the user canceled it, they probably won’t really look at the return code anyway...

Max

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
block/backup.c +1 -0
     } else {
         ret = backup_loop(s);
     }
 
+ out:
     notifier_with_return_remove(&s->before_write);
 
     /* wait until pending backup_do_cow() calls have completed */
author
#20 lore raw

Re: [Qemu-devel] [PATCH v2 09/11] block/backup: teach TOP to never copy unallocated regions

Quoted 62 lines · Max Reitz
Quoted 61 lines · John Snow
Quoted 46 lines · Max Reitz
Quoted 18 lines · John Snow

Presently, If sync=TOP is selected, we mark the entire bitmap as dirty. In the write notifier handler, we dutifully copy out such regions.

Fix this in three parts:

1. Mark the bitmap as being initialized before the first yield.
2. After the first yield but before the backup loop, interrogate the allocation status asynchronously and initialize the bitmap.
3. Teach the write notifier to interrogate allocation status if it is invoked during bitmap initialization.

As an effect of this patch, the job progress for TOP backups now behaves like this:

- total progress starts at bdrv_length.
- As allocation status is interrogated, total progress decreases.
- As blocks are copied, current progress increases.

Quoted 7 lines

Taken together, the floor and ceiling move to meet each other.

---
 block/backup.c     | 78 ++++++++++++++++++++++++++++++++++++++++------
 block/trace-events |  1 +
 2 files changed, 70 insertions(+), 9 deletions(-)

Looks good to me but for a seemingly unrelated change:

Quoted 4 lines
block/backup.c +0 -0
index b407d57954..e28fd23f6a 100644

[...]

block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 568

Quoted 5 lines
block/backup.c +1 -0
@@ -507,10 +565,12 @@ static int coroutine_fn backup_run(Job *job, Error **errp)
              * notify callback service CoW requests. */
             job_yield(job);
         }
+        ret = -ECANCELED;

This one. This doesn’t look like it belongs in this patch, and I’m not even sure it’s correct. Being cancelled is the normal state for sync=none, so I suppose it is correct to just return 0 then.

Quoted 2 lines

Max

Yeah, this is wiggly, so... yes, we can return 0 here. The job infrastructure machinery is going to change it to an ECANCELED for us anyway:

job_completed
job_update_rc

    if (!job->ret && job_is_cancelled(job)) {
        job->ret = -ECANCELED;
    }
Quoted 9 lines
Quoted 6 lines

So in this case I just figured that I might as well make it explicit; this is an error exit.

(I guess just leaving it at 0 means "whatever the job machinery thinks" too, which is probably also fine. The job machinery does not distinguish between "canceled and 0" or "canceled and < 0".)

Hm, OK. I think it should be an own patch, though.

Path of least edits.

I'm removing this one change (since it doesn't change anything anyway), making the error message capitalization fix in the reference output, and staging the series.

--js

author
#21 lore raw

PATCH v2 10/11 block/backup: support bitmap sync modes for non-bitmap backups

In reply to John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---
 block/backup.c       |  8 +-------
 blockdev.c           | 22 ++++++++++++++++++++++
 qapi/block-core.json |  6 ++++--
 3 files changed, 27 insertions(+), 9 deletions(-)
block/backup.c +1 -7
index e28fd23f6a..47628aca24 100644
@@ -686,7 +686,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    if (sync_mode == MIRROR_SYNC_MODE_BITMAP) {
+    if (sync_bitmap) {
         /* If we need to write to this bitmap, check that we can: */
         if (bitmap_mode != BITMAP_SYNC_MODE_NEVER &&
             bdrv_dirty_bitmap_check(sync_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
@@ -697,12 +697,6 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         if (bdrv_dirty_bitmap_create_successor(bs, sync_bitmap, errp) < 0) {
             return NULL;
         }
-    } else if (sync_bitmap) {
-        error_setg(errp,
-                   "a bitmap was given to backup_job_create, "
-                   "but it received an incompatible sync_mode (%s)",
-                   MirrorSyncMode_str(sync_mode));
-        return NULL;
     }
 
     len = bdrv_getlength(bs);
blockdev.c +22 -0
index 3c76c85cb5..29c6c6044a 100644
@@ -3565,6 +3565,28 @@ static BlockJob *do_backup_common(BackupCommon *backup,
         if (bdrv_dirty_bitmap_check(bmap, BDRV_BITMAP_ALLOW_RO, errp)) {
             return NULL;
         }
+
+        /* This does not produce a useful bitmap artifact: */
+        if (backup->sync == MIRROR_SYNC_MODE_NONE) {
+            error_setg(errp, "sync mode '%s' does not produce meaningful bitmap"
+                       " outputs", MirrorSyncMode_str(backup->sync));
+            return NULL;
+        }
+
+        /* If the bitmap isn't used for input or output, this is useless: */
+        if (backup->bitmap_mode == BITMAP_SYNC_MODE_NEVER &&
+            backup->sync != MIRROR_SYNC_MODE_BITMAP) {
+            error_setg(errp, "Bitmap sync mode '%s' has no meaningful effect"
+                       " when combined with sync mode '%s'",
+                       BitmapSyncMode_str(backup->bitmap_mode),
+                       MirrorSyncMode_str(backup->sync));
+            return NULL;
+        }
+    }
+
+    if (!backup->has_bitmap && backup->has_bitmap_mode) {
+        error_setg(errp, "Cannot specify bitmap sync mode without a bitmap");
+        return NULL;
     }
 
     if (!backup->auto_finalize) {
qapi/block-core.json +4 -2
index 5a578806c5..099e4f37b2 100644
@@ -1352,13 +1352,15 @@
 # @speed: the maximum speed, in bytes per second. The default is 0,
 #         for unlimited.
 #
-# @bitmap: the name of a dirty bitmap if sync is "bitmap" or "incremental".
+# @bitmap: The name of a dirty bitmap to use.
 #          Must be present if sync is "bitmap" or "incremental".
+#          Can be present if sync is "full" or "top".
 #          Must not be present otherwise.
 #          (Since 2.4 (drive-backup), 3.1 (blockdev-backup))
 #
 # @bitmap-mode: Specifies the type of data the bitmap should contain after
-#               the operation concludes. Must be present if sync is "bitmap".
+#               the operation concludes.
+#               Must be present if a bitmap was provided,
 #               Must NOT be present otherwise. (Since 4.2)
 #
 # @compress: true to compress data, if the target format supports it.
Signature · 1 line
2.21.0
#22 lore raw

Re: [Qemu-devel] [PATCH v2 10/11] block/backup: support bitmap sync modes for non-bitmap backups

Quoted 22 lines · John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---

[...]

Quoted 12 lines
qapi/block-core.json +2 -1
index 5a578806c5..099e4f37b2 100644
@@ -1352,13 +1352,15 @@
 # @speed: the maximum speed, in bytes per second. The default is 0,
 #         for unlimited.
 #
-# @bitmap: the name of a dirty bitmap if sync is "bitmap" or "incremental".
+# @bitmap: The name of a dirty bitmap to use.
 #          Must be present if sync is "bitmap" or "incremental".
+#          Can be present if sync is "full" or "top".

qapi/block-core.json line 1366

Quoted 10 lines
qapi/block-core.json +2 -1
 #          Must not be present otherwise.
 #          (Since 2.4 (drive-backup), 3.1 (blockdev-backup))
 #
 # @bitmap-mode: Specifies the type of data the bitmap should contain after
-#               the operation concludes. Must be present if sync is "bitmap".
+#               the operation concludes.
+#               Must be present if a bitmap was provided,
 #               Must NOT be present otherwise. (Since 4.2)
 #
 # @compress: true to compress data, if the target format supports it.

Do you expect management applications will want to know about the presence of this patch?

author
#23 lore raw

Re: [Qemu-devel] [PATCH v2 10/11] block/backup: support bitmap sync modes for non-bitmap backups

Quoted 42 lines · Markus Armbruster
Quoted 22 lines · John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

---

[...]

Quoted 17 lines
qapi/block-core.json +2 -2
index 5a578806c5..099e4f37b2 100644
@@ -1352,13 +1352,15 @@
 # @speed: the maximum speed, in bytes per second. The default is 0,
 #         for unlimited.
 #
-# @bitmap: the name of a dirty bitmap if sync is "bitmap" or "incremental".
+# @bitmap: The name of a dirty bitmap to use.
 #          Must be present if sync is "bitmap" or "incremental".
+#          Can be present if sync is "full" or "top".
 #          Must not be present otherwise.
 #          (Since 2.4 (drive-backup), 3.1 (blockdev-backup))
 #
 # @bitmap-mode: Specifies the type of data the bitmap should contain after
-#               the operation concludes. Must be present if sync is "bitmap".
Quoted 9 lines

qapi/block-core.json line 1366

Quoted 5 lines
qapi/block-core.json +2 -0
+#               the operation concludes.
+#               Must be present if a bitmap was provided,
 #               Must NOT be present otherwise. (Since 4.2)
 #
 # @compress: true to compress data, if the target format supports it.

Do you expect management applications will want to know about the presence of this patch?

It's all going to be queued up for 4.2, so management can gate on the presence of the "bitmap-mode" field. The text being replaced here is intermediate only.

Bitmaps can't be used with traditional backup modes without the bitmap-mode parameter, so I believe this is OK.

--js

Reviewed-by on v2 10/11
#24 lore raw

Re: [Qemu-devel] [PATCH v2 10/11] block/backup: support bitmap sync modes for non-bitmap backups

In reply to John Snow
Quoted 20 lines · John Snow

Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.

Some combinations don't make sense, though:

- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.

- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.

Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.

Quoted 6 lines
---
 block/backup.c       |  8 +-------
 blockdev.c           | 22 ++++++++++++++++++++++
 qapi/block-core.json |  6 ++++--
 3 files changed, 27 insertions(+), 9 deletions(-)

(I’ve seen Markus’s concern, but I think management applications can just see whether specifying sync={full,top} + bitmap works if they want to use it.)

author
#25 lore raw

PATCH v2 11/11 iotests/257: test traditional sync modes

In reply to John Snow
---
 tests/qemu-iotests/257     |   41 +-
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3128 insertions(+), 2 deletions(-)
tests/qemu-iotests/257 +39 -2
index 53ab31c92e..c2a72c577a 100755
@@ -283,6 +283,12 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
                       Bitmaps are always synchronized, regardless of failure.
                       (Partial images must be kept.)
 
+    :param msync_mode: The mirror sync mode to use for the first backup.
+                       Can be any one of:
+        - bitmap: Backups based on bitmap manifest.
+        - full:   Full backups.
+        - top:    Full backups of the top layer only.
+
     :param failure: Is the (optional) failure mode, and can be any of:
         - None:         No failure. Test the normative path. Default.
         - simulated:    Cancel the job right before it completes.
@@ -393,7 +399,7 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
             # group 1 gets cleared first, then group two gets written.
             if ((bsync_mode == 'on-success' and not failure) or
                 (bsync_mode == 'always')):
-                ebitmap.clear_group(1)
+                ebitmap.clear()
             ebitmap.dirty_group(2)
 
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
@@ -404,8 +410,19 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         log('')
 
         if bsync_mode == 'always' and failure == 'intermediate':
+            # TOP treats anything allocated as dirty, expect to see:
+            if msync_mode == 'top':
+                ebitmap.dirty_group(0)
+
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)
+
+            # Full returns all bits set except what was copied/skipped
+            if msync_mode == 'full':
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
+
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
@@ -499,10 +516,25 @@ def test_backup_api():
                 'bitmap404': ['on-success', 'always', 'never', None],
                 'bitmap0':   [None],
             },
+            'full': {
+                None:        ['on-success', 'always', 'never'],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['never', None],
+            },
+            'top': {
+                None:        ['on-success', 'always', 'never'],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['never', None],
+            },
+            'none': {
+                None:        ['on-success', 'always', 'never'],
+                'bitmap404': ['on-success', 'always', 'never', None],
+                'bitmap0':   ['on-success', 'always', 'never', None],
+            }
         }
 
         # Dicts, as always, are not stably-ordered prior to 3.7, so use tuples:
-        for sync_mode in ('incremental', 'bitmap'):
+        for sync_mode in ('incremental', 'bitmap', 'full', 'top', 'none'):
             log("-- Sync mode {:s} tests --\n".format(sync_mode))
             for bitmap in (None, 'bitmap404', 'bitmap0'):
                 for policy in error_cases[sync_mode][bitmap]:
@@ -517,6 +549,11 @@ def main():
         for failure in ("simulated", "intermediate", None):
             test_bitmap_sync(bsync_mode, "bitmap", failure)
 
+    for sync_mode in ('full', 'top'):
+        for bsync_mode in ('on-success', 'always'):
+            for failure in ('simulated', 'intermediate', None):
+                test_bitmap_sync(bsync_mode, sync_mode, failure)
+
     test_backup_api()
 
 if __name__ == '__main__':
tests/qemu-iotests/257.out +3089 -0

2,924 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.21.0

Part of the diff is not shown here. View the raw message.

#26 lore raw

Re: [Qemu-devel] [PATCH v2 11/11] iotests/257: test traditional sync modes

Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   41 +-
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3128 insertions(+), 2 deletions(-)

This needs a %s/specify Bitmap sync mode/specify bitmap sync mode/.

Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index 53ab31c92e..c2a72c577a 100755

[...]

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 402

Quoted 6 lines
tests/qemu-iotests/257 +1 -1
@@ -393,7 +399,7 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
             # group 1 gets cleared first, then group two gets written.
             if ((bsync_mode == 'on-success' and not failure) or
                 (bsync_mode == 'always')):
-                ebitmap.clear_group(1)
+                ebitmap.clear()

Hmmm... Why?

Quoted 10 lines
tests/qemu-iotests/257 +3 -0
             ebitmap.dirty_group(2)
 
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
@@ -404,8 +410,19 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         log('')
 
         if bsync_mode == 'always' and failure == 'intermediate':
+            # TOP treats anything allocated as dirty, expect to see:
+            if msync_mode == 'top':
+                ebitmap.dirty_group(0)

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 425

Quoted 10 lines
tests/qemu-iotests/257 +8 -0
+
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)
+
+            # Full returns all bits set except what was copied/skipped
+            if msync_mode == 'full':
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
+

So sync=top didn‘t copy anything? Is that because it now errors out before getting to copy something?

(The rest looks good to me.)

Max

Quoted 3 lines
tests/qemu-iotests/257 +0 -0
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
author
#27 lore raw

Re: [Qemu-devel] [PATCH v2 11/11] iotests/257: test traditional sync modes

Quoted 16 lines · Max Reitz
Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   41 +-
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3128 insertions(+), 2 deletions(-)

This needs a %s/specify Bitmap sync mode/specify bitmap sync mode/.

Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index 53ab31c92e..c2a72c577a 100755

[...]

Quoted 9 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 402

Quoted 6 lines
tests/qemu-iotests/257 +1 -1
@@ -393,7 +399,7 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
             # group 1 gets cleared first, then group two gets written.
             if ((bsync_mode == 'on-success' and not failure) or
                 (bsync_mode == 'always')):
-                ebitmap.clear_group(1)
+                ebitmap.clear()

Hmmm... Why?

From an order of operations standpoint, if we are here, we are expecting the bitmap to be synchronized. We can clear any existing data it holds, and then:

Quoted 2 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 404

Quoted 2 lines
tests/qemu-iotests/257 +0 -0
             ebitmap.dirty_group(2)
 

Add new writes that occurred during the job; which only happen here in this callback.

(The old code cleared specifically only group 1, the new code is just more general. I wound up changing it for a version that didn't make it to the list, but this is still correct.)

Quoted 9 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 416

Quoted 9 lines
tests/qemu-iotests/257 +4 -0
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
@@ -404,8 +410,19 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         log('')
 
         if bsync_mode == 'always' and failure == 'intermediate':
+            # TOP treats anything allocated as dirty, expect to see:
+            if msync_mode == 'top':
+                ebitmap.dirty_group(0)
+

Sorry, this code is definitely in the "cute" category...

If the failure was intermediate, we never call the pre-finalize callback above. So we know that the allocated regions of the file are only from groups 0 and 1.

So, HERE, we can mark the emulated bitmap's group 0 as dirty, to mimic what the copy_bitmap is going to have started the operation with.

Quoted 2 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 418

Quoted 2 lines
tests/qemu-iotests/257 +0 -0
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)

And then right here, we clear the first bit which we did copy out successfully. The emulated bitmap is now correct for sync=top.

Quoted 7 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 425

Quoted 7 lines
tests/qemu-iotests/257 +7 -0
+
+            # Full returns all bits set except what was copied/skipped
+            if msync_mode == 'full':
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
+

The full mode, though, is special. We cleared the first allocated bit just like for sync=top, but we take note of the second bit which is the one that caused the injected failure.

For both 'top' and 'full' modes here we're really using the ebitmap as an allocation record to inform what the output bitmap is going to look like.

Quoted 4 lines

So sync=top didn‘t copy anything? Is that because it now errors out before getting to copy something?

The ebitmap.clear_bit(ebitmap.first_bit) triggers for top, too. The test output should hopefully make sense here.

Quoted 8 lines

(The rest looks good to me.)

Max

Quoted 3 lines
tests/qemu-iotests/257 +0 -0
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
Reviewed-by
#28 lore raw

Re: [Qemu-devel] [PATCH v2 11/11] iotests/257: test traditional sync modes

Quoted 84 lines · John Snow
Quoted 16 lines · Max Reitz
Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   41 +-
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3128 insertions(+), 2 deletions(-)

This needs a %s/specify Bitmap sync mode/specify bitmap sync mode/.

Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index 53ab31c92e..c2a72c577a 100755

[...]

Quoted 9 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 402

Quoted 6 lines
tests/qemu-iotests/257 +1 -1
@@ -393,7 +399,7 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
             # group 1 gets cleared first, then group two gets written.
             if ((bsync_mode == 'on-success' and not failure) or
                 (bsync_mode == 'always')):
-                ebitmap.clear_group(1)
+                ebitmap.clear()

Hmmm... Why?

From an order of operations standpoint, if we are here, we are expecting the bitmap to be synchronized. We can clear any existing data it holds, and then:

Quoted 2 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 404

Quoted 2 lines
tests/qemu-iotests/257 +0 -0
             ebitmap.dirty_group(2)
 

Add new writes that occurred during the job; which only happen here in this callback.

(The old code cleared specifically only group 1, the new code is just more general. I wound up changing it for a version that didn't make it to the list, but this is still correct.)

Quoted 9 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 416

Quoted 9 lines
tests/qemu-iotests/257 +4 -0
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
@@ -404,8 +410,19 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         log('')
 
         if bsync_mode == 'always' and failure == 'intermediate':
+            # TOP treats anything allocated as dirty, expect to see:
+            if msync_mode == 'top':
+                ebitmap.dirty_group(0)
+

Sorry, this code is definitely in the "cute" category...

If the failure was intermediate, we never call the pre-finalize callback above. So we know that the allocated regions of the file are only from groups 0 and 1.

So, HERE, we can mark the emulated bitmap's group 0 as dirty, to mimic what the copy_bitmap is going to have started the operation with.

Quoted 2 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 418

Quoted 2 lines
tests/qemu-iotests/257 +0 -0
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)

And then right here, we clear the first bit which we did copy out successfully. The emulated bitmap is now correct for sync=top.

Quoted 7 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 425

Quoted 7 lines
tests/qemu-iotests/257 +7 -0
+
+            # Full returns all bits set except what was copied/skipped
+            if msync_mode == 'full':
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
+

The full mode, though, is special. We cleared the first allocated bit just like for sync=top, but we take note of the second bit which is the one that caused the injected failure.

For both 'top' and 'full' modes here we're really using the ebitmap as an allocation record to inform what the output bitmap is going to look like.

Quoted 1 line
Quoted 6 lines
Quoted 3 lines

So sync=top didn‘t copy anything? Is that because it now errors out before getting to copy something?

The ebitmap.clear_bit(ebitmap.first_bit) triggers for top, too. The test output should hopefully make sense here.

I... I have no idea how I missed the ebitmap.clear_bit().

So with the test output fixed:

Quoted 9 lines
Quoted 8 lines

(The rest looks good to me.)

Max

Quoted 3 lines
tests/qemu-iotests/257 +0 -0
         ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0'))
 
         # 2 - Writes and Reference Backup
author
#29 lore raw

Re: [Qemu-devel] [PATCH v2 11/11] iotests/257: test traditional sync modes

Quoted 89 lines · Max Reitz
Quoted 88 lines · John Snow
Quoted 16 lines · Max Reitz
Quoted 5 lines · John Snow
---
 tests/qemu-iotests/257     |   41 +-
 tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++
 2 files changed, 3128 insertions(+), 2 deletions(-)

This needs a %s/specify Bitmap sync mode/specify bitmap sync mode/.

Quoted 4 lines
tests/qemu-iotests/257 +0 -0
index 53ab31c92e..c2a72c577a 100755

[...]

Quoted 9 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 402

Quoted 6 lines
tests/qemu-iotests/257 +1 -1
@@ -393,7 +399,7 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
             # group 1 gets cleared first, then group two gets written.
             if ((bsync_mode == 'on-success' and not failure) or
                 (bsync_mode == 'always')):
-                ebitmap.clear_group(1)
+                ebitmap.clear()

Hmmm... Why?

From an order of operations standpoint, if we are here, we are expecting the bitmap to be synchronized. We can clear any existing data it holds, and then:

Quoted 2 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 404

Quoted 2 lines
tests/qemu-iotests/257 +0 -0
             ebitmap.dirty_group(2)
 

Add new writes that occurred during the job; which only happen here in this callback.

(The old code cleared specifically only group 1, the new code is just more general. I wound up changing it for a version that didn't make it to the list, but this is still correct.)

Quoted 9 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 416

Quoted 9 lines
tests/qemu-iotests/257 +4 -0
         vm.run_job(job, auto_dismiss=True, auto_finalize=False,
@@ -404,8 +410,19 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
         log('')
 
         if bsync_mode == 'always' and failure == 'intermediate':
+            # TOP treats anything allocated as dirty, expect to see:
+            if msync_mode == 'top':
+                ebitmap.dirty_group(0)
+

Sorry, this code is definitely in the "cute" category...

If the failure was intermediate, we never call the pre-finalize callback above. So we know that the allocated regions of the file are only from groups 0 and 1.

So, HERE, we can mark the emulated bitmap's group 0 as dirty, to mimic what the copy_bitmap is going to have started the operation with.

Quoted 2 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 418

Quoted 2 lines
tests/qemu-iotests/257 +0 -0
             # We manage to copy one sector (one bit) before the error.
             ebitmap.clear_bit(ebitmap.first_bit)

And then right here, we clear the first bit which we did copy out successfully. The emulated bitmap is now correct for sync=top.

Quoted 7 lines

tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 425

Quoted 7 lines
tests/qemu-iotests/257 +7 -0
+
+            # Full returns all bits set except what was copied/skipped
+            if msync_mode == 'full':
+                fail_bit = ebitmap.first_bit
+                ebitmap.clear()
+                ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY))
+

The full mode, though, is special. We cleared the first allocated bit just like for sync=top, but we take note of the second bit which is the one that caused the injected failure.

For both 'top' and 'full' modes here we're really using the ebitmap as an allocation record to inform what the output bitmap is going to look like.

Quoted 4 lines

So sync=top didn‘t copy anything? Is that because it now errors out before getting to copy something?

Quoted 9 lines
Quoted 2 lines

The ebitmap.clear_bit(ebitmap.first_bit) triggers for top, too. The test output should hopefully make sense here.

I... I have no idea how I missed the ebitmap.clear_bit().

So with the test output fixed:

Whoops. I definitely fixed that typo *after* I ran my tests and didn't re-run them.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

(The rest looks good to me.)

Max

Thank you for your reviews!

author Applied
#30 lore raw
In reply to John Snow
Quoted 76 lines · John Snow

Based-on: https://github.com/jnsnow/qemu/tree/bitmaps

This follows the previous series which adds the 'bitmap' sync mode and uses it to add interactions with bitmaps to the 'full' and 'top' modes to blockdev-backup and drive-backup.

Why?
on-success: Can conveniently synchronize a bitmap to a full backup.
Allows for transactionless anchor backups. Allows us to attempt an anchor backup without damaging
our bitmap until the backup is successful.
Allows for transactional, ungrouped anchor backups.
always: Allows us to resume full/top style backups with a later
invocation to sync=bitmap. Neat!

Summary:
1-3: Refactor iotest 257 to accommodate this;
4-5: Augment 257 to test trivial failure cases
6-9: Refactor sync=top for block/backup
10: Implement feature
11: Test feature

===
V2:
===

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/11:[0010] [FC] 'iotests/257: add Pattern class'
002/11:[0003] [FC] 'iotests/257: add EmulatedBitmap class' 003/11:[0042] [FC] 'iotests/257: Refactor backup helpers' 004/11:[----] [--] 'block/backup: hoist bitmap check into QMP interface' 005/11:[0016] [FC] 'iotests/257: test API failures'
006/11:[down] 'block/backup: improve sync=bitmap work estimates' 007/11:[down] 'block/backup: centralize copy_bitmap initialization' 008/11:[down] 'block/backup: add backup_is_cluster_allocated' 009/11:[down] 'block/backup: teach TOP to never copy unallocated regions' 010/11:[0002] [FC] 'block/backup: support bitmap sync modes for non-bitmap backups' 011/11:[0058] [FC] 'iotests/257: test traditional sync modes'

001: Fallout from changing floor(x / y) to x // y.
Fallout from changing x = x | y to x |= y.
(Decided not to keep RB.)
002: Removed dead variable
Change x = x - y to x -= y in clear_bits()
003: Substantially reorganize patch. Hopefully 10% more clever and 10% less cute. 004: (Added RB.)
005: Added docstring to test_backup_api
Fixed test matrix enumeration to have consistent order in python2/3 (Declined RB.)
---: What was patch 006 was dropped, and became patches 6-9.
6-9: New!

010: Was 007;
     Formatting fix.
     (Added RB.)

011: Was 008;
Adjust bitmap clearing/expected code as a consequence of #9. Fallout from changes to 005.

John Snow (11):
iotests/257: add Pattern class
iotests/257: add EmulatedBitmap class
iotests/257: Refactor backup helpers
block/backup: hoist bitmap check into QMP interface iotests/257: test API failures
block/backup: improve sync=bitmap work estimates block/backup: centralize copy_bitmap initialization block/backup: add backup_is_cluster_allocated block/backup: teach TOP to never copy unallocated regions block/backup: support bitmap sync modes for non-bitmap backups iotests/257: test traditional sync modes

Quoted 9 lines
 block/backup.c             |  188 +-
 block/trace-events         |    1 +
 blockdev.c                 |   32 +
 qapi/block-core.json       |    6 +-
 tests/qemu-iotests/256.out |    4 +-
 tests/qemu-iotests/257     |  342 ++--
 tests/qemu-iotests/257.out | 3366 +++++++++++++++++++++++++++++++++++-
 7 files changed, 3683 insertions(+), 256 deletions(-)

[Applied small edits to 9 and 11, and:]

Thanks, applied to my bitmaps tree:

https://github.com/jnsnow/qemu/commits/bitmaps
https://github.com/jnsnow/qemu.git

--js