bitmaps: allow bitmaps to be used with full and top
cover bitmaps: allow bitmaps to be used with full and top
Requires: <[email protected]> [PATCH v4 00/18] bitmaps: introduce 'bitmap' sync mode
This follows the previous series which adds the 'bitmap' sync mode and uses it to add interactions with bitmaps to the 'full' and 'top' modes to blockdev-backup and drive-backup.
Why?
on-success: Can conveniently synchronize a bitmap to a full backup.
Allows for transactionless anchor backups. Allows us to attempt an anchor backup without damaging
our bitmap until the backup is successful.
Allows for transactional, ungrouped anchor backups.
always: Allows us to resume full/top style backups with a later
invocation to sync=bitmap. Neat!
Summary:
1-3: Refactor iotest 257 to accommodate this;
4-5: Augment 257 to test trivial failure cases
6-7: Implement feature
8: Test new feature
John Snow (8):
iotests/257: add Pattern class
iotests/257: add EmulatedBitmap class
iotests/257: Refactor backup helpers
block/backup: hoist bitmap check into QMP interface iotests/257: test API failures
block/backup: issue progress updates for skipped regions block/backup: support bitmap sync modes for non-bitmap backups iotests/257: test traditional sync modes
block/backup.c | 22 +- blockdev.c | 32 + qapi/block-core.json | 6 +- tests/qemu-iotests/257 | 329 +++- tests/qemu-iotests/257.out | 3366 +++++++++++++++++++++++++++++++++++- 5 files changed, 3548 insertions(+), 207 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/8 iotests/257: add Pattern class
Just kidding, this is easier to manage with a full class instead of a namedtuple.
--- tests/qemu-iotests/257 | 58 +++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 26 deletions(-)
tests/qemu-iotests/257 +32 -26
@@ -19,7 +19,6 @@ # # [email protected] -from collections import namedtuple import math import os
@@ -29,10 +28,18 @@ from iotests import log, qemu_img SIZE = 64 * 1024 * 1024 GRANULARITY = 64 * 1024 -Pattern = namedtuple('Pattern', ['byte', 'offset', 'size']) -def mkpattern(byte, offset, size=GRANULARITY): - """Constructor for Pattern() with default size""" - return Pattern(byte, offset, size) + +class Pattern: + def __init__(self, byte, offset, size=GRANULARITY): + self.byte = byte + self.offset = offset + self.size = size + + def bits(self, granularity): + lower = math.floor(self.offset / granularity) + upper = math.floor((self.offset + self.size - 1) / granularity) + return set(range(lower, upper + 1)) + class PatternGroup: """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
@@ -43,40 +50,39 @@ class PatternGroup: """Calculate the unique bits dirtied by this pattern grouping""" res = set() for pattern in self.patterns: - lower = math.floor(pattern.offset / granularity) - upper = math.floor((pattern.offset + pattern.size - 1) / granularity) - res = res | set(range(lower, upper + 1)) + res = res | pattern.bits(granularity) return res + GROUPS = [ PatternGroup([ # Batch 0: 4 clusters - mkpattern('0x49', 0x0000000), - mkpattern('0x6c', 0x0100000), # 1M - mkpattern('0x6f', 0x2000000), # 32M - mkpattern('0x76', 0x3ff0000)]), # 64M - 64K + Pattern('0x49', 0x0000000), + Pattern('0x6c', 0x0100000), # 1M + Pattern('0x6f', 0x2000000), # 32M + Pattern('0x76', 0x3ff0000)]), # 64M - 64K PatternGroup([ # Batch 1: 6 clusters (3 new) - mkpattern('0x65', 0x0000000), # Full overwrite - mkpattern('0x77', 0x00f8000), # Partial-left (1M-32K) - mkpattern('0x72', 0x2008000), # Partial-right (32M+32K) - mkpattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K) + Pattern('0x65', 0x0000000), # Full overwrite + Pattern('0x77', 0x00f8000), # Partial-left (1M-32K) + Pattern('0x72', 0x2008000), # Partial-right (32M+32K) + Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K) PatternGroup([ # Batch 2: 7 clusters (3 new) - mkpattern('0x74', 0x0010000), # Adjacent-right - mkpattern('0x69', 0x00e8000), # Partial-left (1M-96K) - mkpattern('0x6e', 0x2018000), # Partial-right (32M+96K) - mkpattern('0x67', 0x3fe0000, - 2*GRANULARITY)]), # Overwrite [(64M-128K)-64M) + Pattern('0x74', 0x0010000), # Adjacent-right + Pattern('0x69', 0x00e8000), # Partial-left (1M-96K) + Pattern('0x6e', 0x2018000), # Partial-right (32M+96K) + Pattern('0x67', 0x3fe0000, + 2*GRANULARITY)]), # Overwrite [(64M-128K)-64M) PatternGroup([ # Batch 3: 8 clusters (5 new) # Carefully chosen such that nothing re-dirties the one cluster # that copies out successfully before failure in Group #1. - mkpattern('0xaa', 0x0010000, - 3*GRANULARITY), # Overwrite and 2x Adjacent-right - mkpattern('0xbb', 0x00d8000), # Partial-left (1M-160K) - mkpattern('0xcc', 0x2028000), # Partial-right (32M+160K) - mkpattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right + Pattern('0xaa', 0x0010000, + 3*GRANULARITY), # Overwrite and 2x Adjacent-right + Pattern('0xbb', 0x00d8000), # Partial-left (1M-160K) + Pattern('0xcc', 0x2028000), # Partial-right (32M+160K) + Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right ] class Drive:
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 1/8] iotests/257: add Pattern class
Quoted 7 lines · John Snow
Just kidding, this is easier to manage with a full class instead of a namedtuple.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 58 +++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 26 deletions(-)
Re: [Qemu-devel] [PATCH 1/8] iotests/257: add Pattern class
In reply to John SnowQuoted 29 lines · John Snow
Just kidding, this is easier to manage with a full class instead of a namedtuple.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 58 +++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 26 deletions(-)tests/qemu-iotests/257 +1
-5@@ -19,7 +19,6 @@ # # [email protected] -from collections import namedtuple import math import os@@ -29,10 +28,18 @@ from iotests import log, qemu_img SIZE = 64 * 1024 * 1024 GRANULARITY = 64 * 1024 -Pattern = namedtuple('Pattern', ['byte', 'offset', 'size']) -def mkpattern(byte, offset, size=GRANULARITY): - """Constructor for Pattern() with default size""" - return Pattern(byte, offset, size) +
tests/qemu-iotests/257 from iotests import log, qemu_img · line 41
Quoted 10 lines
tests/qemu-iotests/257 +10
-0+class Pattern: + def __init__(self, byte, offset, size=GRANULARITY): + self.byte = byte + self.offset = offset + self.size = size + + def bits(self, granularity): + lower = math.floor(self.offset / granularity) + upper = math.floor((self.offset + self.size - 1) / granularity) + return set(range(lower, upper + 1))
By the way, this doesn’t work with Python2 (pre-existing in your other series). It complains that these are floats.
Now I don’t know whether you care but there is the fact that the expressions would be shorter if they were of the form x // y instead of math.floor(x / y).
Max
Re: [Qemu-devel] [PATCH 1/8] iotests/257: add Pattern class
Quoted 44 lines · Max Reitz
Quoted 29 lines · John Snow
Just kidding, this is easier to manage with a full class instead of a namedtuple.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 58 +++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 26 deletions(-)tests/qemu-iotests/257 +1
-5@@ -19,7 +19,6 @@ # # [email protected] -from collections import namedtuple import math import os@@ -29,10 +28,18 @@ from iotests import log, qemu_img SIZE = 64 * 1024 * 1024 GRANULARITY = 64 * 1024 -Pattern = namedtuple('Pattern', ['byte', 'offset', 'size']) -def mkpattern(byte, offset, size=GRANULARITY): - """Constructor for Pattern() with default size""" - return Pattern(byte, offset, size) +tests/qemu-iotests/257 from iotests import log, qemu_img · line 41
Quoted 10 lines
tests/qemu-iotests/257 +10
-0+class Pattern: + def __init__(self, byte, offset, size=GRANULARITY): + self.byte = byte + self.offset = offset + self.size = size + + def bits(self, granularity): + lower = math.floor(self.offset / granularity) + upper = math.floor((self.offset + self.size - 1) / granularity) + return set(range(lower, upper + 1))By the way, this doesn’t work with Python2 (pre-existing in your other series). It complains that these are floats.
Quoted 6 lines
Now I don’t know whether you care but there is the fact that the expressions would be shorter if they were of the form x // y instead of
math.floor(x / y).Max
Ah, crud; OK -- I'll play nice with python2 for a while longer. Thanks for pointing this out.
PATCH 2/8 iotests/257: add EmulatedBitmap class
In reply to John SnowRepresent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.
--- tests/qemu-iotests/257 | 125 +++++++++++++++++++++++++---------------- 1 file changed, 76 insertions(+), 49 deletions(-)
tests/qemu-iotests/257 +76 -49
@@ -85,6 +85,60 @@ GROUPS = [ Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right ] + +class EmulatedBitmap: + def __init__(self, granularity=GRANULARITY): + self._bits = set() + self.groups = set() + self.granularity = granularity + + def dirty_bits(self, bits): + self._bits |= set(bits) + + def dirty_group(self, n): + self.dirty_bits(GROUPS[n].bits(self.granularity)) + + def clear(self): + self._bits = set() + + def clear_bits(self, bits): + self._bits = self._bits - set(bits) + + def clear_bit(self, bit): + self.clear_bits({bit}) + + def clear_group(self, n): + self.clear_bits(GROUPS[n].bits(self.granularity)) + + @property + def first_bit(self): + return sorted(self.bits)[0] + + @property + def bits(self): + return self._bits + + @property + def count(self): + return len(self.bits) + + def compare(self, qmp_bitmap): + """ + Print a nice human-readable message checking that a bitmap as reported + by the QMP interface has as many bits set as we expect it to. + """ + + name = qmp_bitmap.get('name', '(anonymous)') + log("= Checking Bitmap {:s} =".format(name)) + + want = self.count + have = qmp_bitmap['count'] // qmp_bitmap['granularity'] + + log("expecting {:d} dirty sectors; have {:d}. {:s}".format( + want, have, "OK!" if want == have else "ERROR!")) + log('') + + class Drive: """Represents, vaguely, a drive attached to a VM. Includes format, graph, and device information."""
@@ -195,27 +249,6 @@ def perform_writes(drive, n): log('') return bitmaps -def calculate_bits(groups=None): - """Calculate how many bits we expect to see dirtied.""" - if groups: - bits = set.union(*(GROUPS[group].bits(GRANULARITY) for group in groups)) - return len(bits) - return 0 - -def bitmap_comparison(bitmap, groups=None, want=0): - """ - Print a nice human-readable message checking that this bitmap has as - many bits set as we expect it to. - """ - log("= Checking Bitmap {:s} =".format(bitmap.get('name', '(anonymous)'))) - - if groups: - want = calculate_bits(groups) - have = bitmap['count'] // bitmap['granularity'] - - log("expecting {:d} dirty sectors; have {:d}. {:s}".format( - want, have, "OK!" if want == have else "ERROR!")) - log('') def compare_images(image, reference, baseimg=None, expected_match=True): """
@@ -321,12 +354,13 @@ def test_bitmap_sync(bsync_mode, failure=None): vm.qmp_log("block-dirty-bitmap-add", node=drive0.name, name="bitmap0", granularity=GRANULARITY) log('') + ebitmap = EmulatedBitmap() # 1 - Writes and Reference Backup bitmaps = perform_writes(drive0, 1) - dirty_groups = {1} + ebitmap.dirty_group(1) bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') - bitmap_comparison(bitmap, groups=dirty_groups) + ebitmap.compare(bitmap) reference_backup(drive0, 1, fbackup1) # 1 - Bitmap Backup (Optional induced failure)
@@ -342,54 +376,47 @@ def test_bitmap_sync(bsync_mode, failure=None): log('') bitmaps = perform_writes(drive0, 2) # Named bitmap (static, should be unchanged) - bitmap_comparison(get_bitmap(bitmaps, drive0.device, 'bitmap0'), - groups=dirty_groups) + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # Anonymous bitmap (dynamic, shows new writes) - bitmap_comparison(get_bitmap(bitmaps, drive0.device, '', - recording=True), groups={2}) - dirty_groups.add(2) + anonymous = EmulatedBitmap() + anonymous.dirty_group(2) + anonymous.compare(get_bitmap(bitmaps, drive0.device, '', + recording=True)) + + # Simulate the order in which this will happen: + # group 1 gets cleared first, then group two gets written. + if ((bsync_mode == 'on-success' and not failure) or + (bsync_mode == 'always')): + ebitmap.clear_group(1) + ebitmap.dirty_group(2) vm.run_job(job, auto_dismiss=True, auto_finalize=False, pre_finalize=_callback, cancel=(failure == 'simulated')) bitmaps = query_bitmaps(vm) - bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') log(bitmaps, indent=2) log('') - if ((bsync_mode == 'on-success' and not failure) or - (bsync_mode == 'always' and failure != 'intermediate')): - dirty_groups.remove(1) - if bsync_mode == 'always' and failure == 'intermediate': # We manage to copy one sector (one bit) before the error. - bitmap_comparison(bitmap, - want=calculate_bits(groups=dirty_groups) - 1) - else: - bitmap_comparison(bitmap, groups=dirty_groups) + ebitmap.clear_bit(ebitmap.first_bit) + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # 2 - Writes and Reference Backup bitmaps = perform_writes(drive0, 3) - dirty_groups.add(3) - bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') - if bsync_mode == 'always' and failure == 'intermediate': - # We're one bit short, still. - bitmap_comparison(bitmap, - want=calculate_bits(groups=dirty_groups) - 1) - else: - bitmap_comparison(bitmap, groups=dirty_groups) + ebitmap.dirty_group(3) + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) reference_backup(drive0, 2, fbackup2) # 2 - Bitmap Backup (In failure modes, this is a recovery.) job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode) vm.run_job(job, auto_dismiss=True, auto_finalize=False) bitmaps = query_bitmaps(vm) - bitmap = get_bitmap(bitmaps, drive0.device, 'bitmap0') log(bitmaps, indent=2) log('') - bitmap_comparison(bitmap, groups={} - if bsync_mode != 'never' - else dirty_groups) + if bsync_mode != 'never': + ebitmap.clear() + ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) log('--- Cleanup ---\n') vm.qmp_log("block-dirty-bitmap-remove",
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 2/8] iotests/257: add EmulatedBitmap class
Quoted 25 lines · John Snow
Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 125 +++++++++++++++++++++++++---------------- 1 file changed, 76 insertions(+), 49 deletions(-)tests/qemu-iotests/257 +8
-0@@ -85,6 +85,60 @@ GROUPS = [ Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right ] + +class EmulatedBitmap: + def __init__(self, granularity=GRANULARITY): + self._bits = set() + self.groups = set() + self.granularity = granularity + + def dirty_bits(self, bits):
tests/qemu-iotests/257 GROUPS = [ · line 105
Quoted 10 lines
tests/qemu-iotests/257 +10
-0+ self._bits |= set(bits) + + def dirty_group(self, n): + self.dirty_bits(GROUPS[n].bits(self.granularity)) + + def clear(self): + self._bits = set() + + def clear_bits(self, bits): + self._bits = self._bits - set(bits)
Does -= not work here?
No real complaints. Sorry.
Re: [Qemu-devel] [PATCH 2/8] iotests/257: add EmulatedBitmap class
Quoted 39 lines · Max Reitz
Quoted 25 lines · John Snow
Represent a bitmap with an object that we can mark and clear bits in. This makes it easier to manage partial writes when we don't write a full group's worth of patterns before an error.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 125 +++++++++++++++++++++++++---------------- 1 file changed, 76 insertions(+), 49 deletions(-)tests/qemu-iotests/257 +8
-0@@ -85,6 +85,60 @@ GROUPS = [ Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right ] + +class EmulatedBitmap: + def __init__(self, granularity=GRANULARITY): + self._bits = set() + self.groups = set() + self.granularity = granularity + + def dirty_bits(self, bits):tests/qemu-iotests/257 GROUPS = [ · line 105
Quoted 10 lines
tests/qemu-iotests/257 +10
-0+ self._bits |= set(bits) + + def dirty_group(self, n): + self.dirty_bits(GROUPS[n].bits(self.granularity)) + + def clear(self): + self._bits = set() + + def clear_bits(self, bits): + self._bits = self._bits - set(bits)Does -= not work here?
Quoted 4 lines
No real complaints. Sorry.
Reviewed-by Max Reitz <[email protected]>
It's okay. I forget which shorthand operators Python has for which types sometimes.
Quoted 3 lines
Quoted 3 lines
tests/qemu-iotests/257 +0
-0> a = {'a', 'b', 'c'} > a -= {'c'}Quoted 1 line
a
{'b', 'a'}
Well, apparently it does. I'll change it.
--js
PATCH 3/8 iotests/257: Refactor backup helpers
In reply to John SnowThis test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.
--- tests/qemu-iotests/257 | 46 +++++---- tests/qemu-iotests/257.out | 192 ++++++++++++++++++------------------- 2 files changed, 124 insertions(+), 114 deletions(-)
tests/qemu-iotests/257 +28 -18
@@ -208,6 +208,14 @@ def get_bitmap(bitmaps, drivename, name, recording=None): return bitmap return None +def blockdev_backup(vm, device, target, sync, **kwargs): + result = vm.qmp_log('blockdev-backup', + device=device, + target=target, + sync=sync, + **kwargs) + return result + def reference_backup(drive, n, filepath): log("--- Reference Backup #{:d} ---\n".format(n)) target_id = "ref_target_{:d}".format(n)
@@ -215,24 +223,26 @@ def reference_backup(drive, n, filepath): target_drive = Drive(filepath, vm=drive.vm) target_drive.create_target(target_id, drive.fmt, drive.size) - drive.vm.qmp_log("blockdev-backup", - job_id=job_id, device=drive.name, - target=target_id, sync="full") + blockdev_backup(drive.vm, drive.name, target_id, "full", job_id=job_id) drive.vm.run_job(job_id, auto_dismiss=True) log('') -def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode): - log("--- Bitmap Backup #{:d} ---\n".format(n)) - target_id = "bitmap_target_{:d}".format(n) - job_id = "bitmap_backup_{:d}".format(n) +def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'): + log("--- Test Backup #{:d} ---\n".format(n)) + target_id = "backup_target_{:d}".format(n) + job_id = "backup_{:d}".format(n) target_drive = Drive(filepath, vm=drive.vm) target_drive.create_target(target_id, drive.fmt, drive.size) - drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name, - target=target_id, sync="bitmap", - bitmap_mode=bitmap_mode, - bitmap=bitmap, - auto_finalize=False) + + kwargs = { + 'job_id': job_id, + 'auto_finalize': False, + 'bitmap': bitmap, + 'bitmap_mode': bitmap_mode, + } + kwargs = {key: val for key, val in kwargs.items() if val is not None} + blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs) return job_id def perform_writes(drive, n):
@@ -264,7 +274,7 @@ def compare_images(image, reference, baseimg=None, expected_match=True): "OK!" if ret == expected_ret else "ERROR!"), filters=[iotests.filter_testfiles]) -def test_bitmap_sync(bsync_mode, failure=None): +def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): """ Test bitmap backup routines.
@@ -292,7 +302,7 @@ def test_bitmap_sync(bsync_mode, failure=None): fbackup0, fbackup1, fbackup2), \ iotests.VM() as vm: - mode = "Bitmap Sync Mode {:s}".format(bsync_mode) + mode = "Mode {:s}; Bitmap Sync {:s}".format(msync_mode, bsync_mode) preposition = "with" if failure else "without" cond = "{:s} {:s}".format(preposition, "{:s} failure".format(failure) if failure
@@ -363,12 +373,12 @@ def test_bitmap_sync(bsync_mode, failure=None): ebitmap.compare(bitmap) reference_backup(drive0, 1, fbackup1) - # 1 - Bitmap Backup (Optional induced failure) + # 1 - Test Backup (w/ Optional induced failure) if failure == 'intermediate': # Activate blkdebug induced failure for second-to-next read log(vm.hmp_qemu_io(drive0.name, 'flush')) log('') - job = bitmap_backup(drive0, 1, bsync1, "bitmap0", bsync_mode) + job = backup(drive0, 1, bsync1, "bitmap0", bsync_mode, sync=msync_mode) def _callback(): """Issue writes while the job is open to test bitmap divergence."""
@@ -409,7 +419,7 @@ def test_bitmap_sync(bsync_mode, failure=None): reference_backup(drive0, 2, fbackup2) # 2 - Bitmap Backup (In failure modes, this is a recovery.) - job = bitmap_backup(drive0, 2, bsync2, "bitmap0", bsync_mode) + job = backup(drive0, 2, bsync2, "bitmap0", bsync_mode) vm.run_job(job, auto_dismiss=True, auto_finalize=False) bitmaps = query_bitmaps(vm) log(bitmaps, indent=2)
@@ -443,7 +453,7 @@ def test_bitmap_sync(bsync_mode, failure=None): def main(): for bsync_mode in ("never", "on-success", "always"): for failure in ("simulated", "intermediate", None): - test_bitmap_sync(bsync_mode, failure) + test_bitmap_sync(bsync_mode, "bitmap", failure) if __name__ == '__main__': iotests.script_main(main, supported_fmts=['qcow2'])
tests/qemu-iotests/257.out +96 -96
523 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 3/8] iotests/257: Refactor backup helpers
Quoted 14 lines · John Snow
This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 46 +++++---- tests/qemu-iotests/257.out | 192 ++++++++++++++++++------------------- 2 files changed, 124 insertions(+), 114 deletions(-)tests/qemu-iotests/257 +0
-0
[...]
Quoted 14 lines
tests/qemu-iotests/257 +4
-7-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode): - log("--- Bitmap Backup #{:d} ---\n".format(n)) - target_id = "bitmap_target_{:d}".format(n) - job_id = "bitmap_backup_{:d}".format(n) +def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'): + log("--- Test Backup #{:d} ---\n".format(n)) + target_id = "backup_target_{:d}".format(n) + job_id = "backup_{:d}".format(n) target_drive = Drive(filepath, vm=drive.vm) target_drive.create_target(target_id, drive.fmt, drive.size) - drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name, - target=target_id, sync="bitmap", - bitmap_mode=bitmap_mode,
tests/qemu-iotests/257 def reference_backup(drive, n, filepath): · line 244
Quoted 10 lines
tests/qemu-iotests/257 +8
-2- bitmap=bitmap, - auto_finalize=False) + + kwargs = { + 'job_id': job_id, + 'auto_finalize': False, + 'bitmap': bitmap, + 'bitmap_mode': bitmap_mode, + } + kwargs = {key: val for key, val in kwargs.items() if val is not None}
I suppose this is to remove items that are None?
Very cute, but why not just
kwargs = {
'job_id': job_id,
'auto_finalize': False,
}
if bitmap is not None:
kwargs['bitmap'] = bitmap
kwargs['bitmap_mode'] = bitmap_modeExactly the same number of lines, but immediately makes it clear what’s going on. Not as cute, I admit.
(Yes, I am indeed actively trying to train you not to write cute code.)
The rest looks good to me:
Re: [Qemu-devel] [PATCH 3/8] iotests/257: Refactor backup helpers
Quoted 55 lines · Max Reitz
Quoted 14 lines · John Snow
This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 46 +++++---- tests/qemu-iotests/257.out | 192 ++++++++++++++++++------------------- 2 files changed, 124 insertions(+), 114 deletions(-)tests/qemu-iotests/257 +0
-0[...]
Quoted 14 lines
tests/qemu-iotests/257 +4
-7-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode): - log("--- Bitmap Backup #{:d} ---\n".format(n)) - target_id = "bitmap_target_{:d}".format(n) - job_id = "bitmap_backup_{:d}".format(n) +def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'): + log("--- Test Backup #{:d} ---\n".format(n)) + target_id = "backup_target_{:d}".format(n) + job_id = "backup_{:d}".format(n) target_drive = Drive(filepath, vm=drive.vm) target_drive.create_target(target_id, drive.fmt, drive.size) - drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name, - target=target_id, sync="bitmap", - bitmap_mode=bitmap_mode,tests/qemu-iotests/257 def reference_backup(drive, n, filepath): · line 244
Quoted 10 lines
tests/qemu-iotests/257 +8
-2- bitmap=bitmap, - auto_finalize=False) + + kwargs = { + 'job_id': job_id, + 'auto_finalize': False, + 'bitmap': bitmap, + 'bitmap_mode': bitmap_mode, + } + kwargs = {key: val for key, val in kwargs.items() if val is not None}I suppose this is to remove items that are None?
Very cute, but why not just
kwargs = { 'job_id': job_id, 'auto_finalize': False, } if bitmap is not None: kwargs['bitmap'] = bitmap kwargs['bitmap_mode'] = bitmap_mode
Quoted 5 lines
Exactly the same number of lines, but immediately makes it clear what’s going on. Not as cute, I admit.
(Yes, I am indeed actively trying to train you not to write cute code.)
It sneaks in. I genuinely struggle with understanding what other people will find readable; I have an authentically hard time reviewing other people's patches too. I'm earnestly not sure how I can help improve this, but I would like to.
I wasn't sure what the easiest way to avoid sending the "None" over the wire was, so I went with a general thing, but yes: it's because bitmap and bitmap_mode are set to None sometimes and I need to omit such keys.
In this case, though, I do test bitmap and bitmap_mode separately, so for the purposes of testing intentionally bad combinations you do need:
if bitmap is not None:
kwargs['bitmap'] = bitmap
if bitmap_mode is not None:
kwargs['bitmap_mode'] = bitmap_mode
And I just looked at this and it did not spark joy; so I went with a generic filter to remove nulled keys. I admit it's /slightly/ cute and not immediately obvious why it needs to be done.
This is even cuter, so maybe I am traveling in the wrong direction:
def backup(drive, n, filepath, sync, **kwargs):
log("--- Test Backup #{:d} ---\n".format(n))
target_id = "backup_target_{:d}".format(n)
job_id = "backup_{:d}".format(n)
target_drive = Drive(filepath, vm=drive.vm)
target_drive.create_target(target_id, drive.fmt, drive.size) kwargs.setdefault('auto_finalize', False)
# Strip any arguments explicitly nulled by the caller:
kwargs = {key: val for key, val in kwargs.items()
if val is not None}
blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs) return job_id
It's quite a bit shorter and also makes backup() more flexible by omitting the bitmap and bitmap_mode arguments entirely, allowing the caller to override the auto_finalize default, etc. In this permutation, we don't know the full extent of kwargs so it makes sense to generically filter it.
Manually conditionally setting arguments is probably also fine. Do you still have a preference for the more static approach?
Quoted 4 lines
The rest looks good to me:
Reviewed-by Max Reitz <[email protected]>
Thanks for reviewing, as always!
Re: [Qemu-devel] [PATCH 3/8] iotests/257: Refactor backup helpers
Quoted 102 lines · John Snow
Quoted 55 lines · Max Reitz
Quoted 14 lines · John Snow
This test needs support for non-bitmap backups and missing or unspecified bitmap sync modes, so rewrite the helpers to be a little more generic.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 46 +++++---- tests/qemu-iotests/257.out | 192 ++++++++++++++++++------------------- 2 files changed, 124 insertions(+), 114 deletions(-)tests/qemu-iotests/257 +0
-0[...]
Quoted 14 lines
tests/qemu-iotests/257 +4
-7-def bitmap_backup(drive, n, filepath, bitmap, bitmap_mode): - log("--- Bitmap Backup #{:d} ---\n".format(n)) - target_id = "bitmap_target_{:d}".format(n) - job_id = "bitmap_backup_{:d}".format(n) +def backup(drive, n, filepath, bitmap, bitmap_mode, sync='bitmap'): + log("--- Test Backup #{:d} ---\n".format(n)) + target_id = "backup_target_{:d}".format(n) + job_id = "backup_{:d}".format(n) target_drive = Drive(filepath, vm=drive.vm) target_drive.create_target(target_id, drive.fmt, drive.size) - drive.vm.qmp_log("blockdev-backup", job_id=job_id, device=drive.name, - target=target_id, sync="bitmap", - bitmap_mode=bitmap_mode,tests/qemu-iotests/257 def reference_backup(drive, n, filepath): · line 244
Quoted 10 lines
tests/qemu-iotests/257 +8
-2- bitmap=bitmap, - auto_finalize=False) + + kwargs = { + 'job_id': job_id, + 'auto_finalize': False, + 'bitmap': bitmap, + 'bitmap_mode': bitmap_mode, + } + kwargs = {key: val for key, val in kwargs.items() if val is not None}I suppose this is to remove items that are None?
Very cute, but why not just
kwargs = { 'job_id': job_id, 'auto_finalize': False, } if bitmap is not None: kwargs['bitmap'] = bitmap kwargs['bitmap_mode'] = bitmap_modeQuoted 5 lines
Exactly the same number of lines, but immediately makes it clear what’s going on. Not as cute, I admit.
(Yes, I am indeed actively trying to train you not to write cute code.)
It sneaks in. I genuinely struggle with understanding what other people will find readable; I have an authentically hard time reviewing other people's patches too. I'm earnestly not sure how I can help improve this, but I would like to.
I wasn't sure what the easiest way to avoid sending the "None" over the wire was, so I went with a general thing, but yes: it's because bitmap and
bitmap_modeare set to None sometimes and I need to omit such keys.In this case, though, I do test bitmap and
bitmap_modeseparately, so for the purposes of testing intentionally bad combinations you do need:if bitmap is not None:
kwargs['bitmap'] = bitmap
ifbitmap_modeis not None:
kwargs['bitmap_mode'] =bitmap_modeAnd I just looked at this and it did not spark joy; so I went with a generic filter to remove nulled keys. I admit it's /slightly/ cute and not immediately obvious why it needs to be done.
This is even cuter, so maybe I am traveling in the wrong direction:
def
backup(drive, n, filepath, sync, **kwargs):
log("--- Test Backup #{:d} ---\n".format(n))
target_id= "backup_target_{:d}".format(n)
job_id= "backup_{:d}".format(n)
target_drive=Drive(filepath, vm=drive.vm)
target_drive.create_target(target_id, drive.fmt, drive.size)kwargs.setdefault('auto_finalize', False)
# Strip any arguments explicitly nulled by the caller:
kwargs = {key: val for key, val inkwargs.items()
if val is not None}
blockdev_backup(drive.vm, drive.name, target_id, sync, **kwargs)returnjob_id
Quoted 8 lines
It's quite a bit shorter and also makes
backup()more flexible by omitting the bitmap andbitmap_modearguments entirely, allowing the caller to override theauto_finalizedefault, etc. In this permutation, we don't know the full extent of kwargs so it makes sense to generically filter it.Manually conditionally setting arguments is probably also fine. Do you still have a preference for the more static approach?
It’s OK with the comment.
Although it needs a kwargs.setdefault('job_id', job_id), too.
(Hm. Shouldn’t 'auto-finalize' and 'job-id' work just fine? It’s not like the QMP scripts swap - and _. They just replace _ by -.)
Max
Quoted 7 lines
Quoted 4 lines
The rest looks good to me:
Reviewed-by Max Reitz <[email protected]>Thanks for reviewing, as always!
PATCH 4/8 block/backup: hoist bitmap check into QMP interface
In reply to John SnowThis is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.
--- block/backup.c | 13 ++++--------- blockdev.c | 10 ++++++++++ 2 files changed, 14 insertions(+), 9 deletions(-)
block/backup.c +4 -9
@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,assert (bs);assert (target); +/* QMP interface protects us from these cases */ +assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL ); +assert (sync_bitmap || sync_mode !=MIRROR_SYNC_MODE_BITMAP ); +if (bs == target) {error_setg (errp,"Source and target cannot be the same" );return NULL ;
@@ -597,16 +601,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,return NULL ; } -/* QMP interface should have handled translating this to bitmap mode */ -assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL ); -if (sync_mode ==MIRROR_SYNC_MODE_BITMAP ) { -if (!sync_bitmap) { -error_setg (errp,"must provide a valid bitmap name for " -"'%s' sync mode" ,MirrorSyncMode_str (sync_mode)); -return NULL ; - } -/* If we need to write to this bitmap, check that we can: */ if (bitmap_mode !=BITMAP_SYNC_MODE_NEVER &&bdrv_dirty_bitmap_check (sync_bitmap,BDRV_BITMAP_DEFAULT , errp)) {
blockdev.c +10 -0
@@ -3464,6 +3464,16 @@ static BlockJob *do_backup_common(BackupCommon *backup,return NULL ; } +if ((backup->sync ==MIRROR_SYNC_MODE_BITMAP ) || + (backup->sync ==MIRROR_SYNC_MODE_INCREMENTAL )) { +/* done before desugaring 'incremental' to print the right message */ +if (!backup->has_bitmap) { +error_setg (errp,"must provide a valid bitmap name for " +"'%s' sync mode" ,MirrorSyncMode_str (backup->sync)); +return NULL ; + } + } +if (backup->sync ==MIRROR_SYNC_MODE_INCREMENTAL ) {if (backup->has_bitmap_mode && backup->bitmap_mode !=BITMAP_SYNC_MODE_ON_SUCCESS ) {
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface
Quoted 14 lines · John Snow
This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.
Signed-off-by John Snow <[email protected]>--- block/backup.c | 13 ++++--------- blockdev.c | 10 ++++++++++ 2 files changed, 14 insertions(+), 9 deletions(-)block/backup.c +0
-0
block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 571
Quoted 6 lines
block/backup.c +3
-0@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,assert (bs);assert (target); +/* QMP interface protects us from these cases */ +assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL ); +assert (sync_bitmap || sync_mode !=MIRROR_SYNC_MODE_BITMAP );
Implication would be a nice operator sometimes.
("assert(sync_mode == MIRROR_SYNC_MODE_BITMAP -> sync_bitmap)")
(Can you do that in C++? No, you can’t overload bool’s operators, right?)
Re: [Qemu-devel] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface
Quoted 24 lines · Max Reitz
Quoted 14 lines · John Snow
This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.
Signed-off-by John Snow <[email protected]>--- block/backup.c | 13 ++++--------- blockdev.c | 10 ++++++++++ 2 files changed, 14 insertions(+), 9 deletions(-)block/backup.c +0
-0block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 571
Quoted 6 lines
block/backup.c +3
-0@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,assert (bs);assert (target); +/* QMP interface protects us from these cases */ +assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL ); +assert (sync_bitmap || sync_mode !=MIRROR_SYNC_MODE_BITMAP );Implication would be a nice operator sometimes.
Quoted 6 lines
("
assert(sync_mode == MIRROR_SYNC_MODE_BITMAP -> sync_bitmap)")(Can you do that in C++? No, you can’t overload bool’s operators, right?)
Reviewed-by Max Reitz <[email protected]>
Yes, I also find this assertion kind of hard to read personally, but it feels somewhat clunky to write:
if (antecedent) {
assert(condition);
}I suppose we can also phrase this as:
assert(sync_mode == MIRROR_SYNC_MODE_BITMAP ? sync_bitmap : true);
Which might honestly be pretty good. Mind if I change it to this?
--js
Re: [Qemu-devel] [PATCH 4/8] block/backup: hoist bitmap check into QMP interface
Quoted 41 lines · John Snow
Quoted 24 lines · Max Reitz
Quoted 14 lines · John Snow
This is nicer to do in the unified QMP interface that we have now, because it lets us use the right terminology back at the user.
Signed-off-by John Snow <[email protected]>--- block/backup.c | 13 ++++--------- blockdev.c | 10 ++++++++++ 2 files changed, 14 insertions(+), 9 deletions(-)block/backup.c +0
-0block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 571
Quoted 6 lines
block/backup.c +3
-0@@ -566,6 +566,10 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,assert (bs);assert (target); +/* QMP interface protects us from these cases */ +assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL ); +assert (sync_bitmap || sync_mode !=MIRROR_SYNC_MODE_BITMAP );Implication would be a nice operator sometimes.
Quoted 6 lines
("
assert(sync_mode == MIRROR_SYNC_MODE_BITMAP -> sync_bitmap)")(Can you do that in C++? No, you can’t overload bool’s operators, right?)
Reviewed-by Max Reitz <[email protected]>Yes, I also find this assertion kind of hard to read personally, but it feels somewhat clunky to write:
if (antecedent) { assert(condition); }
Quoted 5 lines
I suppose we can also phrase this as:
assert(sync_mode == MIRROR_SYNC_MODE_BITMAP ? sync_bitmap : true);Which might honestly be pretty good. Mind if I change it to this?
Looks weird (mostly unfamiliar), but I do not.
Max
PATCH 5/8 iotests/257: test API failures
In reply to John Snow--- tests/qemu-iotests/257 | 69 +++++++++++++++++++++++++++++++ tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 154 insertions(+)
tests/qemu-iotests/257 +69 -0
@@ -450,10 +450,79 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): compare_images(img_path, fbackup2) log('') +def test_backup_api(): + """ + """ + with iotests.FilePaths(['img', 'bsync1']) as \ + (img_path, backup_path), \ + iotests.VM() as vm: + + log("\n=== API failure tests ===\n") + log('--- Preparing image & VM ---\n') + drive0 = Drive(img_path, vm=vm) + drive0.img_create(iotests.imgfmt, SIZE) + vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device())) + vm.launch() + + file_config = { + 'driver': 'file', + 'filename': drive0.path + } + + vm.qmp_log('blockdev-add', + filters=[iotests.filter_qmp_testfiles], + node_name="drive0", + driver=drive0.fmt, + file=file_config) + drive0.node = 'drive0' + drive0.device = 'device0' + vm.qmp_log("device_add", id=drive0.device, + drive=drive0.name, driver="scsi-hd") + log('') + + target0 = Drive(backup_path, vm=vm) + target0.create_target("backup_target", drive0.fmt, drive0.size) + log('') + + vm.qmp_log("block-dirty-bitmap-add", node=drive0.name, + name="bitmap0", granularity=GRANULARITY) + log('') + + log('-- Testing invalid QMP commands --\n') + + error_cases = { + 'incremental': { + None: ['on-success', 'always', 'never', None], + 'bitmap404': ['on-success', 'always', 'never', None], + 'bitmap0': ['always', 'never'] + }, + 'bitmap': { + None: ['on-success', 'always', 'never', None], + 'bitmap404': ['on-success', 'always', 'never', None], + 'bitmap0': [None], + }, + } + + for sync_mode, config in error_cases.items(): + log("-- Sync mode {:s} tests --\n".format(sync_mode)) + for bitmap, policies in config.items(): + for policy in policies: + kwargs = { 'job_id': 'api_job' } + if bitmap is not None: + kwargs['bitmap'] = bitmap + if policy is not None: + kwargs['bitmap-mode'] = policy + blockdev_backup(drive0.vm, drive0.name, "backup_target", + sync_mode, **kwargs) + log('') + + def main(): for bsync_mode in ("never", "on-success", "always"): for failure in ("simulated", "intermediate", None): test_bitmap_sync(bsync_mode, "bitmap", failure) + test_backup_api() + if __name__ == '__main__': iotests.script_main(main, supported_fmts=['qcow2'])
tests/qemu-iotests/257.out +85 -0
88 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 5/8] iotests/257: test API failures
Quoted 11 lines · John Snow
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 69 +++++++++++++++++++++++++++++++ tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 154 insertions(+)tests/qemu-iotests/257 +0
-0
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 455
Quoted 6 lines
tests/qemu-iotests/257 +3
-0@@ -450,10 +450,79 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): compare_images(img_path, fbackup2) log('') +def test_backup_api(): + """ + """
Er, OK?
[...]
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 508
Quoted 3 lines
tests/qemu-iotests/257 +3
-0+ for sync_mode, config in error_cases.items(): + log("-- Sync mode {:s} tests --\n".format(sync_mode)) + for bitmap, policies in config.items():
You might be interested in the fact that the iteration order is different for Python2. Or maybe you aren’t.
Re: [Qemu-devel] [PATCH 5/8] iotests/257: test API failures
Quoted 15 lines · Max Reitz
Quoted 14 lines · John Snow
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 69 +++++++++++++++++++++++++++++++ tests/qemu-iotests/257.out | 85 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 154 insertions(+)tests/qemu-iotests/257 +0
-0@@ -450,10 +450,79 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): compare_images(img_path, fbackup2) log('')
Quoted 8 lines
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 455
Quoted 3 lines
tests/qemu-iotests/257 +3
-0+def test_backup_api(): + """ + """Er, OK?
[...]
Whooooooooops.
Quoted 7 lines
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 508
Quoted 3 lines
tests/qemu-iotests/257 +3
-0+ for sync_mode, config in error_cases.items(): + log("-- Sync mode {:s} tests --\n".format(sync_mode)) + for bitmap, policies in config.items():You might be interested in the fact that the iteration order is different for Python2. Or maybe you aren’t.
asdf. This is undoubtedly the worst thing about Python. I'll have to fix this, because the ordering isn't guaranteed until 3.5 or some such and we're only EOLing Python2.
--js
PATCH 6/8 block/backup: issue progress updates for skipped regions
In reply to John SnowThe way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.
The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.
Update the progress when we skip regions.
--- block/backup.c | 1 + 1 file changed, 1 insertion(+)
block/backup.c +1 -0
@@ -417,6 +417,7 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)if (job->sync_mode ==MIRROR_SYNC_MODE_TOP &&bdrv_is_unallocated_range (bs, offset, job->cluster_size)) { +job_progress_update (&job->common.job, job->cluster_size);bdrv_reset_dirty_bitmap (job->copy_bitmap, offset, job->cluster_size);continue ;
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions
Quoted 2 lines · John Snow
The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.
Although there is this comment:
Quoted 1 line
/* TODO
job_progress_set_remaining()would make more sense */
So...
Quoted 5 lines
The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.
Update the progress when we skip regions.
Wouldn’t it be more correct to decrease the job length?
Max
Quoted 18 lines
Signed-off-by John Snow <[email protected]>--- block/backup.c | 1 + 1 file changed, 1 insertion(+)block/backup.c +1
-0@@ -417,6 +417,7 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)if (job->sync_mode ==MIRROR_SYNC_MODE_TOP &&bdrv_is_unallocated_range (bs, offset, job->cluster_size)) { +job_progress_update (&job->common.job, job->cluster_size);bdrv_reset_dirty_bitmap (job->copy_bitmap, offset, job->cluster_size);continue ;
Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow
The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.
Although there is this comment:
Quoted 1 line
/* TODO
job_progress_set_remaining()would make more sense */So...
Quoted 4 lines
The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.
Quoted 6 lines
Quoted 1 line
Update the progress when we skip regions.
Wouldn’t it be more correct to decrease the job length?
Max
Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:
(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]
This means we can do stuff like:
- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particular
Bitmap sync works by artificially increasing progress for NOP regions, seen in init_copy_bitmap.
Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.
Top sync is the odd one out, which just omits progress for regions it skips.
My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?
¯\_(ツ)_/¯
Quoted 20 lines
Quoted 18 lines
Signed-off-by John Snow <[email protected]>--- block/backup.c | 1 + 1 file changed, 1 insertion(+)block/backup.c +1
-0@@ -417,6 +417,7 @@ static int coroutine_fn backup_loop(BackupBlockJob *job)if (job->sync_mode ==MIRROR_SYNC_MODE_TOP &&bdrv_is_unallocated_range (bs, offset, job->cluster_size)) { +job_progress_update (&job->common.job, job->cluster_size);bdrv_reset_dirty_bitmap (job->copy_bitmap, offset, job->cluster_size);continue ;
Max Reitz
Reviewed-byRe: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions
Quoted 31 lines · John Snow
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow
The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.
Although there is this comment:
Quoted 1 line
/* TODO
job_progress_set_remaining()would make more sense */So...
Quoted 4 lines
The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.
Quoted 6 lines
Quoted 1 line
Update the progress when we skip regions.
Wouldn’t it be more correct to decrease the job length?
Max
Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:
(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]
Quoted 8 lines
This means we can do stuff like:
- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particularBitmap sync works by artificially increasing progress for NOP regions, seen in
init_copy_bitmap.
Yes, and it has a TODO comment that says it should be done differently.
Quoted 11 lines
Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.
Top sync is the odd one out, which just omits progress for regions it skips.
My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?
If you look at the commit (05df8a6a2b4), I suppose it was because that commit simply did not intend to change behavior. It just touched that piece of code and noted that maybe there should be a follow-up commit to change it.
But yeah, whatever.
Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions
Quoted 55 lines · Max Reitz
Quoted 31 lines · John Snow
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow
The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.
Although there is this comment:
Quoted 1 line
/* TODO
job_progress_set_remaining()would make more sense */So...
Quoted 4 lines
The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.
Quoted 6 lines
Quoted 1 line
Update the progress when we skip regions.
Wouldn’t it be more correct to decrease the job length?
Max
Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:
(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]Quoted 8 lines
This means we can do stuff like:
- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particularBitmap sync works by artificially increasing progress for NOP regions, seen in
init_copy_bitmap.Yes, and it has a TODO comment that says it should be done differently.
Quoted 11 lines
Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.
Top sync is the odd one out, which just omits progress for regions it skips.
My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?
Quoted 9 lines
If you look at the commit (
05df8a6a2b4), I suppose it was because that commit simply did not intend to change behavior. It just touched that piece of code and noted that maybe there should be a follow-up commit to change it.But yeah, whatever.
Reviewed-by Max Reitz <[email protected]>
I mean. I'll make it consistent either way, but I actually don't know which way I should make it go. I just think that all the modes should work the same if we can help it.
Flip a coin?
--js
Re: [Qemu-devel] [PATCH 6/8] block/backup: issue progress updates for skipped regions
Quoted 65 lines · John Snow
Quoted 62 lines · Max Reitz
Quoted 31 lines · John Snow
Quoted 14 lines · Max Reitz
Quoted 2 lines · John Snow
The way bitmap backups work is by starting at 75% if it needs to copy just 25% of the disk.
Although there is this comment:
Quoted 1 line
/* TODO
job_progress_set_remaining()would make more sense */So...
Quoted 4 lines
The way sync=top currently works, however, is to start at 0% and then never update the progress if it doesn't copy a region. If it needs to copy 25% of the disk, we'll finish at 25%.
Quoted 6 lines
Quoted 1 line
Update the progress when we skip regions.
Wouldn’t it be more correct to decrease the job length?
Max
Admittedly I have precisely no idea. Maybe? As far as I understand it, we guarantee only:
(1) Progress monotonically increases
(2) Upon completion, progress will equal the total work estimate. [Trying to fix that to be true here.]Quoted 8 lines
This means we can do stuff like:
- Total work estimate can increase or decrease arbitrarily
- Neither value has to mean anything in particularBitmap sync works by artificially increasing progress for NOP regions, seen in
init_copy_bitmap.Yes, and it has a TODO comment that says it should be done differently.
Quoted 11 lines
Full sync also tends to increase progress regardless of it actually did a copy or not; offload support also counts as progress here. So if you full sync an empty image, you'll see it increasing the progress as it doesn't actually do anything.
Top sync is the odd one out, which just omits progress for regions it skips.
My only motivation here was to make them consistent. Can I do it the other way? Yeah, probably. Is one way better than the other? I legitimately have no idea. I guess whoever wrote the last comment felt that it should all be the other way instead. Why'd they not do that?
If you look at the commit (
05df8a6a2b4), I suppose it was because that commit simply did not intend to change behavior. It just touched that piece of code and noted that maybe there should be a follow-up commit to change it.But yeah, whatever.
Quoted 8 lines
Quoted 2 lines
Reviewed-by Max Reitz <[email protected]>I mean. I'll make it consistent either way, but I actually don't know which way I should make it go. I just think that all the modes should work the same if we can help it.
Flip a coin?
If you’d flip a coin, I can say that I’d find it a bit more meaningful to reduce the length. Especially if you change sync=top to calculate beforehand how much we need to copy, so the length isn’t even reduced while the job is running.
And it’s easier to remove the TODO comment this way. If we decide against it, you’d have to remove the TODO comment aso (because we decided against it), but you’d need to justify it in the commit message.
And we all know that writing explanations and documentation is the
hardest thing of all.
Max
PATCH 7/8 block/backup: support bitmap sync modes for non-bitmap backups
In reply to John SnowAccept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.
Some combinations don't make sense, though:
- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.
- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.
Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.
--- block/backup.c | 8 +------- blockdev.c | 22 ++++++++++++++++++++++ qapi/block-core.json | 6 ++++-- 3 files changed, 27 insertions(+), 9 deletions(-)
block/backup.c +1 -7
@@ -602,7 +602,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,return NULL ; } -if (sync_mode ==MIRROR_SYNC_MODE_BITMAP ) { +if (sync_bitmap) {/* If we need to write to this bitmap, check that we can: */ if (bitmap_mode !=BITMAP_SYNC_MODE_NEVER &&bdrv_dirty_bitmap_check (sync_bitmap,BDRV_BITMAP_DEFAULT , errp)) {
@@ -613,12 +613,6 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,if (bdrv_dirty_bitmap_create_successor (bs, sync_bitmap, errp) <0 ) {return NULL ; } - }else if (sync_bitmap) { -error_setg (errp, -"a bitmap was given to backup_job_create, " -"but it received an incompatible sync_mode (%s)" , -MirrorSyncMode_str (sync_mode)); -return NULL ; } len =bdrv_getlength (bs);
blockdev.c +22 -0
@@ -3502,6 +3502,28 @@ static BlockJob *do_backup_common(BackupCommon *backup,if (bdrv_dirty_bitmap_check (bmap,BDRV_BITMAP_ALLOW_RO , errp)) {return NULL ; } + +/* This does not produce a useful bitmap artifact: */ +if (backup->sync ==MIRROR_SYNC_MODE_NONE ) { +error_setg (errp,"sync mode '%s' does not produce meaningful bitmap" +" outputs" ,MirrorSyncMode_str (backup->sync)); +return NULL ; + } + +/* If the bitmap isn't used for input or output, this is useless: */ +if (backup->bitmap_mode ==BITMAP_SYNC_MODE_NEVER && + backup->sync !=MIRROR_SYNC_MODE_BITMAP ) { +error_setg (errp,"Bitmap sync mode '%s' has no meaningful effect" +" when combined with sync mode '%s'" , +BitmapSyncMode_str (backup->bitmap_mode), +MirrorSyncMode_str (backup->sync)); +return NULL ; + } + } + +if (!backup->has_bitmap && backup->has_bitmap_mode) { +error_setg (errp,"Cannot specify Bitmap sync mode without a bitmap" ); +return NULL ; }if (!backup->auto_finalize) {
qapi/block-core.json +4 -2
@@ -1352,13 +1352,15 @@ # @speed: the maximum speed, in bytes per second. The default is 0, # for unlimited. # -# @bitmap: the name of a dirty bitmap if sync is "bitmap" or "incremental". +# @bitmap: The name of a dirty bitmap to use. # Must be present if sync is "bitmap" or "incremental". +# Can be present if sync is "full" or "top". # Must not be present otherwise. # (Since 2.4 (drive-backup), 3.1 (blockdev-backup)) # # @bitmap-mode: Specifies the type of data the bitmap should contain after -# the operation concludes. Must be present if sync is "bitmap". +# the operation concludes. +# Must be present if a bitmap was provided, # Must NOT be present otherwise. (Since 4.2) # # @compress: true to compress data, if the target format supports it.
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups
Quoted 26 lines · John Snow
Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.
Some combinations don't make sense, though:
- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.
Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.
Signed-off-by John Snow <[email protected]>--- block/backup.c | 8 +------- blockdev.c | 22 ++++++++++++++++++++++ qapi/block-core.json | 6 ++++-- 3 files changed, 27 insertions(+), 9 deletions(-)
[...]
Quoted 4 lines
blockdev.c +0
-0
[...]
blockdev.c static BlockJob *do_backup_common(BackupCommon *backup, · line 3525
Quoted 2 lines
blockdev.c +2
-0+if (!backup->has_bitmap && backup->has_bitmap_mode) { +error_setg (errp,"Cannot specify Bitmap sync mode without a bitmap" );
Any reason for capitalizing the first “Bitmap”?
With a reason or it fixed:
Re: [Qemu-devel] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups
Quoted 37 lines · Max Reitz
Quoted 26 lines · John Snow
Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.
Some combinations don't make sense, though:
- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.
Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.
Signed-off-by John Snow <[email protected]>--- block/backup.c | 8 +------- blockdev.c | 22 ++++++++++++++++++++++ qapi/block-core.json | 6 ++++-- 3 files changed, 27 insertions(+), 9 deletions(-)[...]
Quoted 4 lines
blockdev.c +0
-0[...]
Quoted 9 lines
blockdev.c static BlockJob *do_backup_common(BackupCommon *backup, · line 3525
Quoted 2 lines
blockdev.c +2
-0+if (!backup->has_bitmap && backup->has_bitmap_mode) { +error_setg (errp,"Cannot specify Bitmap sync mode without a bitmap" );Any reason for capitalizing the first “Bitmap”?
With a reason or it fixed:
Reviewed-by Max Reitz <[email protected]>
Hanging around Germans too much?
Actually, I can explain why: because a "bitmap" is a generic term, but whenever I capitalize it as "Bitmap" I am referring to a Block Dirty Bitmap which is a specific sort of thing. I do this unconsciously.
But in that case, I should actually be consistent in the interface (and docstrings and docs and error strings) and always call it that specific thing, which I don't.
"bitmap" is probably more correct for now, but I ought to go through all the interface and make it consistent in some way or another.
(Actually: I'd like to see if I can't rename the "BdrvDirtyBitmap" object to something more generic and shorter so I can rename many of the functions we have something shorter.
Because the structure is "BdrvDirtyBitmap", there's some confusion when we name functions like bdrv_dirty_bitmap_{verb} because it's not clear if this is a bdrv function that does something with dirty bitmaps, or if it's a "BdrvDirtyBitmap" function that does something with that object.
I guess that seems like a subtle point, but it's why the naming conventions in dirty-bitmap.c are all over the place. I think at one point, the idea was that:
bdrv_{verb}_dirty_bitmap was an action applied to a BDS that happened to do something with dirty bitmaps. bdrv_dirty_bitmap_{verb} was an action that applied to a "BdrvDirtyBitmap". Crystal clear and not confusing at all, right?
It'd be nice to have functions that operate on a node be named bdrv_dbitmap_foo() and functions that operate on the bitmap structure itself named just dbitmap_bar().
Would it be okay if I named them such a thing, I wonder?
we have "bitmap" and "hbitmap" already, I could do something like "dbitmap" for "dirty bitmap" or some such. Kind of an arbitrary change I admit, but I'm itching to do a big spring cleaning in dirty-bitmap.c right after this series is done.)
Re: [Qemu-devel] [PATCH 7/8] block/backup: support bitmap sync modes for non-bitmap backups
Quoted 43 lines · John Snow
Quoted 40 lines · Max Reitz
Quoted 26 lines · John Snow
Accept bitmaps and sync policies for the other backup modes. This allows us to do things like create a bitmap synced to a full backup without a transaction, or start a resumable backup process.
Some combinations don't make sense, though:
- NEVER policy combined with any non-BITMAP mode doesn't do anything, because the bitmap isn't used for input or output.
It's harmless, but is almost certainly never what the user wanted.- sync=NONE is more questionable. It can't use on-success because this job never completes with success anyway, and the resulting artifact of 'always' is suspect: because we start with a full bitmap and only copy out segments that get written to, the final output bitmap will always be ... a fully set bitmap.
Maybe there's contexts in which bitmaps make sense for sync=none, but not without more severe changes to the current job, and omitting it here doesn't prevent us from adding it later.
Signed-off-by John Snow <[email protected]>--- block/backup.c | 8 +------- blockdev.c | 22 ++++++++++++++++++++++ qapi/block-core.json | 6 ++++-- 3 files changed, 27 insertions(+), 9 deletions(-)[...]
Quoted 4 lines
blockdev.c +0
-0[...]
Quoted 2 lines
blockdev.c +2
-0+if (!backup->has_bitmap && backup->has_bitmap_mode) { +error_setg (errp,"Cannot specify Bitmap sync mode without a bitmap" );
Quoted 8 lines
Quoted 6 lines
Any reason for capitalizing the first “Bitmap”?
With a reason or it fixed:
Reviewed-by Max Reitz <[email protected]>Hanging around Germans too much?
You should know then that the korrekt way to write it would be:
„Specifying Binarydigitmapsynchronizationmode without a Binarydigitmap is absolutely verboten!“
Quoted 8 lines
Actually, I can explain why: because a "bitmap" is a generic term, but whenever I capitalize it as "Bitmap" I am referring to a Block Dirty Bitmap which is a specific sort of thing. I do this unconsciously.
But in that case, I should actually be consistent in the interface (and docstrings and docs and error strings) and always call it that specific thing, which I don't.
Quoted 7 lines
"bitmap" is probably more correct for now, but I ought to go through all the interface and make it consistent in some way or another.
(Actually: I'd like to see if I can't rename the "BdrvDirtyBitmap" object to something more generic and shorter so I can rename many of the functions we have something shorter.
Well, BDB is free. That path has worked fine for BlockDriverStates.
Or what I said on IRC, but you know.
Quoted 5 lines
Because the structure is "BdrvDirtyBitmap", there's some confusion when we name functions like bdrv_dirty_bitmap_{verb} because it's not clear if this is a bdrv function that does something with dirty bitmaps, or if it's a "BdrvDirtyBitmap" function that does something with that object.
Quoted 8 lines
I guess that seems like a subtle point, but it's why the naming conventions in
dirty-bitmap.care all over the place. I think at one point, the idea was that:bdrv_{verb}_dirty_bitmap was an action applied to a BDS that happened to do something with dirty bitmaps. bdrv_dirty_bitmap_{verb} was an action that applied to a "BdrvDirtyBitmap". Crystal clear and not confusing at all, right?
I just thought people named their functions whatever they felt like at the time.
Quoted 10 lines
It'd be nice to have functions that operate on a node be named
bdrv_dbitmap_foo()and functions that operate on the bitmap structure itself named justdbitmap_bar().Would it be okay if I named them such a thing, I wonder?
we have "bitmap" and "hbitmap" already, I could do something like "dbitmap" for "dirty bitmap" or some such. Kind of an arbitrary change I admit, but I'm itching to do a big spring cleaning in
dirty-bitmap.cright after this series is done.)
HBitmaps are generally used to track dirty areas, so I’d find this a misnomer. BDBitmap would be OK. The “block” part should be in there somewhere.
Max
PATCH 8/8 iotests/257: test traditional sync modes
In reply to John Snow--- tests/qemu-iotests/257 | 31 + tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++ 2 files changed, 3120 insertions(+)
tests/qemu-iotests/257 +31 -0
@@ -288,6 +288,12 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): Bitmaps are always synchronized, regardless of failure. (Partial images must be kept.) + :param msync_mode: The mirror sync mode to use for the first backup. + Can be any one of: + - bitmap: Backups based on bitmap manifest. + - full: Full backups. + - top: Full backups of the top layer only. + :param failure: Is the (optional) failure mode, and can be any of: - None: No failure. Test the normative path. Default. - simulated: Cancel the job right before it completes.
@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): if bsync_mode == 'always' and failure == 'intermediate': # We manage to copy one sector (one bit) before the error. ebitmap.clear_bit(ebitmap.first_bit) + if msync_mode in ('full', 'top'): + # These modes return all bits set except what was copied/skipped + fail_bit = ebitmap.first_bit + ebitmap.clear() + ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY)) ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # 2 - Writes and Reference Backup
@@ -501,6 +512,21 @@ def test_backup_api(): 'bitmap404': ['on-success', 'always', 'never', None], 'bitmap0': [None], }, + 'full': { + None: ['on-success', 'always', 'never'], + 'bitmap404': ['on-success', 'always', 'never', None], + 'bitmap0': ['never', None], + }, + 'top': { + None: ['on-success', 'always', 'never'], + 'bitmap404': ['on-success', 'always', 'never', None], + 'bitmap0': ['never', None], + }, + 'none': { + None: ['on-success', 'always', 'never'], + 'bitmap404': ['on-success', 'always', 'never', None], + 'bitmap0': ['on-success', 'always', 'never', None], + } } for sync_mode, config in error_cases.items():
@@ -522,6 +548,11 @@ def main(): for failure in ("simulated", "intermediate", None): test_bitmap_sync(bsync_mode, "bitmap", failure) + for sync_mode in ('full', 'top'): + for bsync_mode in ('on-success', 'always'): + for failure in ('simulated', 'intermediate', None): + test_bitmap_sync(bsync_mode, sync_mode, failure) + test_backup_api() if __name__ == '__main__':
tests/qemu-iotests/257.out +3089 -0
2,945 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.0
Part of the diff is not shown here. View the raw message.
Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes
Quoted 5 lines · John Snow
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 31 + tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++ 2 files changed, 3120 insertions(+)
Oof.
Quoted 4 lines
tests/qemu-iotests/257 +0
-0
[...]
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 420
Quoted 6 lines
tests/qemu-iotests/257 +2
-0@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): if bsync_mode == 'always' and failure == 'intermediate': # We manage to copy one sector (one bit) before the error. ebitmap.clear_bit(ebitmap.first_bit) + if msync_mode in ('full', 'top'): + # These modes return all bits set except what was copied/skipped
Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.
Max
Quoted 6 lines
tests/qemu-iotests/257 +3
-0+ fail_bit = ebitmap.first_bit + ebitmap.clear() + ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY)) ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # 2 - Writes and Reference Backup
[...]
Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes
Quoted 9 lines · Max Reitz
Quoted 5 lines · John Snow
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 31 + tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++ 2 files changed, 3120 insertions(+)Oof.
Yeah, it's... a lot of test output. We probably shouldn't count the reference test output against any kind of SLOC metrics.
Quoted 18 lines
Quoted 4 lines
tests/qemu-iotests/257 +0
-0[...]
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 420
Quoted 6 lines
tests/qemu-iotests/257 +2
-0@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): if bsync_mode == 'always' and failure == 'intermediate': # We manage to copy one sector (one bit) before the error. ebitmap.clear_bit(ebitmap.first_bit) + if msync_mode in ('full', 'top'): + # These modes return all bits set except what was copied/skippedHm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.
Quoted 2 lines
Max
Good point!
I think this can be fixed by doing an initialization pass of the copy_bitmap when sync=top to set only the allocated regions in the bitmap.
This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.
So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.
This doesn't touch sync=none at all, which needs more invasive fixes if we wanted it to have useful bitmap return values (it needs to differentiate the idea between must-copy and can-copy, and I still don't know if this is worthwhile to do, so until I hear otherwise, I'm not gonna.)
Quoted 8 lines
Quoted 6 lines
tests/qemu-iotests/257 +3
-0+ fail_bit = ebitmap.first_bit + ebitmap.clear() + ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY)) ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # 2 - Writes and Reference Backup[...]
Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes
Quoted 39 lines · John Snow
Quoted 9 lines · Max Reitz
Quoted 5 lines · John Snow
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/257 | 31 + tests/qemu-iotests/257.out | 3089 ++++++++++++++++++++++++++++++++++++ 2 files changed, 3120 insertions(+)Oof.
Yeah, it's... a lot of test output. We probably shouldn't count the reference test output against any kind of SLOC metrics.
Quoted 18 lines
Quoted 4 lines
tests/qemu-iotests/257 +0
-0[...]
tests/qemu-iotests/257 def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): · line 420
Quoted 6 lines
tests/qemu-iotests/257 +2
-0@@ -410,6 +416,11 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None): if bsync_mode == 'always' and failure == 'intermediate': # We manage to copy one sector (one bit) before the error. ebitmap.clear_bit(ebitmap.first_bit) + if msync_mode in ('full', 'top'): + # These modes return all bits set except what was copied/skippedHm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.
Quoted 2 lines
Max
Good point!
Quoted 7 lines
I think this can be fixed by doing an initialization pass of the
copy_bitmapwhen sync=top to set only the allocated regions in the bitmap.This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.
Now that you mention it... I didn’t realize that. Yes, you’re right.
Quoted 9 lines
So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.This doesn't touch sync=none at all, which needs more invasive fixes if we wanted it to have useful bitmap return values (it needs to differentiate the idea between must-copy and can-copy, and I still don't know if this is worthwhile to do, so until I hear otherwise, I'm not gonna.)
No, I’m with you on that one.
Max
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
tests/qemu-iotests/257 +3
-0+ fail_bit = ebitmap.first_bit + ebitmap.clear() + ebitmap.dirty_bits(range(fail_bit, SIZE // GRANULARITY)) ebitmap.compare(get_bitmap(bitmaps, drive0.device, 'bitmap0')) # 2 - Writes and Reference Backup[...]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes
Quoted 31 lines · John Snow
Quoted 16 lines · Max Reitz
Quoted 15 lines · John Snow
Quoted 8 lines · Max Reitz
On 10.07.19 03:05, John Snow wrote:
Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.
Max
Good point!
I think this can be fixed by doing an initialization pass of the
copy_bitmapwhen sync=top to set only the allocated regions in the bitmap.Quoted 12 lines
Quoted 4 lines
This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.
Now that you mention it... I didn’t realize that. Yes, you’re right.
Quoted 5 lines
So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.
Quoted 7 lines
This might be harder than I first thought.
initializing the
copy_bitmapgenerally happens before we install the write notifier, which means that it occurs before the first yield.However, checking the allocation status can potentially be very slow, can't it? I can't just hog the thread while I check.
I was thinking about that myself. It isn’t that bad, because you aren’t doing the full block_status dance but just checking allocation status, which is reasonably quick (it just needs to look at the image format metadata, it doesn’t go down to the protocol layer).
But it’s probably not so good to halt the monitor for this, yes.
Quoted 7 lines
There are ways to cooperatively process write notifier interrupts and continue to check allocated status once we enter the main loop, but the problem there becomes: if we fail early, how can we tell if the backup is worth resuming?
We might not have reached a convergence point for the
copy_bitmapbefore we failed, and still have a lot of extra bits set.
Is that so bad?
Quoted 4 lines
I suppose at least in the case where we aren't trying to save the
copy_bitmapand need it to mean something specific, this is a reasonable approach to fixing sync=TOP.
Quoted 8 lines
As far as resume is concerned, I don't think I have good ideas. I could emit an event or something if you're using sync=top with a bitmap for output, but that feels *so* specialized for a niche(?) command that I don't know if it's worth pursuing.
(Plus, even then, what do you do if it fails before you see that event? You just have to give up on what we copied out? That seems like a waste and not the point of this exercise.)
Before that event, the bitmap can still be usable, as long as all “unknown” areas are set to dirty. Sure, your resumed backup will then copy too much data. But who cares.
So I don’t think you even need an event.
Quoted 3 lines
The only way I can think of at all to get a retry on sync=top is to take an always policy, and to allow a special invocation with something like mode=bitmap+top:
Yes, that was my first idea, too. But I didn’t even write about it, because of...
Quoted 5 lines
"Assume we need to copy anything set in the bitmap, unless it's not in the top layer, and then skip it."
Which seems awful, because it would be a specialty mode for the exclusive purpose of re-trying sync=top backups.
...exactly this.
Quoted 1 line
Meh.
I don’t think it’s all that bad.
Max
Re: [Qemu-devel] [PATCH 8/8] iotests/257: test traditional sync modes
Quoted 86 lines · Max Reitz
Quoted 31 lines · John Snow
Quoted 16 lines · Max Reitz
Quoted 15 lines · John Snow
Quoted 8 lines · Max Reitz
On 10.07.19 03:05, John Snow wrote:
Hm. How useful is bitmap support for 'top' then, anyway? That means that if you want to resume a top backup, you always have to resume it like it was a full backup. Which sounds kind of useless.
Max
Good point!
I think this can be fixed by doing an initialization pass of the
copy_bitmapwhen sync=top to set only the allocated regions in the bitmap.Quoted 12 lines
Quoted 4 lines
This means that the write notifier won't copy out regions that are written to that weren't already in the top layer. I believe this is actually a bugfix; the data we'd copy out in such cases is actually in the backing layer and shouldn't be copied with sync=top.
Now that you mention it... I didn’t realize that. Yes, you’re right.
Quoted 5 lines
So this would have two effects:
(1) sync=top gets a little more judicious about what it copies out on sync=top, and
(2) the bitmap return value is more meaningful again.Quoted 7 lines
This might be harder than I first thought.
initializing the
copy_bitmapgenerally happens before we install the write notifier, which means that it occurs before the first yield.However, checking the allocation status can potentially be very slow, can't it? I can't just hog the thread while I check.
I was thinking about that myself. It isn’t that bad, because you aren’t doing the full
block_statusdance but just checking allocation status, which is reasonably quick (it just needs to look at the image format metadata, it doesn’t go down to the protocol layer).But it’s probably not so good to halt the monitor for this, yes.
Quoted 7 lines
There are ways to cooperatively process write notifier interrupts and continue to check allocated status once we enter the main loop, but the problem there becomes: if we fail early, how can we tell if the backup is worth resuming?
We might not have reached a convergence point for the
copy_bitmapbefore we failed, and still have a lot of extra bits set.Is that so bad?
Quoted 4 lines
I suppose at least in the case where we aren't trying to save the
copy_bitmapand need it to mean something specific, this is a reasonable approach to fixing sync=TOP.Quoted 8 lines
As far as resume is concerned, I don't think I have good ideas. I could emit an event or something if you're using sync=top with a bitmap for output, but that feels *so* specialized for a niche(?) command that I don't know if it's worth pursuing.
(Plus, even then, what do you do if it fails before you see that event? You just have to give up on what we copied out? That seems like a waste and not the point of this exercise.)
Before that event, the bitmap can still be usable, as long as all “unknown” areas are set to dirty. Sure, your resumed backup will then copy too much data. But who cares.
So I don’t think you even need an event.
Quoted 3 lines
The only way I can think of at all to get a retry on sync=top is to take an always policy, and to allow a special invocation with something like mode=bitmap+top:
Yes, that was my first idea, too. But I didn’t even write about it, because of...
Quoted 3 lines
"Assume we need to copy anything set in the bitmap, unless it's not in the top layer, and then skip it."
Quoted 9 lines
Quoted 2 lines
Which seems awful, because it would be a specialty mode for the exclusive purpose of re-trying sync=top backups.
...exactly this.
Quoted 1 line
Meh.
I don’t think it’s all that bad.
No, it's just not ideal and it's something I'd have to defend in a patch. It's a caveat that would need documenting.
"Hey, depending on how far the job got before it failed, you might not want to resume it because it may not have finished determining which segments held allocated data. There's no way to tell if this happened or not."
It makes the decision making process by e.g. libvirt harder, though there are still some heuristics you could use, like:
- Is the bitmap count less than the size of the top image?
- Is it bigger?
And that might be good enough when deciding how to proceed. I suppose if we want to give more precise mechanisms for this we'd always be within our right to continue refining it and just document that it MIGHT have extra bits set.
--js