block: add block-dirty-bitmap-populate job
v2 cover block: add block-dirty-bitmap-populate job
Hi,
This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.
This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.
This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.
v2:
- Addressed some, but not all feedback
- Rebased on latest 'job-runner' series; but it's not clear if it should be kept.
- This version doesn't address all of the feedback from v1, but I am posting it to the list as an RFC.
John Snow (5):
block: add bitmap-populate job
blockdev: combine DriveBackupState and BlockdevBackupState
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add 287 for block-dirty-bitmap-populate
qapi/block-core.json | 66 + qapi/job.json | 2 +- qapi/transaction.json | 2 + include/block/block_int.h | 21 + block/bitmap-alloc.c | 207 ++ blockdev.c | 104 +- blockjob.c | 3 +- block/Makefile.objs | 1 + tests/qemu-iotests/257 | 110 +- tests/qemu-iotests/287 | 242 ++ tests/qemu-iotests/287.out | 4544 +++++++++++++++++++++++++++++++++ tests/qemu-iotests/bitmaps.py | 131 + tests/qemu-iotests/group | 1 + 13 files changed, 5305 insertions(+), 129 deletions(-) create mode 100644 block/bitmap-alloc.c create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out create mode 100644 tests/qemu-iotests/bitmaps.py
Signature · 1 line
2.21.1
RFC PATCH v2 1/5 block: add bitmap-populate job
This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.
It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.
--- qapi/block-core.json | 48 +++++++++ qapi/job.json | 2 +- include/block/block_int.h | 21 ++++ block/bitmap-alloc.c | 207 ++++++++++++++++++++++++++++++++++++++ blockjob.c | 3 +- block/Makefile.objs | 1 + 6 files changed, 280 insertions(+), 2 deletions(-) create mode 100644 block/bitmap-alloc.c
qapi/block-core.json +48 -0
@@ -2202,6 +2202,54 @@ { 'command': 'block-dirty-bitmap-merge', 'data': 'BlockDirtyBitmapMerge' } +## +# @BitmapPattern: +# +# An enumeration of possible patterns that can be written into a bitmap. +# +# @allocation-top: The allocation status of the top layer +# of the attached storage node. +# +# Since: 5.0 +## +{ 'enum': 'BitmapPattern', + 'data': ['allocation-top'] } + +## +# @BlockDirtyBitmapPopulate: +# +# @job-id: identifier for the newly-created block job. +# +# @pattern: What pattern should be written into the bitmap? +# +# @on-error: the action to take if an error is encountered on a bitmap's +# attached node, default 'report'. +# 'stop' and 'enospc' can only be used if the block device supports +# io-status (see BlockInfo). +# +# @auto-finalize: When false, this job will wait in a PENDING state after it has +# finished its work, waiting for @block-job-finalize before +# making any block graph changes. +# When true, this job will automatically +# perform its abort or commit actions. +# Defaults to true. +# +# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it +# has completely ceased all work, and awaits @block-job-dismiss. +# When true, this job will automatically disappear from the query +# list without user intervention. +# Defaults to true. +# +# Since: 5.0 +## +{ 'struct': 'BlockDirtyBitmapPopulate', + 'base': 'BlockDirtyBitmap', + 'data': { 'job-id': 'str', + 'pattern': 'BitmapPattern', + '*on-error': 'BlockdevOnError', + '*auto-finalize': 'bool', + '*auto-dismiss': 'bool' } } + ## # @BlockDirtyBitmapSha256: #
qapi/job.json +1 -1
@@ -22,7 +22,7 @@ # Since: 1.7 ## { 'enum': 'JobType', - 'data': ['commit', 'stream', 'mirror', 'backup', 'create'] } + 'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'bitmap-populate'] } ## # @JobStatus:
include/block/block_int.h +21 -0
@@ -1222,6 +1222,27 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BlockCompletionFunc *cb, void *opaque, JobTxn *txn, Error **errp); +/* +* bitpop_job_create: Create a new bitmap population job. +* +* @job_id: The id of the newly-created job. +* @bs: Block device associated with the @target_bitmap. +* @target_bitmap: The bitmap to populate. +* @on_error: What to do if an error on @bs is encountered. +* @creation_flags: Flags that control the behavior of the Job lifetime. +* See @BlockJobCreateFlags +* @cb: Completion function for the job. +* @opaque: Opaque pointer value passed to @cb. +* @txn: Transaction that this job is part of (may be NULL). +*/ +BlockJob *bitpop_job_create (const char *job_id,BlockDriverState *bs, +BdrvDirtyBitmap *target_bitmap, +BitmapPattern pattern, +BlockdevOnError on_error, +int creation_flags, +BlockCompletionFunc *cb,void *opaque, +JobTxn *txn,Error **errp); + BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs, const char *child_name, const BdrvChildRole *child_role,
block/bitmap-alloc.c new file +207 -0
207 lines not on this page · raw message
blockjob.c +2 -1
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)return job_type (job) ==JOB_TYPE_BACKUP ||job_type (job) ==JOB_TYPE_COMMIT ||job_type (job) ==JOB_TYPE_MIRROR || -job_type (job) ==JOB_TYPE_STREAM ; +job_type (job) ==JOB_TYPE_STREAM || +job_type (job) ==JOB_TYPE_BITMAP_POPULATE ; }BlockJob *block_job_next (BlockJob *bjob)
block/Makefile.objs +1 -0
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o block-obj-y += accounting.o dirty-bitmap.o block-obj-y += write-threshold.o block-obj-y += backup.o +block-obj-y += bitmap-alloc.o block-obj-$(CONFIG_REPLICATION) += replication.o block-obj-y += throttle.o copy-on-read.o block-obj-y += block-copy.o
Signature · 1 line
2.21.1
RFC PATCH v2 2/5 blockdev: combine DriveBackupState and BlockdevBackupState
In reply to John SnowThey have the same fields -- rename it BlockJobState.
--- blockdev.c | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-)
blockdev.c +12 -18
@@ -1702,11 +1702,11 @@ static void external_snapshot_clean(BlkActionState *common)aio_context_release (aio_context); } -typedef struct DriveBackupState { +typedef struct BlockJobActionState {BlkActionState common;BlockDriverState *bs;BlockJob *job; -}DriveBackupState ; +}BlockJobActionState ; static BlockJob *do_backup_common(BackupCommon *backup, BlockDriverState *bs,
@@ -1716,7 +1716,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,static void drive_backup_prepare (BlkActionState *common,Error **errp) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);DriveBackup *backup;BlockDriverState *bs;BlockDriverState *target_bs;
@@ -1853,7 +1853,7 @@ out:static void drive_backup_commit (BlkActionState *common) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context; aio_context =bdrv_get_aio_context (state->bs);
@@ -1867,7 +1867,7 @@ static void drive_backup_commit(BlkActionState *common)static void drive_backup_abort (BlkActionState *common) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);if (state->job) {AioContext *aio_context;
@@ -1883,7 +1883,7 @@ static void drive_backup_abort(BlkActionState *common)static void drive_backup_clean (BlkActionState *common) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context;if (!state->bs) {
@@ -1898,15 +1898,9 @@ static void drive_backup_clean(BlkActionState *common)aio_context_release (aio_context); } -typedef struct BlockdevBackupState { -BlkActionState common; -BlockDriverState *bs; -BlockJob *job; -}BlockdevBackupState ; -static void blockdev_backup_prepare (BlkActionState *common,Error **errp) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);BlockdevBackup *backup;BlockDriverState *bs;BlockDriverState *target_bs;
@@ -1954,7 +1948,7 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)static void blockdev_backup_commit (BlkActionState *common) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context; aio_context =bdrv_get_aio_context (state->bs);
@@ -1968,7 +1962,7 @@ static void blockdev_backup_commit(BlkActionState *common)static void blockdev_backup_abort (BlkActionState *common) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);if (state->job) {AioContext *aio_context;
@@ -1984,7 +1978,7 @@ static void blockdev_backup_abort(BlkActionState *common)static void blockdev_backup_clean (BlkActionState *common) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context;if (!state->bs) {
@@ -2265,14 +2259,14 @@ static const BlkActionOps actions[] = { .clean = external_snapshot_clean, }, [TRANSACTION_ACTION_KIND_DRIVE_BACKUP] = { - .instance_size = sizeof(DriveBackupState), + .instance_size = sizeof(BlockJobActionState), .prepare = drive_backup_prepare, .commit = drive_backup_commit, .abort = drive_backup_abort, .clean = drive_backup_clean, }, [TRANSACTION_ACTION_KIND_BLOCKDEV_BACKUP] = { - .instance_size = sizeof(BlockdevBackupState), + .instance_size = sizeof(BlockJobActionState), .prepare = blockdev_backup_prepare, .commit = blockdev_backup_commit, .abort = blockdev_backup_abort,
Signature · 1 line
2.21.1
RFC PATCH v2 5/5 iotests: add 287 for block-dirty-bitmap-populate
In reply to John SnowGive block-dirty-bitmap-populate a workout.
--- tests/qemu-iotests/287 | 242 ++ tests/qemu-iotests/287.out | 4544 ++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 4787 insertions(+) create mode 100755 tests/qemu-iotests/287 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +242 -0
@@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +# +# Test block-dirty-bitmap-populate +# +# Copyright (c) 2020 John Snow for Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +from collections import namedtuple +import itertools +import math +import os + +import iotests +from iotests import log, qemu_img +from bitmaps import EmulatedBitmap, GROUPS + +SIZE = 64 * 1024 * 1024 +GRANULARITY = 64 * 1024 + + +class Drive: + def __init__(self, path, vm): + self.path = path + self.vm = vm + self.fmt = None + self.size = None + self.node = None + + def img_create(self, fmt, size): + self.fmt = fmt + self.size = size + iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size)) + + +def block_dirty_bitmap_populate(vm, node, bitmap, job_id, pattern, **kwargs): + # Strip any arguments explicitly nulled by the caller: + kwargs = {key: val for key, val in kwargs.items() if val is not None} + result = vm.qmp_log('block-dirty-bitmap-populate', + node=node, + name=bitmap, + job_id=job_id, + pattern=pattern, + **kwargs) + return result + + +def populate(drive, bitmap, job_id, pattern='allocation-top', **kwargs): + kwargs.setdefault('pattern', pattern) + kwargs.setdefault('auto-finalize', False) + kwargs.setdefault('auto-dismiss', False) + ret = block_dirty_bitmap_populate(drive.vm, drive.node, + bitmap, job_id, **kwargs) + return { + 'id': job_id, + 'auto-finalize': kwargs['auto-finalize'], + 'auto-dismiss': kwargs['auto-dismiss'], + 'return': ret, + } + + +def perform_writes(drive, n, filter_node_name=None): + log("-- Write #{:d}:".format(n)) + node_name = filter_node_name or drive.node + for pattern in GROUPS[n].patterns: + cmd = "write -P{:s} 0x{:07x} 0x{:x}".format( + pattern.byte, + pattern.offset, + pattern.size) + drive.vm.hmp_qemu_io(node_name, cmd, use_log=True) + log('') + + +TestConfig = namedtuple('TestConfig', [ + 'base_pattern', + 'disabled', + 'pre_writes', + 'mid_writes', + 'cancel', + 'post_writes', +]) + + +def test_bitmap_populate(config): + """ + Test bitmap populate. + + :param base_pattern: Write a base pattern? + :param disabled: Disable the target bitmap? + :param pre_writes: Write a pattern after bitmap creation, but before job? + :param mid_writes: Write a pattern before job finalizes? + :param cancel: Cancel the job instead of finalizing it? + :param post_writes: Write a pattern after the job? + """ + with iotests.FilePaths(['img']) as (img_path,), iotests.VM() as vm: + log("\n=== Bitmap Populate {:s} ===\n".format(str(config))) + + log('-- Prepare image & VM:') + drive0 = Drive(img_path, vm=vm) + drive0.img_create(iotests.imgfmt, SIZE) + vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device())) + vm.launch() + + file_config = { + 'driver': 'file', + 'filename': drive0.path + } + + drive0.node = 'drive0' + vm.qmp_log('blockdev-add', + filters=[iotests.filter_qmp_testfiles], + node_name=drive0.node, + driver=drive0.fmt, + file=file_config) + log('') + + + # Step 0: Prepare & Base Allocation Pattern + + if config.base_pattern: + perform_writes(drive0, 0) + + + # Step 1: Add test bitmap + + log('-- Add Bitmap:') + vm.qmp_log('block-dirty-bitmap-add', + node=drive0.node, + name="target", + granularity=GRANULARITY, + disabled=config.disabled) + ebitmap = EmulatedBitmap() + log('') + + # Step 2: Pre-Writes + + if config.pre_writes: + perform_writes(drive0, 1) + if not config.disabled: + ebitmap.dirty_group(1) + bitmap = vm.get_bitmap(drive0.node, 'target') + ebitmap.compare(bitmap) + + + # Step 3: Launch job & Mid-Writes + + log('-- Test block-dirty-bitmap-populate (bitpop0):') + def pre_finalize(): + # Writes issued prior to job finalization: + if config.mid_writes: + perform_writes(drive0, 2) + if not config.disabled: + ebitmap.dirty_group(2) + + + class TestJobRunner(iotests.JobRunner): + def on_pending(self, event): + if config.mid_writes: + perform_writes(drive0, 2) + if not config.disabled: + ebitmap.dirty_group(2) + super().on_pending(event) + + job = populate(drive0, 'target', 'bitpop0') + assert job['return'] == {'return': {}} + job_runner = TestJobRunner(vm, job['id'], + auto_dismiss=job['auto-dismiss'], + auto_finalize=job['auto-finalize'], + cancel=config.cancel) + job_runner.run() + log('') + + + # Step 4: Post-job verification + + if not config.cancel: + # Any writes made prior to the job finishing should now be visible. + if config.base_pattern: + ebitmap.dirty_group(0) + if config.pre_writes: + ebitmap.dirty_group(1) + if config.mid_writes: + ebitmap.dirty_group(2) + + bitmap = vm.get_bitmap(drive0.node, 'target') + ebitmap.compare(bitmap) + + + # Step 5: Post-Writes + + if config.post_writes: + perform_writes(drive0, 3) + if not config.disabled: + ebitmap.dirty_group(3) + + + # Step 6: Final Verification + + log('-- Verification:') + bitmaps = vm.query_bitmaps() + log({'bitmaps': bitmaps}, indent=2) + log('') + bitmap = vm.get_bitmap(drive0.node, 'target', bitmaps=bitmaps) + ebitmap.compare(bitmap) + + + log('-- Cleanup:') + vm.qmp_log("block-dirty-bitmap-remove", + node=drive0.node, name="target") + + bitmaps = vm.query_bitmaps() + if bitmaps: + log("ERROR: bitmaps unaccounted for:") + log(bitmaps) + else: + log('OK: All bitmaps removed') + vm.shutdown() + log('') + + +def main(): + for args in itertools.product((True, False), repeat=6): + cfg = TestConfig(*args) + test_bitmap_populate(cfg) + + +if __name__ == '__main__': + iotests.script_main(main, supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/287.out new file +4544 -0
2,758 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.21.1
Part of the diff is not shown here. View the raw message.
RFC PATCH v2 3/5 qmp: expose block-dirty-bitmap-populate
In reply to John SnowThis is a new job-creating command.
--- qapi/block-core.json | 18 +++++++++++ qapi/transaction.json | 2 ++ blockdev.c | 74 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+)
qapi/block-core.json +18 -0
@@ -2250,6 +2250,24 @@ '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } +## +# @block-dirty-bitmap-populate: +# +# Creates a new job that writes a pattern into a dirty bitmap. +# +# Since: 5.0 +# +# Example: +# +# -> { "execute": "block-dirty-bitmap-populate", +# "arguments": { "node": "drive0", "target": "bitmap0", +# "job-id": "job0", "pattern": "allocate-top" } } +# <- { "return": {} } +# +## +{ 'command': 'block-dirty-bitmap-populate', 'boxed': true, + 'data': 'BlockDirtyBitmapPopulate' } + ## # @BlockDirtyBitmapSha256: #
qapi/transaction.json +2 -0
@@ -50,6 +50,7 @@ # - @block-dirty-bitmap-enable: since 4.0 # - @block-dirty-bitmap-disable: since 4.0 # - @block-dirty-bitmap-merge: since 4.0 +# - @block-dirty-bitmap-populate: since 5.0 # - @blockdev-backup: since 2.3 # - @blockdev-snapshot: since 2.5 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -67,6 +68,7 @@ 'block-dirty-bitmap-enable': 'BlockDirtyBitmap', 'block-dirty-bitmap-disable': 'BlockDirtyBitmap', 'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge', + 'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate', 'blockdev-backup': 'BlockdevBackup', 'blockdev-snapshot': 'BlockdevSnapshot', 'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +74 -0
92 lines not on this page · raw message
Signature · 1 line
2.21.1
RFC PATCH v2 4/5 iotests: move bitmap helpers into their own file
In reply to John Snow--- tests/qemu-iotests/257 | 110 +--------------------------- tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+), 109 deletions(-) create mode 100644 tests/qemu-iotests/bitmaps.py
tests/qemu-iotests/257 +1 -109
@@ -24,120 +24,12 @@ import os import iotests from iotests import log, qemu_img +from bitmaps import EmulatedBitmap, GROUPS SIZE = 64 * 1024 * 1024 GRANULARITY = 64 * 1024 -class Pattern: - def __init__(self, byte, offset, size=GRANULARITY): - self.byte = byte - self.offset = offset - self.size = size - - def bits(self, granularity): - lower = self.offset // granularity - upper = (self.offset + self.size - 1) // granularity - return set(range(lower, upper + 1)) - - -class PatternGroup: - """Grouping of Pattern objects. Initialize with an iterable of Patterns.""" - def __init__(self, patterns): - self.patterns = patterns - - def bits(self, granularity): - """Calculate the unique bits dirtied by this pattern grouping""" - res = set() - for pattern in self.patterns: - res |= pattern.bits(granularity) - return res - - -GROUPS = [ - PatternGroup([ - # Batch 0: 4 clusters - Pattern('0x49', 0x0000000), - Pattern('0x6c', 0x0100000), # 1M - Pattern('0x6f', 0x2000000), # 32M - Pattern('0x76', 0x3ff0000)]), # 64M - 64K - PatternGroup([ - # Batch 1: 6 clusters (3 new) - Pattern('0x65', 0x0000000), # Full overwrite - Pattern('0x77', 0x00f8000), # Partial-left (1M-32K) - Pattern('0x72', 0x2008000), # Partial-right (32M+32K) - Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K) - PatternGroup([ - # Batch 2: 7 clusters (3 new) - Pattern('0x74', 0x0010000), # Adjacent-right - Pattern('0x69', 0x00e8000), # Partial-left (1M-96K) - Pattern('0x6e', 0x2018000), # Partial-right (32M+96K) - Pattern('0x67', 0x3fe0000, - 2*GRANULARITY)]), # Overwrite [(64M-128K)-64M) - PatternGroup([ - # Batch 3: 8 clusters (5 new) - # Carefully chosen such that nothing re-dirties the one cluster - # that copies out successfully before failure in Group #1. - Pattern('0xaa', 0x0010000, - 3*GRANULARITY), # Overwrite and 2x Adjacent-right - Pattern('0xbb', 0x00d8000), # Partial-left (1M-160K) - Pattern('0xcc', 0x2028000), # Partial-right (32M+160K) - Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right -] - - -class EmulatedBitmap: - def __init__(self, granularity=GRANULARITY): - self._bits = set() - self.granularity = granularity - - def dirty_bits(self, bits): - self._bits |= set(bits) - - def dirty_group(self, n): - self.dirty_bits(GROUPS[n].bits(self.granularity)) - - def clear(self): - self._bits = set() - - def clear_bits(self, bits): - self._bits -= set(bits) - - def clear_bit(self, bit): - self.clear_bits({bit}) - - def clear_group(self, n): - self.clear_bits(GROUPS[n].bits(self.granularity)) - - @property - def first_bit(self): - return sorted(self.bits)[0] - - @property - def bits(self): - return self._bits - - @property - def count(self): - return len(self.bits) - - def compare(self, qmp_bitmap): - """ - Print a nice human-readable message checking that a bitmap as reported - by the QMP interface has as many bits set as we expect it to. - """ - - name = qmp_bitmap.get('name', '(anonymous)') - log("= Checking Bitmap {:s} =".format(name)) - - want = self.count - have = qmp_bitmap['count'] // qmp_bitmap['granularity'] - - log("expecting {:d} dirty sectors; have {:d}. {:s}".format( - want, have, "OK!" if want == have else "ERROR!")) - log('') - - class Drive: """Represents, vaguely, a drive attached to a VM. Includes format, graph, and device information."""
tests/qemu-iotests/bitmaps.py new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.21.1
Quoted 5 lines · John Snow
Hi,
This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.
Quoted 7 lines
This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.
This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.
I've dusted-off my patches for using this blockjob for this very specific case and it works for me.
For now I'll continue the integration with other blockjobs where we merge bitmaps.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to John SnowQuoted 3 lines · John Snow
This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.
retrofitted
Quoted 9 lines
It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.
Signed-off-by John Snow <[email protected]>---
+++
b/qapi/block-core.json
qapi/block-core.json line 2211
Quoted 10 lines
qapi/block-core.json +7
-0@@ -2202,6 +2202,54 @@ { 'command': 'block-dirty-bitmap-merge', 'data': 'BlockDirtyBitmapMerge' } +## +# @BitmapPattern: +# +# An enumeration of possible patterns that can be written into a bitmap. +# +# @allocation-top: The allocation status of the top layer +# of the attached storage node.
Presumably, allocated => bits are set.
Review line 2213
Quoted 2 lines
(quoted diff) +2
-0+# +# Since: 5.0
5.1, now
Quoted 20 lines
qapi/block-core.json +20
-0+## +{ 'enum': 'BitmapPattern', + 'data': ['allocation-top'] } + +## +# @BlockDirtyBitmapPopulate: +# +# @job-id: identifier for the newly-created block job. +# +# @pattern: What pattern should be written into the bitmap? +# +# @on-error: the action to take if an error is encountered on a bitmap's +# attached node, default 'report'. +# 'stop' and 'enospc' can only be used if the block device supports +# io-status (see BlockInfo). +# +# @auto-finalize: When false, this job will wait in a PENDING state after it has +# finished its work, waiting for @block-job-finalize before +# making any block graph changes. +# When true, this job will automatically
qapi/block-core.json line 2243
Quoted 10 lines
qapi/block-core.json +10
-0+# perform its abort or commit actions. +# Defaults to true. +# +# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it +# has completely ceased all work, and awaits @block-job-dismiss. +# When true, this job will automatically disappear from the query +# list without user intervention. +# Defaults to true. +# +# Since: 5.0
5.1
Quoted 17 lines
qapi/block-core.json +9
-0+## +{ 'struct': 'BlockDirtyBitmapPopulate', + 'base': 'BlockDirtyBitmap', + 'data': { 'job-id': 'str', + 'pattern': 'BitmapPattern', + '*on-error': 'BlockdevOnError', + '*auto-finalize': 'bool', + '*auto-dismiss': 'bool' } } + ## # @BlockDirtyBitmapSha256: #qapi/job.json +0
-0
qapi/job.json line 25
Quoted 5 lines
qapi/job.json +1
-1@@ -22,7 +22,7 @@ # Since: 1.7 ## { 'enum': 'JobType', - 'data': ['commit', 'stream', 'mirror', 'backup', 'create'] } + 'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'bitmap-populate'] }
Missing docs that 'bitmap-populate' is since 5.1.
qapi/job.json
Quoted 2 lines
qapi/job.json +1
-0+++ b/block/bitmap-alloc.c@@ -0,0 +1,207 @@
New file, but no MAINTAINERS update. It is covered by 'Block layer core', but shouldn't it also be covered by 'Dirty Bitmaps'?
Quoted 10 lines
qapi/job.json +10
-0+BlockJob *bitpop_job_create( + const char *job_id, + BlockDriverState *bs, + BdrvDirtyBitmap *target_bitmap, + BitmapPattern pattern, + BlockdevOnError on_error, + int creation_flags, + BlockCompletionFunc *cb, + void *opaque, + JobTxn *txn,
qapi/job.json
Quoted 10 lines
qapi/job.json +9
-0+ Error **errp) +{ + + /* NB: new bitmap is anonymous and enabled */ + cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap); + new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp); + if (!new_bitmap) { + return NULL; + }
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Do we need to worry about the converse case where the job started with something allocated but runs in parallel with the guest trimming, such that our bitmap marks something as set even though at the conclusion of our job it is no longer allocated?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 2/5] blockdev: combine DriveBackupState and BlockdevBackupState
In reply to John SnowQuoted 1 line · John Snow
They have the same fields -- rename it BlockJobState.
This says BlockJobState...
Quoted 11 lines
Signed-off-by John Snow <[email protected]>--- blockdev.c | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-)blockdev.c +0
-0
blockdev.c static void external_snapshot_clean(BlkActionState *common) · line 1705
Quoted 5 lines
blockdev.c +1
-1@@ -1702,11 +1702,11 @@ static void external_snapshot_clean(BlkActionState *common)aio_context_release (aio_context); } -typedef struct DriveBackupState { +typedef struct BlockJobActionState {
...but this does not. I'm assuming it is just a typo in the commit message?
Otherwise,
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 3/5] qmp: expose block-dirty-bitmap-populate
In reply to John SnowQuoted 14 lines · John Snow
This is a new job-creating command.
Signed-off-by John Snow <[email protected]>--- qapi/block-core.json | 18 +++++++++++ qapi/transaction.json | 2 ++ blockdev.c | 74 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+)qapi/block-core.json +0
-0
qapi/block-core.json line 2258
Quoted 9 lines
qapi/block-core.json +6
-0@@ -2250,6 +2250,24 @@ '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } +## +# @block-dirty-bitmap-populate: +# +# Creates a new job that writes a pattern into a dirty bitmap. +# +# Since: 5.0
5.1
qapi/block-core.json line 53
Quoted 6 lines
qapi/block-core.json +2
-0+++ b/qapi/transaction.json@@ -50,6 +50,7 @@ # - @block-dirty-bitmap-enable: since 4.0 # - @block-dirty-bitmap-disable: since 4.0 # - @block-dirty-bitmap-merge: since 4.0 +# - @block-dirty-bitmap-populate: since 5.0
ditto
Quoted 7 lines
qapi/block-core.json +3
-0+++ b/blockdev.c@@ -2233,6 +2233,63 @@ static void block_dirty_bitmap_remove_commit(BlkActionState *common) bdrv_release_dirty_bitmap(state->bitmap); } +static void block_dirty_bitmap_populate_prepare(BlkActionState *common, + Error **errp)
qapi/block-core.json static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2247
Quoted 10 lines
qapi/block-core.json +10
-0+{ + BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common); + BlockDirtyBitmapPopulate *bitpop; + BlockDriverState *bs; + AioContext *aio_context; + BdrvDirtyBitmap *bmap = NULL; + int job_flags = JOB_DEFAULT; + + assert(common->action->type == \ + TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE);
\ is not necessary here.
qapi/block-core.json static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2253
Quoted 6 lines
qapi/block-core.json +6
-0+ bitpop = common->action->u.block_dirty_bitmap_populate.data; + + bmap = block_dirty_bitmap_lookup(bitpop->node, bitpop->name, &bs, errp); + if (!bmap) { + return; + }
So the bitmap has to already exist, and we are just merging into it, correct?
Otherwise looks good.
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 5/5] iotests: add 287 for block-dirty-bitmap-populate
In reply to John SnowQuoted 6 lines · John Snow
Give block-dirty-bitmap-populate a workout.
Signed-off-by John Snow <[email protected]>--- tests/qemu-iotests/287 | 242 ++ tests/qemu-iotests/287.out | 4544 ++++++++++++++++++++++++++++++++++++
Sheesh, that output file is a workout indeed.
Quoted 4 lines
tests/qemu-iotests/group | 1 + 3 files changed, 4787 insertions(+)create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
287 has been created in the meantime (commit dd488fc1c00); we'll have to rebase. (Vladimir's series to rename iotests to something sensible instead of three digits may help...)
tests/qemu-iotests/287 line 46
Quoted 13 lines
tests/qemu-iotests/287 new file +13
-0+ +class Drive: + def __init__(self, path, vm): + self.path = path + self.vm = vm + self.fmt = None + self.size = None + self.node = None + + def img_create(self, fmt, size): + self.fmt = fmt + self.size = size + iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size))
Are we creating images with backing files anywhere in this test? Should we be? (The question of what is allocated in the top layer is more interesting to answer when there is a backing layer)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Eric BlakeQuoted 1 line · Eric Blake
On 5/13/20 10:49 PM, John Snow wrote:
[...]
Quoted 8 lines
block/bitmap-alloc.c line 183
Quoted 7 lines
block/bitmap-alloc.c new file +7
-0+ +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap); + new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + }
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Well, this could be made problem of the caller by skipping any newly allocated sectors to be written to the bitmap. The caller then can decide whether a snapshot of the allocation map is needed and thus a new inactive bitmap should be used as the destination, or whether new writes should be tracked by using an active bitmap.
Quoted 4 lines
Do we need to worry about the converse case where the job started with something allocated but runs in parallel with the guest trimming, such that our bitmap marks something as set even though at the conclusion of our job it is no longer allocated?
Given the semantics above this would conveniently not be a problem of the population job. If you create a snapshot of the allocation map any any point we'd care about that state.
Anyways, from the point of view of the bitmap code any write to a sector sets the bit so the trimming should not be treated differently.
Speicifically libvirt plans to use it on overlay(snapshot) images where the btimaps are not present so in that case even trimmed sectors need to mask the data in the backing image so they can technically be considered as allocated too.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to John SnowAm 14.05.2020 um 05:49 hat John Snow geschrieben:
Quoted 7 lines
This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.
It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.
Quoted 6 lines
Signed-off-by John Snow <[email protected]>--- qapi/block-core.json | 48 +++++++++ qapi/job.json | 2 +- include/block/block_int.h | 21 ++++ block/bitmap-alloc.c | 207 ++++++++++++++++++++++++++++++++++++++
bitmap-populate.c to be more consistent with the actual job name?
Quoted 4 lines
blockjob.c | 3 +- block/Makefile.objs | 1 + 6 files changed, 280 insertions(+), 2 deletions(-) create mode 100644 block/bitmap-alloc.c
[...]
Quoted 19 lines
block/bitmap-alloc.c new file +19
-0+BlockJob *bitpop_job_create ( +const char *job_id, +BlockDriverState *bs, +BdrvDirtyBitmap *target_bitmap, +BitmapPattern pattern, +BlockdevOnError on_error, +int creation_flags, +BlockCompletionFunc *cb, +void *opaque, +JobTxn *txn, +Error **errp) +{ +int64_t len; +BitpopBlockJob *job =NULL ; +int64_t cluster_size; +BdrvDirtyBitmap *new_bitmap =NULL ; + +assert (bs); +assert (target_bitmap);
block/bitmap-alloc.c line 160
Quoted 10 lines
block/bitmap-alloc.c new file +10
-0+ +if (!bdrv_is_inserted (bs)) { +error_setg (errp,"Device is not inserted: %s" , +bdrv_get_device_name (bs)); +return NULL ; + } + +if (bdrv_op_is_blocked (bs,BLOCK_OP_TYPE_BACKUP_SOURCE , errp)) { +return NULL ; + }
What does this protect? And why does BACKUP_SOURCE describe acccurately what this job does?
block/bitmap-alloc.c line 174
Quoted 13 lines
block/bitmap-alloc.c new file +13
-0+if (bdrv_dirty_bitmap_check (target_bitmap,BDRV_BITMAP_DEFAULT , errp)) { +return NULL ; + } + +if (pattern !=BITMAP_PATTERN_ALLOCATION_TOP ) { +error_setg (errp,"Unrecognized bitmap pattern" ); +return NULL ; + } + + len =bdrv_getlength (bs); +if (len <0 ) { +error_setg_errno (errp, -len,"unable to get length for '%s'" , +bdrv_get_device_name (bs));
This operates on the node level, so bdrv_get_device_or_node_name() is necessary to avoid empty strings in the message.
Quoted 5 lines
block/bitmap-alloc.c new file +5
-0+return NULL ; + } + +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap);
block/bitmap-alloc.c line 189
Quoted 10 lines
block/bitmap-alloc.c new file +10
-0+ new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + } + +/* Take ownership; we reserve the right to write into this on-commit. */ +bdrv_dirty_bitmap_set_busy (target_bitmap, true); + + job = block_job_create(job_id, &bitpop_job_driver, txn, bs, + BLK_PERM_CONSISTENT_READ,
I don't think we actually rely on CONSISTENT_READ, but then, using the job on inconsistent nodes probably makes little sense and we can always relax the restriction later if necessary.
Quoted 8 lines
block/bitmap-alloc.c new file +8
-0+BLK_PERM_ALL & ~BLK_PERM_RESIZE , +0 , creation_flags, + cb, opaque, errp); +if (!job) { +bdrv_dirty_bitmap_set_busy (target_bitmap, false); +bdrv_release_dirty_bitmap (new_bitmap); +return NULL ; + }
block/bitmap-alloc.c line 207
Quoted 10 lines
block/bitmap-alloc.c new file +10
-0+ + job->bs = bs; + job->on_error = on_error; + job->target_bitmap = target_bitmap; + job->new_bitmap = new_bitmap; + job->len = len; +job_progress_set_remaining (&job->common.job, job->len); + +return &job->common; +}
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Eric BlakeAm 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
block/bitmap-alloc.c line 183
Quoted 7 lines
block/bitmap-alloc.c new file +7
-0+ +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap); + new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + }
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from new_bitmap or are the bits already set in it anyway?
If it relies on new_bitmap, I think this deserves a comment here.
In the documentation of BitmapPattern and specifically @allocation-top, we should probably also be more explicit what this means for requests that the guest makes while the job is running.
Quoted 4 lines
Do we need to worry about the converse case where the job started with something allocated but runs in parallel with the guest trimming, such that our bitmap marks something as set even though at the conclusion of our job it is no longer allocated?
Discard shouldn't make the backing file visible (didn't we fix this for qcow2 v2 only recently?), so in the sense of bdrv_is_allocated(), they will still be allocated, even though they are probably zero clusters now.
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 9 lines · Kevin Wolf
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
block/bitmap-alloc.c new file +7
-0+ +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap); + new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + }
Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Re: [PATCH RFC v2 5/5] iotests: add 287 for block-dirty-bitmap-populate
In reply to John SnowAm 14.05.2020 um 05:49 hat John Snow geschrieben:
Quoted 3 lines
Give block-dirty-bitmap-populate a workout.
Signed-off-by John Snow <[email protected]>
I think it would be good to test on-error behaviour, too. Maybe that would be a different test file, though, considering that the output for this one is already huge?
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Peter KrempaAm 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
block/bitmap-alloc.c new file +7
-0+ +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap); + new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?
Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 26 lines · Kevin Wolf
Am 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
block/bitmap-alloc.c new file +7
-0+ +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap); + new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Quoted 5 lines
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Possibly yes, but I'll leave that for later. All of this is done when executin very expensive operations anyways so for our first implementation it IMO won't matter that much.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Am 04.06.2020 um 18:22 hat Peter Krempa geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 26 lines
Am 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
block/bitmap-alloc.c new file +7
-0+ +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap); + new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Quoted 5 lines
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Quoted 3 lines
Possibly yes, but I'll leave that for later. All of this is done when executin very expensive operations anyways so for our first implementation it IMO won't matter that much.
I'm not necessarily saying that the change is needed on the libvirt side. It could also be that the block job should directly work with the given bitmap instead of having its internal temporary bitmap. Changing this later would mean changing the semantics of the block job, so it would be somewhat problematic.
It would be good to have a clear picture of what we want the final result to look like.
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 38 lines · Kevin Wolf
Am 04.06.2020 um 18:22 hat Peter Krempa geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 26 lines
Am 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
(quoted diff) +7
-0+ + /* NB: new bitmap is anonymous and enabled */ + cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap); + new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp); + if (!new_bitmap) { + return NULL; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Quoted 5 lines
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Quoted 3 lines
Possibly yes, but I'll leave that for later. All of this is done when executin very expensive operations anyways so for our first implementation it IMO won't matter that much.
Quoted 8 lines
I'm not necessarily saying that the change is needed on the libvirt side. It could also be that the block job should directly work with the given bitmap instead of having its internal temporary bitmap. Changing this later would mean changing the semantics of the block job, so it would be somewhat problematic.
It would be good to have a clear picture of what we want the final result to look like.
Well with current semantics of the 'nodename' argument controling both where the populated bitmap is located and also which node's allocation bitmap to take I don't think we can optimize it further in libvirt.
Current usage scenario is that we use a temporary bitmap populated with the job to merge with bitmaps present in nodes which are removed by blockjobs into the destination node of the block job. This means that the real destination of the bits populated is in a different node than it was originally and the above job semantics don't allow that.
Either way I'd strongly prefer to be able to kick off all the populate jobs at once rather than having to sequence them so any semantic change towards making it possible to target bitmaps in a different node would also require that multiple jobs can run in parallel with a single bitmap as destination. I'm not sure if that doesn't overcomplicate things though.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Am 05.06.2020 um 11:24 hat Peter Krempa geschrieben:
Quoted 52 lines · Kevin Wolf
Quoted 38 lines
Am 04.06.2020 um 18:22 hat Peter Krempa geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 26 lines
Am 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
(quoted diff) +7
-0+ + /* NB: new bitmap is anonymous and enabled */ + cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap); + new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp); + if (!new_bitmap) { + return NULL; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Quoted 5 lines
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Quoted 3 lines
Possibly yes, but I'll leave that for later. All of this is done when executin very expensive operations anyways so for our first implementation it IMO won't matter that much.
Quoted 8 lines
I'm not necessarily saying that the change is needed on the libvirt side. It could also be that the block job should directly work with the given bitmap instead of having its internal temporary bitmap. Changing this later would mean changing the semantics of the block job, so it would be somewhat problematic.
It would be good to have a clear picture of what we want the final result to look like.
Well with current semantics of the 'nodename' argument controling both where the populated bitmap is located and also which node's allocation bitmap to take I don't think we can optimize it further in libvirt.
Quoted 5 lines
Current usage scenario is that we use a temporary bitmap populated with the job to merge with bitmaps present in nodes which are removed by blockjobs into the destination node of the block job. This means that the real destination of the bits populated is in a different node than it was originally and the above job semantics don't allow that.
So does this mean that a better API wouldn't only take a node-name and bitmap name (where the node identified by node-name is not only where the target bitmap is, but also the node whose allocation status is queried), but that it should take two different node-names for source (= reading allocation status) and target (= owner of the bitmap)?
Quoted 6 lines
Either way I'd strongly prefer to be able to kick off all the populate jobs at once rather than having to sequence them so any semantic change towards making it possible to target bitmaps in a different node would also require that multiple jobs can run in parallel with a single bitmap as destination. I'm not sure if that doesn't overcomplicate things though.
Other people are more familiar with the dirty bitmap code, so I may be wrong, but intuitively, I don't see any problem with multiple jobs dirtying blocks in the same bitmap. Or, with the internal temporary bitmap as used in this version of the series, multiple jobs that, one after another, merge their result to the same bitmap on completion.
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 59 lines · Kevin Wolf
Am 05.06.2020 um 11:24 hat Peter Krempa geschrieben:
Quoted 52 lines · Kevin Wolf
Quoted 38 lines
Am 04.06.2020 um 18:22 hat Peter Krempa geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 26 lines
Am 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
(quoted diff) +7
-0+ + /* NB: new bitmap is anonymous and enabled */ + cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap); + new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp); + if (!new_bitmap) { + return NULL; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Quoted 5 lines
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Quoted 3 lines
Possibly yes, but I'll leave that for later. All of this is done when executin very expensive operations anyways so for our first implementation it IMO won't matter that much.
Quoted 8 lines
I'm not necessarily saying that the change is needed on the libvirt side. It could also be that the block job should directly work with the given bitmap instead of having its internal temporary bitmap. Changing this later would mean changing the semantics of the block job, so it would be somewhat problematic.
It would be good to have a clear picture of what we want the final result to look like.
Well with current semantics of the 'nodename' argument controling both where the populated bitmap is located and also which node's allocation bitmap to take I don't think we can optimize it further in libvirt.
Quoted 5 lines
Current usage scenario is that we use a temporary bitmap populated with the job to merge with bitmaps present in nodes which are removed by blockjobs into the destination node of the block job. This means that the real destination of the bits populated is in a different node than it was originally and the above job semantics don't allow that.
Quoted 5 lines
So does this mean that a better API wouldn't only take a node-name and bitmap name (where the node identified by node-name is not only where the target bitmap is, but also the node whose allocation status is queried), but that it should take two different node-names for source (= reading allocation status) and target (= owner of the bitmap)?
Yes. That way one of the merges would be merged (heh) into the operation itself preventing us from the need to have an extra temporary bitmap.
Quoted 7 lines
Quoted 6 lines
Either way I'd strongly prefer to be able to kick off all the populate jobs at once rather than having to sequence them so any semantic change towards making it possible to target bitmaps in a different node would also require that multiple jobs can run in parallel with a single bitmap as destination. I'm not sure if that doesn't overcomplicate things though.
Quoted 5 lines
Other people are more familiar with the dirty bitmap code, so I may be wrong, but intuitively, I don't see any problem with multiple jobs dirtying blocks in the same bitmap. Or, with the internal temporary bitmap as used in this version of the series, multiple jobs that, one after another, merge their result to the same bitmap on completion.
I don't see a problem with the bitmaps itself since we are only OR-ing results together. I just wanted to state the desired usage.
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 85 lines · Kevin Wolf
Quoted 59 lines
Am 05.06.2020 um 11:24 hat Peter Krempa geschrieben:
Quoted 52 lines · Kevin Wolf
Quoted 38 lines
Am 04.06.2020 um 18:22 hat Peter Krempa geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 26 lines
Am 04.06.2020 um 11:16 hat Peter Krempa geschrieben:
Quoted 19 lines · Kevin Wolf
Quoted 9 lines
Am 18.05.2020 um 22:49 hat Eric Blake geschrieben:
Quoted 8 lines
Quoted 7 lines
(quoted diff) +7
-0+ + /* NB: new bitmap is anonymous and enabled */ + cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap); + new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp); + if (!new_bitmap) { + return NULL; + }Quoted 8 lines
Quoted 4 lines
This means if the guest writes to the disk while the job is ongoing, the bitmap will be updated to mark that portion of the bitmap as set, even if it was not allocated at the time the job started. But then again, the guest writes are causing allocation, so this seems like the right thing to do.
Is the target bitmap active at the same time, i.e. will it get the correct information only from
new_bitmapor are the bits already set in it anyway?Quoted 5 lines
Yes, libvirt plans to use it with an active non-persistent bitmap which will in subsequent steps be merged into others. The bitmap is added in the same transaction. The bitmap must be active, because we need to wait for the block jobs to finish before it becomes usable and thus can't sequence in other operations until later.
Quoted 5 lines
A lot of bitmap merging then, because the block job in this series already creates a temporary internal bitmap that is merged into the target bitmap on completion. But if the target bitmap is only libvirt's temporary bitmap to be merged to yet another bitmap, I wonder if this process shouldn't be simplified.
Quoted 3 lines
Possibly yes, but I'll leave that for later. All of this is done when executin very expensive operations anyways so for our first implementation it IMO won't matter that much.
Quoted 8 lines
I'm not necessarily saying that the change is needed on the libvirt side. It could also be that the block job should directly work with the given bitmap instead of having its internal temporary bitmap. Changing this later would mean changing the semantics of the block job, so it would be somewhat problematic.
It would be good to have a clear picture of what we want the final result to look like.
Well with current semantics of the 'nodename' argument controling both where the populated bitmap is located and also which node's allocation bitmap to take I don't think we can optimize it further in libvirt.
Quoted 5 lines
Current usage scenario is that we use a temporary bitmap populated with the job to merge with bitmaps present in nodes which are removed by blockjobs into the destination node of the block job. This means that the real destination of the bits populated is in a different node than it was originally and the above job semantics don't allow that.
Quoted 5 lines
So does this mean that a better API wouldn't only take a node-name and bitmap name (where the node identified by node-name is not only where the target bitmap is, but also the node whose allocation status is queried), but that it should take two different node-names for source (= reading allocation status) and target (= owner of the bitmap)?
Yes. That way one of the merges would be merged (heh) into the operation itself preventing us from the need to have an extra temporary bitmap.
Quoted 7 lines
Quoted 6 lines
Either way I'd strongly prefer to be able to kick off all the populate jobs at once rather than having to sequence them so any semantic change towards making it possible to target bitmaps in a different node would also require that multiple jobs can run in parallel with a single bitmap as destination. I'm not sure if that doesn't overcomplicate things though.
Quoted 5 lines
Other people are more familiar with the dirty bitmap code, so I may be wrong, but intuitively, I don't see any problem with multiple jobs dirtying blocks in the same bitmap. Or, with the internal temporary bitmap as used in this version of the series, multiple jobs that, one after another, merge their result to the same bitmap on completion.
I don't see a problem with the bitmaps itself since we are only OR-ing results together. I just wanted to state the desired usage.
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 2 lines · Kevin Wolf
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 1 line · Kevin Wolf
[...]
Quoted 8 lines
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Quoted 4 lines
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
I certainly can document the way we'll use it but that in turn depends on how the job behaves.
With the current state of the job I plan to use it in two scenarios:
Preface: I'm currently changing libvirt to use one active bitmap per checkpoint (checkpoint is name for the point in time we want to take backup from). This means that a layer of the backing chain can have multiple active bitmaps depending on how many checkpoints were created in the current top layer. (previously we've tried to optimize things by having just one bitmap active, but the semantics were getting too crazy to be maintainable long-term)
Bitmaps are no longer propagated over to upper layers when creating snapshots as we can use block-dirty-bitmap-populate instead.
1) backup
Prior to doing the backup I'm figuring out the final backup bitmap, this involves creating a temporary bitmap populated by the job for every layer of the backing chain above of the one which contains the bitmap we want to take a backup from and then merge all of them together as a base for the backup.
2) blockjobs
Note: This is currently an outline how the things should be as I've hit the snag with attempting to run the population jobs during 'ready' state of a active-layer block-commit/blockdev-mirror job only an hour ago and I need to change a few things.
2.1) active layer block-commit/blockdev-mirror
When the job reaches 'ready' state I'll create bitmaps in the destination/base image of the job for every bitmap in the images dropped/merged (we use blockdev-mirror in full-sync mode) by the blockjob. This will capture the writes that happen after 'job-complete'.
The job will then be completed and the 2.2. will be executed as well.
2.2) non-active commit and also continuation of active layer block-commit/blockdev-mirror
After the job is completed succesfully I'll create temporary non-persistent bitmaps for/in the images removed by the blockjob and merge them into the destination image's bitmaps depending on their original location in the backing chain so that the bitmap state still properly describes which blocks have changed.
After that the original images willbe blockdev-del-eted. The above is partialy in use today and since the job is already completed also requires blockdev-reopen to successfuly write to the bitmaps.
----
While writing the above down I've actually realized that controling the destination of the bitmap might not be as useful as I've thought originally as in 2.2. step I might need the allocation bitmap merged into multiple bitmaps, so I'd either need a temporary bitmap anyways or would have to re-run the job multiple times which seems wasteful. I'm no longer fully persuaded that adding the 'merge' step to the dirty populate blockjob will be the best thing since sliced bread.
Quoted 10 lines · John Snow
Hi,
This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.
This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.
Quoted 7 lines
This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.
v2:
- Addressed some, but not all feedback
- Rebased on latest 'job-runner' series; but it's not clear if it should be kept.
Message-id for that series? I'm not finding a message with a subject containing a literal 'job-runner', but am not sure which subject to look for instead.
I also couldn't find an obvious tag or branch at https://github.com/jnsnow/qemu/branches where you might have stashed this including prerequisites.
Quoted 2 lines
- This version doesn't address all of the feedback from v1, but I am posting it to the list as an RFC.
I'm happy to try and take over these patches to prepare a v3, but only if I can get them to build by finding the prerequisites :)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Peter KrempaQuoted 14 lines · Peter Krempa
Quoted 2 lines · Kevin Wolf
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 1 line · Kevin Wolf
[...]
Quoted 8 lines
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Quoted 5 lines
Quoted 4 lines
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
Hi! Sorry me missing the discussion for a long time.
About new job semantics: if you create temporary bitmaps anyway, I do think that we should allow to populate into target bitmap directly, without creating any internal temporary bitmaps. I suggested it when reviewing v1, John argued for more transaction-like semantics to look like other jobs. Still, we can support both modes if we want.
Allowing to use one target for several populating job is an interesting idea. Current series does "bdrv_dirty_bitmap_set_busy(target_bitmap, true);", which forbids it.. Hmm. If we just drop it, nothing prevents user just remove target bitmap during the job. So, we'll need something like collective-busy bitmap..
Quoted 5 lines
I certainly can document the way we'll use it but that in turn depends on how the job behaves.
With the current state of the job I plan to use it in two scenarios:
Quoted 7 lines
Preface: I'm currently changing libvirt to use one active bitmap per checkpoint (checkpoint is name for the point in time we want to take backup from). This means that a layer of the backing chain can have multiple active bitmaps depending on how many checkpoints were created in the current top layer. (previously we've tried to optimize things by having just one bitmap active, but the semantics were getting too crazy to be maintainable long-term)
Hmm. I had a plan of creating "lazy" disabled bitmaps, to optimize scenario with one active bitmap, so that disabled bitmaps are not loaded into RAM on start, but only on demand. But how to do it with "many active bitmaps" scenario? I don't think that's a good idea.. Possibly, we can implement laziness by internally make only one active bitmap and merge it here and there when you request some active bitmap which we actually didn't load yet..
Could you describe, what is the exact problem with "several disabled - one active" scheme, and how is it solved by "several active"?
Quoted 3 lines
Bitmaps are no longer propagated over to upper layers when creating snapshots as we can use block-dirty-bitmap-populate instead.
Unexpected turn. When all this topic only started, it was reasoned more like "if user forget to create bitmap at start, let's help him".. But now it becomes the common scenario. Hmm.
What do you think of granularity? We in Virtuozzo use 1M cluster as a default for qcow2 images. But we use 64k granularity (default) for bitmaps, to have smaller incremental backups. So, this is advantage of creating bitmap over relaying on block-status capturing by block-dirty-bitmap-populate: you don't control dirtiness granularity. So, I think that bitmap propagation, or just creating new dirty bitmap to track dirtiness from start of new snapshot is better.
Quoted 8 lines
1) backup
Prior to doing the backup I'm figuring out the final backup bitmap, this involves creating a temporary bitmap populated by the job for every layer of the backing chain above of the one which contains the bitmap we want to take a backup from and then merge all of them together as a base for the backup.
(just thinking out loud)
So, assume the sequence top -> middle -> base
If we have a backup, which was done when we were in base, than bitmap is stored in base. And is loaded, and is active, but don't changes really, as base is opened read-only.] We merge block-status information of top and middle together with this bitmap, and aggregate difference between last backup and current state.
Quoted 8 lines
2) blockjobs
Note: This is currently an outline how the things should be as I've hit the snag with attempting to run the population jobs during 'ready' state of a active-layer block-commit/blockdev-mirror job only an hour ago and I need to change a few things.
Quoted 8 lines
2.1) active layer block-commit/blockdev-mirror
When the job reaches 'ready' state I'll create bitmaps in the destination/base image of the job for every bitmap in the images dropped/merged (we use blockdev-mirror in full-sync mode) by the blockjob. This will capture the writes that happen after 'job-complete'.
The job will then be completed and the 2.2. will be executed as well.
So, the aim is not to miss any new writes after switching to new bs, but do not capture into bitmaps writes which are copying the whole disk during mirror.
Quoted 8 lines
2.2) non-active commit and also continuation of active layer block-commit/blockdev-mirror
After the job is completed succesfully I'll create temporary non-persistent bitmaps for/in the images removed by the blockjob and merge them into the destination image's bitmaps depending on their original location in the backing chain so that the bitmap state still properly describes which blocks have changed.
I don't follow. How do you populate these new temporary bitmaps? They are empty after creation..
Quoted 7 lines
After that the original images willbe blockdev-del-eted. The above is partialy in use today and since the job is already completed also requires blockdev-reopen to successfuly write to the bitmaps.
----
Quoted 8 lines
While writing the above down I've actually realized that controling the destination of the bitmap might not be as useful as I've thought originally as in 2.2. step I might need the allocation bitmap merged into multiple bitmaps, so I'd either need a temporary bitmap anyways or would have to re-run the job multiple times which seems wasteful. I'm no longer fully persuaded that adding the 'merge' step to the dirty populate blockjob will be the best thing since sliced bread.
What is 'merge' step? Do you mean that populating directly into target bitmap is not really needed?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Am 06.06.2020 um 08:55 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 21 lines
Quoted 14 lines · Peter Krempa
Quoted 2 lines · Kevin Wolf
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 1 line · Kevin Wolf
[...]
Quoted 8 lines
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Quoted 5 lines
Quoted 4 lines
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
Quoted 7 lines
Hi! Sorry me missing the discussion for a long time.
About new job semantics: if you create temporary bitmaps anyway, I do think that we should allow to populate into target bitmap directly, without creating any internal temporary bitmaps. I suggested it when reviewing v1, John argued for more transaction-like semantics to look like other jobs. Still, we can support both modes if we want.
But don't all other jobs allow you to see intermediate states? Like, when you look at the target node in the middle of a mirror job, you'll see a half-updated target image.
If we have an actual use case for both modes, we can certainly support that, but do we have one?
Quoted 6 lines
Allowing to use one target for several populating job is an interesting idea. Current series does "
bdrv_dirty_bitmap_set_busy(target_bitmap, true);", which forbids it.. Hmm. If we just drop it, nothing prevents user just remove target bitmap during the job. So, we'll need something like collective-busy bitmap..
I'm not sure for what the busy flag is used in detail, but if this is the problem, maybe it's possible to just replace it with a counter?
Kevin
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Vladimir Sementsov-OgievskiyQuoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Peter Krempa
Quoted 2 lines · Kevin Wolf
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 1 line · Kevin Wolf
[...]
Quoted 8 lines
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Quoted 5 lines
Quoted 4 lines
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
Hi! Sorry me missing the discussion for a long time.
About new job semantics: if you create temporary bitmaps anyway, I do think that we should allow to populate into target bitmap directly, without creating any internal temporary bitmaps. I suggested it when reviewing v1, John argued for more transaction-like semantics to look like other jobs. Still, we can support both modes if we want.
Allowing to use one target for several populating job is an interesting idea. Current series does "
bdrv_dirty_bitmap_set_busy(target_bitmap, true);", which forbids it.. Hmm. If we just drop it, nothing prevents user just remove target bitmap during the job. So, we'll need something like collective-busy bitmap..Quoted 5 lines
I certainly can document the way we'll use it but that in turn depends on how the job behaves.
With the current state of the job I plan to use it in two scenarios:
Quoted 7 lines
Preface: I'm currently changing libvirt to use one active bitmap per checkpoint (checkpoint is name for the point in time we want to take backup from). This means that a layer of the backing chain can have multiple active bitmaps depending on how many checkpoints were created in the current top layer. (previously we've tried to optimize things by having just one bitmap active, but the semantics were getting too crazy to be maintainable long-term)
Quoted 3 lines
Hmm. I had a plan of creating "lazy" disabled bitmaps, to optimize scenario with one active bitmap, so that disabled bitmaps are not loaded into RAM on start, but only on demand. But how to do it with "many active bitmaps" scenario? I don't think that's a good idea.. Possibly, we can implement laziness by internally make only one active bitmap and merge it here and there when you request some active bitmap which we actually didn't load yet..
Could you describe, what is the exact problem with "several disabled - one active" scheme, and how is it solved by "several active"?
The 'several disabled one active' semantics _heavily_ depend on metadata which must be tracked outside of qemu and is more prone to break. If any of the intermediate bitmaps is broken or missing everything breaks.
Then there's the complexity of the code which handles merging of the bitmaps during block jobs. Jobs such as blockdev-mirror in full mode and block-commit squash together the data and we need to do something about the bitmaps for the backups to work properly afterwards.
Without considering overlays which were created without propagating bitmaps, the code was already getting hairy especially in the case of backups where we needed to stitch together bitmaps for all the bitmaps corresponding to the given point in time where the backup is taken from.
When we add overlays without any bitmaps into the mix the code for resolving which bitmaps to merge the code is becoming very unpleasant, hard to understand and maintain and that is the main point for the switch.
I don't want to add unnecessary complexity to the libvirt code which will make it more fragile or hard to understand and fix in the future.
Both points which I heard for now (performance, and backup granularity in case of non-default qcow2 block size) don't seem compelling enough to me to make my life of implementing the feature in libvirt so much harder.
Also users really can just remove the point in time they wish to backup from after a successful backup which will also remove the corresponding active bitmap.
Quoted 4 lines
Quoted 2 lines
Bitmaps are no longer propagated over to upper layers when creating snapshots as we can use block-dirty-bitmap-populate instead.
Unexpected turn. When all this topic only started, it was reasoned more like "if user forget to create bitmap at start, let's help him".. But now it becomes the common scenario. Hmm.
It's not only a "user forgot" thing, but more that a systemic change would be required.
Additionally until _very_ recently it wasn't possible to create bitmaps using qemu-img, which made it impossible to create overlays for inactive VMs. Arguably this has changed so we could require it. It still adds a moving part which can break if the user doesn't add the bitmap or requires yet another special case handling if we want to compensate for that.
As of such, in libvirt's tech-preview implementation that is present currently upstream, if you create a qcow2 overlay without adding the appropriate bitmaps, the backups simply won't work.
Quoted 1 line
What do you think of granularity? We in Virtuozzo use 1M cluster as a default for qcow2 images. But we use 64k granularity (default) for bitmaps, to have smaller incremental backups. So, this is advantage of creating bitmap over relaying on block-status capturing by block-dirty-bitmap-populate: you don't control dirtiness granularity. So, I think that bitmap propagation, or just creating new dirty bitmap to track dirtiness from start of new snapshot is better.
This is a valid argument. Backups in this scenario will be bigger. I still don't feel like the code needs to be made much more complex because of it though.
Quoted 37 lines
Quoted 7 lines
1) backup
Prior to doing the backup I'm figuring out the final backup bitmap, this involves creating a temporary bitmap populated by the job for every layer of the backing chain above of the one which contains the bitmap we want to take a backup from and then merge all of them together as a base for the backup.
(just thinking out loud)
So, assume the sequence top -> middle -> base
If we have a backup, which was done when we were in base, than bitmap is stored in base. And is loaded, and is active, but don't changes really, as base is opened read-only.] We merge block-status information of top and middle together with this bitmap, and aggregate difference between last backup and current state.
Quoted 8 lines
2) blockjobs
Note: This is currently an outline how the things should be as I've hit the snag with attempting to run the population jobs during 'ready' state of a active-layer block-commit/blockdev-mirror job only an hour ago and I need to change a few things.
Quoted 8 lines
2.1) active layer block-commit/blockdev-mirror
When the job reaches 'ready' state I'll create bitmaps in the destination/base image of the job for every bitmap in the images dropped/merged (we use blockdev-mirror in full-sync mode) by the blockjob. This will capture the writes that happen after 'job-complete'.
The job will then be completed and the 2.2. will be executed as well.
So, the aim is not to miss any new writes after switching to new bs, but do not capture into bitmaps writes which are copying the whole disk during mirror.
Quoted 3 lines
2.2) non-active commit and also continuation of active layer block-commit/blockdev-mirror
Quoted 7 lines
Quoted 5 lines
After the job is completed succesfully I'll create temporary non-persistent bitmaps for/in the images removed by the blockjob and merge them into the destination image's bitmaps depending on their original location in the backing chain so that the bitmap state still properly describes which blocks have changed.
I don't follow. How do you populate these new temporary bitmaps? They are empty after creation..
With the 'block-dirty-bitmap-populate' block job.
Quoted 15 lines
Quoted 6 lines
After that the original images willbe blockdev-del-eted. The above is partialy in use today and since the job is already completed also requires blockdev-reopen to successfuly write to the bitmaps.
----
Quoted 8 lines
While writing the above down I've actually realized that controling the destination of the bitmap might not be as useful as I've thought originally as in 2.2. step I might need the allocation bitmap merged into multiple bitmaps, so I'd either need a temporary bitmap anyways or would have to re-run the job multiple times which seems wasteful. I'm no longer fully persuaded that adding the 'merge' step to the dirty populate blockjob will be the best thing since sliced bread.
Quoted 1 line
What is 'merge' step?
In some previous replies to Kevin, we discussed that it might be worth optimizing 'block-dirty-bitmap-populate' to directly populate the bits in the target bitmap rather than after the job is complete, so efectively directly mering it. I probably described it wrong here.
Quoted 1 line
Do you mean that populating directly into target bitmap is not really needed?
I need the bitmap populated by 'block-dirty-bitmap-populate' to be merged into multiple bitmaps in the new semantics. If the job itself doesn't support that sematics, changing it to just to directly populate one bitmap doesn't seem to be worth it since I'll be using intermediate bitmaps anyways.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Kevin WolfQuoted 30 lines · Kevin Wolf
Am 06.06.2020 um 08:55 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 21 lines
Quoted 14 lines · Peter Krempa
Quoted 2 lines · Kevin Wolf
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 1 line · Kevin Wolf
[...]
Quoted 8 lines
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Quoted 5 lines
Quoted 4 lines
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
Quoted 7 lines
Hi! Sorry me missing the discussion for a long time.
About new job semantics: if you create temporary bitmaps anyway, I do think that we should allow to populate into target bitmap directly, without creating any internal temporary bitmaps. I suggested it when reviewing v1, John argued for more transaction-like semantics to look like other jobs. Still, we can support both modes if we want.
Quoted 3 lines
But don't all other jobs allow you to see intermediate states? Like, when you look at the target node in the middle of a mirror job, you'll see a half-updated target image.
That was my argument on v1 :)
Quoted 3 lines
If we have an actual use case for both modes, we can certainly support that, but do we have one?
No. We have only one: without extra temporary bitmap, let's implement it. We can always add transaction-like case later on demand.
Quoted 11 lines
Quoted 7 lines
Allowing to use one target for several populating job is an interesting idea. Current series does "
bdrv_dirty_bitmap_set_busy(target_bitmap, true);", which forbids it.. Hmm. If we just drop it, nothing prevents user just remove target bitmap during the job. So, we'll need something like collective-busy bitmap..I'm not sure for what the busy flag is used in detail, but if this is the problem, maybe it's possible to just replace it with a counter?
busy flag means that bitmap is already in-use by some process (for example backup, or exported through NBD, or being migrated). User can't change or use busy bitmaps for another jobs. So multiple jobs writing into one bitmaps should be an exclusion from this rule (we want to allow several similar block-jobs, but still don't want to allow migration or NBD-export in the same time, or bitmap removing). I think we can just hardcode this case, add a new flag, which says that bitmap is used as target of populating jobs and being busy still allowed as a target for another populating job.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Peter KrempaQuoted 67 lines · Peter Krempa
Quoted 40 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Peter Krempa
Quoted 2 lines · Kevin Wolf
Am 05.06.2020 um 11:58 hat Peter Krempa geschrieben:
Quoted 1 line · Kevin Wolf
[...]
Quoted 8 lines
Quoted 7 lines
The above was actually inspired by a very recent problem I have in my attempt to use the dirty bitmap populate job to refactor how libvirt handles bitmaps. I've just figured out that I need to shuffle around some stuff as I can't run the dirty-bitmap-populate job while an active layer commit is in synchronised phase and I wanted to do the merging at that point. That reminded me of a possible gotcha in having to sequence the blockjobs which certainly would be more painful.
Quoted 5 lines
Quoted 4 lines
It would probably be good to have not only an iotests case that tests the low-level functionality of the block job, but also one that resembles the way libvirt would actually use it in combination with other jobs.
Hi! Sorry me missing the discussion for a long time.
About new job semantics: if you create temporary bitmaps anyway, I do think that we should allow to populate into target bitmap directly, without creating any internal temporary bitmaps. I suggested it when reviewing v1, John argued for more transaction-like semantics to look like other jobs. Still, we can support both modes if we want.
Allowing to use one target for several populating job is an interesting idea. Current series does "
bdrv_dirty_bitmap_set_busy(target_bitmap, true);", which forbids it.. Hmm. If we just drop it, nothing prevents user just remove target bitmap during the job. So, we'll need something like collective-busy bitmap..Quoted 5 lines
I certainly can document the way we'll use it but that in turn depends on how the job behaves.
With the current state of the job I plan to use it in two scenarios:
Quoted 7 lines
Preface: I'm currently changing libvirt to use one active bitmap per checkpoint (checkpoint is name for the point in time we want to take backup from). This means that a layer of the backing chain can have multiple active bitmaps depending on how many checkpoints were created in the current top layer. (previously we've tried to optimize things by having just one bitmap active, but the semantics were getting too crazy to be maintainable long-term)
Quoted 3 lines
Hmm. I had a plan of creating "lazy" disabled bitmaps, to optimize scenario with one active bitmap, so that disabled bitmaps are not loaded into RAM on start, but only on demand. But how to do it with "many active bitmaps" scenario? I don't think that's a good idea.. Possibly, we can implement laziness by internally make only one active bitmap and merge it here and there when you request some active bitmap which we actually didn't load yet..
Could you describe, what is the exact problem with "several disabled - one active" scheme, and how is it solved by "several active"?
The 'several disabled one active' semantics _heavily_ depend on metadata which must be tracked outside of qemu and is more prone to break. If any of the intermediate bitmaps is broken or missing everything breaks.
Then there's the complexity of the code which handles merging of the bitmaps during block jobs. Jobs such as blockdev-mirror in full mode and block-commit squash together the data and we need to do something about the bitmaps for the backups to work properly afterwards.
Without considering overlays which were created without propagating bitmaps, the code was already getting hairy especially in the case of backups where we needed to stitch together bitmaps for all the bitmaps corresponding to the given point in time where the backup is taken from.
When we add overlays without any bitmaps into the mix the code for resolving which bitmaps to merge the code is becoming very unpleasant, hard to understand and maintain and that is the main point for the switch.
I don't want to add unnecessary complexity to the libvirt code which will make it more fragile or hard to understand and fix in the future.
Quoted 8 lines
Both points which I heard for now (performance, and backup granularity in case of non-default qcow2 block size) don't seem compelling enough to me to make my life of implementing the feature in libvirt so much harder.
Also users really can just remove the point in time they wish to backup from after a successful backup which will also remove the corresponding active bitmap.
That's all is reasonable enough..
Than, we really need to refactor the code around bitmap support. Currently we do keep all active bitmaps in RAM and update them in a loop on each write. But it's obvious, the we can keep only one (with smallest granularity of existing dirty bitmaps) to track guest writes.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Bitmaps are no longer propagated over to upper layers when creating snapshots as we can use block-dirty-bitmap-populate instead.
Unexpected turn. When all this topic only started, it was reasoned more like "if user forget to create bitmap at start, let's help him".. But now it becomes the common scenario. Hmm.
It's not only a "user forgot" thing, but more that a systemic change would be required.
Additionally until _very_ recently it wasn't possible to create bitmaps using qemu-img, which made it impossible to create overlays for inactive
Didn't you consider to use qemu started in stopped mode to do block operations in same manner as for running vm? What's wrong with it? Also, there is qemu-storage-daemon, which is developed as separated binary, where block-layer is compiled in together with QMP interface.
Quoted 9 lines
VMs. Arguably this has changed so we could require it. It still adds a moving part which can break if the user doesn't add the bitmap or requires yet another special case handling if we want to compensate for that.
As of such, in libvirt's tech-preview implementation that is present currently upstream, if you create a qcow2 overlay without adding the appropriate bitmaps, the backups simply won't work.
Quoted 5 lines
Quoted 1 line
What do you think of granularity? We in Virtuozzo use 1M cluster as a default for qcow2 images. But we use 64k granularity (default) for bitmaps, to have smaller incremental backups. So, this is advantage of creating bitmap over relaying on block-status capturing by block-dirty-bitmap-populate: you don't control dirtiness granularity. So, I think that bitmap propagation, or just creating new dirty bitmap to track dirtiness from start of new snapshot is better.
This is a valid argument. Backups in this scenario will be bigger. I still don't feel like the code needs to be made much more complex because of it though.
May be, there is a simple solution? an option for blockdev-snapshot-sync to create a bitmap in a new image (or if you create image by qemu-img, just create bitmap by qemu-img as well, using new functionality).
Isn't it simpler than to just use existing block-status-bitmap, than run a job?
Quoted 70 lines
Quoted 38 lines
Quoted 8 lines
1) backup
Prior to doing the backup I'm figuring out the final backup bitmap, this involves creating a temporary bitmap populated by the job for every layer of the backing chain above of the one which contains the bitmap we want to take a backup from and then merge all of them together as a base for the backup.
(just thinking out loud)
So, assume the sequence top -> middle -> base
If we have a backup, which was done when we were in base, than bitmap is stored in base. And is loaded, and is active, but don't changes really, as base is opened read-only.] We merge block-status information of top and middle together with this bitmap, and aggregate difference between last backup and current state.
Quoted 8 lines
2) blockjobs
Note: This is currently an outline how the things should be as I've hit the snag with attempting to run the population jobs during 'ready' state of a active-layer block-commit/blockdev-mirror job only an hour ago and I need to change a few things.
Quoted 8 lines
2.1) active layer block-commit/blockdev-mirror
When the job reaches 'ready' state I'll create bitmaps in the destination/base image of the job for every bitmap in the images dropped/merged (we use blockdev-mirror in full-sync mode) by the blockjob. This will capture the writes that happen after 'job-complete'.
The job will then be completed and the 2.2. will be executed as well.
So, the aim is not to miss any new writes after switching to new bs, but do not capture into bitmaps writes which are copying the whole disk during mirror.
Quoted 3 lines
2.2) non-active commit and also continuation of active layer block-commit/blockdev-mirror
Quoted 7 lines
Quoted 5 lines
After the job is completed succesfully I'll create temporary non-persistent bitmaps for/in the images removed by the blockjob and merge them into the destination image's bitmaps depending on their original location in the backing chain so that the bitmap state still properly describes which blocks have changed.
I don't follow. How do you populate these new temporary bitmaps? They are empty after creation..
With the 'block-dirty-bitmap-populate' block job.
Quoted 15 lines
Quoted 6 lines
After that the original images willbe blockdev-del-eted. The above is partialy in use today and since the job is already completed also requires blockdev-reopen to successfuly write to the bitmaps.
----
Quoted 8 lines
While writing the above down I've actually realized that controling the destination of the bitmap might not be as useful as I've thought originally as in 2.2. step I might need the allocation bitmap merged into multiple bitmaps, so I'd either need a temporary bitmap anyways or would have to re-run the job multiple times which seems wasteful. I'm no longer fully persuaded that adding the 'merge' step to the dirty populate blockjob will be the best thing since sliced bread.
Quoted 1 line
What is 'merge' step?
In some previous replies to Kevin, we discussed that it might be worth optimizing 'block-dirty-bitmap-populate' to directly populate the bits in the target bitmap rather than after the job is complete, so efectively directly mering it. I probably described it wrong here.
Quoted 8 lines
Quoted 1 line
Do you mean that populating directly into target bitmap is not really needed?
I need the bitmap populated by 'block-dirty-bitmap-populate' to be merged into multiple bitmaps in the new semantics. If the job itself doesn't support that sematics, changing it to just to directly populate one bitmap doesn't seem to be worth it since I'll be using intermediate bitmaps anyways.
Hmm, if the main use case of populating job is to merge changes since snapshot to several bitmaps (all active bitmaps?), than I think it's correct to implement exactly this semantics, allowing a list of targets as well as list of source bitmaps. We even can reuse same structure for target-list which we use for source-list. And it's simple to implement in Qemu.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 3 lines · Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Peter Krempa
Quoted 1 line · Vladimir Sementsov-Ogievskiy
05.06.2020 13:59, Peter Krempa wrote:
[...]
Quoted 11 lines
Quoted 6 lines
It's not only a "user forgot" thing, but more that a systemic change would be required.
Additionally until _very_ recently it wasn't possible to create bitmaps using qemu-img, which made it impossible to create overlays for inactive
Didn't you consider to use qemu started in stopped mode to do block operations in same manner as for running vm? What's wrong with it? Also, there is qemu-storage-daemon, which is developed as separated binary, where block-layer is compiled in together with QMP interface.
It's still considered, but I didn't have time to implement anything related to it and nobody else implemented it either.
Additionally the problem isn't in libvirt's handling but more in other apps handling. We still due to historical reasons support if users create overlays outside of libvirt.
This would mean that either backups break if the overlay is not done properly or we have to have a fallback to use 'block-dirty-bitmap-populate'. At this point for both my sanity and actually finishing all the details in regards to incremental backup
Quoted 9 lines
Quoted 9 lines
VMs. Arguably this has changed so we could require it. It still adds a moving part which can break if the user doesn't add the bitmap or requires yet another special case handling if we want to compensate for that.
As of such, in libvirt's tech-preview implementation that is present currently upstream, if you create a qcow2 overlay without adding the appropriate bitmaps, the backups simply won't work.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
What do you think of granularity? We in Virtuozzo use 1M cluster as a default for qcow2 images. But we use 64k granularity (default) for bitmaps, to have smaller incremental backups. So, this is advantage of creating bitmap over relaying on block-status capturing by block-dirty-bitmap-populate: you don't control dirtiness granularity. So, I think that bitmap propagation, or just creating new dirty bitmap to track dirtiness from start of new snapshot is better.
This is a valid argument. Backups in this scenario will be bigger. I still don't feel like the code needs to be made much more complex because of it though.
May be, there is a simple solution? an option for blockdev-snapshot-sync to create a bitmap in a new image (or if you create image by qemu-img, just create bitmap by qemu-img as well, using new functionality).
You mean like we do now?:
https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c#L15020
That is slated to be deleted with my patchset though.
Good thing is that we can theoretically re-add it later.
For now I'd go with the simpler option that has fewer side effects.
Quoted 1 line
Isn't it simpler than to just use existing block-status-bitmap, than run a job?
No. Because we'd still need to support cases where user added an overlay without the appropriate bitmap. As said I'd prefer to keep the code simple and then work on optimizations. Good thing is that with 'one active bitmap per point in time' this is simpler to achieve.
[...]
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
What is 'merge' step?
In some previous replies to Kevin, we discussed that it might be worth optimizing 'block-dirty-bitmap-populate' to directly populate the bits in the target bitmap rather than after the job is complete, so efectively directly mering it. I probably described it wrong here.
Quoted 1 line
Do you mean that populating directly into target bitmap is not really needed?
Quoted 8 lines
Quoted 6 lines
I need the bitmap populated by 'block-dirty-bitmap-populate' to be merged into multiple bitmaps in the new semantics. If the job itself doesn't support that sematics, changing it to just to directly populate one bitmap doesn't seem to be worth it since I'll be using intermediate bitmaps anyways.
Hmm, if the main use case of populating job is to merge changes since snapshot to several bitmaps (all active bitmaps?), than I think it's correct to implement exactly this semantics, allowing a list of targets as well as list of source bitmaps. We even can reuse same structure for target-list which we use for source-list. And it's simple to implement in Qemu.
I'd mainly like for the design to converge. I actually have almost complete patches which use the current semantics. I'm okay with reworking it but at some point there should be a line when it won't change.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Vladimir Sementsov-OgievskiyAm 08.06.2020 um 12:00 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 9 lines
Quoted 8 lines · Kevin Wolf
Am 06.06.2020 um 08:55 hat Vladimir Sementsov-Ogievskiy geschrieben:
Quoted 6 lines
Allowing to use one target for several populating job is an interesting idea. Current series does "
bdrv_dirty_bitmap_set_busy(target_bitmap, true);", which forbids it.. Hmm. If we just drop it, nothing prevents user just remove target bitmap during the job. So, we'll need something like collective-busy bitmap..
Quoted 6 lines
Quoted 2 lines
I'm not sure for what the busy flag is used in detail, but if this is the problem, maybe it's possible to just replace it with a counter?
busy flag means that bitmap is already in-use by some process (for example backup, or exported through NBD, or being migrated). User can't change or use busy bitmaps for another jobs.
I think this rule is overly restrictive. That you can't delete a bitmap that is in use is obvious. That you can't have more than one "consumer" of dirty bits that clears bits after processing them seems at least reasonable. But why shouldn't you be able to have more than one "producer" of dirty bits?
Quoted 6 lines
So multiple jobs writing into one bitmaps should be an exclusion from this rule (we want to allow several similar block-jobs, but still don't want to allow migration or NBD-export in the same time, or bitmap removing). I think we can just hardcode this case, add a new flag, which says that bitmap is used as target of populating jobs and being busy still allowed as a target for another populating job.
I think we need to find a more general pattern. Maybe the distinction I made above between producer (setting dirty bits) and consumer (clearing dirty bits) is what we need, or maybe we need something else to distinguish roles. But I don't think the definition should have any reference to bitmap-populate.
Kevin
Quoted 14 lines · Peter Krempa
Quoted 5 lines · John Snow
Hi,
This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.
Quoted 7 lines
This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.
This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.
Quoted 7 lines
I've dusted-off my patches for using this blockjob for this very specific case and it works for me.
Tested-by Peter Krempa <[email protected]>For now I'll continue the integration with other blockjobs where we merge bitmaps.
I've posted the libvirt patches which make use of this blockjob as RFC on the libvir-list:
https://www.redhat.com/archives/libvir-list/2020-June/msg00292.html
I also have a branch with the patchset rebased to master (except for one of the test commits IIRC) linked from the cover-letter.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Kevin WolfQuoted 7 lines · Kevin Wolf
Am 14.05.2020 um 05:49 hat John Snow geschrieben:
Quoted 6 lines
This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.
It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.
Other patterns that might make sense someday:
all-ones (we already have bitmap-clear for all zeroes) an inverse flag (set bits for all unallocated portions) compressed (set bits for portions that are compressed)
but yeah, I don't see it worth implementing any of them without a client.
Quoted 9 lines
Quoted 7 lines
Signed-off-by John Snow <[email protected]>--- qapi/block-core.json | 48 +++++++++ qapi/job.json | 2 +- include/block/block_int.h | 21 ++++ block/bitmap-alloc.c | 207 ++++++++++++++++++++++++++++++++++++++
bitmap-populate.cto be more consistent with the actual job name?
Seems reasonable to me.
Quoted 8 lines
Quoted 5 lines
blockjob.c | 3 +- block/Makefile.objs | 1 + 6 files changed, 280 insertions(+), 2 deletions(-)create mode 100644
block/bitmap-alloc.c[...]
Quoted 8 lines
block/bitmap-alloc.c line 160
Quoted 5 lines
block/bitmap-alloc.c new file +4
-0+BlockJob *bitpop_job_create ( + if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { +return NULL ; + }What does this protect? And why does
BACKUP_SOURCEdescribe acccurately what this job does?
I'm less certain what the BLOCK_OP_TYPE_* constants are supposed to block, or if this is just copy/paste from backup.c. Does BlockOpType in block.h need a new entry?
Quoted 17 lines
Quoted 4 lines
block/bitmap-alloc.c new file +3
-0+if (bdrv_dirty_bitmap_check (target_bitmap,BDRV_BITMAP_DEFAULT , errp)) { +return NULL ; + }block/bitmap-alloc.c line 174
Quoted 10 lines
block/bitmap-alloc.c new file +10
-0+ +if (pattern !=BITMAP_PATTERN_ALLOCATION_TOP ) { +error_setg (errp,"Unrecognized bitmap pattern" ); +return NULL ; + } + + len =bdrv_getlength (bs); +if (len <0 ) { +error_setg_errno (errp, -len,"unable to get length for '%s'" , +bdrv_get_device_name (bs));This operates on the node level, so
bdrv_get_device_or_node_name()is necessary to avoid empty strings in the message.
Easy to fix.
Quoted 42 lines
Quoted 6 lines
block/bitmap-alloc.c new file +5
-0+return NULL ; + } + +/* NB: new bitmap is anonymous and enabled */ + cluster_size =bdrv_dirty_bitmap_granularity (target_bitmap);block/bitmap-alloc.c line 189
Quoted 10 lines
block/bitmap-alloc.c new file +10
-0+ new_bitmap =bdrv_create_dirty_bitmap (bs, cluster_size,NULL , errp); +if (!new_bitmap) { +return NULL ; + } + +/* Take ownership; we reserve the right to write into this on-commit. */ +bdrv_dirty_bitmap_set_busy (target_bitmap, true); + + job = block_job_create(job_id, &bitpop_job_driver, txn, bs, + BLK_PERM_CONSISTENT_READ,I don't think we actually rely on
CONSISTENT_READ, but then, using the job on inconsistent nodes probably makes little sense and we can always relax the restriction later if necessary.Quoted 8 lines
block/bitmap-alloc.c new file +8
-0+BLK_PERM_ALL & ~BLK_PERM_RESIZE , +0 , creation_flags, + cb, opaque, errp); +if (!job) { +bdrv_dirty_bitmap_set_busy (target_bitmap, false); +bdrv_release_dirty_bitmap (new_bitmap); +return NULL ; + }block/bitmap-alloc.c line 207
Quoted 10 lines
block/bitmap-alloc.c new file +10
-0+ + job->bs = bs; + job->on_error = on_error; + job->target_bitmap = target_bitmap; + job->new_bitmap = new_bitmap; + job->len = len; +job_progress_set_remaining (&job->common.job, job->len); + +return &job->common; +}Kevin
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Quoted 3 lines · Eric Blake
I'm less certain what the BLOCK_OP_TYPE_* constants are supposed to block, or if this is just copy/paste from
backup.c. Does BlockOpType inblock.hneed a new entry?
Copy paste. I wasn't sure myself.
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
In reply to Eric BlakeQuoted 10 lines · Eric Blake
Quoted 8 lines
Quoted 5 lines
+BlockJob *bitpop_job_create(
+ if (
bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { + return NULL;
+ }What does this protect? And why does
BACKUP_SOURCEdescribe acccurately what this job does?I'm less certain what the BLOCK_OP_TYPE_* constants are supposed to block, or if this is just copy/paste from
backup.c. Does BlockOpType inblock.hneed a new entry?
As it is, our code base has slowly moved away from op_blockers. We no longer have any explicit bdrv_op_block() except when blocking everything, then immediately followed up with unblocking a mere subset of all of the defined op types:
block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_COMMIT_TARGET, block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_STREAM, block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_BACKUP_SOURCE, block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_BACKUP_TARGET, block/replication.c: bdrv_op_unblock(top_bs, BLOCK_OP_TYPE_DATAPLANE, s->blocker); blockjob.c: bdrv_op_unblock(bs, BLOCK_OP_TYPE_DATAPLANE, job->blocker);
Are we at the point where we can ditch op_blockers altogether in favor of the block permissions system?
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [PATCH RFC v2 1/5] block: add bitmap-populate job
Am 16.06.2020 um 22:02 hat Eric Blake geschrieben:
Quoted 8 lines · Eric Blake
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
+BlockJob *bitpop_job_create(
+Â Â Â if (
bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { +Â Â Â Â Â Â Â return NULL;
+Â Â Â }
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
What does this protect? And why does
BACKUP_SOURCEdescribe acccurately what this job does?I'm less certain what the BLOCK_OP_TYPE_* constants are supposed to block, or if this is just copy/paste from
backup.c. Does BlockOpType inblock.hneed a new entry?As it is, our code base has slowly moved away from
op_blockers.
Yes, this is true. We're now trying to express conflicts with the permission system instead.
Quoted 3 lines
We no longer have any explicit
bdrv_op_block()except when blocking everything, then immediately followed up with unblocking a mere subset of all of the defined op types:
I believe we never had any other pattern because nobody ever could be bothered to think about specific conflicts. If we had one example of conflicting uses, we just blocked everything and only unblocked cases when we needed them (usually not really knowing whether they were really always safe). So op blocker are almost always overblocking, but sometimes too permissive, too.
This is why I was asking what we actually protect against here.
Quoted 11 lines
block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_COMMIT_TARGET, block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_STREAM, block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_BACKUP_SOURCE, block.c: bdrv_op_unblock(backing_hd, BLOCK_OP_TYPE_BACKUP_TARGET, block/replication.c: bdrv_op_unblock(top_bs, BLOCK_OP_TYPE_DATAPLANE,
s->blocker);blockjob.c:bdrv_op_unblock(bs, BLOCK_OP_TYPE_DATAPLANE, job->blocker);Are we at the point where we can ditch
op_blockersaltogether in favor of the block permissions system?
I actually started some patches to remove op blockers the other day. The tricky part is proving that each type is really unnecessary now. I only did this for RESIZE and DATAPLANE, which seemed relatively easy to prove, though I guess I could send at least these patches.
Kevin