block: add block-dirty-bitmap-populate job
Subsystems (2)
v4 cover block: add block-dirty-bitmap-populate job
This is NOT the final version of this patch series, but I'm posting it to revive conversation on the topic while fixing it to compile on top of meson changes.
v3 was: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06258.html
001/5:[0025] [FC] 'block: add bitmap-populate job'
002/5:[----] [--] 'blockdev: combine DriveBackupState and BlockdevBackupState' 003/5:[0004] [FC] 'qmp: expose block-dirty-bitmap-populate' 004/5:[----] [--] 'iotests: move bitmap helpers into their own file' 005/5:[----] [-C] 'iotests: add 298 for block-dirty-bitmap-populate'
I'm still trying to find the right QAPI contract (affects patch 1 and 3 for the qapi, and 5 for invoking the command in iotests), but right now, I'm leaning towards:
{ "execute": "block-dirty-bitmap-populate", "arguments": {
"job-id": "job0", "node": "target_node", "name": "target_bitmap",
"sources": [ { "pattern": "allocation", "node": "from_node" } ] } }which allows expansion into multiple sources, so that we can combine a populate action with a bitmap merge rather than having to do those as separate commands, like:
{ "execute": "block-dirty-bitmap-populate", "arguments": {
"job-id": "job0", "node": "target_node", "name": "target_bitmap",
"sources": [ { "pattern": "allocation", "node": "from_node" },
{ "pattern": "bitmap", "node": "from_node",
"bitmap": "from_bitmap" } ] } }John Snow (5):
block: add bitmap-populate job
blockdev: combine DriveBackupState and BlockdevBackupState
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add 298 for block-dirty-bitmap-populate
qapi/block-core.json | 66 + qapi/job.json | 6 +- qapi/transaction.json | 2 + include/block/block.h | 1 + include/block/block_int.h | 21 + block/bitmap-populate.c | 207 ++ blockdev.c | 104 +- blockjob.c | 3 +- MAINTAINERS | 1 + block/meson.build | 1 + tests/qemu-iotests/257 | 110 +- tests/qemu-iotests/298 | 232 ++ tests/qemu-iotests/298.out | 4544 +++++++++++++++++++++++++++++++++ tests/qemu-iotests/bitmaps.py | 131 + tests/qemu-iotests/group | 1 + 15 files changed, 5301 insertions(+), 129 deletions(-) create mode 100644 block/bitmap-populate.c create mode 100755 tests/qemu-iotests/298 create mode 100644 tests/qemu-iotests/298.out create mode 100644 tests/qemu-iotests/bitmaps.py
Signature · 1 line
2.28.0
PATCH v4 1/5 block: add bitmap-populate job
From: John Snow <[email protected]>
This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.
It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.
--- qapi/block-core.json | 48 +++++++++ qapi/job.json | 6 +- include/block/block.h | 1 + include/block/block_int.h | 21 ++++ block/bitmap-populate.c | 207 ++++++++++++++++++++++++++++++++++++++ blockjob.c | 3 +- MAINTAINERS | 1 + block/meson.build | 1 + 8 files changed, 286 insertions(+), 2 deletions(-) create mode 100644 block/bitmap-populate.c
qapi/block-core.json +48 -0
@@ -2216,6 +2216,54 @@ { 'command': 'block-dirty-bitmap-merge', 'data': 'BlockDirtyBitmapMerge' } +## +# @BitmapPattern: +# +# An enumeration of possible patterns that can be written into a bitmap. +# +# @allocation-top: The allocation status of the top layer +# of the attached storage node. +# +# Since: 5.2 +## +{ 'enum': 'BitmapPattern', + 'data': ['allocation-top'] } + +## +# @BlockDirtyBitmapPopulate: +# +# @job-id: identifier for the newly-created block job. +# +# @pattern: What pattern should be written into the bitmap? +# +# @on-error: the action to take if an error is encountered on a bitmap's +# attached node, default 'report'. +# 'stop' and 'enospc' can only be used if the block device supports +# io-status (see BlockInfo). +# +# @auto-finalize: When false, this job will wait in a PENDING state after it +# has finished its work, waiting for @block-job-finalize +# before making any block graph changes. +# When true, this job will automatically +# perform its abort or commit actions. +# Defaults to true. +# +# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it +# has completely ceased all work, and awaits @block-job-dismiss. +# When true, this job will automatically disappear from the +# query list without user intervention. +# Defaults to true. +# +# Since: 5.2 +## +{ 'struct': 'BlockDirtyBitmapPopulate', + 'base': 'BlockDirtyBitmap', + 'data': { 'job-id': 'str', + 'pattern': 'BitmapPattern', + '*on-error': 'BlockdevOnError', + '*auto-finalize': 'bool', + '*auto-dismiss': 'bool' } } + ## # @BlockDirtyBitmapSha256: #
qapi/job.json +5 -1
@@ -22,10 +22,14 @@ # # @amend: image options amend job type, see "x-blockdev-amend" (since 5.1) # +# @bitmap-populate: drive bitmap population job type, see +# "block-dirty-bitmap-populate" (since 5.2) +# # Since: 1.7 ## { 'enum': 'JobType', - 'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend'] } + 'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend', + 'bitmap-populate'] } ## # @JobStatus:
include/block/block.h +1 -0
@@ -200,6 +200,7 @@ typedef struct BDRVReopenState { typedef enum BlockOpType { BLOCK_OP_TYPE_BACKUP_SOURCE, BLOCK_OP_TYPE_BACKUP_TARGET, + BLOCK_OP_TYPE_BITMAP_POPULATE, BLOCK_OP_TYPE_CHANGE, BLOCK_OP_TYPE_COMMIT_SOURCE, BLOCK_OP_TYPE_COMMIT_TARGET,
include/block/block_int.h +21 -0
@@ -1257,6 +1257,27 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, BlockCompletionFunc *cb, void *opaque, JobTxn *txn, Error **errp); +/* +* bitpop_job_create: Create a new bitmap population job. +* +* @job_id: The id of the newly-created job. +* @bs: Block device associated with the @target_bitmap. +* @target_bitmap: The bitmap to populate. +* @on_error: What to do if an error on @bs is encountered. +* @creation_flags: Flags that control the behavior of the Job lifetime. +* See @BlockJobCreateFlags +* @cb: Completion function for the job. +* @opaque: Opaque pointer value passed to @cb. +* @txn: Transaction that this job is part of (may be NULL). +*/ +BlockJob *bitpop_job_create (const char *job_id,BlockDriverState *bs, +BdrvDirtyBitmap *target_bitmap, +BitmapPattern pattern, +BlockdevOnError on_error, +int creation_flags, +BlockCompletionFunc *cb,void *opaque, +JobTxn *txn,Error **errp); + BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs, const char *child_name, const BdrvChildClass *child_class,
block/bitmap-populate.c new file +207 -0
207 lines not on this page · raw message
blockjob.c +2 -1
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)return job_type (job) ==JOB_TYPE_BACKUP ||job_type (job) ==JOB_TYPE_COMMIT ||job_type (job) ==JOB_TYPE_MIRROR || -job_type (job) ==JOB_TYPE_STREAM ; +job_type (job) ==JOB_TYPE_STREAM || +job_type (job) ==JOB_TYPE_BITMAP_POPULATE ; }BlockJob *block_job_next (BlockJob *bjob)
MAINTAINERS +1 -0
@@ -2156,6 +2156,7 @@ S: Supported F: include/qemu/hbitmap.h F: include/block/dirty-bitmap.h F: block/monitor/bitmap-qmp-cmds.c +F: block/bitmap-populate.c F: block/dirty-bitmap.c F: block/qcow2-bitmap.c F: migration/block-dirty-bitmap.c
block/meson.build +1 -0
@@ -5,6 +5,7 @@ block_ss.add(files( 'amend.c', 'backup.c', 'backup-top.c', + 'bitmap-populate.c', 'blkdebug.c', 'blklogwrites.c', 'blkreplay.c',
Signature · 1 line
2.28.0
Re: [PATCH v4 1/5] block: add bitmap-populate job
Quoted 28 lines · Eric Blake
From: John Snow <[email protected]>
This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.
It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.
Signed-off-by John Snow <[email protected]>Signed-off-by Eric Blake <[email protected]>--- qapi/block-core.json | 48 +++++++++ qapi/job.json | 6 +- include/block/block.h | 1 + include/block/block_int.h | 21 ++++ block/bitmap-populate.c | 207 ++++++++++++++++++++++++++++++++++++++ blockjob.c | 3 +- MAINTAINERS | 1 + block/meson.build | 1 + 8 files changed, 286 insertions(+), 2 deletions(-) create mode 100644 block/bitmap-populate.cqapi/block-core.json +0
-0
qapi/block-core.json line 2225
Quoted 10 lines
qapi/block-core.json +7
-0@@ -2216,6 +2216,54 @@ { 'command': 'block-dirty-bitmap-merge', 'data': 'BlockDirtyBitmapMerge' } +## +# @BitmapPattern: +# +# An enumeration of possible patterns that can be written into a bitmap. +# +# @allocation-top: The allocation status of the top layer +# of the attached storage node.
Greek to me.
Quoted 49 lines
qapi/block-core.json +41
-0+# +# Since: 5.2 +## +{ 'enum': 'BitmapPattern', + 'data': ['allocation-top'] } + +## +# @BlockDirtyBitmapPopulate: +# +# @job-id: identifier for the newly-created block job. +# +# @pattern: What pattern should be written into the bitmap? +# +# @on-error: the action to take if an error is encountered on a bitmap's +# attached node, default 'report'. +# 'stop' and 'enospc' can only be used if the block device supports +# io-status (see BlockInfo). +# +# @auto-finalize: When false, this job will wait in a PENDING state after it +# has finished its work, waiting for @block-job-finalize +# before making any block graph changes. +# When true, this job will automatically +# perform its abort or commit actions. +# Defaults to true. +# +# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it +# has completely ceased all work, and awaits @block-job-dismiss. +# When true, this job will automatically disappear from the +# query list without user intervention. +# Defaults to true. +# +# Since: 5.2 +## +{ 'struct': 'BlockDirtyBitmapPopulate', + 'base': 'BlockDirtyBitmap', + 'data': { 'job-id': 'str', + 'pattern': 'BitmapPattern', + '*on-error': 'BlockdevOnError', + '*auto-finalize': 'bool', + '*auto-dismiss': 'bool' } } + ## # @BlockDirtyBitmapSha256: #qapi/job.json +0
-0
qapi/job.json line 26
Quoted 5 lines
qapi/job.json +2
-0@@ -22,10 +22,14 @@ # # @amend: image options amend job type, see "x-blockdev-amend" (since 5.1) # +# @bitmap-populate: drive bitmap population job type, see +# "block-dirty-bitmap-populate" (since 5.2)
Dangling reference, healed in PATCH 3. Job appearing before the command it wraps is unusual.
What's a "drive bitmap population job"?
The reference provides a clue:
##
# @block-dirty-bitmap-populate:
#
# Creates a new job that writes a pattern into a dirty bitmap.Do you mean "dirty bitmap population job"?
By the way, I think the doc comment would be easier to read with s/job type/job/.
Quoted 10 lines
qapi/job.json +3
-1+# # Since: 1.7 ## { 'enum': 'JobType', - 'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend'] } + 'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend', + 'bitmap-populate'] } ## # @JobStatus:
[...]
PATCH v4 2/5 blockdev: combine DriveBackupState and BlockdevBackupState
In reply to Eric BlakeFrom: John Snow <[email protected]>
They have the same fields -- rename it BlockJobActionState.
--- blockdev.c | 30 ++++++++++++------------------ 1 file changed, 12 insertions(+), 18 deletions(-)
blockdev.c +12 -18
@@ -1668,11 +1668,11 @@ static void external_snapshot_clean(BlkActionState *common)aio_context_release (aio_context); } -typedef struct DriveBackupState { +typedef struct BlockJobActionState {BlkActionState common;BlockDriverState *bs;BlockJob *job; -}DriveBackupState ; +}BlockJobActionState ; static BlockJob *do_backup_common(BackupCommon *backup, BlockDriverState *bs,
@@ -1682,7 +1682,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,static void drive_backup_prepare (BlkActionState *common,Error **errp) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);DriveBackup *backup;BlockDriverState *bs;BlockDriverState *target_bs;
@@ -1819,7 +1819,7 @@ out:static void drive_backup_commit (BlkActionState *common) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context; aio_context =bdrv_get_aio_context (state->bs);
@@ -1833,7 +1833,7 @@ static void drive_backup_commit(BlkActionState *common)static void drive_backup_abort (BlkActionState *common) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);if (state->job) {AioContext *aio_context;
@@ -1849,7 +1849,7 @@ static void drive_backup_abort(BlkActionState *common)static void drive_backup_clean (BlkActionState *common) { -DriveBackupState *state =DO_UPCAST (DriveBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context;if (!state->bs) {
@@ -1864,15 +1864,9 @@ static void drive_backup_clean(BlkActionState *common)aio_context_release (aio_context); } -typedef struct BlockdevBackupState { -BlkActionState common; -BlockDriverState *bs; -BlockJob *job; -}BlockdevBackupState ; -static void blockdev_backup_prepare (BlkActionState *common,Error **errp) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);BlockdevBackup *backup;BlockDriverState *bs;BlockDriverState *target_bs;
@@ -1920,7 +1914,7 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)static void blockdev_backup_commit (BlkActionState *common) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context; aio_context =bdrv_get_aio_context (state->bs);
@@ -1934,7 +1928,7 @@ static void blockdev_backup_commit(BlkActionState *common)static void blockdev_backup_abort (BlkActionState *common) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);if (state->job) {AioContext *aio_context;
@@ -1950,7 +1944,7 @@ static void blockdev_backup_abort(BlkActionState *common)static void blockdev_backup_clean (BlkActionState *common) { -BlockdevBackupState *state =DO_UPCAST (BlockdevBackupState, common, common); +BlockJobActionState *state =DO_UPCAST (BlockJobActionState, common, common);AioContext *aio_context;if (!state->bs) {
@@ -2222,14 +2216,14 @@ static const BlkActionOps actions[] = { .clean = external_snapshot_clean, }, [TRANSACTION_ACTION_KIND_DRIVE_BACKUP] = { - .instance_size = sizeof(DriveBackupState), + .instance_size = sizeof(BlockJobActionState), .prepare = drive_backup_prepare, .commit = drive_backup_commit, .abort = drive_backup_abort, .clean = drive_backup_clean, }, [TRANSACTION_ACTION_KIND_BLOCKDEV_BACKUP] = { - .instance_size = sizeof(BlockdevBackupState), + .instance_size = sizeof(BlockJobActionState), .prepare = blockdev_backup_prepare, .commit = blockdev_backup_commit, .abort = blockdev_backup_abort,
Signature · 1 line
2.28.0
Re: [PATCH v4 2/5] blockdev: combine DriveBackupState and BlockdevBackupState
Quoted 3 lines · Eric Blake
From: John Snow<[email protected]>
They have the same fields -- rename it BlockJobActionState.
commit/abort/clean functions are identical after it. I think better to combine them as well here
(no changes from v3)
Signature · 2 lines
Best regards, Vladimir
PATCH v4 3/5 qmp: expose block-dirty-bitmap-populate
In reply to Eric BlakeFrom: John Snow <[email protected]>
This is a new job-creating command.
--- qapi/block-core.json | 18 +++++++++++ qapi/transaction.json | 2 ++ blockdev.c | 74 +++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 94 insertions(+)
qapi/block-core.json +18 -0
@@ -2264,6 +2264,24 @@ '*auto-finalize': 'bool', '*auto-dismiss': 'bool' } } +## +# @block-dirty-bitmap-populate: +# +# Creates a new job that writes a pattern into a dirty bitmap. +# +# Since: 5.2 +# +# Example: +# +# -> { "execute": "block-dirty-bitmap-populate", +# "arguments": { "node": "drive0", "target": "bitmap0", +# "job-id": "job0", "pattern": "allocate-top" } } +# <- { "return": {} } +# +## +{ 'command': 'block-dirty-bitmap-populate', 'boxed': true, + 'data': 'BlockDirtyBitmapPopulate' } + ## # @BlockDirtyBitmapSha256: #
qapi/transaction.json +2 -0
@@ -51,6 +51,7 @@ # - @block-dirty-bitmap-enable: since 4.0 # - @block-dirty-bitmap-disable: since 4.0 # - @block-dirty-bitmap-merge: since 4.0 +# - @block-dirty-bitmap-populate: since 5.2 # - @blockdev-backup: since 2.3 # - @blockdev-snapshot: since 2.5 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -68,6 +69,7 @@ 'block-dirty-bitmap-enable': 'BlockDirtyBitmap', 'block-dirty-bitmap-disable': 'BlockDirtyBitmap', 'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge', + 'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate', 'blockdev-backup': 'BlockdevBackup', 'blockdev-snapshot': 'BlockdevSnapshot', 'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +74 -0
92 lines not on this page · raw message
Signature · 1 line
2.28.0
PATCH v4 4/5 iotests: move bitmap helpers into their own file
In reply to Eric BlakeFrom: John Snow <[email protected]>
Message-Id: <[email protected]>
--- tests/qemu-iotests/257 | 110 +--------------------------- tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++ 2 files changed, 132 insertions(+), 109 deletions(-) create mode 100644 tests/qemu-iotests/bitmaps.py
tests/qemu-iotests/257 +1 -109
@@ -24,120 +24,12 @@ import os import iotests from iotests import log, qemu_img +from bitmaps import EmulatedBitmap, GROUPS SIZE = 64 * 1024 * 1024 GRANULARITY = 64 * 1024 -class Pattern: - def __init__(self, byte, offset, size=GRANULARITY): - self.byte = byte - self.offset = offset - self.size = size - - def bits(self, granularity): - lower = self.offset // granularity - upper = (self.offset + self.size - 1) // granularity - return set(range(lower, upper + 1)) - - -class PatternGroup: - """Grouping of Pattern objects. Initialize with an iterable of Patterns.""" - def __init__(self, patterns): - self.patterns = patterns - - def bits(self, granularity): - """Calculate the unique bits dirtied by this pattern grouping""" - res = set() - for pattern in self.patterns: - res |= pattern.bits(granularity) - return res - - -GROUPS = [ - PatternGroup([ - # Batch 0: 4 clusters - Pattern('0x49', 0x0000000), - Pattern('0x6c', 0x0100000), # 1M - Pattern('0x6f', 0x2000000), # 32M - Pattern('0x76', 0x3ff0000)]), # 64M - 64K - PatternGroup([ - # Batch 1: 6 clusters (3 new) - Pattern('0x65', 0x0000000), # Full overwrite - Pattern('0x77', 0x00f8000), # Partial-left (1M-32K) - Pattern('0x72', 0x2008000), # Partial-right (32M+32K) - Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K) - PatternGroup([ - # Batch 2: 7 clusters (3 new) - Pattern('0x74', 0x0010000), # Adjacent-right - Pattern('0x69', 0x00e8000), # Partial-left (1M-96K) - Pattern('0x6e', 0x2018000), # Partial-right (32M+96K) - Pattern('0x67', 0x3fe0000, - 2*GRANULARITY)]), # Overwrite [(64M-128K)-64M) - PatternGroup([ - # Batch 3: 8 clusters (5 new) - # Carefully chosen such that nothing re-dirties the one cluster - # that copies out successfully before failure in Group #1. - Pattern('0xaa', 0x0010000, - 3*GRANULARITY), # Overwrite and 2x Adjacent-right - Pattern('0xbb', 0x00d8000), # Partial-left (1M-160K) - Pattern('0xcc', 0x2028000), # Partial-right (32M+160K) - Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right -] - - -class EmulatedBitmap: - def __init__(self, granularity=GRANULARITY): - self._bits = set() - self.granularity = granularity - - def dirty_bits(self, bits): - self._bits |= set(bits) - - def dirty_group(self, n): - self.dirty_bits(GROUPS[n].bits(self.granularity)) - - def clear(self): - self._bits = set() - - def clear_bits(self, bits): - self._bits -= set(bits) - - def clear_bit(self, bit): - self.clear_bits({bit}) - - def clear_group(self, n): - self.clear_bits(GROUPS[n].bits(self.granularity)) - - @property - def first_bit(self): - return sorted(self.bits)[0] - - @property - def bits(self): - return self._bits - - @property - def count(self): - return len(self.bits) - - def compare(self, qmp_bitmap): - """ - Print a nice human-readable message checking that a bitmap as reported - by the QMP interface has as many bits set as we expect it to. - """ - - name = qmp_bitmap.get('name', '(anonymous)') - log("= Checking Bitmap {:s} =".format(name)) - - want = self.count - have = qmp_bitmap['count'] // qmp_bitmap['granularity'] - - log("expecting {:d} dirty sectors; have {:d}. {:s}".format( - want, have, "OK!" if want == have else "ERROR!")) - log('') - - class Drive: """Represents, vaguely, a drive attached to a VM. Includes format, graph, and device information."""
tests/qemu-iotests/bitmaps.py new file +131 -0
131 lines not on this page · raw message
Signature · 1 line
2.28.0
Re: [PATCH v4 4/5] iotests: move bitmap helpers into their own file
Quoted 5 lines · Eric Blake
From: John Snow<[email protected]>
Signed-off-by John Snow<[email protected]>Message-Id:<[email protected]>
Signed-off-by Eric Blake<[email protected]>
You forget my:
Signature · 2 lines
Best regards, Vladimir
PATCH v4 5/5 iotests: add 298 for block-dirty-bitmap-populate
In reply to Eric BlakeFrom: John Snow <[email protected]>
Give block-dirty-bitmap-populate a workout.
--- tests/qemu-iotests/298 | 232 ++ tests/qemu-iotests/298.out | 4544 ++++++++++++++++++++++++++++++++++++ tests/qemu-iotests/group | 1 + 3 files changed, 4777 insertions(+) create mode 100755 tests/qemu-iotests/298 create mode 100644 tests/qemu-iotests/298.out
tests/qemu-iotests/298 new file +232 -0
@@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +# +# Test block-dirty-bitmap-populate +# +# Copyright (c) 2020 John Snow for Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# +# [email protected] + +from collections import namedtuple +import itertools +import math +import os + +import iotests +from iotests import log, qemu_img +from bitmaps import EmulatedBitmap, GROUPS + +SIZE = 64 * 1024 * 1024 +GRANULARITY = 64 * 1024 + + +class Drive: + def __init__(self, path, vm): + self.path = path + self.vm = vm + self.fmt = None + self.size = None + self.node = None + + def img_create(self, fmt, size): + self.fmt = fmt + self.size = size + iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size)) + + +def block_dirty_bitmap_populate(vm, node, bitmap, job_id, pattern, **kwargs): + # Strip any arguments explicitly nulled by the caller: + kwargs = {key: val for key, val in kwargs.items() if val is not None} + result = vm.qmp_log('block-dirty-bitmap-populate', + node=node, + name=bitmap, + job_id=job_id, + pattern=pattern, + **kwargs) + return result + + +def populate(drive, bitmap, job_id, pattern='allocation-top', **kwargs): + kwargs.setdefault('pattern', pattern) + kwargs.setdefault('auto-finalize', False) + kwargs.setdefault('auto-dismiss', False) + ret = block_dirty_bitmap_populate(drive.vm, drive.node, + bitmap, job_id, **kwargs) + return { + 'id': job_id, + 'auto-finalize': kwargs['auto-finalize'], + 'auto-dismiss': kwargs['auto-dismiss'], + 'return': ret, + } + + +def perform_writes(drive, n, filter_node_name=None): + log("-- Write #{:d}:".format(n)) + node_name = filter_node_name or drive.node + for pattern in GROUPS[n].patterns: + cmd = "write -P{:s} 0x{:07x} 0x{:x}".format( + pattern.byte, + pattern.offset, + pattern.size) + drive.vm.hmp_qemu_io(node_name, cmd, use_log=True) + log('') + + +TestConfig = namedtuple('TestConfig', [ + 'base_pattern', + 'disabled', + 'pre_writes', + 'mid_writes', + 'cancel', + 'post_writes', +]) + + +def test_bitmap_populate(config): + """ + Test bitmap populate. + + :param base_pattern: Write a base pattern? + :param disabled: Disable the target bitmap? + :param pre_writes: Write a pattern after bitmap creation, but before job? + :param mid_writes: Write a pattern before job finalizes? + :param cancel: Cancel the job instead of finalizing it? + :param post_writes: Write a pattern after the job? + """ + with iotests.FilePaths(['img']) as (img_path,), iotests.VM() as vm: + log("\n=== Bitmap Populate {:s} ===\n".format(str(config))) + + log('-- Prepare image & VM:') + drive0 = Drive(img_path, vm=vm) + drive0.img_create(iotests.imgfmt, SIZE) + vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device())) + vm.launch() + + file_config = { + 'driver': 'file', + 'filename': drive0.path + } + + drive0.node = 'drive0' + vm.qmp_log('blockdev-add', + filters=[iotests.filter_qmp_testfiles], + node_name=drive0.node, + driver=drive0.fmt, + file=file_config) + log('') + + + # Step 0: Prepare & Base Allocation Pattern + + if config.base_pattern: + perform_writes(drive0, 0) + + + # Step 1: Add test bitmap + + log('-- Add Bitmap:') + vm.qmp_log('block-dirty-bitmap-add', + node=drive0.node, + name="target", + granularity=GRANULARITY, + disabled=config.disabled) + ebitmap = EmulatedBitmap() + log('') + + # Step 2: Pre-Writes + + if config.pre_writes: + perform_writes(drive0, 1) + if not config.disabled: + ebitmap.dirty_group(1) + bitmap = vm.get_bitmap(drive0.node, 'target') + ebitmap.compare(bitmap) + + + # Step 3: Launch job & Mid-Writes + + log('-- Test block-dirty-bitmap-populate (bitpop0):') + def pre_finalize(): + # Writes issued prior to job finalization: + if config.mid_writes: + perform_writes(drive0, 2) + if not config.disabled: + ebitmap.dirty_group(2) + + job = populate(drive0, 'target', 'bitpop0') + assert job['return'] == {'return': {}} + vm.run_job(job['id'], auto_dismiss=job['auto-dismiss'], + auto_finalize=job['auto-finalize'], + pre_finalize=pre_finalize, + cancel=config.cancel) + log('') + + + # Step 4: Post-job verification + + if not config.cancel: + # Any writes made prior to the job finishing should now be visible. + if config.base_pattern: + ebitmap.dirty_group(0) + if config.pre_writes: + ebitmap.dirty_group(1) + if config.mid_writes: + ebitmap.dirty_group(2) + + bitmap = vm.get_bitmap(drive0.node, 'target') + ebitmap.compare(bitmap) + + + # Step 5: Post-Writes + + if config.post_writes: + perform_writes(drive0, 3) + if not config.disabled: + ebitmap.dirty_group(3) + + + # Step 6: Final Verification + + log('-- Verification:') + bitmaps = vm.query_bitmaps() + log({'bitmaps': bitmaps}, indent=2) + log('') + bitmap = vm.get_bitmap(drive0.node, 'target', bitmaps=bitmaps) + ebitmap.compare(bitmap) + + + log('-- Cleanup:') + vm.qmp_log("block-dirty-bitmap-remove", + node=drive0.node, name="target") + + bitmaps = vm.query_bitmaps() + if bitmaps: + log("ERROR: bitmaps unaccounted for:") + log(bitmaps) + else: + log('OK: All bitmaps removed') + vm.shutdown() + log('') + + +def main(): + for args in itertools.product((True, False), repeat=6): + cfg = TestConfig(*args) + test_bitmap_populate(cfg) + + +if __name__ == '__main__': + iotests.script_main(main, supported_fmts=['qcow2'], + supported_protocols=['file'])
tests/qemu-iotests/298.out new file +4544 -0
2,768 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.28.0
Part of the diff is not shown here. View the raw message.
Quoted 24 lines · Eric Blake
This is NOT the final version of this patch series, but I'm posting it to revive conversation on the topic while fixing it to compile on top of meson changes.
v3 was: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06258.html
001/5:[0025] [FC] 'block: add bitmap-populate job'
002/5:[----] [--] 'blockdev: combine DriveBackupState and BlockdevBackupState' 003/5:[0004] [FC] 'qmp: expose block-dirty-bitmap-populate' 004/5:[----] [--] 'iotests: move bitmap helpers into their own file' 005/5:[----] [-C] 'iotests: add 298 for block-dirty-bitmap-populate'I'm still trying to find the right QAPI contract (affects patch 1 and 3 for the qapi, and 5 for invoking the command in iotests), but right now, I'm leaning towards:
{ "execute": "block-dirty-bitmap-populate", "arguments": { "job-id": "job0", "node": "target_node", "name": "target_bitmap", "sources": [ { "pattern": "allocation", "node": "from_node" } ] } }which allows expansion into multiple sources, so that we can combine a populate action with a bitmap merge rather than having to do those as separate commands, like:
Quoted 5 lines
{ "execute": "block-dirty-bitmap-populate", "arguments": {
"job-id": "job0", "node": "target_node", "name": "target_bitmap",
"sources": [ { "pattern": "allocation", "node": "from_node" },
{ "pattern": "bitmap", "node": "from_node",
"bitmap": "from_bitmap" } ] } }(I see, this is not realized in v4)
What about multiple targets as well, as we discussed in v3?
Quoted 29 lines
John Snow (5):
block: add bitmap-populate job
blockdev: combine DriveBackupState and BlockdevBackupState
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add 298 for block-dirty-bitmap-populateqapi/block-core.json | 66 + qapi/job.json | 6 +- qapi/transaction.json | 2 + include/block/block.h | 1 + include/block/block_int.h | 21 + block/bitmap-populate.c | 207 ++ blockdev.c | 104 +- blockjob.c | 3 +- MAINTAINERS | 1 + block/meson.build | 1 + tests/qemu-iotests/257 | 110 +- tests/qemu-iotests/298 | 232 ++ tests/qemu-iotests/298.out | 4544 +++++++++++++++++++++++++++++++++ tests/qemu-iotests/bitmaps.py | 131 + tests/qemu-iotests/group | 1 + 15 files changed, 5301 insertions(+), 129 deletions(-)create mode 100644
block/bitmap-populate.ccreate mode 100755 tests/qemu-iotests/298 create mode 100644 tests/qemu-iotests/298.out create mode 100644tests/qemu-iotests/bitmaps.py
Signature · 2 lines
Best regards, Vladimir
Hi!
Any plans on v5? Now I have a request for this feature.. If no plans, I can take care of it and make a v5
Quoted 29 lines · Eric Blake
This is NOT the final version of this patch series, but I'm posting it to revive conversation on the topic while fixing it to compile on top of meson changes.
v3 was:https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06258.html
001/5:[0025] [FC] 'block: add bitmap-populate job'
002/5:[----] [--] 'blockdev: combine DriveBackupState and BlockdevBackupState' 003/5:[0004] [FC] 'qmp: expose block-dirty-bitmap-populate' 004/5:[----] [--] 'iotests: move bitmap helpers into their own file' 005/5:[----] [-C] 'iotests: add 298 for block-dirty-bitmap-populate'I'm still trying to find the right QAPI contract (affects patch 1 and 3 for the qapi, and 5 for invoking the command in iotests), but right now, I'm leaning towards:
{ "execute": "block-dirty-bitmap-populate", "arguments": { "job-id": "job0", "node": "target_node", "name": "target_bitmap", "sources": [ { "pattern": "allocation", "node": "from_node" } ] } }which allows expansion into multiple sources, so that we can combine a populate action with a bitmap merge rather than having to do those as separate commands, like:
{ "execute": "block-dirty-bitmap-populate", "arguments": { "job-id": "job0", "node": "target_node", "name": "target_bitmap", "sources": [ { "pattern": "allocation", "node": "from_node" }, { "pattern": "bitmap", "node": "from_node", "bitmap": "from_bitmap" } ] } }
Signature · 2 lines
Best regards, Vladimir