Skip to contentSkip to conversation

block: add block-dirty-bitmap-populate job

PATCH In review Eric Blake 3 participants 11 messages Sep 2, 2020 – Mar 19, 2021

Subsystems (2)
lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v4 cover block: add block-dirty-bitmap-populate job

This is NOT the final version of this patch series, but I'm posting it to revive conversation on the topic while fixing it to compile on top of meson changes.

v3 was: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06258.html

001/5:[0025] [FC] 'block: add bitmap-populate job'
002/5:[----] [--] 'blockdev: combine DriveBackupState and BlockdevBackupState' 003/5:[0004] [FC] 'qmp: expose block-dirty-bitmap-populate' 004/5:[----] [--] 'iotests: move bitmap helpers into their own file' 005/5:[----] [-C] 'iotests: add 298 for block-dirty-bitmap-populate'

I'm still trying to find the right QAPI contract (affects patch 1 and 3 for the qapi, and 5 for invoking the command in iotests), but right now, I'm leaning towards:

{ "execute": "block-dirty-bitmap-populate", "arguments": {
    "job-id": "job0", "node": "target_node", "name": "target_bitmap",
    "sources": [ { "pattern": "allocation", "node": "from_node" } ] } }

which allows expansion into multiple sources, so that we can combine a populate action with a bitmap merge rather than having to do those as separate commands, like:

{ "execute": "block-dirty-bitmap-populate", "arguments": {
    "job-id": "job0", "node": "target_node", "name": "target_bitmap",
    "sources": [ { "pattern": "allocation", "node": "from_node" },
                 { "pattern": "bitmap", "node": "from_node",
		     "bitmap": "from_bitmap" } ] } }

John Snow (5):
block: add bitmap-populate job
blockdev: combine DriveBackupState and BlockdevBackupState
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add 298 for block-dirty-bitmap-populate

 qapi/block-core.json          |   66 +
 qapi/job.json                 |    6 +-
 qapi/transaction.json         |    2 +
 include/block/block.h         |    1 +
 include/block/block_int.h     |   21 +
 block/bitmap-populate.c       |  207 ++
 blockdev.c                    |  104 +-
 blockjob.c                    |    3 +-
 MAINTAINERS                   |    1 +
 block/meson.build             |    1 +
 tests/qemu-iotests/257        |  110 +-
 tests/qemu-iotests/298        |  232 ++
 tests/qemu-iotests/298.out    | 4544 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/bitmaps.py |  131 +
 tests/qemu-iotests/group      |    1 +
 15 files changed, 5301 insertions(+), 129 deletions(-)
 create mode 100644 block/bitmap-populate.c
 create mode 100755 tests/qemu-iotests/298
 create mode 100644 tests/qemu-iotests/298.out
 create mode 100644 tests/qemu-iotests/bitmaps.py
Signature · 1 line
2.28.0
author
#1 lore raw

PATCH v4 1/5 block: add bitmap-populate job

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
 qapi/block-core.json      |  48 +++++++++
 qapi/job.json             |   6 +-
 include/block/block.h     |   1 +
 include/block/block_int.h |  21 ++++
 block/bitmap-populate.c   | 207 ++++++++++++++++++++++++++++++++++++++
 blockjob.c                |   3 +-
 MAINTAINERS               |   1 +
 block/meson.build         |   1 +
 8 files changed, 286 insertions(+), 2 deletions(-)
 create mode 100644 block/bitmap-populate.c
qapi/block-core.json +48 -0
index db08c58d788c..1cac9a9a8207 100644
@@ -2216,6 +2216,54 @@
 { 'command': 'block-dirty-bitmap-merge',
   'data': 'BlockDirtyBitmapMerge' }
 
+##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.2
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device supports
+#            io-status (see BlockInfo).
+#
+# @auto-finalize: When false, this job will wait in a PENDING state after it
+#                 has finished its work, waiting for @block-job-finalize
+#                 before making any block graph changes.
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the
+#                query list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.2
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/job.json +5 -1
index 280c2f76f136..fb0b606e868d 100644
@@ -22,10 +22,14 @@
 #
 # @amend: image options amend job type, see "x-blockdev-amend" (since 5.1)
 #
+# @bitmap-populate: drive bitmap population job type, see
+#                   "block-dirty-bitmap-populate" (since 5.2)
+#
 # Since: 1.7
 ##
 { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend',
+           'bitmap-populate'] }
 
 ##
 # @JobStatus:
include/block/block.h +1 -0
index 6e36154061c1..f4b740857725 100644
@@ -200,6 +200,7 @@ typedef struct BDRVReopenState {
 typedef enum BlockOpType {
     BLOCK_OP_TYPE_BACKUP_SOURCE,
     BLOCK_OP_TYPE_BACKUP_TARGET,
+    BLOCK_OP_TYPE_BITMAP_POPULATE,
     BLOCK_OP_TYPE_CHANGE,
     BLOCK_OP_TYPE_COMMIT_SOURCE,
     BLOCK_OP_TYPE_COMMIT_TARGET,
include/block/block_int.h +21 -0
index 9da7a42927eb..c85a79666524 100644
@@ -1257,6 +1257,27 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
                             BlockCompletionFunc *cb, void *opaque,
                             JobTxn *txn, Error **errp);
 
+/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+
 BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,
                                   const char *child_name,
                                   const BdrvChildClass *child_class,
block/bitmap-populate.c new file +207 -0

207 lines not on this page · raw message

blockjob.c +2 -1
index 470facfd47a0..e6ee8376645d 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)
     return job_type(job) == JOB_TYPE_BACKUP ||
            job_type(job) == JOB_TYPE_COMMIT ||
            job_type(job) == JOB_TYPE_MIRROR ||
-           job_type(job) == JOB_TYPE_STREAM;
+           job_type(job) == JOB_TYPE_STREAM ||
+           job_type(job) == JOB_TYPE_BITMAP_POPULATE;
 }
 
 BlockJob *block_job_next(BlockJob *bjob)
MAINTAINERS +1 -0
index b5ba553ee01a..b13b00774130 100644
@@ -2156,6 +2156,7 @@ S: Supported
 F: include/qemu/hbitmap.h
 F: include/block/dirty-bitmap.h
 F: block/monitor/bitmap-qmp-cmds.c
+F: block/bitmap-populate.c
 F: block/dirty-bitmap.c
 F: block/qcow2-bitmap.c
 F: migration/block-dirty-bitmap.c
block/meson.build +1 -0
index a3e56b7cd11d..8be5fd3c34fa 100644
@@ -5,6 +5,7 @@ block_ss.add(files(
   'amend.c',
   'backup.c',
   'backup-top.c',
+  'bitmap-populate.c',
   'blkdebug.c',
   'blklogwrites.c',
   'blkreplay.c',
Signature · 1 line
2.28.0
#2 lore raw

Re: [PATCH v4 1/5] block: add bitmap-populate job

Quoted 28 lines · Eric Blake

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
 qapi/block-core.json      |  48 +++++++++
 qapi/job.json             |   6 +-
 include/block/block.h     |   1 +
 include/block/block_int.h |  21 ++++
 block/bitmap-populate.c   | 207 ++++++++++++++++++++++++++++++++++++++
 blockjob.c                |   3 +-
 MAINTAINERS               |   1 +
 block/meson.build         |   1 +
 8 files changed, 286 insertions(+), 2 deletions(-)
 create mode 100644 block/bitmap-populate.c
qapi/block-core.json +0 -0
index db08c58d788c..1cac9a9a8207 100644

qapi/block-core.json line 2225

Quoted 10 lines
qapi/block-core.json +7 -0
@@ -2216,6 +2216,54 @@
 { 'command': 'block-dirty-bitmap-merge',
   'data': 'BlockDirtyBitmapMerge' }
 
+##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.

Greek to me.

Quoted 49 lines
qapi/block-core.json +41 -0
+#
+# Since: 5.2
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device supports
+#            io-status (see BlockInfo).
+#
+# @auto-finalize: When false, this job will wait in a PENDING state after it
+#                 has finished its work, waiting for @block-job-finalize
+#                 before making any block graph changes.
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the
+#                query list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.2
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/job.json +0 -0
index 280c2f76f136..fb0b606e868d 100644

qapi/job.json line 26

Quoted 5 lines
qapi/job.json +2 -0
@@ -22,10 +22,14 @@
 #
 # @amend: image options amend job type, see "x-blockdev-amend" (since 5.1)
 #
+# @bitmap-populate: drive bitmap population job type, see
+#                   "block-dirty-bitmap-populate" (since 5.2)

Dangling reference, healed in PATCH 3. Job appearing before the command it wraps is unusual.

What's a "drive bitmap population job"?

The reference provides a clue:

    ##
    # @block-dirty-bitmap-populate:
    #
    # Creates a new job that writes a pattern into a dirty bitmap.

Do you mean "dirty bitmap population job"?

By the way, I think the doc comment would be easier to read with s/job type/job/.

Quoted 10 lines
qapi/job.json +3 -1
+#
 # Since: 1.7
 ##
 { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'amend',
+           'bitmap-populate'] }
 
 ##
 # @JobStatus:

[...]

author
#3 lore raw

PATCH v4 2/5 blockdev: combine DriveBackupState and BlockdevBackupState

In reply to Eric Blake

From: John Snow <[email protected]>

They have the same fields -- rename it BlockJobActionState.

---
 blockdev.c | 30 ++++++++++++------------------
 1 file changed, 12 insertions(+), 18 deletions(-)
blockdev.c +12 -18
index 3848a9c8ab86..f177048a4fe8 100644
@@ -1668,11 +1668,11 @@ static void external_snapshot_clean(BlkActionState *common)
     aio_context_release(aio_context);
 }
 
-typedef struct DriveBackupState {
+typedef struct BlockJobActionState {
     BlkActionState common;
     BlockDriverState *bs;
     BlockJob *job;
-} DriveBackupState;
+} BlockJobActionState;
 
 static BlockJob *do_backup_common(BackupCommon *backup,
                                   BlockDriverState *bs,
@@ -1682,7 +1682,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,
 
 static void drive_backup_prepare(BlkActionState *common, Error **errp)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     DriveBackup *backup;
     BlockDriverState *bs;
     BlockDriverState *target_bs;
@@ -1819,7 +1819,7 @@ out:
 
 static void drive_backup_commit(BlkActionState *common)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     aio_context = bdrv_get_aio_context(state->bs);
@@ -1833,7 +1833,7 @@ static void drive_backup_commit(BlkActionState *common)
 
 static void drive_backup_abort(BlkActionState *common)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
 
     if (state->job) {
         AioContext *aio_context;
@@ -1849,7 +1849,7 @@ static void drive_backup_abort(BlkActionState *common)
 
 static void drive_backup_clean(BlkActionState *common)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     if (!state->bs) {
@@ -1864,15 +1864,9 @@ static void drive_backup_clean(BlkActionState *common)
     aio_context_release(aio_context);
 }
 
-typedef struct BlockdevBackupState {
-    BlkActionState common;
-    BlockDriverState *bs;
-    BlockJob *job;
-} BlockdevBackupState;
-
 static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     BlockdevBackup *backup;
     BlockDriverState *bs;
     BlockDriverState *target_bs;
@@ -1920,7 +1914,7 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
 
 static void blockdev_backup_commit(BlkActionState *common)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     aio_context = bdrv_get_aio_context(state->bs);
@@ -1934,7 +1928,7 @@ static void blockdev_backup_commit(BlkActionState *common)
 
 static void blockdev_backup_abort(BlkActionState *common)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
 
     if (state->job) {
         AioContext *aio_context;
@@ -1950,7 +1944,7 @@ static void blockdev_backup_abort(BlkActionState *common)
 
 static void blockdev_backup_clean(BlkActionState *common)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     if (!state->bs) {
@@ -2222,14 +2216,14 @@ static const BlkActionOps actions[] = {
         .clean = external_snapshot_clean,
     },
     [TRANSACTION_ACTION_KIND_DRIVE_BACKUP] = {
-        .instance_size = sizeof(DriveBackupState),
+        .instance_size = sizeof(BlockJobActionState),
         .prepare = drive_backup_prepare,
         .commit = drive_backup_commit,
         .abort = drive_backup_abort,
         .clean = drive_backup_clean,
     },
     [TRANSACTION_ACTION_KIND_BLOCKDEV_BACKUP] = {
-        .instance_size = sizeof(BlockdevBackupState),
+        .instance_size = sizeof(BlockJobActionState),
         .prepare = blockdev_backup_prepare,
         .commit = blockdev_backup_commit,
         .abort = blockdev_backup_abort,
Signature · 1 line
2.28.0
#4 lore raw

Re: [PATCH v4 2/5] blockdev: combine DriveBackupState and BlockdevBackupState

Quoted 3 lines · Eric Blake

From: John Snow<[email protected]>

They have the same fields -- rename it BlockJobActionState.

commit/abort/clean functions are identical after it. I think better to combine them as well here

(no changes from v3)

Signature · 2 lines
Best regards,
Vladimir
author
#5 lore raw

PATCH v4 3/5 qmp: expose block-dirty-bitmap-populate

In reply to Eric Blake

From: John Snow <[email protected]>

This is a new job-creating command.

---
 qapi/block-core.json  | 18 +++++++++++
 qapi/transaction.json |  2 ++
 blockdev.c            | 74 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+)
qapi/block-core.json +18 -0
index 1cac9a9a8207..ed05c0bfa720 100644
@@ -2264,6 +2264,24 @@
             '*auto-finalize': 'bool',
             '*auto-dismiss': 'bool' } }
 
+##
+# @block-dirty-bitmap-populate:
+#
+# Creates a new job that writes a pattern into a dirty bitmap.
+#
+# Since: 5.2
+#
+# Example:
+#
+# -> { "execute": "block-dirty-bitmap-populate",
+#      "arguments": { "node": "drive0", "target": "bitmap0",
+#                     "job-id": "job0", "pattern": "allocate-top" } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'block-dirty-bitmap-populate', 'boxed': true,
+  'data': 'BlockDirtyBitmapPopulate' }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/transaction.json +2 -0
index 15ddebdbc360..8302b824e4d5 100644
@@ -51,6 +51,7 @@
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0
 # - @block-dirty-bitmap-merge: since 4.0
+# - @block-dirty-bitmap-populate: since 5.2
 # - @blockdev-backup: since 2.3
 # - @blockdev-snapshot: since 2.5
 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -68,6 +69,7 @@
        'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge',
+       'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
        'blockdev-backup': 'BlockdevBackup',
        'blockdev-snapshot': 'BlockdevSnapshot',
        'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +74 -0

92 lines not on this page · raw message

Signature · 1 line
2.28.0
author
#6 lore raw

PATCH v4 4/5 iotests: move bitmap helpers into their own file

In reply to Eric Blake

From: John Snow <[email protected]>

Message-Id: <[email protected]>

---
 tests/qemu-iotests/257        | 110 +---------------------------
 tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++
 2 files changed, 132 insertions(+), 109 deletions(-)
 create mode 100644 tests/qemu-iotests/bitmaps.py
tests/qemu-iotests/257 +1 -109
index e1e60772195e..c12191ec422f 100755
@@ -24,120 +24,12 @@ import os
 
 import iotests
 from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS
 
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
 
-class Pattern:
-    def __init__(self, byte, offset, size=GRANULARITY):
-        self.byte = byte
-        self.offset = offset
-        self.size = size
-
-    def bits(self, granularity):
-        lower = self.offset // granularity
-        upper = (self.offset + self.size - 1) // granularity
-        return set(range(lower, upper + 1))
-
-
-class PatternGroup:
-    """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
-    def __init__(self, patterns):
-        self.patterns = patterns
-
-    def bits(self, granularity):
-        """Calculate the unique bits dirtied by this pattern grouping"""
-        res = set()
-        for pattern in self.patterns:
-            res |= pattern.bits(granularity)
-        return res
-
-
-GROUPS = [
-    PatternGroup([
-        # Batch 0: 4 clusters
-        Pattern('0x49', 0x0000000),
-        Pattern('0x6c', 0x0100000),   # 1M
-        Pattern('0x6f', 0x2000000),   # 32M
-        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
-    PatternGroup([
-        # Batch 1: 6 clusters (3 new)
-        Pattern('0x65', 0x0000000),   # Full overwrite
-        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
-        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
-        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
-    PatternGroup([
-        # Batch 2: 7 clusters (3 new)
-        Pattern('0x74', 0x0010000),   # Adjacent-right
-        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
-        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
-        Pattern('0x67', 0x3fe0000,
-                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
-    PatternGroup([
-        # Batch 3: 8 clusters (5 new)
-        # Carefully chosen such that nothing re-dirties the one cluster
-        # that copies out successfully before failure in Group #1.
-        Pattern('0xaa', 0x0010000,
-                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
-        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
-        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
-        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
-]
-
-
-class EmulatedBitmap:
-    def __init__(self, granularity=GRANULARITY):
-        self._bits = set()
-        self.granularity = granularity
-
-    def dirty_bits(self, bits):
-        self._bits |= set(bits)
-
-    def dirty_group(self, n):
-        self.dirty_bits(GROUPS[n].bits(self.granularity))
-
-    def clear(self):
-        self._bits = set()
-
-    def clear_bits(self, bits):
-        self._bits -= set(bits)
-
-    def clear_bit(self, bit):
-        self.clear_bits({bit})
-
-    def clear_group(self, n):
-        self.clear_bits(GROUPS[n].bits(self.granularity))
-
-    @property
-    def first_bit(self):
-        return sorted(self.bits)[0]
-
-    @property
-    def bits(self):
-        return self._bits
-
-    @property
-    def count(self):
-        return len(self.bits)
-
-    def compare(self, qmp_bitmap):
-        """
-        Print a nice human-readable message checking that a bitmap as reported
-        by the QMP interface has as many bits set as we expect it to.
-        """
-
-        name = qmp_bitmap.get('name', '(anonymous)')
-        log("= Checking Bitmap {:s} =".format(name))
-
-        want = self.count
-        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
-
-        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
-            want, have, "OK!" if want == have else "ERROR!"))
-        log('')
-
-
 class Drive:
     """Represents, vaguely, a drive attached to a VM.
     Includes format, graph, and device information."""
tests/qemu-iotests/bitmaps.py new file +131 -0

131 lines not on this page · raw message

Signature · 1 line
2.28.0
author
#8 lore raw

PATCH v4 5/5 iotests: add 298 for block-dirty-bitmap-populate

In reply to Eric Blake

From: John Snow <[email protected]>

Give block-dirty-bitmap-populate a workout.

---
 tests/qemu-iotests/298     |  232 ++
 tests/qemu-iotests/298.out | 4544 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |    1 +
 3 files changed, 4777 insertions(+)
 create mode 100755 tests/qemu-iotests/298
 create mode 100644 tests/qemu-iotests/298.out
tests/qemu-iotests/298 new file +232 -0
new file mode 100755
index 000000000000..4bfcecd3bc88
@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+#
+# Test block-dirty-bitmap-populate
+#
+# Copyright (c) 2020 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+from collections import namedtuple
+import itertools
+import math
+import os
+
+import iotests
+from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS
+
+SIZE = 64 * 1024 * 1024
+GRANULARITY = 64 * 1024
+
+
+class Drive:
+    def __init__(self, path, vm):
+        self.path = path
+        self.vm = vm
+        self.fmt = None
+        self.size = None
+        self.node = None
+
+    def img_create(self, fmt, size):
+        self.fmt = fmt
+        self.size = size
+        iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size))
+
+
+def block_dirty_bitmap_populate(vm, node, bitmap, job_id, pattern, **kwargs):
+    # Strip any arguments explicitly nulled by the caller:
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}
+    result = vm.qmp_log('block-dirty-bitmap-populate',
+                        node=node,
+                        name=bitmap,
+                        job_id=job_id,
+                        pattern=pattern,
+                        **kwargs)
+    return result
+
+
+def populate(drive, bitmap, job_id, pattern='allocation-top', **kwargs):
+    kwargs.setdefault('pattern', pattern)
+    kwargs.setdefault('auto-finalize', False)
+    kwargs.setdefault('auto-dismiss', False)
+    ret = block_dirty_bitmap_populate(drive.vm, drive.node,
+                                      bitmap, job_id, **kwargs)
+    return {
+        'id': job_id,
+        'auto-finalize': kwargs['auto-finalize'],
+        'auto-dismiss': kwargs['auto-dismiss'],
+        'return': ret,
+    }
+
+
+def perform_writes(drive, n, filter_node_name=None):
+    log("-- Write #{:d}:".format(n))
+    node_name = filter_node_name or drive.node
+    for pattern in GROUPS[n].patterns:
+        cmd = "write -P{:s} 0x{:07x} 0x{:x}".format(
+            pattern.byte,
+            pattern.offset,
+            pattern.size)
+        drive.vm.hmp_qemu_io(node_name, cmd, use_log=True)
+    log('')
+
+
+TestConfig = namedtuple('TestConfig', [
+    'base_pattern',
+    'disabled',
+    'pre_writes',
+    'mid_writes',
+    'cancel',
+    'post_writes',
+])
+
+
+def test_bitmap_populate(config):
+    """
+    Test bitmap populate.
+
+    :param base_pattern: Write a base pattern?
+    :param disabled:     Disable the target bitmap?
+    :param pre_writes:   Write a pattern after bitmap creation, but before job?
+    :param mid_writes:   Write a pattern before job finalizes?
+    :param cancel:       Cancel the job instead of finalizing it?
+    :param post_writes:  Write a pattern after the job?
+    """
+    with iotests.FilePaths(['img']) as (img_path,), iotests.VM() as vm:
+        log("\n=== Bitmap Populate {:s} ===\n".format(str(config)))
+
+        log('-- Prepare image & VM:')
+        drive0 = Drive(img_path, vm=vm)
+        drive0.img_create(iotests.imgfmt, SIZE)
+        vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device()))
+        vm.launch()
+
+        file_config = {
+            'driver': 'file',
+            'filename': drive0.path
+        }
+
+        drive0.node = 'drive0'
+        vm.qmp_log('blockdev-add',
+                   filters=[iotests.filter_qmp_testfiles],
+                   node_name=drive0.node,
+                   driver=drive0.fmt,
+                   file=file_config)
+        log('')
+
+
+        # Step 0: Prepare & Base Allocation Pattern
+
+        if config.base_pattern:
+            perform_writes(drive0, 0)
+
+
+        # Step 1: Add test bitmap
+
+        log('-- Add Bitmap:')
+        vm.qmp_log('block-dirty-bitmap-add',
+                   node=drive0.node,
+                   name="target",
+                   granularity=GRANULARITY,
+                   disabled=config.disabled)
+        ebitmap = EmulatedBitmap()
+        log('')
+
+        # Step 2: Pre-Writes
+
+        if config.pre_writes:
+            perform_writes(drive0, 1)
+            if not config.disabled:
+                ebitmap.dirty_group(1)
+            bitmap = vm.get_bitmap(drive0.node, 'target')
+            ebitmap.compare(bitmap)
+
+
+        # Step 3: Launch job & Mid-Writes
+
+        log('-- Test block-dirty-bitmap-populate (bitpop0):')
+        def pre_finalize():
+            # Writes issued prior to job finalization:
+            if config.mid_writes:
+                perform_writes(drive0, 2)
+                if not config.disabled:
+                    ebitmap.dirty_group(2)
+
+        job = populate(drive0, 'target', 'bitpop0')
+        assert job['return'] == {'return': {}}
+        vm.run_job(job['id'], auto_dismiss=job['auto-dismiss'],
+                   auto_finalize=job['auto-finalize'],
+                   pre_finalize=pre_finalize,
+                   cancel=config.cancel)
+        log('')
+
+
+        # Step 4: Post-job verification
+
+        if not config.cancel:
+            # Any writes made prior to the job finishing should now be visible.
+            if config.base_pattern:
+                ebitmap.dirty_group(0)
+            if config.pre_writes:
+                ebitmap.dirty_group(1)
+            if config.mid_writes:
+                ebitmap.dirty_group(2)
+
+        bitmap = vm.get_bitmap(drive0.node, 'target')
+        ebitmap.compare(bitmap)
+
+
+        # Step 5: Post-Writes
+
+        if config.post_writes:
+            perform_writes(drive0, 3)
+            if not config.disabled:
+                ebitmap.dirty_group(3)
+
+
+        # Step 6: Final Verification
+
+        log('-- Verification:')
+        bitmaps = vm.query_bitmaps()
+        log({'bitmaps': bitmaps}, indent=2)
+        log('')
+        bitmap = vm.get_bitmap(drive0.node, 'target', bitmaps=bitmaps)
+        ebitmap.compare(bitmap)
+
+
+        log('-- Cleanup:')
+        vm.qmp_log("block-dirty-bitmap-remove",
+                   node=drive0.node, name="target")
+
+        bitmaps = vm.query_bitmaps()
+        if bitmaps:
+            log("ERROR: bitmaps unaccounted for:")
+            log(bitmaps)
+        else:
+            log('OK: All bitmaps removed')
+        vm.shutdown()
+        log('')
+
+
+def main():
+    for args in itertools.product((True, False), repeat=6):
+        cfg = TestConfig(*args)
+        test_bitmap_populate(cfg)
+
+
+if __name__ == '__main__':
+    iotests.script_main(main, supported_fmts=['qcow2'],
+                        supported_protocols=['file'])
tests/qemu-iotests/298.out new file +4544 -0

2,768 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 5cad0152317a..f2c84c7b77c0 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.28.0

Part of the diff is not shown here. View the raw message.

#9 lore raw
In reply to Eric Blake
Quoted 24 lines · Eric Blake

This is NOT the final version of this patch series, but I'm posting it to revive conversation on the topic while fixing it to compile on top of meson changes.

v3 was: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06258.html

001/5:[0025] [FC] 'block: add bitmap-populate job'
002/5:[----] [--] 'blockdev: combine DriveBackupState and BlockdevBackupState' 003/5:[0004] [FC] 'qmp: expose block-dirty-bitmap-populate' 004/5:[----] [--] 'iotests: move bitmap helpers into their own file' 005/5:[----] [-C] 'iotests: add 298 for block-dirty-bitmap-populate'

I'm still trying to find the right QAPI contract (affects patch 1 and 3 for the qapi, and 5 for invoking the command in iotests), but right now, I'm leaning towards:

{ "execute": "block-dirty-bitmap-populate", "arguments": {
     "job-id": "job0", "node": "target_node", "name": "target_bitmap",
     "sources": [ { "pattern": "allocation", "node": "from_node" } ] } }

which allows expansion into multiple sources, so that we can combine a populate action with a bitmap merge rather than having to do those as separate commands, like:

Quoted 5 lines
{ "execute": "block-dirty-bitmap-populate", "arguments": {
     "job-id": "job0", "node": "target_node", "name": "target_bitmap",
     "sources": [ { "pattern": "allocation", "node": "from_node" },
                  { "pattern": "bitmap", "node": "from_node",
		     "bitmap": "from_bitmap" } ] } }

(I see, this is not realized in v4)

What about multiple targets as well, as we discussed in v3?

Quoted 29 lines

John Snow (5):
block: add bitmap-populate job
blockdev: combine DriveBackupState and BlockdevBackupState
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add 298 for block-dirty-bitmap-populate

  qapi/block-core.json          |   66 +
  qapi/job.json                 |    6 +-
  qapi/transaction.json         |    2 +
  include/block/block.h         |    1 +
  include/block/block_int.h     |   21 +
  block/bitmap-populate.c       |  207 ++
  blockdev.c                    |  104 +-
  blockjob.c                    |    3 +-
  MAINTAINERS                   |    1 +
  block/meson.build             |    1 +
  tests/qemu-iotests/257        |  110 +-
  tests/qemu-iotests/298        |  232 ++
  tests/qemu-iotests/298.out    | 4544 +++++++++++++++++++++++++++++++++
  tests/qemu-iotests/bitmaps.py |  131 +
  tests/qemu-iotests/group      |    1 +
  15 files changed, 5301 insertions(+), 129 deletions(-)

create mode 100644 block/bitmap-populate.c create mode 100755 tests/qemu-iotests/298 create mode 100644 tests/qemu-iotests/298.out create mode 100644 tests/qemu-iotests/bitmaps.py

Signature · 2 lines
Best regards,
Vladimir
#10 lore raw
In reply to Eric Blake

Hi!

Any plans on v5? Now I have a request for this feature.. If no plans, I can take care of it and make a v5

Quoted 29 lines · Eric Blake

This is NOT the final version of this patch series, but I'm posting it to revive conversation on the topic while fixing it to compile on top of meson changes.

v3 was:https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06258.html

001/5:[0025] [FC] 'block: add bitmap-populate job'
002/5:[----] [--] 'blockdev: combine DriveBackupState and BlockdevBackupState' 003/5:[0004] [FC] 'qmp: expose block-dirty-bitmap-populate' 004/5:[----] [--] 'iotests: move bitmap helpers into their own file' 005/5:[----] [-C] 'iotests: add 298 for block-dirty-bitmap-populate'

I'm still trying to find the right QAPI contract (affects patch 1 and 3 for the qapi, and 5 for invoking the command in iotests), but right now, I'm leaning towards:

{ "execute": "block-dirty-bitmap-populate", "arguments": {
     "job-id": "job0", "node": "target_node", "name": "target_bitmap",
     "sources": [ { "pattern": "allocation", "node": "from_node" } ] } }

which allows expansion into multiple sources, so that we can combine a populate action with a bitmap merge rather than having to do those as separate commands, like:

{ "execute": "block-dirty-bitmap-populate", "arguments": {
     "job-id": "job0", "node": "target_node", "name": "target_bitmap",
     "sources": [ { "pattern": "allocation", "node": "from_node" },
                  { "pattern": "bitmap", "node": "from_node",
		     "bitmap": "from_bitmap" } ] } }
Signature · 2 lines
Best regards,
Vladimir