Skip to contentSkip to conversation

block: add block-dirty-bitmap-populate job

PATCH Superseded John Snow 2 participants 20 messages Feb 25, 2020 – Mar 3, 2020 Block layer

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

John Snow

author
#0 lore raw

cover block: add block-dirty-bitmap-populate job

Hi,

This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.

This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.

This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.

Patches 1-2: The new job.
Patch 3: iotest prerequisite
Patch 4-5: completely optional cleanup.
Patch 6: Test.

John Snow (6):
block: add bitmap-populate job
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add hmp helper with logging qmp.py: change event_wait to use a dict
iotests: add 287 for block-dirty-bitmap-populate

 qapi/block-core.json          |   66 +
 qapi/job.json                 |    2 +-
 qapi/transaction.json         |    2 +
 include/block/block_int.h     |   21 +
 block/bitmap-alloc.c          |  207 ++
 blockdev.c                    |   78 +
 blockjob.c                    |    3 +-
 block/Makefile.objs           |    1 +
 python/qemu/machine.py        |   10 +-
 tests/qemu-iotests/040        |   12 +-
 tests/qemu-iotests/257        |  110 +-
 tests/qemu-iotests/260        |    5 +-
 tests/qemu-iotests/287        |  233 ++
 tests/qemu-iotests/287.out    | 4544 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/bitmaps.py |  131 +
 tests/qemu-iotests/group      |    1 +
 tests/qemu-iotests/iotests.py |   34 +-
 17 files changed, 5321 insertions(+), 139 deletions(-)
 create mode 100644 block/bitmap-alloc.c
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
 create mode 100644 tests/qemu-iotests/bitmaps.py
Signature · 1 line
2.21.1
author
#1 lore raw

PATCH 1/6 block: add bitmap-populate job

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
 qapi/block-core.json      |  48 +++++++++
 qapi/job.json             |   2 +-
 include/block/block_int.h |  21 ++++
 block/bitmap-alloc.c      | 207 ++++++++++++++++++++++++++++++++++++++
 blockjob.c                |   3 +-
 block/Makefile.objs       |   1 +
 6 files changed, 280 insertions(+), 2 deletions(-)
 create mode 100644 block/bitmap-alloc.c
qapi/block-core.json +48 -0
index 85e27bb61f..df1797681a 100644
@@ -2245,6 +2245,54 @@
       { 'command': 'block-dirty-bitmap-merge',
         'data': 'BlockDirtyBitmapMerge' }
 
+##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.0
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device supports
+#            io-status (see BlockInfo).
+#
+# @auto-finalize: When false, this job will wait in a PENDING state after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/job.json +1 -1
index 5e658281f5..5f496d4630 100644
@@ -22,7 +22,7 @@
 # Since: 1.7
 ##
 { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'bitmap-populate'] }
 
 ##
 # @JobStatus:
include/block/block_int.h +21 -0
index 6f9fd5e20e..a5884b597e 100644
@@ -1215,6 +1215,27 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
                             BlockCompletionFunc *cb, void *opaque,
                             JobTxn *txn, Error **errp);
 
+/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+
 void hmp_drive_add_node(Monitor *mon, const char *optstr);
 
 BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,
block/bitmap-alloc.c new file +207 -0

207 lines not on this page · raw message

blockjob.c +2 -1
index 5d63b1e89d..7e450372bd 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)
     return job_type(job) == JOB_TYPE_BACKUP ||
            job_type(job) == JOB_TYPE_COMMIT ||
            job_type(job) == JOB_TYPE_MIRROR ||
-           job_type(job) == JOB_TYPE_STREAM;
+           job_type(job) == JOB_TYPE_STREAM ||
+           job_type(job) == JOB_TYPE_BITMAP_POPULATE;
 }
 
 BlockJob *block_job_next(BlockJob *bjob)
block/Makefile.objs +1 -0
index 3bcb35c81d..f3cfc89d90 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o
 block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 block-obj-y += backup.o
+block-obj-y += bitmap-alloc.o
 block-obj-$(CONFIG_REPLICATION) += replication.o
 block-obj-y += throttle.o copy-on-read.o
 block-obj-y += block-copy.o
Signature · 1 line
2.21.1
#2 lore raw

Re: [PATCH 1/6] block: add bitmap-populate job

Quoted 44 lines · John Snow

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
  qapi/block-core.json      |  48 +++++++++
  qapi/job.json             |   2 +-
  include/block/block_int.h |  21 ++++
  block/bitmap-alloc.c      | 207 ++++++++++++++++++++++++++++++++++++++
  blockjob.c                |   3 +-
  block/Makefile.objs       |   1 +
  6 files changed, 280 insertions(+), 2 deletions(-)

create mode 100644 block/bitmap-alloc.c

qapi/block-core.json +18 -0
index 85e27bb61f..df1797681a 100644
@@ -2245,6 +2245,54 @@
        { 'command': 'block-dirty-bitmap-merge',
          'data': 'BlockDirtyBitmapMerge' }
 
+##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.0
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#

qapi/block-core.json line 2275

Quoted 10 lines
qapi/block-core.json +10 -0
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device supports
+#            io-status (see BlockInfo).
+#
+# @auto-finalize: When false, this job will wait in a PENDING state after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.

sounds a bit strange in context of bitmap-population job

Quoted 85 lines
qapi/block-core.json +20 -0
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
  ##
  # @BlockDirtyBitmapSha256:
  #
qapi/job.json +1 -1
index 5e658281f5..5f496d4630 100644
@@ -22,7 +22,7 @@
  # Since: 1.7
  ##
  { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create', 'bitmap-populate'] }
 
  ##
  # @JobStatus:
include/block/block_int.h +21 -0
index 6f9fd5e20e..a5884b597e 100644
@@ -1215,6 +1215,27 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
                              BlockCompletionFunc *cb, void *opaque,
                              JobTxn *txn, Error **errp);
 
+/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+
  void hmp_drive_add_node(Monitor *mon, const char *optstr);
 
  BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,
block/bitmap-alloc.c new file +11 -0
new file mode 100644
index 0000000000..47d542dc12
@@ -0,0 +1,207 @@
+/*
+ * Async Dirty Bitmap Populator
+ *
+ * Copyright (C) 2020 Red Hat, Inc.
+ *
+ * Authors:
+ *  John Snow <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ *

block/bitmap-alloc.c line 21

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+ */
+
+#include "qemu/osdep.h"
+
+#include "trace.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "block/blockjob_int.h"
+#include "block/block_backup.h"
+#include "block/block-copy.h"

I hope, not all includes are needed :)

Quoted 15 lines
block/bitmap-alloc.c new file +15 -0
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/ratelimit.h"
+#include "qemu/cutils.h"
+#include "sysemu/block-backend.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+
+typedef struct BitpopBlockJob {
+    BlockJob common;
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *target_bitmap;
+    BdrvDirtyBitmap *new_bitmap;
+    BlockdevOnError on_error;
+    uint64_t len;

block/bitmap-alloc.c line 46

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+} BitpopBlockJob;
+
+static const BlockJobDriver bitpop_job_driver;
+
+static void bitpop_commit(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+
+    bdrv_dirty_bitmap_merge_internal(s->target_bitmap, s->new_bitmap,
+                                     NULL, true);

Hmm, so you populate new_bitmap, and then merge to target. Why can't we work directly with target bitmap? The most probable case is that libvirt will create bitmap specifically to use as target in this operation, or not?

Hmm, just to make it possible to cancel the job and keep the target bitmap in original state? Is it really needed? I think on failure target bitmap will be removed anyway..

Quoted 47 lines
block/bitmap-alloc.c new file +47 -0
+}
+
+/* no abort needed; just clean without committing. */
+
+static void bitpop_clean(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+
+    bdrv_release_dirty_bitmap(s->new_bitmap);
+    bdrv_dirty_bitmap_set_busy(s->target_bitmap, false);
+}
+
+static BlockErrorAction bitpop_error_action(BitpopBlockJob *job, int error)
+{
+    return block_job_error_action(&job->common, job->on_error, true, error);
+}
+
+static bool coroutine_fn yield_and_check(Job *job)
+{
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    job_sleep_ns(job, 0);
+
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    return false;
+}
+
+static int coroutine_fn bitpop_run(Job *job, Error **errp)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+    int ret = 0;
+    int64_t offset;
+    int64_t count;
+    int64_t bytes;
+
+    for (offset = 0; offset < s->len; ) {
+        if (yield_and_check(job)) {
+            ret = -ECANCELED;
+            break;
+        }
+
+        bytes = s->len - offset;

block/bitmap-alloc.c line 103

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+        ret = bdrv_is_allocated(s->bs, offset, bytes, &count);
+        if (ret < 0) {
+            if (bitpop_error_action(s, -ret) == BLOCK_ERROR_ACTION_REPORT) {
+                break;
+            }
+            continue;
+        }
+
+        if (!count) {
+            ret = 0;

Hmm, I think it's impossible case.. If so, better to make an assertion or ignore..

Quoted 43 lines
block/bitmap-alloc.c new file +43 -0
+            break;
+        }
+
+        if (ret) {
+            bdrv_set_dirty_bitmap(s->new_bitmap, offset, count);
+            ret = 0;
+        }
+
+        job_progress_update(job, count);
+        offset += count;
+    }
+
+    return ret;
+}
+
+static const BlockJobDriver bitpop_job_driver = {
+    .job_driver = {
+        .instance_size          = sizeof(BitpopBlockJob),
+        .job_type               = JOB_TYPE_BITMAP_POPULATE,
+        .free                   = block_job_free,
+        .user_resume            = block_job_user_resume,
+        .run                    = bitpop_run,
+        .commit                 = bitpop_commit,
+        .clean                  = bitpop_clean,
+    }
+};
+
+
+BlockJob *bitpop_job_create(
+    const char *job_id,
+    BlockDriverState *bs,
+    BdrvDirtyBitmap *target_bitmap,
+    BitmapPattern pattern,
+    BlockdevOnError on_error,
+    int creation_flags,
+    BlockCompletionFunc *cb,
+    void *opaque,
+    JobTxn *txn,
+    Error **errp)
+{
+    int64_t len;
+    BitpopBlockJob *job = NULL;
+    int64_t cluster_size;

block/bitmap-alloc.c line 156

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+    BdrvDirtyBitmap *new_bitmap = NULL;
+
+    assert(bs);
+    assert(target_bitmap);
+
+    if (!bdrv_is_inserted(bs)) {
+        error_setg(errp, "Device is not inserted: %s",
+                   bdrv_get_device_name(bs));
+        return NULL;
+    }

Why this?

block/bitmap-alloc.c line 160

Quoted 4 lines
block/bitmap-alloc.c new file +4 -0
+
+    if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) {
+        return NULL;
+    }

and this?

Quoted 16 lines
block/bitmap-alloc.c new file +16 -0
+
+    if (bdrv_dirty_bitmap_check(target_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
+        return NULL;
+    }
+
+    if (pattern != BITMAP_PATTERN_ALLOCATION_TOP) {
+        error_setg(errp, "Unrecognized bitmap pattern");
+        return NULL;
+    }
+
+    len = bdrv_getlength(bs);
+    if (len < 0) {
+        error_setg_errno(errp, -len, "unable to get length for '%s'",
+                         bdrv_get_device_name(bs));
+        return NULL;
+    }

block/bitmap-alloc.c line 186

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+
+    /* NB: new bitmap is anonymous and enabled */
+    cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap);
+    new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
+    if (!new_bitmap) {
+        return NULL;
+    }
+
+    /* Take ownership; we reserve the right to write into this on-commit. */
+    bdrv_dirty_bitmap_set_busy(target_bitmap, true);

Honestly, I still have bad understanding about how should we use dirty bitmap mutex, but note that bdrv_dirty_bitmap_set_busy locks the mutex. And it is (may be) possible, that busy status of the bitmap is changed after bdrv_dirty_bitmap_check but before bdrv_dirty_bitmap_set_busy. So, more correct would be do both operation under one critical section. Still, I don't know is the situation possible.

block/bitmap-alloc.c line 189

Quoted 3 lines
block/bitmap-alloc.c new file +3 -0
+
+    job = block_job_create(job_id, &bitpop_job_driver, txn, bs,
+                           BLK_PERM_CONSISTENT_READ,

Do we need it? We are not going to read..

Quoted 45 lines
block/bitmap-alloc.c new file +18 -0
+                           BLK_PERM_ALL & ~BLK_PERM_RESIZE,
+                           0, creation_flags,
+                           cb, opaque, errp);
+    if (!job) {
+        bdrv_dirty_bitmap_set_busy(target_bitmap, false);
+        bdrv_release_dirty_bitmap(new_bitmap);
+        return NULL;
+    }
+
+    job->bs = bs;
+    job->on_error = on_error;
+    job->target_bitmap = target_bitmap;
+    job->new_bitmap = new_bitmap;
+    job->len = len;
+    job_progress_set_remaining(&job->common.job, job->len);
+
+    return &job->common;
+}
blockjob.c +2 -1
index 5d63b1e89d..7e450372bd 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)
      return job_type(job) == JOB_TYPE_BACKUP ||
             job_type(job) == JOB_TYPE_COMMIT ||
             job_type(job) == JOB_TYPE_MIRROR ||
-           job_type(job) == JOB_TYPE_STREAM;
+           job_type(job) == JOB_TYPE_STREAM ||
+           job_type(job) == JOB_TYPE_BITMAP_POPULATE;
  }
 
  BlockJob *block_job_next(BlockJob *bjob)
block/Makefile.objs +1 -0
index 3bcb35c81d..f3cfc89d90 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o
  block-obj-y += accounting.o dirty-bitmap.o
  block-obj-y += write-threshold.o
  block-obj-y += backup.o
+block-obj-y += bitmap-alloc.o
  block-obj-$(CONFIG_REPLICATION) += replication.o
  block-obj-y += throttle.o copy-on-read.o
  block-obj-y += block-copy.o
Signature · 2 lines
Best regards,
Vladimir
author
#3 lore raw

Re: [PATCH 1/6] block: add bitmap-populate job

Quoted 51 lines · Vladimir Sementsov-Ogievskiy
Quoted 50 lines · John Snow

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
  qapi/block-core.json      |  48 +++++++++
  qapi/job.json             |   2 +- include/block/block_int.h |  21 ++++
  block/bitmap-alloc.c      | 207 ++++++++++++++++++++++++++++++++++++++ blockjob.c                |   3 +-
  block/Makefile.objs       |   1 +

  6 files changed, 280 insertions(+), 2 deletions(-)

  create mode 100644 block/bitmap-alloc.c

qapi/block-core.json +0 -0
index 85e27bb61f..df1797681a 100644
@@ -2245,6 +2245,54 @@
         { 'command': 'block-dirty-bitmap-merge',
           'data': 'BlockDirtyBitmapMerge' }

  +##

qapi/block-core.json +22 -0
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.0
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device

supports
+#            io-status (see BlockInfo).

Quoted 9 lines
Quoted 6 lines
qapi/block-core.json +2 -0
+#
+# @auto-finalize: When false, this job will wait in a PENDING state

after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.

sounds a bit strange in context of bitmap-population job

Yeah, you're right. Copy-pasted for "consistency".

Quoted 92 lines
Quoted 92 lines
qapi/block-core.json +5 -0
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state

after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query

qapi/block-core.json +13 -0
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+

  ##
  # @BlockDirtyBitmapSha256:
  #

qapi/job.json +1 -1
index 5e658281f5..5f496d4630 100644
@@ -22,7 +22,7 @@
   # Since: 1.7
   ##
   { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create',
 'bitmap-populate'] }
     ##
   # @JobStatus:
include/block/block_int.h +0 -0
index 6f9fd5e20e..a5884b597e 100644
@@ -1215,6 +1215,27 @@ BlockJob *backup_job_create(const char *job_id,
 BlockDriverState *bs,
                               BlockCompletionFunc *cb, void *opaque,
                               JobTxn *txn, Error **errp);

  +/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */

include/block/block_int.h +8 -0
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+

  void hmp_drive_add_node(Monitor *mon, const char *optstr); BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,

block/bitmap-alloc.c new file +9 -0
new file mode 100644
index 0000000000..47d542dc12
@@ -0,0 +1,207 @@
+/*
+ * Async Dirty Bitmap Populator
+ *
+ * Copyright (C) 2020 Red Hat, Inc.
+ *
+ * Authors:
+ *  John Snow <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or

later.
+ * See the COPYING file in the top-level directory.
+ *
+ */

block/bitmap-alloc.c new file +3 -0
+
+#include "qemu/osdep.h"
+
Quoted 8 lines

block/bitmap-alloc.c line 21

Quoted 6 lines
block/bitmap-alloc.c new file +6 -0
+#include "trace.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "block/blockjob_int.h"
+#include "block/block_backup.h"
+#include "block/block-copy.h"

I hope, not all includes are needed :)

Whoops, no, of course not. I copied the skeleton from backup, as you can tell ;)

Quoted 23 lines
Quoted 23 lines
block/bitmap-alloc.c new file +22 -0
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/ratelimit.h"
+#include "qemu/cutils.h"
+#include "sysemu/block-backend.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+
+typedef struct BitpopBlockJob {
+    BlockJob common;
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *target_bitmap;
+    BdrvDirtyBitmap *new_bitmap;
+    BlockdevOnError on_error;
+    uint64_t len;
+} BitpopBlockJob;
+
+static const BlockJobDriver bitpop_job_driver;
+
+static void bitpop_commit(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
Quoted 9 lines
Quoted 3 lines

+
+    bdrv_dirty_bitmap_merge_internal(s->target_bitmap, s->new_bitmap, +                                     NULL, true);

Hmm, so you populate new_bitmap, and then merge to target. Why can't we work
directly with target bitmap? The most probable case is that libvirt will create bitmap specifically to use as target in this operation, or not?

Most likely case, yes. Odds are very good it will be a brand new bitmap.

However, we already have a creation command -- I didn't want to make a second job-version of the command and then maintain two interfaces, so I made it a "merge into existing" style command instead.

Quoted 6 lines

Hmm, just to make it possible to cancel the job and keep the target bitmap in
original state? Is it really needed? I think on failure target bitmap will be
removed anyway..

You caught me being *lazy*. I copy the bitmap so I can unconditionally enable it to catch in-flight writes without having to create block graph modifications.

But, yes, to undo changes if we cancel.

I didn't want to make a job that was not able to be canceled. The alternative is pursuing the design where we allow new bitmaps only -- because then on cancel we can just delete them.

Quoted 55 lines
Quoted 55 lines
block/bitmap-alloc.c new file +13 -0
+}
+
+/* no abort needed; just clean without committing. */
+
+static void bitpop_clean(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+
+    bdrv_release_dirty_bitmap(s->new_bitmap);
+    bdrv_dirty_bitmap_set_busy(s->target_bitmap, false);
+}
+
+static BlockErrorAction bitpop_error_action(BitpopBlockJob *job, int

error)
+{
+    return block_job_error_action(&job->common, job->on_error, true, error);

block/bitmap-alloc.c new file +35 -0
+}
+
+static bool coroutine_fn yield_and_check(Job *job)
+{
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    job_sleep_ns(job, 0);
+
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    return false;
+}
+
+static int coroutine_fn bitpop_run(Job *job, Error **errp)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+    int ret = 0;
+    int64_t offset;
+    int64_t count;
+    int64_t bytes;
+
+    for (offset = 0; offset < s->len; ) {
+        if (yield_and_check(job)) {
+            ret = -ECANCELED;
+            break;
+        }
+
+        bytes = s->len - offset;
+        ret = bdrv_is_allocated(s->bs, offset, bytes, &count);
+        if (ret < 0) {
+            if (bitpop_error_action(s, -ret) ==

BLOCK_ERROR_ACTION_REPORT) {
+                break;
+            }

Quoted 9 lines
Quoted 5 lines

+            continue;
+        }
+
+        if (!count) {
+            ret = 0;

Hmm, I think it's impossible case.. If so, better to make an assertion or ignore..

OK, agreed.

Quoted 47 lines
Quoted 47 lines

+            break;
+        }

block/bitmap-alloc.c new file +45 -0
+
+        if (ret) {
+            bdrv_set_dirty_bitmap(s->new_bitmap, offset, count);
+            ret = 0;
+        }
+
+        job_progress_update(job, count);
+        offset += count;
+    }
+
+    return ret;
+}
+
+static const BlockJobDriver bitpop_job_driver = {
+    .job_driver = {
+        .instance_size          = sizeof(BitpopBlockJob),
+        .job_type               = JOB_TYPE_BITMAP_POPULATE,
+        .free                   = block_job_free,
+        .user_resume            = block_job_user_resume,
+        .run                    = bitpop_run,
+        .commit                 = bitpop_commit,
+        .clean                  = bitpop_clean,
+    }
+};
+
+
+BlockJob *bitpop_job_create(
+    const char *job_id,
+    BlockDriverState *bs,
+    BdrvDirtyBitmap *target_bitmap,
+    BitmapPattern pattern,
+    BlockdevOnError on_error,
+    int creation_flags,
+    BlockCompletionFunc *cb,
+    void *opaque,
+    JobTxn *txn,
+    Error **errp)
+{
+    int64_t len;
+    BitpopBlockJob *job = NULL;
+    int64_t cluster_size;
+    BdrvDirtyBitmap *new_bitmap = NULL;
+
+    assert(bs);
+    assert(target_bitmap);
Quoted 9 lines
Quoted 6 lines

+
+    if (!bdrv_is_inserted(bs)) {
+        error_setg(errp, "Device is not inserted: %s", +                   bdrv_get_device_name(bs)); +        return NULL;
+    }

Why this?

I assumed there was nothing to read the allocation map *of* if there wasn't a media present.

Am I mistaken?

Quoted 7 lines
Quoted 4 lines

+
+    if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { +        return NULL;
+    }

and this?

Copy-paste: I don't understand if I want a new op blocker, to re-use an op-blocker, or to have no op blocker.

Genuinely I have no idea. I should left a review comment here, I forgot about this part, sorry.

Quoted 30 lines
Quoted 26 lines

+
+    if (bdrv_dirty_bitmap_check(target_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
+        return NULL;
+    }

block/bitmap-alloc.c new file +21 -0
+
+    if (pattern != BITMAP_PATTERN_ALLOCATION_TOP) {
+        error_setg(errp, "Unrecognized bitmap pattern");
+        return NULL;
+    }
+
+    len = bdrv_getlength(bs);
+    if (len < 0) {
+        error_setg_errno(errp, -len, "unable to get length for '%s'",
+                         bdrv_get_device_name(bs));
+        return NULL;
+    }
+
+    /* NB: new bitmap is anonymous and enabled */
+    cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap);
+    new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
+    if (!new_bitmap) {
+        return NULL;
+    }
+
+    /* Take ownership; we reserve the right to write into this
Quoted 2 lines

on-commit. */
+    bdrv_dirty_bitmap_set_busy(target_bitmap, true);

Honestly, I still have bad understanding about how should we use dirty

Quoted 9 lines

bitmap mutex,
but note that bdrv_dirty_bitmap_set_busy locks the mutex. And it is (may be) possible,
that busy status of the bitmap is changed after bdrv_dirty_bitmap_check but before
bdrv_dirty_bitmap_set_busy.  So, more correct would be do both operation under one
critical section. Still, I don't know is the situation possible.

Aren't we under the BQL here? Can we be pre-empted? :(

Quoted 6 lines
Quoted 3 lines

+
+    job = block_job_create(job_id, &bitpop_job_driver, txn, bs, +                           BLK_PERM_CONSISTENT_READ,

Do we need it? We are not going to read..

Copy-paste / leftover from an earlier draft where I was trying to achieve atomicity. It can be removed if we don't want the stricter atomicity.

Quoted 36 lines
Quoted 36 lines

+                           BLK_PERM_ALL & ~BLK_PERM_RESIZE, +                           0, creation_flags,
+                           cb, opaque, errp);
+    if (!job) {
+        bdrv_dirty_bitmap_set_busy(target_bitmap, false); +        bdrv_release_dirty_bitmap(new_bitmap);
+        return NULL;
+    }

block/bitmap-alloc.c new file +10 -0
+
+    job->bs = bs;
+    job->on_error = on_error;
+    job->target_bitmap = target_bitmap;
+    job->new_bitmap = new_bitmap;
+    job->len = len;
+    job_progress_set_remaining(&job->common.job, job->len);
+
+    return &job->common;
+}
blockjob.c +0 -0
index 5d63b1e89d..7e450372bd 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)

      return job_type(job) == JOB_TYPE_BACKUP ||

             job_type(job) == JOB_TYPE_COMMIT ||
             job_type(job) == JOB_TYPE_MIRROR ||

-           job_type(job) == JOB_TYPE_STREAM; +           job_type(job) == JOB_TYPE_STREAM || +           job_type(job) == JOB_TYPE_BITMAP_POPULATE; }
    BlockJob *block_job_next(BlockJob *bjob)

block/Makefile.objs +0 -0
index 3bcb35c81d..f3cfc89d90 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o
Quoted 10 lines
Quoted 8 lines

  block-obj-y += accounting.o dirty-bitmap.o block-obj-y += write-threshold.o block-obj-y += backup.o
+block-obj-y += bitmap-alloc.o block-obj-$(CONFIG_REPLICATION) += replication.o block-obj-y += throttle.o copy-on-read.o block-obj-y += block-copy.o

Thanks for the review. I'll start making changes, but won't send V2 just yet.

--js

#4 lore raw

Re: [PATCH 1/6] block: add bitmap-populate job

Quoted 220 lines · John Snow
Quoted 51 lines · Vladimir Sementsov-Ogievskiy
Quoted 50 lines · John Snow

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
   qapi/block-core.json      |  48 +++++++++
   qapi/job.json             |   2 +-
   include/block/block_int.h |  21 ++++
   block/bitmap-alloc.c      | 207 ++++++++++++++++++++++++++++++++++++++
   blockjob.c                |   3 +-
   block/Makefile.objs       |   1 +
   6 files changed, 280 insertions(+), 2 deletions(-)

  create mode 100644 block/bitmap-alloc.c

qapi/block-core.json +22 -0
index 85e27bb61f..df1797681a 100644
@@ -2245,6 +2245,54 @@
         { 'command': 'block-dirty-bitmap-merge',
           'data': 'BlockDirtyBitmapMerge' }
   +##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.0
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device

supports
+#            io-status (see BlockInfo).

Quoted 9 lines
Quoted 6 lines
qapi/block-core.json +2 -0
+#
+# @auto-finalize: When false, this job will wait in a PENDING state

after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.

sounds a bit strange in context of bitmap-population job

Yeah, you're right. Copy-pasted for "consistency".

Quoted 92 lines
Quoted 92 lines
qapi/block-core.json +5 -0
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state

after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query

qapi/block-core.json +13 -0
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
   ##
   # @BlockDirtyBitmapSha256:
   #
qapi/job.json +1 -1
index 5e658281f5..5f496d4630 100644
@@ -22,7 +22,7 @@
   # Since: 1.7
   ##
   { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create',
 'bitmap-populate'] }
     ##
   # @JobStatus:
include/block/block_int.h +20 -0
index 6f9fd5e20e..a5884b597e 100644
@@ -1215,6 +1215,27 @@ BlockJob *backup_job_create(const char *job_id,
 BlockDriverState *bs,
                               BlockCompletionFunc *cb, void *opaque,
                               JobTxn *txn, Error **errp);
   +/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+
   void hmp_drive_add_node(Monitor *mon, const char *optstr);
     BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,
block/bitmap-alloc.c new file +9 -0
new file mode 100644
index 0000000000..47d542dc12
@@ -0,0 +1,207 @@
+/*
+ * Async Dirty Bitmap Populator
+ *
+ * Copyright (C) 2020 Red Hat, Inc.
+ *
+ * Authors:
+ *  John Snow <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or

later.
+ * See the COPYING file in the top-level directory.
+ *
+ */

block/bitmap-alloc.c new file +3 -0
+
+#include "qemu/osdep.h"
+
Quoted 8 lines

block/bitmap-alloc.c line 21

Quoted 6 lines
block/bitmap-alloc.c new file +6 -0
+#include "trace.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "block/blockjob_int.h"
+#include "block/block_backup.h"
+#include "block/block-copy.h"

I hope, not all includes are needed :)

Whoops, no, of course not. I copied the skeleton from backup, as you can tell ;)

Quoted 23 lines
Quoted 23 lines
block/bitmap-alloc.c new file +22 -0
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/ratelimit.h"
+#include "qemu/cutils.h"
+#include "sysemu/block-backend.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+
+typedef struct BitpopBlockJob {
+    BlockJob common;
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *target_bitmap;
+    BdrvDirtyBitmap *new_bitmap;
+    BlockdevOnError on_error;
+    uint64_t len;
+} BitpopBlockJob;
+
+static const BlockJobDriver bitpop_job_driver;
+
+static void bitpop_commit(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
Quoted 9 lines
Quoted 3 lines

+
+    bdrv_dirty_bitmap_merge_internal(s->target_bitmap, s->new_bitmap, +                                     NULL, true);

Hmm, so you populate new_bitmap, and then merge to target. Why can't we work
directly with target bitmap? The most probable case is that libvirt will create bitmap specifically to use as target in this operation, or not?

Most likely case, yes. Odds are very good it will be a brand new bitmap.

However, we already have a creation command -- I didn't want to make a second job-version of the command and then maintain two interfaces, so I made it a "merge into existing" style command instead.

Quoted 6 lines

Hmm, just to make it possible to cancel the job and keep the target bitmap in
original state? Is it really needed? I think on failure target bitmap will be
removed anyway..

You caught me being *lazy*. I copy the bitmap so I can unconditionally enable it to catch in-flight writes without having to create block graph modifications.

Quoted 5 lines

But, yes, to undo changes if we cancel.

I didn't want to make a job that was not able to be canceled. The alternative is pursuing the design where we allow new bitmaps only -- because then on cancel we can just delete them.

On backup job (and any other) we can't rollback target changes on cancel. So, I think it would be OK to take same semantics for the new job, keeping in mind that it would be most probable usage case and no sense in creating additional bitmaps. And if caller needs to use existent non-empty bitmap as target and wants correct cancel, it always can create additional bitmap by itself and then merge it to actual target.

And why new? It's up to user, what to use as target. And user knows, that on failure or cancel target becomes invalid and will workaround this if needed.

Quoted 125 lines
Quoted 56 lines
Quoted 56 lines
block/bitmap-alloc.c new file +13 -0
+}
+
+/* no abort needed; just clean without committing. */
+
+static void bitpop_clean(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+
+    bdrv_release_dirty_bitmap(s->new_bitmap);
+    bdrv_dirty_bitmap_set_busy(s->target_bitmap, false);
+}
+
+static BlockErrorAction bitpop_error_action(BitpopBlockJob *job, int

error)
+{
+    return block_job_error_action(&job->common, job->on_error, true, error);

block/bitmap-alloc.c new file +35 -0
+}
+
+static bool coroutine_fn yield_and_check(Job *job)
+{
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    job_sleep_ns(job, 0);
+
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    return false;
+}
+
+static int coroutine_fn bitpop_run(Job *job, Error **errp)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+    int ret = 0;
+    int64_t offset;
+    int64_t count;
+    int64_t bytes;
+
+    for (offset = 0; offset < s->len; ) {
+        if (yield_and_check(job)) {
+            ret = -ECANCELED;
+            break;
+        }
+
+        bytes = s->len - offset;
+        ret = bdrv_is_allocated(s->bs, offset, bytes, &count);
+        if (ret < 0) {
+            if (bitpop_error_action(s, -ret) ==

BLOCK_ERROR_ACTION_REPORT) {
+                break;
+            }

Quoted 9 lines
Quoted 5 lines

+            continue;
+        }
+
+        if (!count) {
+            ret = 0;

Hmm, I think it's impossible case.. If so, better to make an assertion or ignore..

OK, agreed.

Quoted 47 lines
Quoted 47 lines

+            break;
+        }

block/bitmap-alloc.c new file +45 -0
+
+        if (ret) {
+            bdrv_set_dirty_bitmap(s->new_bitmap, offset, count);
+            ret = 0;
+        }
+
+        job_progress_update(job, count);
+        offset += count;
+    }
+
+    return ret;
+}
+
+static const BlockJobDriver bitpop_job_driver = {
+    .job_driver = {
+        .instance_size          = sizeof(BitpopBlockJob),
+        .job_type               = JOB_TYPE_BITMAP_POPULATE,
+        .free                   = block_job_free,
+        .user_resume            = block_job_user_resume,
+        .run                    = bitpop_run,
+        .commit                 = bitpop_commit,
+        .clean                  = bitpop_clean,
+    }
+};
+
+
+BlockJob *bitpop_job_create(
+    const char *job_id,
+    BlockDriverState *bs,
+    BdrvDirtyBitmap *target_bitmap,
+    BitmapPattern pattern,
+    BlockdevOnError on_error,
+    int creation_flags,
+    BlockCompletionFunc *cb,
+    void *opaque,
+    JobTxn *txn,
+    Error **errp)
+{
+    int64_t len;
+    BitpopBlockJob *job = NULL;
+    int64_t cluster_size;
+    BdrvDirtyBitmap *new_bitmap = NULL;
+
+    assert(bs);
+    assert(target_bitmap);
Quoted 9 lines
Quoted 6 lines

+
+    if (!bdrv_is_inserted(bs)) {
+        error_setg(errp, "Device is not inserted: %s", +                   bdrv_get_device_name(bs)); +        return NULL;
+    }

Why this?

Quoted 4 lines

I assumed there was nothing to read the allocation map *of* if there wasn't a media present.

Am I mistaken?

is_inserted checks existing of bs->drv, but bitmap operations actually doesn't need any drv.. Hmm. I'm not against this check anyway.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

+
+    if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { +        return NULL;
+    }

Quoted 8 lines
Quoted 2 lines

and this?

Copy-paste: I don't understand if I want a new op blocker, to re-use an op-blocker, or to have no op blocker.

Genuinely I have no idea. I should left a review comment here, I forgot about this part, sorry.

I'm for no op blocker. As I understand, op-blockers are old and should be replaced by permissions and frozen children. So, if we don't know, do we need any blocking here, better go on without it. Also, op-blockers can't block forbidden usage through filters anyway.

Quoted 41 lines
Quoted 31 lines
Quoted 27 lines

+
+    if (bdrv_dirty_bitmap_check(target_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
+        return NULL;
+    }

block/bitmap-alloc.c new file +21 -0
+
+    if (pattern != BITMAP_PATTERN_ALLOCATION_TOP) {
+        error_setg(errp, "Unrecognized bitmap pattern");
+        return NULL;
+    }
+
+    len = bdrv_getlength(bs);
+    if (len < 0) {
+        error_setg_errno(errp, -len, "unable to get length for '%s'",
+                         bdrv_get_device_name(bs));
+        return NULL;
+    }
+
+    /* NB: new bitmap is anonymous and enabled */
+    cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap);
+    new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL, errp);
+    if (!new_bitmap) {
+        return NULL;
+    }
+
+    /* Take ownership; we reserve the right to write into this
Quoted 2 lines

on-commit. */
+    bdrv_dirty_bitmap_set_busy(target_bitmap, true);

Honestly, I still have bad understanding about how should we use dirty

Quoted 9 lines

bitmap mutex,
but note that bdrv_dirty_bitmap_set_busy locks the mutex. And it is (may be) possible,
that busy status of the bitmap is changed after bdrv_dirty_bitmap_check but before
bdrv_dirty_bitmap_set_busy.  So, more correct would be do both operation under one
critical section. Still, I don't know is the situation possible.

Quoted 1 line

Aren't we under the BQL here? Can we be pre-empted? :(

Seems we are. But, as it's said above dirty_bitmap_mutex declaration:

/* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
* Reading from the list can be done with either the BQL or the
* dirty_bitmap_mutex. Modifying a bitmap only requires
* dirty_bitmap_mutex. */

It means, that another thread may modify bitmap (for example its 'busy' field) taking only dirty_bitmap_mutex, which will lead to the case I described.

Quoted 64 lines
Quoted 7 lines
Quoted 4 lines

+
+    job = block_job_create(job_id, &bitpop_job_driver, txn, bs, +                           BLK_PERM_CONSISTENT_READ,

Do we need it? We are not going to read..

Copy-paste / leftover from an earlier draft where I was trying to achieve atomicity. It can be removed if we don't want the stricter atomicity.

Quoted 36 lines
Quoted 36 lines

+                           BLK_PERM_ALL & ~BLK_PERM_RESIZE, +                           0, creation_flags,
+                           cb, opaque, errp);
+    if (!job) {
+        bdrv_dirty_bitmap_set_busy(target_bitmap, false); +        bdrv_release_dirty_bitmap(new_bitmap);
+        return NULL;
+    }

block/bitmap-alloc.c new file +10 -0
+
+    job->bs = bs;
+    job->on_error = on_error;
+    job->target_bitmap = target_bitmap;
+    job->new_bitmap = new_bitmap;
+    job->len = len;
+    job_progress_set_remaining(&job->common.job, job->len);
+
+    return &job->common;
+}
blockjob.c +2 -1
index 5d63b1e89d..7e450372bd 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)
       return job_type(job) == JOB_TYPE_BACKUP ||
              job_type(job) == JOB_TYPE_COMMIT ||
              job_type(job) == JOB_TYPE_MIRROR ||
-           job_type(job) == JOB_TYPE_STREAM;
+           job_type(job) == JOB_TYPE_STREAM ||
+           job_type(job) == JOB_TYPE_BITMAP_POPULATE;
   }
     BlockJob *block_job_next(BlockJob *bjob)
block/Makefile.objs +0 -0
index 3bcb35c81d..f3cfc89d90 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o
Quoted 10 lines

block/Makefile.objs block-obj-$(CONFIG_LIBSSH) += ssh.o · line 42

Quoted 8 lines
block/Makefile.objs +1 -0
   block-obj-y += accounting.o dirty-bitmap.o
   block-obj-y += write-threshold.o
   block-obj-y += backup.o
+block-obj-y += bitmap-alloc.o
   block-obj-$(CONFIG_REPLICATION) += replication.o
   block-obj-y += throttle.o copy-on-read.o
   block-obj-y += block-copy.o

Thanks for the review. I'll start making changes, but won't send V2 just yet.

--js

Signature · 2 lines
Best regards,
Vladimir
author
#5 lore raw

Re: [PATCH 1/6] block: add bitmap-populate job

Quoted 231 lines · Vladimir Sementsov-Ogievskiy
Quoted 224 lines · John Snow
Quoted 54 lines · Vladimir Sementsov-Ogievskiy
Quoted 53 lines · John Snow

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
   qapi/block-core.json      |  48 +++++++++
   qapi/job.json             |   2 +-
   include/block/block_int.h |  21 ++++
   block/bitmap-alloc.c      | 207
++++++++++++++++++++++++++++++++++++++
   blockjob.c                |   3 +-
   block/Makefile.objs       |   1 +

   6 files changed, 280 insertions(+), 2 deletions(-)

   create mode 100644 block/bitmap-alloc.c

qapi/block-core.json +0 -0
index 85e27bb61f..df1797681a 100644
@@ -2245,6 +2245,54 @@
          { 'command': 'block-dirty-bitmap-merge',
            'data': 'BlockDirtyBitmapMerge' }

   +##

qapi/block-core.json +3 -0
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a

bitmap.

qapi/block-core.json +17 -0
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.0
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a

bitmap's

qapi/block-core.json +2 -0
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device

supports
+#            io-status (see BlockInfo).

Quoted 9 lines
Quoted 6 lines
qapi/block-core.json +2 -0
+#
+# @auto-finalize: When false, this job will wait in a PENDING state

after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.

sounds a bit strange in context of bitmap-population job

Yeah, you're right. Copy-pasted for "consistency".

Quoted 93 lines
Quoted 93 lines
qapi/block-core.json +5 -0
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state

after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query

qapi/block-core.json +13 -0
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+

   ##
   # @BlockDirtyBitmapSha256:
   #

qapi/job.json +1 -1
index 5e658281f5..5f496d4630 100644
@@ -22,7 +22,7 @@
    # Since: 1.7
    ##
    { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create',
 'bitmap-populate'] }
      ##
    # @JobStatus:
include/block/block_int.h +0 -0
index 6f9fd5e20e..a5884b597e 100644
@@ -1215,6 +1215,27 @@ BlockJob *backup_job_create(const char *job_id,
 BlockDriverState *bs,
                                BlockCompletionFunc *cb, void *opaque,
                                JobTxn *txn, Error **errp);

   +/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */

include/block/block_int.h +8 -0
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+

   void hmp_drive_add_node(Monitor *mon, const char *optstr); BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,

block/bitmap-alloc.c new file +9 -0
new file mode 100644
index 0000000000..47d542dc12
@@ -0,0 +1,207 @@
+/*
+ * Async Dirty Bitmap Populator
+ *
+ * Copyright (C) 2020 Red Hat, Inc.
+ *
+ * Authors:
+ *  John Snow <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or

later.
+ * See the COPYING file in the top-level directory.
+ *
+ */

block/bitmap-alloc.c new file +3 -0
+
+#include "qemu/osdep.h"
+
Quoted 8 lines

block/bitmap-alloc.c line 21

Quoted 6 lines
block/bitmap-alloc.c new file +6 -0
+#include "trace.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "block/blockjob_int.h"
+#include "block/block_backup.h"
+#include "block/block-copy.h"

I hope, not all includes are needed :)

Whoops, no, of course not. I copied the skeleton from backup, as you can tell ;)

Quoted 23 lines
Quoted 23 lines
block/bitmap-alloc.c new file +22 -0
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/ratelimit.h"
+#include "qemu/cutils.h"
+#include "sysemu/block-backend.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+
+typedef struct BitpopBlockJob {
+    BlockJob common;
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *target_bitmap;
+    BdrvDirtyBitmap *new_bitmap;
+    BlockdevOnError on_error;
+    uint64_t len;
+} BitpopBlockJob;
+
+static const BlockJobDriver bitpop_job_driver;
+
+static void bitpop_commit(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
Quoted 9 lines
Quoted 3 lines

+
+    bdrv_dirty_bitmap_merge_internal(s->target_bitmap, s->new_bitmap, +                                     NULL, true);

Hmm, so you populate new_bitmap, and then merge to target. Why can't we work
directly with target bitmap? The most probable case is that libvirt will create bitmap specifically to use as target in this operation, or not?

Most likely case, yes. Odds are very good it will be a brand new bitmap.

However, we already have a creation command -- I didn't want to make a second job-version of the command and then maintain two interfaces, so I made it a "merge into existing" style command instead.

Quoted 6 lines

Hmm, just to make it possible to cancel the job and keep the target bitmap in
original state? Is it really needed? I think on failure target bitmap will be
removed anyway..

You caught me being *lazy*. I copy the bitmap so I can unconditionally enable it to catch in-flight writes without having to create block graph modifications.

Quoted 5 lines

But, yes, to undo changes if we cancel.

I didn't want to make a job that was not able to be canceled. The alternative is pursuing the design where we allow new bitmaps only -- because then on cancel we can just delete them.

Quoted 8 lines

On backup job (and any other) we can't rollback target changes on cancel. So, I think it would be OK to take same semantics for the new job, keeping in
mind that it would be most probable usage case and no sense in creating additional bitmaps. And if caller needs to use existent non-empty bitmap as target and wants correct cancel, it always can create additional bitmap by itself and then merge it to actual target.

While backup can't roll back data changes, it does avoid modifying bitmaps if it didn't succeed. This is a bitmap-centric job and we have the capability to do a complete rollback.

i.e. I am thinking more about consistency with bitmap behavior than I am consistency with job behavior.

That this job becomes the only one to be able to be *fully* reversed on cancel is unique to jobs -- but is shared by all transactionable bitmap commands.

Quoted 5 lines

And why new? It's up to user, what to use as target. And user knows, that on
failure or cancel target becomes invalid and will workaround this if needed.

I think you're advocating for writing directly into the bitmap and just accepting that it's trashed afterwards, but I don't like this behavior because it limits what we can do with this job in the future.

Is this just a matter of taste? Since every other bitmap command offers some kind of rollback I felt more comfortable doing the same here.

I'm worried that if I decide to treat the result bitmap as disposable, that I will need to provide greater atomicity for the allocation status read. At the moment I rely on the temporary bitmap being *enabled* to make sure that if any new writes happen in regions I have already read that they are reflected in the bitmap anyway.

If the user passes a disabled bitmap, that guarantee is lost -- we lose point in time semantics entirely. Writes that occur further in the file are captured, but ones that occur earlier are missed.

If I make a temporary enabled bitmap, I catch everything.

I could prohibit *disabled* bitmaps from being passed in to this job, but that again makes it of a more limited use for other patterns in the future.

So I guess I would prefer to just leave this as-is for now if it's not that harmful.

Quoted 125 lines
Quoted 125 lines
Quoted 56 lines
Quoted 56 lines
block/bitmap-alloc.c new file +13 -0
+}
+
+/* no abort needed; just clean without committing. */
+
+static void bitpop_clean(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+
+    bdrv_release_dirty_bitmap(s->new_bitmap);
+    bdrv_dirty_bitmap_set_busy(s->target_bitmap, false);
+}
+
+static BlockErrorAction bitpop_error_action(BitpopBlockJob *job, int

error)
+{
+    return block_job_error_action(&job->common, job->on_error, true, error);

block/bitmap-alloc.c new file +35 -0
+}
+
+static bool coroutine_fn yield_and_check(Job *job)
+{
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    job_sleep_ns(job, 0);
+
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    return false;
+}
+
+static int coroutine_fn bitpop_run(Job *job, Error **errp)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+    int ret = 0;
+    int64_t offset;
+    int64_t count;
+    int64_t bytes;
+
+    for (offset = 0; offset < s->len; ) {
+        if (yield_and_check(job)) {
+            ret = -ECANCELED;
+            break;
+        }
+
+        bytes = s->len - offset;
+        ret = bdrv_is_allocated(s->bs, offset, bytes, &count);
+        if (ret < 0) {
+            if (bitpop_error_action(s, -ret) ==

BLOCK_ERROR_ACTION_REPORT) {
+                break;
+            }

Quoted 9 lines
Quoted 5 lines

+            continue;
+        }
+
+        if (!count) {
+            ret = 0;

Hmm, I think it's impossible case.. If so, better to make an assertion or ignore..

OK, agreed.

Quoted 56 lines
Quoted 43 lines

+            break;
+        }

block/bitmap-alloc.c new file +41 -0
+
+        if (ret) {
+            bdrv_set_dirty_bitmap(s->new_bitmap, offset, count);
+            ret = 0;
+        }
+
+        job_progress_update(job, count);
+        offset += count;
+    }
+
+    return ret;
+}
+
+static const BlockJobDriver bitpop_job_driver = {
+    .job_driver = {
+        .instance_size          = sizeof(BitpopBlockJob),
+        .job_type               = JOB_TYPE_BITMAP_POPULATE,
+        .free                   = block_job_free,
+        .user_resume            = block_job_user_resume,
+        .run                    = bitpop_run,
+        .commit                 = bitpop_commit,
+        .clean                  = bitpop_clean,
+    }
+};
+
+
+BlockJob *bitpop_job_create(
+    const char *job_id,
+    BlockDriverState *bs,
+    BdrvDirtyBitmap *target_bitmap,
+    BitmapPattern pattern,
+    BlockdevOnError on_error,
+    int creation_flags,
+    BlockCompletionFunc *cb,
+    void *opaque,
+    JobTxn *txn,
+    Error **errp)
+{
+    int64_t len;
+    BitpopBlockJob *job = NULL;
+    int64_t cluster_size;

block/bitmap-alloc.c line 156

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+    BdrvDirtyBitmap *new_bitmap = NULL;
+
+    assert(bs);
+    assert(target_bitmap);
+
+    if (!bdrv_is_inserted(bs)) {
+        error_setg(errp, "Device is not inserted: %s",
+                   bdrv_get_device_name(bs));
+        return NULL;
+    }

Why this?

Quoted 8 lines
Quoted 4 lines

I assumed there was nothing to read the allocation map *of* if there wasn't a media present.

Am I mistaken?

is_inserted checks existing of bs->drv, but bitmap operations actually doesn't need any drv.. Hmm. I'm not against this check anyway.

Maybe it would be clearer to say this:

if (pattern = BITMAP_PATTERN_ALLOCATION_TOP) {
    if (!bdrv_is_inserted(bs)) {
        ...
    }
} else {
    ...
}

We need to make sure bs->drv exists for the allocation map reading -- not for every theoretical pattern, but for this one we do.

Quoted 12 lines
Quoted 12 lines
Quoted 8 lines
Quoted 5 lines

+
+    if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { +        return NULL;
+    }

and this?

Copy-paste: I don't understand if I want a new op blocker, to re-use an op-blocker, or to have no op blocker.

Quoted 8 lines
Quoted 2 lines

Genuinely I have no idea. I should left a review comment here, I forgot about this part, sorry.

I'm for no op blocker. As I understand, op-blockers are old and should be replaced by permissions and frozen children. So, if we don't know, do we need any blocking here, better go on without it. Also, op-blockers can't block forbidden usage through filters anyway.

Sounds good to me, honestly.

Quoted 46 lines
Quoted 42 lines
Quoted 32 lines
Quoted 22 lines

+
+    if (bdrv_dirty_bitmap_check(target_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
+        return NULL;
+    }

block/bitmap-alloc.c new file +16 -0
+
+    if (pattern != BITMAP_PATTERN_ALLOCATION_TOP) {
+        error_setg(errp, "Unrecognized bitmap pattern");
+        return NULL;
+    }
+
+    len = bdrv_getlength(bs);
+    if (len < 0) {
+        error_setg_errno(errp, -len, "unable to get length for '%s'",
+                         bdrv_get_device_name(bs));
+        return NULL;
+    }
+
+    /* NB: new bitmap is anonymous and enabled */
+    cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap);
+    new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL,
Quoted 8 lines

errp);
+    if (!new_bitmap) {
+        return NULL;
+    }
+
+    /* Take ownership; we reserve the right to write into this on-commit. */
+    bdrv_dirty_bitmap_set_busy(target_bitmap, true);

Honestly, I still have bad understanding about how should we use dirty

Quoted 9 lines

bitmap mutex,
but note that bdrv_dirty_bitmap_set_busy locks the mutex. And it is (may be) possible,
that busy status of the bitmap is changed after bdrv_dirty_bitmap_check but before
bdrv_dirty_bitmap_set_busy.  So, more correct would be do both operation under one
critical section. Still, I don't know is the situation possible.

Quoted 1 line

Aren't we under the BQL here? Can we be pre-empted? :(

Seems we are. But, as it's said above dirty_bitmap_mutex declaration:

Quoted 9 lines

    /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
     * Reading from the list can be done with either the BQL or the
     * dirty_bitmap_mutex.  Modifying a bitmap only requires
     * dirty_bitmap_mutex.  */

It means, that another thread may modify bitmap (for example its 'busy' field)
taking only dirty_bitmap_mutex, which will lead to the case I described.

Alright, I'll add the locking change. Atomic check-and-lock for the busy bit sounds like a good model anyway.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

+
+    job = block_job_create(job_id, &bitpop_job_driver, txn, bs, +                           BLK_PERM_CONSISTENT_READ,

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Do we need it? We are not going to read..

Copy-paste / leftover from an earlier draft where I was trying to achieve atomicity. It can be removed if we don't want the stricter atomicity.

I guess I really don't need READ here. I guess ~RESIZE is the only important one.

Quoted 39 lines
Quoted 39 lines
Quoted 39 lines
Quoted 39 lines

+                           BLK_PERM_ALL & ~BLK_PERM_RESIZE, +                           0, creation_flags,
+                           cb, opaque, errp);
+    if (!job) {
+        bdrv_dirty_bitmap_set_busy(target_bitmap, false); +        bdrv_release_dirty_bitmap(new_bitmap);
+        return NULL;
+    }

block/bitmap-alloc.c new file +10 -0
+
+    job->bs = bs;
+    job->on_error = on_error;
+    job->target_bitmap = target_bitmap;
+    job->new_bitmap = new_bitmap;
+    job->len = len;
+    job_progress_set_remaining(&job->common.job, job->len);
+
+    return &job->common;
+}
blockjob.c +0 -0
index 5d63b1e89d..7e450372bd 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)

       return job_type(job) == JOB_TYPE_BACKUP ||

              job_type(job) == JOB_TYPE_COMMIT ||
              job_type(job) == JOB_TYPE_MIRROR ||

-           job_type(job) == JOB_TYPE_STREAM; +           job_type(job) == JOB_TYPE_STREAM || +           job_type(job) == JOB_TYPE_BITMAP_POPULATE; }
     BlockJob *block_job_next(BlockJob *bjob)

block/Makefile.objs +0 -0
index 3bcb35c81d..f3cfc89d90 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o

   block-obj-y += accounting.o dirty-bitmap.o
   block-obj-y += write-threshold.o
   block-obj-y += backup.o

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

+block-obj-y += bitmap-alloc.o block-obj-$(CONFIG_REPLICATION) += replication.o block-obj-y += throttle.o copy-on-read.o block-obj-y += block-copy.o

Thanks!

#6 lore raw

Re: [PATCH 1/6] block: add bitmap-populate job

Quoted 247 lines · John Snow
Quoted 231 lines · Vladimir Sementsov-Ogievskiy
Quoted 224 lines · John Snow
Quoted 54 lines · Vladimir Sementsov-Ogievskiy
Quoted 53 lines · John Snow

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofit into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
    qapi/block-core.json      |  48 +++++++++
    qapi/job.json             |   2 +-
    include/block/block_int.h |  21 ++++
    block/bitmap-alloc.c      | 207
++++++++++++++++++++++++++++++++++++++
    blockjob.c                |   3 +-
    block/Makefile.objs       |   1 +
    6 files changed, 280 insertions(+), 2 deletions(-)

   create mode 100644 block/bitmap-alloc.c

qapi/block-core.json +3 -0
index 85e27bb61f..df1797681a 100644
@@ -2245,6 +2245,54 @@
          { 'command': 'block-dirty-bitmap-merge',
            'data': 'BlockDirtyBitmapMerge' }
    +##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a

bitmap.

qapi/block-core.json +17 -0
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.0
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a

bitmap's

qapi/block-core.json +2 -0
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device

supports
+#            io-status (see BlockInfo).

Quoted 9 lines
Quoted 6 lines
qapi/block-core.json +2 -0
+#
+# @auto-finalize: When false, this job will wait in a PENDING state

after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.

sounds a bit strange in context of bitmap-population job

Yeah, you're right. Copy-pasted for "consistency".

Quoted 93 lines
Quoted 93 lines
qapi/block-core.json +5 -0
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state

after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query

qapi/block-core.json +13 -0
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.0
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
    ##
    # @BlockDirtyBitmapSha256:
    #
qapi/job.json +1 -1
index 5e658281f5..5f496d4630 100644
@@ -22,7 +22,7 @@
    # Since: 1.7
    ##
    { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create',
 'bitmap-populate'] }
      ##
    # @JobStatus:
include/block/block_int.h +7 -0
index 6f9fd5e20e..a5884b597e 100644
@@ -1215,6 +1215,27 @@ BlockJob *backup_job_create(const char *job_id,
 BlockDriverState *bs,
                                BlockCompletionFunc *cb, void *opaque,
                                JobTxn *txn, Error **errp);
    +/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job

lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */

include/block/block_int.h +8 -0
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+
    void hmp_drive_add_node(Monitor *mon, const char *optstr);
      BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,
block/bitmap-alloc.c new file +9 -0
new file mode 100644
index 0000000000..47d542dc12
@@ -0,0 +1,207 @@
+/*
+ * Async Dirty Bitmap Populator
+ *
+ * Copyright (C) 2020 Red Hat, Inc.
+ *
+ * Authors:
+ *  John Snow <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or

later.
+ * See the COPYING file in the top-level directory.
+ *
+ */

block/bitmap-alloc.c new file +3 -0
+
+#include "qemu/osdep.h"
+
Quoted 8 lines

block/bitmap-alloc.c line 21

Quoted 6 lines
block/bitmap-alloc.c new file +6 -0
+#include "trace.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "block/blockjob_int.h"
+#include "block/block_backup.h"
+#include "block/block-copy.h"

I hope, not all includes are needed :)

Whoops, no, of course not. I copied the skeleton from backup, as you can tell ;)

Quoted 23 lines
Quoted 23 lines
block/bitmap-alloc.c new file +22 -0
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/ratelimit.h"
+#include "qemu/cutils.h"
+#include "sysemu/block-backend.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+
+typedef struct BitpopBlockJob {
+    BlockJob common;
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *target_bitmap;
+    BdrvDirtyBitmap *new_bitmap;
+    BlockdevOnError on_error;
+    uint64_t len;
+} BitpopBlockJob;
+
+static const BlockJobDriver bitpop_job_driver;
+
+static void bitpop_commit(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
Quoted 9 lines
Quoted 3 lines

+
+    bdrv_dirty_bitmap_merge_internal(s->target_bitmap, s->new_bitmap, +                                     NULL, true);

Hmm, so you populate new_bitmap, and then merge to target. Why can't we work
directly with target bitmap? The most probable case is that libvirt will create bitmap specifically to use as target in this operation, or not?

Most likely case, yes. Odds are very good it will be a brand new bitmap.

However, we already have a creation command -- I didn't want to make a second job-version of the command and then maintain two interfaces, so I made it a "merge into existing" style command instead.

Quoted 6 lines

Hmm, just to make it possible to cancel the job and keep the target bitmap in
original state? Is it really needed? I think on failure target bitmap will be
removed anyway..

You caught me being *lazy*. I copy the bitmap so I can unconditionally enable it to catch in-flight writes without having to create block graph modifications.

Quoted 5 lines

But, yes, to undo changes if we cancel.

I didn't want to make a job that was not able to be canceled. The alternative is pursuing the design where we allow new bitmaps only -- because then on cancel we can just delete them.

Quoted 8 lines

On backup job (and any other) we can't rollback target changes on cancel. So, I think it would be OK to take same semantics for the new job, keeping in
mind that it would be most probable usage case and no sense in creating additional bitmaps. And if caller needs to use existent non-empty bitmap as target and wants correct cancel, it always can create additional bitmap by itself and then merge it to actual target.

While backup can't roll back data changes, it does avoid modifying bitmaps if it didn't succeed. This is a bitmap-centric job and we have the capability to do a complete rollback.

Quoted 6 lines

i.e. I am thinking more about consistency with bitmap behavior than I am consistency with job behavior.

That this job becomes the only one to be able to be *fully* reversed on cancel is unique to jobs -- but is shared by all transactionable bitmap commands.

Hmm, yes.. But it will be useless without a possibility to put job-completion into transaction. Still we'll need such thing anyway at some point.

Quoted 7 lines
Quoted 6 lines

And why new? It's up to user, what to use as target. And user knows, that on
failure or cancel target becomes invalid and will workaround this if needed.

Quoted 6 lines

I think you're advocating for writing directly into the bitmap and just accepting that it's trashed afterwards, but I don't like this behavior because it limits what we can do with this job in the future.

Is this just a matter of taste? Since every other bitmap command offers some kind of rollback I felt more comfortable doing the same here.

OK

Quoted 10 lines

I'm worried that if I decide to treat the result bitmap as disposable, that I will need to provide greater atomicity for the allocation status read. At the moment I rely on the temporary bitmap being *enabled* to make sure that if any new writes happen in regions I have already read that they are reflected in the bitmap anyway.

If the user passes a disabled bitmap, that guarantee is lost -- we lose point in time semantics entirely. Writes that occur further in the file are captured, but ones that occur earlier are missed.

Still, if user pass disabled bitmap, it will be invalid immediately after job finish.

Quoted 9 lines

If I make a temporary enabled bitmap, I catch everything.

I could prohibit *disabled* bitmaps from being passed in to this job, but that again makes it of a more limited use for other patterns in the future.

So I guess I would prefer to just leave this as-is for now if it's not that harmful.

OK, I agreed, because of semantics of other bitmap-related commands.

Quoted 145 lines
Quoted 126 lines
Quoted 126 lines
Quoted 57 lines
Quoted 57 lines
block/bitmap-alloc.c new file +13 -0
+}
+
+/* no abort needed; just clean without committing. */
+
+static void bitpop_clean(Job *job)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+
+    bdrv_release_dirty_bitmap(s->new_bitmap);
+    bdrv_dirty_bitmap_set_busy(s->target_bitmap, false);
+}
+
+static BlockErrorAction bitpop_error_action(BitpopBlockJob *job, int

error)
+{
+    return block_job_error_action(&job->common, job->on_error, true, error);

block/bitmap-alloc.c new file +35 -0
+}
+
+static bool coroutine_fn yield_and_check(Job *job)
+{
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    job_sleep_ns(job, 0);
+
+    if (job_is_cancelled(job)) {
+        return true;
+    }
+
+    return false;
+}
+
+static int coroutine_fn bitpop_run(Job *job, Error **errp)
+{
+    BitpopBlockJob *s = container_of(job, BitpopBlockJob, common.job);
+    int ret = 0;
+    int64_t offset;
+    int64_t count;
+    int64_t bytes;
+
+    for (offset = 0; offset < s->len; ) {
+        if (yield_and_check(job)) {
+            ret = -ECANCELED;
+            break;
+        }
+
+        bytes = s->len - offset;
+        ret = bdrv_is_allocated(s->bs, offset, bytes, &count);
+        if (ret < 0) {
+            if (bitpop_error_action(s, -ret) ==

BLOCK_ERROR_ACTION_REPORT) {
+                break;
+            }

Quoted 9 lines
Quoted 5 lines

+            continue;
+        }
+
+        if (!count) {
+            ret = 0;

Hmm, I think it's impossible case.. If so, better to make an assertion or ignore..

OK, agreed.

Quoted 56 lines
Quoted 43 lines

+            break;
+        }

block/bitmap-alloc.c new file +41 -0
+
+        if (ret) {
+            bdrv_set_dirty_bitmap(s->new_bitmap, offset, count);
+            ret = 0;
+        }
+
+        job_progress_update(job, count);
+        offset += count;
+    }
+
+    return ret;
+}
+
+static const BlockJobDriver bitpop_job_driver = {
+    .job_driver = {
+        .instance_size          = sizeof(BitpopBlockJob),
+        .job_type               = JOB_TYPE_BITMAP_POPULATE,
+        .free                   = block_job_free,
+        .user_resume            = block_job_user_resume,
+        .run                    = bitpop_run,
+        .commit                 = bitpop_commit,
+        .clean                  = bitpop_clean,
+    }
+};
+
+
+BlockJob *bitpop_job_create(
+    const char *job_id,
+    BlockDriverState *bs,
+    BdrvDirtyBitmap *target_bitmap,
+    BitmapPattern pattern,
+    BlockdevOnError on_error,
+    int creation_flags,
+    BlockCompletionFunc *cb,
+    void *opaque,
+    JobTxn *txn,
+    Error **errp)
+{
+    int64_t len;
+    BitpopBlockJob *job = NULL;
+    int64_t cluster_size;

block/bitmap-alloc.c line 156

Quoted 10 lines
block/bitmap-alloc.c new file +10 -0
+    BdrvDirtyBitmap *new_bitmap = NULL;
+
+    assert(bs);
+    assert(target_bitmap);
+
+    if (!bdrv_is_inserted(bs)) {
+        error_setg(errp, "Device is not inserted: %s",
+                   bdrv_get_device_name(bs));
+        return NULL;
+    }

Why this?

Quoted 8 lines
Quoted 4 lines

I assumed there was nothing to read the allocation map *of* if there wasn't a media present.

Am I mistaken?

is_inserted checks existing of bs->drv, but bitmap operations actually doesn't need any drv.. Hmm. I'm not against this check anyway.

Maybe it would be clearer to say this:

if (pattern = BITMAP_PATTERN_ALLOCATION_TOP) {
     if (!bdrv_is_inserted(bs)) {
         ...
     }
} else {
     ...
}
Quoted 2 lines

We need to make sure bs->drv exists for the allocation map reading -- not for every theoretical pattern, but for this one we do.

We can leave this as is now, I don't want to care much. Still I just doubt that this check is needed in any job. What are the cases when we have bs without drv? Corrupted qcow2? And where is guarantee that we will not lose drv during the job?

Quoted 91 lines
Quoted 13 lines
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines

+
+    if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) { +        return NULL;
+    }

and this?

Copy-paste: I don't understand if I want a new op blocker, to re-use an op-blocker, or to have no op blocker.

Quoted 8 lines
Quoted 2 lines

Genuinely I have no idea. I should left a review comment here, I forgot about this part, sorry.

I'm for no op blocker. As I understand, op-blockers are old and should be replaced by permissions and frozen children. So, if we don't know, do we need any blocking here, better go on without it. Also, op-blockers can't block forbidden usage through filters anyway.

Sounds good to me, honestly.

Quoted 46 lines
Quoted 42 lines
Quoted 32 lines
Quoted 22 lines

+
+    if (bdrv_dirty_bitmap_check(target_bitmap, BDRV_BITMAP_DEFAULT, errp)) {
+        return NULL;
+    }

block/bitmap-alloc.c new file +16 -0
+
+    if (pattern != BITMAP_PATTERN_ALLOCATION_TOP) {
+        error_setg(errp, "Unrecognized bitmap pattern");
+        return NULL;
+    }
+
+    len = bdrv_getlength(bs);
+    if (len < 0) {
+        error_setg_errno(errp, -len, "unable to get length for '%s'",
+                         bdrv_get_device_name(bs));
+        return NULL;
+    }
+
+    /* NB: new bitmap is anonymous and enabled */
+    cluster_size = bdrv_dirty_bitmap_granularity(target_bitmap);
+    new_bitmap = bdrv_create_dirty_bitmap(bs, cluster_size, NULL,
Quoted 8 lines

errp);
+    if (!new_bitmap) {
+        return NULL;
+    }
+
+    /* Take ownership; we reserve the right to write into this on-commit. */
+    bdrv_dirty_bitmap_set_busy(target_bitmap, true);

Honestly, I still have bad understanding about how should we use dirty

Quoted 9 lines

bitmap mutex,
but note that bdrv_dirty_bitmap_set_busy locks the mutex. And it is (may be) possible,
that busy status of the bitmap is changed after bdrv_dirty_bitmap_check but before
bdrv_dirty_bitmap_set_busy.  So, more correct would be do both operation under one
critical section. Still, I don't know is the situation possible.

Quoted 1 line

Aren't we under the BQL here? Can we be pre-empted? :(

Seems we are. But, as it's said above dirty_bitmap_mutex declaration:

Quoted 9 lines

    /* Writing to the list requires the BQL _and_ the dirty_bitmap_mutex.
     * Reading from the list can be done with either the BQL or the
     * dirty_bitmap_mutex.  Modifying a bitmap only requires
     * dirty_bitmap_mutex.  */

It means, that another thread may modify bitmap (for example its 'busy' field)
taking only dirty_bitmap_mutex, which will lead to the case I described.

Alright, I'll add the locking change. Atomic check-and-lock for the busy bit sounds like a good model anyway.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

+
+    job = block_job_create(job_id, &bitpop_job_driver, txn, bs, +                           BLK_PERM_CONSISTENT_READ,

Do we need it? We are not going to read..

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

Copy-paste / leftover from an earlier draft where I was trying to achieve atomicity. It can be removed if we don't want the stricter atomicity.

I guess I really don't need READ here. I guess ~RESIZE is the only important one.

Agree

Quoted 51 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines
Quoted 40 lines

+                           BLK_PERM_ALL & ~BLK_PERM_RESIZE, +                           0, creation_flags,
+                           cb, opaque, errp);
+    if (!job) {
+        bdrv_dirty_bitmap_set_busy(target_bitmap, false); +        bdrv_release_dirty_bitmap(new_bitmap);
+        return NULL;
+    }

block/bitmap-alloc.c new file +10 -0
+
+    job->bs = bs;
+    job->on_error = on_error;
+    job->target_bitmap = target_bitmap;
+    job->new_bitmap = new_bitmap;
+    job->len = len;
+    job_progress_set_remaining(&job->common.job, job->len);
+
+    return &job->common;
+}
blockjob.c +2 -1
index 5d63b1e89d..7e450372bd 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)
        return job_type(job) == JOB_TYPE_BACKUP ||
               job_type(job) == JOB_TYPE_COMMIT ||
               job_type(job) == JOB_TYPE_MIRROR ||
-           job_type(job) == JOB_TYPE_STREAM;
+           job_type(job) == JOB_TYPE_STREAM ||
+           job_type(job) == JOB_TYPE_BITMAP_POPULATE;
    }
      BlockJob *block_job_next(BlockJob *bjob)
block/Makefile.objs +0 -0
index 3bcb35c81d..f3cfc89d90 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o
    block-obj-y += accounting.o dirty-bitmap.o
    block-obj-y += write-threshold.o
    block-obj-y += backup.o
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

block/Makefile.objs block-obj-$(CONFIG_LIBSSH) += ssh.o · line 42

Quoted 5 lines
block/Makefile.objs +1 -0
+block-obj-y += bitmap-alloc.o
    block-obj-$(CONFIG_REPLICATION) += replication.o
    block-obj-y += throttle.o copy-on-read.o
    block-obj-y += block-copy.o

Thanks!

Signature · 2 lines
Best regards,
Vladimir
author
#7 lore raw

Re: [PATCH 1/6] block: add bitmap-populate job

Quoted 3 lines · Vladimir Sementsov-Ogievskiy

Still, if user pass disabled bitmap, it will be invalid immediately after job finish.

Yes ... In truth, I really want to augment this job to provide a defined point-in-time semantic so it can be fully useful in all circumstances.

At the moment, the point-in-time it provides is "at finalize", which may or may not be the single most useful semantic. (It matches mirror -- because that was easier.)

The other option is "at start" which matches backup, but will require a new filter and some changes to permissions. That was more invasive, so I avoided it for now.

(In the interest of a speedy resolution for libvirt, can we add a critical_moment=[START|FINALIZE] option to this job *later*?)

author
#8 lore raw

PATCH 2/6 qmp: expose block-dirty-bitmap-populate

In reply to John Snow

This is a new job-creating command.

---
 qapi/block-core.json  | 18 ++++++++++
 qapi/transaction.json |  2 ++
 blockdev.c            | 78 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 98 insertions(+)
qapi/block-core.json +18 -0
index df1797681a..a8be1fb36b 100644
@@ -2293,6 +2293,24 @@
             '*auto-finalize': 'bool',
             '*auto-dismiss': 'bool' } }
 
+##
+# @block-dirty-bitmap-populate:
+#
+# Creates a new job that writes a pattern into a dirty bitmap.
+#
+# Since: 5.0
+#
+# Example:
+#
+# -> { "execute": "block-dirty-bitmap-populate",
+#      "arguments": { "node": "drive0", "target": "bitmap0",
+#                     "job-id": "job0", "pattern": "allocate-top" } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'block-dirty-bitmap-populate', 'boxed': true,
+  'data': 'BlockDirtyBitmapPopulate' }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/transaction.json +2 -0
index 04301f1be7..28521d5c7f 100644
@@ -50,6 +50,7 @@
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0
 # - @block-dirty-bitmap-merge: since 4.0
+# - @block-dirty-bitmap-populate: since 5.0
 # - @blockdev-backup: since 2.3
 # - @blockdev-snapshot: since 2.5
 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -67,6 +68,7 @@
        'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge',
+       'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
        'blockdev-backup': 'BlockdevBackup',
        'blockdev-snapshot': 'BlockdevSnapshot',
        'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +78 -0

96 lines not on this page · raw message

Signature · 1 line
2.21.1
#9 lore raw

Re: [PATCH 2/6] qmp: expose block-dirty-bitmap-populate

Quoted 63 lines · John Snow

This is a new job-creating command.

---
  qapi/block-core.json  | 18 ++++++++++
  qapi/transaction.json |  2 ++
  blockdev.c            | 78 +++++++++++++++++++++++++++++++++++++++++++
  3 files changed, 98 insertions(+)
qapi/block-core.json +18 -0
index df1797681a..a8be1fb36b 100644
@@ -2293,6 +2293,24 @@
              '*auto-finalize': 'bool',
              '*auto-dismiss': 'bool' } }
 
+##
+# @block-dirty-bitmap-populate:
+#
+# Creates a new job that writes a pattern into a dirty bitmap.
+#
+# Since: 5.0
+#
+# Example:
+#
+# -> { "execute": "block-dirty-bitmap-populate",
+#      "arguments": { "node": "drive0", "target": "bitmap0",
+#                     "job-id": "job0", "pattern": "allocate-top" } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'block-dirty-bitmap-populate', 'boxed': true,
+  'data': 'BlockDirtyBitmapPopulate' }
+
  ##
  # @BlockDirtyBitmapSha256:
  #
qapi/transaction.json +2 -0
index 04301f1be7..28521d5c7f 100644
@@ -50,6 +50,7 @@
  # - @block-dirty-bitmap-enable: since 4.0
  # - @block-dirty-bitmap-disable: since 4.0
  # - @block-dirty-bitmap-merge: since 4.0
+# - @block-dirty-bitmap-populate: since 5.0
  # - @blockdev-backup: since 2.3
  # - @blockdev-snapshot: since 2.5
  # - @blockdev-snapshot-internal-sync: since 1.7
@@ -67,6 +68,7 @@
         'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
         'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
         'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge',
+       'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
         'blockdev-backup': 'BlockdevBackup',
         'blockdev-snapshot': 'BlockdevSnapshot',
         'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +0 -0
index 011dcfec27..33c0e35399 100644

blockdev.c static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2317

Quoted 4 lines
blockdev.c +1 -0
@@ -2314,6 +2314,67 @@ static void block_dirty_bitmap_remove_commit(BlkActionState *common)
      bdrv_release_dirty_bitmap(state->bitmap);
  }
 
+static void block_dirty_bitmap_populate_prepare(BlkActionState *common, Error **errp)

over80 line (not the only)

blockdev.c static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2319

Quoted 2 lines
blockdev.c +2 -0
+{
+    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);

At first glance using *Backup* looks like a mistake. May be rename it, or at least add a comment.

Quoted 9 lines
blockdev.c +9 -0
+    BlockDirtyBitmapPopulate *bitpop;
+    BlockDriverState *bs;
+    AioContext *aio_context;
+    BdrvDirtyBitmap *bmap = NULL;
+    int job_flags = JOB_DEFAULT;
+
+    assert(common->action->type == TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE);
+    bitpop = common->action->u.block_dirty_bitmap_populate.data;
+

blockdev.c static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2338

Quoted 10 lines
blockdev.c +10 -0
+    bs = bdrv_lookup_bs(bitpop->node, bitpop->node, errp);
+    if (!bs) {
+        return;
+    }
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+    state->bs = bs;
+
+    bmap = bdrv_find_dirty_bitmap(bs, bitpop->name);

Could we use block_dirty_bitmap_lookup ?

blockdev.c static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2341

Quoted 3 lines
blockdev.c +3 -0
+    if (!bmap) {
+        error_setg(errp, "Bitmap '%s' could not be found", bitpop->name);
+        return;

aio context lock leaked

Quoted 71 lines
blockdev.c +53 -0
+    }
+
+    /* Paired with .clean() */
+    bdrv_drained_begin(state->bs);
+
+    if (!bitpop->has_on_error) {
+        bitpop->on_error = BLOCKDEV_ON_ERROR_REPORT;
+    }
+    if (!bitpop->has_auto_finalize) {
+        bitpop->auto_finalize = true;
+    }
+    if (!bitpop->has_auto_dismiss) {
+        bitpop->auto_dismiss = true;
+    }
+
+    if (!bitpop->auto_finalize) {
+        job_flags |= JOB_MANUAL_FINALIZE;
+    }
+    if (!bitpop->auto_dismiss) {
+        job_flags |= JOB_MANUAL_DISMISS;
+    }
+
+    state->job = bitpop_job_create(
+        bitpop->job_id,
+        bs,
+        bmap,
+        bitpop->pattern,
+        bitpop->on_error,
+        job_flags,
+        NULL, NULL,
+        common->block_job_txn,
+        errp);
+
+    aio_context_release(aio_context);
+}
+
  static void abort_prepare(BlkActionState *common, Error **errp)
  {
      error_setg(errp, "Transaction aborted using Abort action");
@@ -2397,6 +2458,13 @@ static const BlkActionOps actions[] = {
          .commit = block_dirty_bitmap_remove_commit,
          .abort = block_dirty_bitmap_remove_abort,
      },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE] = {
+        .instance_size = sizeof(BlockdevBackupState),
+        .prepare = block_dirty_bitmap_populate_prepare,
+        .commit = blockdev_backup_commit,
+        .abort = blockdev_backup_abort,
+        .clean = blockdev_backup_clean,
+    },
      /* Where are transactions for MIRROR, COMMIT and STREAM?
       * Although these blockjobs use transaction callbacks like the backup job,
       * these jobs do not necessarily adhere to transaction semantics.
@@ -3225,6 +3293,16 @@ void qmp_block_dirty_bitmap_merge(const char *node, const char *target,
      do_block_dirty_bitmap_merge(node, target, bitmaps, NULL, errp);
  }
 
+void qmp_block_dirty_bitmap_populate(BlockDirtyBitmapPopulate *bitpop,
+                                     Error **errp)
+{
+    TransactionAction action = {
+        .type = TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE,
+        .u.block_dirty_bitmap_populate.data = bitpop,
+    };
+    blockdev_do_action(&action, errp);
+}
+
  BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256(const char *node,
                                                                const char *name,
                                                                Error **errp)
Signature · 2 lines
Best regards,
Vladimir
author
#10 lore raw

Re: [PATCH 2/6] qmp: expose block-dirty-bitmap-populate

Quoted 62 lines · Vladimir Sementsov-Ogievskiy
Quoted 61 lines · John Snow

This is a new job-creating command.

---
  qapi/block-core.json  | 18 ++++++++++ qapi/transaction.json |  2 ++
  blockdev.c            | 78 +++++++++++++++++++++++++++++++++++++++++++

  3 files changed, 98 insertions(+)
qapi/block-core.json +0 -0
index df1797681a..a8be1fb36b 100644
@@ -2293,6 +2293,24 @@
               '*auto-finalize': 'bool',
               '*auto-dismiss': 'bool' } }

  +##

qapi/block-core.json +17 -0
+# @block-dirty-bitmap-populate:
+#
+# Creates a new job that writes a pattern into a dirty bitmap.
+#
+# Since: 5.0
+#
+# Example:
+#
+# -> { "execute": "block-dirty-bitmap-populate",
+#      "arguments": { "node": "drive0", "target": "bitmap0",
+#                     "job-id": "job0", "pattern": "allocate-top" } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'block-dirty-bitmap-populate', 'boxed': true,
+  'data': 'BlockDirtyBitmapPopulate' }
+

  ##
  # @BlockDirtyBitmapSha256:
  #

qapi/transaction.json +2 -0
index 04301f1be7..28521d5c7f 100644
@@ -50,6 +50,7 @@
   # - @block-dirty-bitmap-enable: since 4.0
   # - @block-dirty-bitmap-disable: since 4.0
   # - @block-dirty-bitmap-merge: since 4.0
+# - @block-dirty-bitmap-populate: since 5.0
   # - @blockdev-backup: since 2.3
   # - @blockdev-snapshot: since 2.5
   # - @blockdev-snapshot-internal-sync: since 1.7
@@ -67,6 +68,7 @@
          'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
          'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
          'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge',
+       'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
          'blockdev-backup': 'BlockdevBackup',
          'blockdev-snapshot': 'BlockdevSnapshot',
          'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +0 -0
index 011dcfec27..33c0e35399 100644
Quoted 9 lines
Quoted 6 lines
blockdev.c +0 -0
@@ -2314,6 +2314,67 @@ static void
 block_dirty_bitmap_remove_commit(BlkActionState *common)
       bdrv_release_dirty_bitmap(state->bitmap);
   }

  +static void block_dirty_bitmap_populate_prepare(BlkActionState *common, Error **errp)

over80 line (not the only)

OK, will fix all instances.

Quoted 7 lines
Quoted 3 lines

+{
+    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);

At first glance using *Backup* looks like a mistake. May be rename it, or at least add a comment.

You're right, it's tricky. A rename would be helpful.

Quoted 14 lines
Quoted 14 lines

+    BlockDirtyBitmapPopulate *bitpop;
+    BlockDriverState *bs;
+    AioContext *aio_context;
+    BdrvDirtyBitmap *bmap = NULL;
+    int job_flags = JOB_DEFAULT;
+
+    assert(common->action->type == TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE);
+    bitpop = common->action->u.block_dirty_bitmap_populate.data; +
+    bs = bdrv_lookup_bs(bitpop->node, bitpop->node, errp); +    if (!bs) {
+        return;
+    }

Quoted 9 lines

blockdev.c static void block_dirty_bitmap_remove_commit(BlkActionState *common) · line 2338

Quoted 6 lines
blockdev.c +6 -0
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
+    state->bs = bs;
+
+    bmap = bdrv_find_dirty_bitmap(bs, bitpop->name);

Could we use block_dirty_bitmap_lookup ?

Yes.

Quoted 7 lines
Quoted 4 lines

+    if (!bmap) {
+        error_setg(errp, "Bitmap '%s' could not be found", bitpop->name);
+        return;

aio context lock leaked

Good spot.

Quoted 77 lines
Quoted 75 lines

+    }

blockdev.c +35 -0
+
+    /* Paired with .clean() */
+    bdrv_drained_begin(state->bs);
+
+    if (!bitpop->has_on_error) {
+        bitpop->on_error = BLOCKDEV_ON_ERROR_REPORT;
+    }
+    if (!bitpop->has_auto_finalize) {
+        bitpop->auto_finalize = true;
+    }
+    if (!bitpop->has_auto_dismiss) {
+        bitpop->auto_dismiss = true;
+    }
+
+    if (!bitpop->auto_finalize) {
+        job_flags |= JOB_MANUAL_FINALIZE;
+    }
+    if (!bitpop->auto_dismiss) {
+        job_flags |= JOB_MANUAL_DISMISS;
+    }
+
+    state->job = bitpop_job_create(
+        bitpop->job_id,
+        bs,
+        bmap,
+        bitpop->pattern,
+        bitpop->on_error,
+        job_flags,
+        NULL, NULL,
+        common->block_job_txn,
+        errp);
+
+    aio_context_release(aio_context);
+}
+

  static void abort_prepare(BlkActionState *common, Error **errp) {
      error_setg(errp, "Transaction aborted using Abort action");

blockdev.c +7 -0
@@ -2397,6 +2458,13 @@ static const BlkActionOps actions[] = {
           .commit = block_dirty_bitmap_remove_commit,
           .abort = block_dirty_bitmap_remove_abort,
       },
+    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE] = {
+        .instance_size = sizeof(BlockdevBackupState),
+        .prepare = block_dirty_bitmap_populate_prepare,
+        .commit = blockdev_backup_commit,
+        .abort = blockdev_backup_abort,
+        .clean = blockdev_backup_clean,
+    },
       /* Where are transactions for MIRROR, COMMIT and STREAM?
        * Although these blockjobs use transaction callbacks like the
 backup job,

       * these jobs do not necessarily adhere to transaction semantics.

blockdev.c +9 -0
@@ -3225,6 +3293,16 @@ void qmp_block_dirty_bitmap_merge(const char
 *node, const char *target,
       do_block_dirty_bitmap_merge(node, target, bitmaps, NULL, errp);
   }
   +void qmp_block_dirty_bitmap_populate(BlockDirtyBitmapPopulate *bitpop,
+                                     Error **errp)
+{
+    TransactionAction action = {
+        .type = TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE,
+        .u.block_dirty_bitmap_populate.data = bitpop,
+    };
+    blockdev_do_action(&action, errp);
+}
+

  BlockDirtyBitmapSha256 *qmp_x_debug_block_dirty_bitmap_sha256(const char *node,
                                                                const char *name,
                                                                Error **errp)

Signature · 1 line
—js
author
#11 lore raw

PATCH 3/6 iotests: move bitmap helpers into their own file

In reply to John Snow
---
 tests/qemu-iotests/257        | 110 +---------------------------
 tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++
 2 files changed, 132 insertions(+), 109 deletions(-)
 create mode 100644 tests/qemu-iotests/bitmaps.py
tests/qemu-iotests/257 +1 -109
index 004a433b8b..2a81f9e30c 100755
@@ -24,120 +24,12 @@ import os
 
 import iotests
 from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS
 
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
 
-class Pattern:
-    def __init__(self, byte, offset, size=GRANULARITY):
-        self.byte = byte
-        self.offset = offset
-        self.size = size
-
-    def bits(self, granularity):
-        lower = self.offset // granularity
-        upper = (self.offset + self.size - 1) // granularity
-        return set(range(lower, upper + 1))
-
-
-class PatternGroup:
-    """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
-    def __init__(self, patterns):
-        self.patterns = patterns
-
-    def bits(self, granularity):
-        """Calculate the unique bits dirtied by this pattern grouping"""
-        res = set()
-        for pattern in self.patterns:
-            res |= pattern.bits(granularity)
-        return res
-
-
-GROUPS = [
-    PatternGroup([
-        # Batch 0: 4 clusters
-        Pattern('0x49', 0x0000000),
-        Pattern('0x6c', 0x0100000),   # 1M
-        Pattern('0x6f', 0x2000000),   # 32M
-        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
-    PatternGroup([
-        # Batch 1: 6 clusters (3 new)
-        Pattern('0x65', 0x0000000),   # Full overwrite
-        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
-        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
-        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
-    PatternGroup([
-        # Batch 2: 7 clusters (3 new)
-        Pattern('0x74', 0x0010000),   # Adjacent-right
-        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
-        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
-        Pattern('0x67', 0x3fe0000,
-                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
-    PatternGroup([
-        # Batch 3: 8 clusters (5 new)
-        # Carefully chosen such that nothing re-dirties the one cluster
-        # that copies out successfully before failure in Group #1.
-        Pattern('0xaa', 0x0010000,
-                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
-        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
-        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
-        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
-]
-
-
-class EmulatedBitmap:
-    def __init__(self, granularity=GRANULARITY):
-        self._bits = set()
-        self.granularity = granularity
-
-    def dirty_bits(self, bits):
-        self._bits |= set(bits)
-
-    def dirty_group(self, n):
-        self.dirty_bits(GROUPS[n].bits(self.granularity))
-
-    def clear(self):
-        self._bits = set()
-
-    def clear_bits(self, bits):
-        self._bits -= set(bits)
-
-    def clear_bit(self, bit):
-        self.clear_bits({bit})
-
-    def clear_group(self, n):
-        self.clear_bits(GROUPS[n].bits(self.granularity))
-
-    @property
-    def first_bit(self):
-        return sorted(self.bits)[0]
-
-    @property
-    def bits(self):
-        return self._bits
-
-    @property
-    def count(self):
-        return len(self.bits)
-
-    def compare(self, qmp_bitmap):
-        """
-        Print a nice human-readable message checking that a bitmap as reported
-        by the QMP interface has as many bits set as we expect it to.
-        """
-
-        name = qmp_bitmap.get('name', '(anonymous)')
-        log("= Checking Bitmap {:s} =".format(name))
-
-        want = self.count
-        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
-
-        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
-            want, have, "OK!" if want == have else "ERROR!"))
-        log('')
-
-
 class Drive:
     """Represents, vaguely, a drive attached to a VM.
     Includes format, graph, and device information."""
tests/qemu-iotests/bitmaps.py new file +131 -0

131 lines not on this page · raw message

Signature · 1 line
2.21.1
Reviewed-by on v1 3/6
#12 lore raw

Re: [PATCH 3/6] iotests: move bitmap helpers into their own file

Quoted 12 lines · John Snow
---
  tests/qemu-iotests/257        | 110 +---------------------------
  tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++
  2 files changed, 132 insertions(+), 109 deletions(-)

create mode 100644 tests/qemu-iotests/bitmaps.py

tests/qemu-iotests/257 +0 -0
index 004a433b8b..2a81f9e30c 100755

tests/qemu-iotests/257 import os · line 27

Quoted 4 lines
tests/qemu-iotests/257 +1 -0
@@ -24,120 +24,12 @@ import os
 
  import iotests
  from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS

Clean code movement, no changes. If test passes, it should be correct :)

The only thing: I'd prefer not exporting global variables and use bitmaps.GROUPS instead (even then, it's not very good interface but..)

with or without it:

Quoted 255 lines
tests/qemu-iotests/257 +0 -109
  SIZE = 64 * 1024 * 1024
  GRANULARITY = 64 * 1024
 
 
-class Pattern:
-    def __init__(self, byte, offset, size=GRANULARITY):
-        self.byte = byte
-        self.offset = offset
-        self.size = size
-
-    def bits(self, granularity):
-        lower = self.offset // granularity
-        upper = (self.offset + self.size - 1) // granularity
-        return set(range(lower, upper + 1))
-
-
-class PatternGroup:
-    """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
-    def __init__(self, patterns):
-        self.patterns = patterns
-
-    def bits(self, granularity):
-        """Calculate the unique bits dirtied by this pattern grouping"""
-        res = set()
-        for pattern in self.patterns:
-            res |= pattern.bits(granularity)
-        return res
-
-
-GROUPS = [
-    PatternGroup([
-        # Batch 0: 4 clusters
-        Pattern('0x49', 0x0000000),
-        Pattern('0x6c', 0x0100000),   # 1M
-        Pattern('0x6f', 0x2000000),   # 32M
-        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
-    PatternGroup([
-        # Batch 1: 6 clusters (3 new)
-        Pattern('0x65', 0x0000000),   # Full overwrite
-        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
-        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
-        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
-    PatternGroup([
-        # Batch 2: 7 clusters (3 new)
-        Pattern('0x74', 0x0010000),   # Adjacent-right
-        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
-        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
-        Pattern('0x67', 0x3fe0000,
-                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
-    PatternGroup([
-        # Batch 3: 8 clusters (5 new)
-        # Carefully chosen such that nothing re-dirties the one cluster
-        # that copies out successfully before failure in Group #1.
-        Pattern('0xaa', 0x0010000,
-                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
-        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
-        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
-        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
-]
-
-
-class EmulatedBitmap:
-    def __init__(self, granularity=GRANULARITY):
-        self._bits = set()
-        self.granularity = granularity
-
-    def dirty_bits(self, bits):
-        self._bits |= set(bits)
-
-    def dirty_group(self, n):
-        self.dirty_bits(GROUPS[n].bits(self.granularity))
-
-    def clear(self):
-        self._bits = set()
-
-    def clear_bits(self, bits):
-        self._bits -= set(bits)
-
-    def clear_bit(self, bit):
-        self.clear_bits({bit})
-
-    def clear_group(self, n):
-        self.clear_bits(GROUPS[n].bits(self.granularity))
-
-    @property
-    def first_bit(self):
-        return sorted(self.bits)[0]
-
-    @property
-    def bits(self):
-        return self._bits
-
-    @property
-    def count(self):
-        return len(self.bits)
-
-    def compare(self, qmp_bitmap):
-        """
-        Print a nice human-readable message checking that a bitmap as reported
-        by the QMP interface has as many bits set as we expect it to.
-        """
-
-        name = qmp_bitmap.get('name', '(anonymous)')
-        log("= Checking Bitmap {:s} =".format(name))
-
-        want = self.count
-        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
-
-        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
-            want, have, "OK!" if want == have else "ERROR!"))
-        log('')
-
-
  class Drive:
      """Represents, vaguely, a drive attached to a VM.
      Includes format, graph, and device information."""
tests/qemu-iotests/bitmaps.py new file +131 -0
new file mode 100644
index 0000000000..522fc25171
@@ -0,0 +1,131 @@
+# Bitmap-related helper utilities
+#
+# Copyright (c) 2020 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+from iotests import log
+
+GRANULARITY = 64 * 1024
+
+
+class Pattern:
+    def __init__(self, byte, offset, size=GRANULARITY):
+        self.byte = byte
+        self.offset = offset
+        self.size = size
+
+    def bits(self, granularity):
+        lower = self.offset // granularity
+        upper = (self.offset + self.size - 1) // granularity
+        return set(range(lower, upper + 1))
+
+
+class PatternGroup:
+    """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
+    def __init__(self, patterns):
+        self.patterns = patterns
+
+    def bits(self, granularity):
+        """Calculate the unique bits dirtied by this pattern grouping"""
+        res = set()
+        for pattern in self.patterns:
+            res |= pattern.bits(granularity)
+        return res
+
+
+GROUPS = [
+    PatternGroup([
+        # Batch 0: 4 clusters
+        Pattern('0x49', 0x0000000),
+        Pattern('0x6c', 0x0100000),   # 1M
+        Pattern('0x6f', 0x2000000),   # 32M
+        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
+    PatternGroup([
+        # Batch 1: 6 clusters (3 new)
+        Pattern('0x65', 0x0000000),   # Full overwrite
+        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
+        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
+        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
+    PatternGroup([
+        # Batch 2: 7 clusters (3 new)
+        Pattern('0x74', 0x0010000),   # Adjacent-right
+        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
+        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
+        Pattern('0x67', 0x3fe0000,
+                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
+    PatternGroup([
+        # Batch 3: 8 clusters (5 new)
+        # Carefully chosen such that nothing re-dirties the one cluster
+        # that copies out successfully before failure in Group #1.
+        Pattern('0xaa', 0x0010000,
+                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
+        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
+        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
+        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
+]
+
+
+class EmulatedBitmap:
+    def __init__(self, granularity=GRANULARITY):
+        self._bits = set()
+        self.granularity = granularity
+
+    def dirty_bits(self, bits):
+        self._bits |= set(bits)
+
+    def dirty_group(self, n):
+        self.dirty_bits(GROUPS[n].bits(self.granularity))
+
+    def clear(self):
+        self._bits = set()
+
+    def clear_bits(self, bits):
+        self._bits -= set(bits)
+
+    def clear_bit(self, bit):
+        self.clear_bits({bit})
+
+    def clear_group(self, n):
+        self.clear_bits(GROUPS[n].bits(self.granularity))
+
+    @property
+    def first_bit(self):
+        return sorted(self.bits)[0]
+
+    @property
+    def bits(self):
+        return self._bits
+
+    @property
+    def count(self):
+        return len(self.bits)
+
+    def compare(self, qmp_bitmap):
+        """
+        Print a nice human-readable message checking that a bitmap as reported
+        by the QMP interface has as many bits set as we expect it to.
+        """
+
+        name = qmp_bitmap.get('name', '(anonymous)')
+        log("= Checking Bitmap {:s} =".format(name))
+
+        want = self.count
+        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
+
+        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
+            want, have, "OK!" if want == have else "ERROR!"))
+        log('')
Signature · 2 lines
Best regards,
Vladimir
author
#13 lore raw

Re: [PATCH 3/6] iotests: move bitmap helpers into their own file

Quoted 8 lines · Vladimir Sementsov-Ogievskiy

Clean code movement, no changes. If test passes, it should be correct :)

The only thing: I'd prefer not exporting global variables and use bitmaps.GROUPS instead (even then, it's not very good interface but..)

with or without it:

Hm, yeah -- it's not great, it's definitely just a bit of a quick shuffle instead of a more serious effort at a refactor to provide bitmap services for all tests.

I don't think it's worth the time just yet to make a serious attempt at making this module bigger -- but maybe there is work that can be done at providing bitmap management services targeted for use outside of testing code. (i.e. into the python/ folder somewhere.)

Well, I can just do this:
"
import bitmaps
from bitmaps import EmulatedBitmap
"

and bitmaps.GROUPS makes it a little more obvious in-context at the expense of one extra import line, so I'll do that.

author
#14 lore raw

PATCH 4/6 iotests: add hmp helper with logging

In reply to John Snow

Just a mild cleanup while I was here.

---
 tests/qemu-iotests/iotests.py | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
tests/qemu-iotests/iotests.py +11 -7
index 8815052eb5..5d2990a0e4 100644
@@ -525,23 +525,27 @@ def add_incoming(self, addr):
         self._args.append(addr)
         return self
 
+    def hmp(self, command_line, log=False):
+        cmd = 'human-monitor-command'
+        kwargs = { 'command-line': command_line }
+        if log:
+            return self.qmp_log(cmd, **kwargs)
+        return self.qmp(cmd, **kwargs)
+
     def pause_drive(self, drive, event=None):
         '''Pause drive r/w operations'''
         if not event:
             self.pause_drive(drive, "read_aio")
             self.pause_drive(drive, "write_aio")
             return
-        self.qmp('human-monitor-command',
-                    command_line='qemu-io %s "break %s bp_%s"' % (drive, event, drive))
+        self.hmp('qemu-io %s "break %s bp_%s"' % (drive, event, drive))
 
     def resume_drive(self, drive):
-        self.qmp('human-monitor-command',
-                    command_line='qemu-io %s "remove_break bp_%s"' % (drive, drive))
+        self.hmp('qemu-io %s "remove_break bp_%s"' % (drive, drive))
 
-    def hmp_qemu_io(self, drive, cmd):
+    def hmp_qemu_io(self, drive, cmd, log=False):
         '''Write to a given drive using an HMP command'''
-        return self.qmp('human-monitor-command',
-                        command_line='qemu-io %s "%s"' % (drive, cmd))
+        return self.hmp('qemu-io %s "%s"' % (drive, cmd), log=log)
 
     def flatten_qmp_object(self, obj, output=None, basestr=''):
         if output is None:
Signature · 1 line
2.21.1
author
#16 lore raw

PATCH 5/6 qmp.py: change event_wait to use a dict

In reply to John Snow

It's easier to work with than a list of tuples, because we can check the keys for membership.

---
 python/qemu/machine.py        | 10 +++++-----
 tests/qemu-iotests/040        | 12 ++++++------
 tests/qemu-iotests/260        |  5 +++--
 tests/qemu-iotests/iotests.py | 16 ++++++++--------
 4 files changed, 22 insertions(+), 21 deletions(-)
python/qemu/machine.py +5 -5
index 183d8f3d38..748de5f322 100644
@@ -476,21 +476,21 @@ def event_wait(self, name, timeout=60.0, match=None):
         timeout: QEMUMonitorProtocol.pull_event timeout parameter.
         match: Optional match criteria. See event_match for details.
         """
-        return self.events_wait([(name, match)], timeout)
+        return self.events_wait({name: match}, timeout)
 
     def events_wait(self, events, timeout=60.0):
         """
         events_wait waits for and returns a named event from QMP with a timeout.
 
-        events: a sequence of (name, match_criteria) tuples.
+        events: a mapping containing {name: match_criteria}.
                 The match criteria are optional and may be None.
                 See event_match for details.
         timeout: QEMUMonitorProtocol.pull_event timeout parameter.
         """
         def _match(event):
-            for name, match in events:
-                if event['event'] == name and self.event_match(event, match):
-                    return True
+            name = event['event']
+            if name in events:
+                return self.event_match(event, events[name])
             return False
 
         # Search cached events
tests/qemu-iotests/040 +6 -6
index 32c82b4ec6..90b59081ff 100755
@@ -485,12 +485,12 @@ class TestErrorHandling(iotests.QMPTestCase):
 
     def run_job(self, expected_events, error_pauses_job=False):
         match_device = {'data': {'device': 'job0'}}
-        events = [
-            ('BLOCK_JOB_COMPLETED', match_device),
-            ('BLOCK_JOB_CANCELLED', match_device),
-            ('BLOCK_JOB_ERROR', match_device),
-            ('BLOCK_JOB_READY', match_device),
-        ]
+        events = {
+            'BLOCK_JOB_COMPLETED': match_device,
+            'BLOCK_JOB_CANCELLED': match_device,
+            'BLOCK_JOB_ERROR': match_device,
+            'BLOCK_JOB_READY': match_device,
+        }
 
         completed = False
         log = []
tests/qemu-iotests/260 +3 -2
index 30c0de380d..b2fb045ddd 100755
@@ -65,8 +65,9 @@ def test(persistent, restart):
 
     vm.qmp_log('block-commit', device='drive0', top=top,
                filters=[iotests.filter_qmp_testfiles])
-    ev = vm.events_wait((('BLOCK_JOB_READY', None),
-                         ('BLOCK_JOB_COMPLETED', None)))
+    ev = vm.events_wait({
+        'BLOCK_JOB_READY': None,
+        'BLOCK_JOB_COMPLETED': None })
     log(filter_qmp_event(ev))
     if (ev['event'] == 'BLOCK_JOB_COMPLETED'):
         vm.shutdown()
tests/qemu-iotests/iotests.py +8 -8
index 5d2990a0e4..3390fab021 100644
@@ -604,14 +604,14 @@ def run_job(self, job, auto_finalize=True, auto_dismiss=False,
         """
         match_device = {'data': {'device': job}}
         match_id = {'data': {'id': job}}
-        events = [
-            ('BLOCK_JOB_COMPLETED', match_device),
-            ('BLOCK_JOB_CANCELLED', match_device),
-            ('BLOCK_JOB_ERROR', match_device),
-            ('BLOCK_JOB_READY', match_device),
-            ('BLOCK_JOB_PENDING', match_id),
-            ('JOB_STATUS_CHANGE', match_id)
-        ]
+        events = {
+            'BLOCK_JOB_COMPLETED': match_device,
+            'BLOCK_JOB_CANCELLED': match_device,
+            'BLOCK_JOB_ERROR': match_device,
+            'BLOCK_JOB_READY': match_device,
+            'BLOCK_JOB_PENDING': match_id,
+            'JOB_STATUS_CHANGE': match_id,
+        }
         error = None
         while True:
             ev = filter_qmp_event(self.events_wait(events, timeout=wait))
Signature · 1 line
2.21.1
Reviewed-by on v1 5/6
#17 lore raw

Re: [PATCH 5/6] qmp.py: change event_wait to use a dict

Quoted 70 lines · John Snow

It's easier to work with than a list of tuples, because we can check the keys for membership.

---
  python/qemu/machine.py        | 10 +++++-----
  tests/qemu-iotests/040        | 12 ++++++------
  tests/qemu-iotests/260        |  5 +++--
  tests/qemu-iotests/iotests.py | 16 ++++++++--------
  4 files changed, 22 insertions(+), 21 deletions(-)
python/qemu/machine.py +5 -5
index 183d8f3d38..748de5f322 100644
@@ -476,21 +476,21 @@ def event_wait(self, name, timeout=60.0, match=None):
          timeout: QEMUMonitorProtocol.pull_event timeout parameter.
          match: Optional match criteria. See event_match for details.
          """
-        return self.events_wait([(name, match)], timeout)
+        return self.events_wait({name: match}, timeout)
 
      def events_wait(self, events, timeout=60.0):
          """
          events_wait waits for and returns a named event from QMP with a timeout.
 
-        events: a sequence of (name, match_criteria) tuples.
+        events: a mapping containing {name: match_criteria}.
                  The match criteria are optional and may be None.
                  See event_match for details.
          timeout: QEMUMonitorProtocol.pull_event timeout parameter.
          """
          def _match(event):
-            for name, match in events:
-                if event['event'] == name and self.event_match(event, match):
-                    return True
+            name = event['event']
+            if name in events:
+                return self.event_match(event, events[name])
              return False
 
          # Search cached events
tests/qemu-iotests/040 +6 -6
index 32c82b4ec6..90b59081ff 100755
@@ -485,12 +485,12 @@ class TestErrorHandling(iotests.QMPTestCase):
 
      def run_job(self, expected_events, error_pauses_job=False):
          match_device = {'data': {'device': 'job0'}}
-        events = [
-            ('BLOCK_JOB_COMPLETED', match_device),
-            ('BLOCK_JOB_CANCELLED', match_device),
-            ('BLOCK_JOB_ERROR', match_device),
-            ('BLOCK_JOB_READY', match_device),
-        ]
+        events = {
+            'BLOCK_JOB_COMPLETED': match_device,
+            'BLOCK_JOB_CANCELLED': match_device,
+            'BLOCK_JOB_ERROR': match_device,
+            'BLOCK_JOB_READY': match_device,
+        }
 
          completed = False
          log = []
tests/qemu-iotests/260 +0 -0
index 30c0de380d..b2fb045ddd 100755

tests/qemu-iotests/260 def test(persistent, restart): · line 70

Quoted 8 lines
tests/qemu-iotests/260 +3 -2
@@ -65,8 +65,9 @@ def test(persistent, restart):
 
      vm.qmp_log('block-commit', device='drive0', top=top,
                 filters=[iotests.filter_qmp_testfiles])
-    ev = vm.events_wait((('BLOCK_JOB_READY', None),
-                         ('BLOCK_JOB_COMPLETED', None)))
+    ev = vm.events_wait({
+        'BLOCK_JOB_READY': None,
+        'BLOCK_JOB_COMPLETED': None })

may be better to keep it 2-lines. Or, otherwise, put closing "})" on a separate line too.

Quoted 21 lines
tests/qemu-iotests/260 +0 -0
      log(filter_qmp_event(ev))
      if (ev['event'] == 'BLOCK_JOB_COMPLETED'):
          vm.shutdown()
tests/qemu-iotests/iotests.py +1 -8
index 5d2990a0e4..3390fab021 100644
@@ -604,14 +604,14 @@ def run_job(self, job, auto_finalize=True, auto_dismiss=False,
          """
          match_device = {'data': {'device': job}}
          match_id = {'data': {'id': job}}
-        events = [
-            ('BLOCK_JOB_COMPLETED', match_device),
-            ('BLOCK_JOB_CANCELLED', match_device),
-            ('BLOCK_JOB_ERROR', match_device),
-            ('BLOCK_JOB_READY', match_device),
-            ('BLOCK_JOB_PENDING', match_id),
-            ('JOB_STATUS_CHANGE', match_id)
-        ]
+        events = {

tests/qemu-iotests/iotests.py def run_job(self, job, auto_finalize=True, auto_dismiss=False, · line 617

Quoted 10 lines
tests/qemu-iotests/iotests.py +7 -0
+            'BLOCK_JOB_COMPLETED': match_device,
+            'BLOCK_JOB_CANCELLED': match_device,
+            'BLOCK_JOB_ERROR': match_device,
+            'BLOCK_JOB_READY': match_device,
+            'BLOCK_JOB_PENDING': match_id,
+            'JOB_STATUS_CHANGE': match_id,
+        }
          error = None
          while True:
              ev = filter_qmp_event(self.events_wait(events, timeout=wait))

Not sure that I like new interface more (neither that it is faster), but it works too:

Signature · 2 lines
Best regards,
Vladimir
author
#18 lore raw

Re: [PATCH 5/6] qmp.py: change event_wait to use a dict

Quoted 71 lines · Vladimir Sementsov-Ogievskiy
Quoted 70 lines · John Snow

It's easier to work with than a list of tuples, because we can check the keys for membership.

---

  python/qemu/machine.py        | 10 +++++-----
  tests/qemu-iotests/040        | 12 ++++++------
  tests/qemu-iotests/260        |  5 +++--
  tests/qemu-iotests/iotests.py | 16 ++++++++--------
  4 files changed, 22 insertions(+), 21 deletions(-)
python/qemu/machine.py +0 -0
index 183d8f3d38..748de5f322 100644
@@ -476,21 +476,21 @@ def event_wait(self, name, timeout=60.0,

match=None):
          timeout: QEMUMonitorProtocol.pull_event timeout parameter.
          match: Optional match criteria. See event_match for details. """
-        return self.events_wait([(name, match)], timeout) +        return self.events_wait({name: match}, timeout) def events_wait(self, events, timeout=60.0):
          """
          events_wait waits for and returns a named event from QMP with a timeout.
  -        events: a sequence of (name, match_criteria) tuples. +        events: a mapping containing {name: match_criteria}. The match criteria are optional and may be None. See event_match for details.
          timeout: QEMUMonitorProtocol.pull_event timeout parameter. """
          def _match(event):
-            for name, match in events:
-                if event['event'] == name and self.event_match(event, match):
-                    return True
+            name = event['event']
+            if name in events:
+                return self.event_match(event, events[name]) return False
            # Search cached events

tests/qemu-iotests/040 +0 -0
index 32c82b4ec6..90b59081ff 100755
@@ -485,12 +485,12 @@ class TestErrorHandling(iotests.QMPTestCase):

        def run_job(self, expected_events, error_pauses_job=False):
          match_device = {'data': {'device': 'job0'}}
-        events = [
-            ('BLOCK_JOB_COMPLETED', match_device),
-            ('BLOCK_JOB_CANCELLED', match_device),
-            ('BLOCK_JOB_ERROR', match_device),
-            ('BLOCK_JOB_READY', match_device),
-        ]
+        events = {
+            'BLOCK_JOB_COMPLETED': match_device,
+            'BLOCK_JOB_CANCELLED': match_device,
+            'BLOCK_JOB_ERROR': match_device,
+            'BLOCK_JOB_READY': match_device,
+        }
            completed = False
          log = []

tests/qemu-iotests/260 +0 -0
index 30c0de380d..b2fb045ddd 100755
@@ -65,8 +65,9 @@ def test(persistent, restart):
         vm.qmp_log('block-commit', device='drive0', top=top,
                  filters=[iotests.filter_qmp_testfiles])
Quoted 9 lines

tests/qemu-iotests/260 def test(persistent, restart): · line 70

Quoted 5 lines
tests/qemu-iotests/260 +3 -2
-    ev = vm.events_wait((('BLOCK_JOB_READY', None),
-                         ('BLOCK_JOB_COMPLETED', None)))
+    ev = vm.events_wait({
+        'BLOCK_JOB_READY': None,
+        'BLOCK_JOB_COMPLETED': None })

may be better to keep it 2-lines. Or, otherwise, put closing "})" on a separate line too.

Sure.

Quoted 38 lines
Quoted 25 lines
tests/qemu-iotests/260 +0 -0
       log(filter_qmp_event(ev))

      if (ev['event'] == 'BLOCK_JOB_COMPLETED'):
          vm.shutdown()

tests/qemu-iotests/iotests.py +0 -0
index 5d2990a0e4..3390fab021 100644
@@ -604,14 +604,14 @@ def run_job(self, job, auto_finalize=True,
 auto_dismiss=False,

          """
          match_device = {'data': {'device': job}} match_id = {'data': {'id': job}} -        events = [
-            ('BLOCK_JOB_COMPLETED', match_device), -            ('BLOCK_JOB_CANCELLED', match_device), -            ('BLOCK_JOB_ERROR', match_device), -            ('BLOCK_JOB_READY', match_device), -            ('BLOCK_JOB_PENDING', match_id), -            ('JOB_STATUS_CHANGE', match_id) -        ]
+        events = {
+            'BLOCK_JOB_COMPLETED': match_device, +            'BLOCK_JOB_CANCELLED': match_device, +            'BLOCK_JOB_ERROR': match_device,

Quoted 9 lines

+            'BLOCK_JOB_READY': match_device,
+            'BLOCK_JOB_PENDING': match_id,
+            'JOB_STATUS_CHANGE': match_id,
+        }
          error = None
          while True:
              ev = filter_qmp_event(self.events_wait(events, timeout=wait))

Not sure that I like new interface more (neither that it is faster), but it works too:

Quoted 2 lines

It's a little bit of cross-contamination from the JobRunner series I posted. This patch really belongs with that series instead.

It's not likely to be faster on the order of ~6 events, but logistically it's just a little cleaner to read.

Thanks for the reviews!

author
#19 lore raw

PATCH 6/6 iotests: add 287 for block-dirty-bitmap-populate

In reply to John Snow

Give block-dirty-bitmap-populate a workout.

---
 tests/qemu-iotests/287     |  233 ++
 tests/qemu-iotests/287.out | 4544 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |    1 +
 3 files changed, 4778 insertions(+)
 create mode 100755 tests/qemu-iotests/287
 create mode 100644 tests/qemu-iotests/287.out
tests/qemu-iotests/287 new file +233 -0
new file mode 100755
index 0000000000..0ab58dc011
@@ -0,0 +1,233 @@
+#!/usr/bin/env python3
+#
+# Test block-dirty-bitmap-populate
+#
+# Copyright (c) 2020 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+from collections import namedtuple
+import itertools
+import math
+import os
+
+import iotests
+from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS
+
+SIZE = 64 * 1024 * 1024
+GRANULARITY = 64 * 1024
+
+
+class Drive:
+    def __init__(self, path, vm):
+        self.path = path
+        self.vm = vm
+        self.fmt = None
+        self.size = None
+        self.node = None
+
+    def img_create(self, fmt, size):
+        self.fmt = fmt
+        self.size = size
+        iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size))
+
+
+def block_dirty_bitmap_populate(vm, node, bitmap, job_id, pattern, **kwargs):
+    # Strip any arguments explicitly nulled by the caller:
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}
+    result = vm.qmp_log('block-dirty-bitmap-populate',
+                        node=node,
+                        name=bitmap,
+                        job_id=job_id,
+                        pattern=pattern,
+                        **kwargs)
+    return result
+
+
+def populate(drive, bitmap, job_id, pattern='allocation-top', **kwargs):
+    kwargs.setdefault('pattern', pattern)
+    kwargs.setdefault('auto-finalize', False)
+    kwargs.setdefault('auto-dismiss', False)
+    ret = block_dirty_bitmap_populate(drive.vm, drive.node,
+                                      bitmap, job_id, **kwargs)
+    return {
+        'id': job_id,
+        'auto-finalize': kwargs['auto-finalize'],
+        'auto-dismiss': kwargs['auto-dismiss'],
+        'return': ret,
+    }
+
+
+def perform_writes(drive, n, filter_node_name=None):
+    log("-- Write #{:d}:".format(n))
+    node_name = filter_node_name or drive.node
+    for pattern in GROUPS[n].patterns:
+        cmd = "write -P{:s} 0x{:07x} 0x{:x}".format(
+            pattern.byte,
+            pattern.offset,
+            pattern.size)
+        drive.vm.hmp_qemu_io(node_name, cmd, log=True)
+    log('')
+
+
+TestConfig = namedtuple('TestConfig', [
+    'base_pattern',
+    'disabled',
+    'pre_writes',
+    'mid_writes',
+    'cancel',
+    'post_writes',
+])
+
+
+def test_bitmap_populate(config):
+    """
+    Test bitmap populate.
+
+    :param base_pattern: Write a base pattern?
+    :param disabled:     Disable the target bitmap?
+    :param pre_writes:   Write a pattern after bitmap creation, but before job?
+    :param mid_writes:   Write a pattern before job finalizes?
+    :param cancel:       Cancel the job instead of finalizing it?
+    :param post_writes:  Write a pattern after the job?
+    """
+    with iotests.FilePaths(['img']) as (img_path,), iotests.VM() as vm:
+        log("\n=== Bitmap Populate {:s} ===\n".format(str(config)))
+
+        log('-- Prepare image & VM:')
+        drive0 = Drive(img_path, vm=vm)
+        drive0.img_create(iotests.imgfmt, SIZE)
+        vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device()))
+        vm.launch()
+
+        file_config = {
+            'driver': 'file',
+            'filename': drive0.path
+        }
+
+        drive0.node = 'drive0'
+        vm.qmp_log('blockdev-add',
+                   filters=[iotests.filter_qmp_testfiles],
+                   node_name=drive0.node,
+                   driver=drive0.fmt,
+                   file=file_config)
+        log('')
+
+
+        # Step 0: Prepare & Base Allocation Pattern
+
+        if config.base_pattern:
+            perform_writes(drive0, 0)
+
+
+        # Step 1: Add test bitmap
+
+        log('-- Add Bitmap:')
+        vm.qmp_log('block-dirty-bitmap-add',
+                   node=drive0.node,
+                   name="target",
+                   granularity=GRANULARITY,
+                   disabled=config.disabled)
+        ebitmap = EmulatedBitmap()
+        log('')
+
+        # Step 2: Pre-Writes
+
+        if config.pre_writes:
+            perform_writes(drive0, 1)
+            if not config.disabled:
+                ebitmap.dirty_group(1)
+            bitmap = vm.get_bitmap(drive0.node, 'target')
+            ebitmap.compare(bitmap)
+
+
+        # Step 3: Launch job & Mid-Writes
+
+        log('-- Test block-dirty-bitmap-populate (bitpop0):')
+        def pre_finalize():
+            # Writes issued prior to job finalization:
+            if config.mid_writes:
+                perform_writes(drive0, 2)
+                if not config.disabled:
+                    ebitmap.dirty_group(2)
+
+        job = populate(drive0, 'target', 'bitpop0')
+        assert job['return'] == {'return': {}}
+        vm.run_job(job['id'],
+                   auto_dismiss=job['auto-dismiss'],
+                   auto_finalize=job['auto-finalize'],
+                   pre_finalize=pre_finalize,
+                   cancel=config.cancel)
+        log('')
+
+
+        # Step 4: Post-job verification
+
+        if not config.cancel:
+            # Any writes made prior to the job finishing should now be visible.
+            if config.base_pattern:
+                ebitmap.dirty_group(0)
+            if config.pre_writes:
+                ebitmap.dirty_group(1)
+            if config.mid_writes:
+                ebitmap.dirty_group(2)
+
+        bitmap = vm.get_bitmap(drive0.node, 'target')
+        ebitmap.compare(bitmap)
+
+
+        # Step 5: Post-Writes
+
+        if config.post_writes:
+            perform_writes(drive0, 3)
+            if not config.disabled:
+                ebitmap.dirty_group(3)
+
+
+        # Step 6: Final Verification
+
+        log('-- Verification:')
+        bitmaps = vm.query_bitmaps()
+        log({'bitmaps': bitmaps}, indent=2)
+        log('')
+        bitmap = vm.get_bitmap(drive0.node, 'target', bitmaps=bitmaps)
+        ebitmap.compare(bitmap)
+
+
+        log('-- Cleanup:')
+        vm.qmp_log("block-dirty-bitmap-remove",
+                   node=drive0.node, name="target")
+
+        bitmaps = vm.query_bitmaps()
+        if bitmaps:
+            log("ERROR: bitmaps unaccounted for:")
+            log(bitmaps)
+        else:
+            log('OK: All bitmaps removed')
+        vm.shutdown()
+        log('')
+
+
+def main():
+    for args in itertools.product((True, False), repeat=6):
+        cfg = TestConfig(*args)
+        test_bitmap_populate(cfg)
+
+
+if __name__ == '__main__':
+    iotests.script_main(main, supported_fmts=['qcow2'],
+                        supported_protocols=['file'])
tests/qemu-iotests/287.out new file +4544 -0

2,767 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index 0317667695..5e80d7deab 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.21.1

Part of the diff is not shown here. View the raw message.