Skip to contentSkip to conversation

block: add block-dirty-bitmap-populate job

PATCH Superseded Eric Blake 3 participants 17 messages Jun 19, 2020 – Sep 2, 2020 Block layer

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Eric Blake

author
#0 lore raw

v3 cover block: add block-dirty-bitmap-populate job

[From John's original cover letter:]
This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.

This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.

This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.

v3:
- Addressed a bit more feedback
- Make it easier to decide if we want an x- prefix if we think there are more tweaks to be made to the interface
- Drop dependency on John's JobRunner iotest series
- Renumber the new iotest

I know there was a lot of discussion about whether there are optimizations to be made with populating directly into the target bitmap rather than into a temporary that then gets merged in at the completion of the job, but the QMP aspect seems fairly stable. Even so, we may still want to consider using an x- prefix until we know for sure whether libvirt can make decent use of the interface.

Eric Blake (1):
bitmaps: Use x- prefix for block-dirty-bitmap-popluate

John Snow (5):
block: add bitmap-populate job
blockdev: combine DriveBackupState and BlockdevBackupState
qmp: expose block-dirty-bitmap-populate
iotests: move bitmap helpers into their own file
iotests: add 298 for block-dirty-bitmap-populate

 qapi/block-core.json          |   66 +
 qapi/job.json                 |    6 +-
 qapi/transaction.json         |    2 +
 include/block/block_int.h     |   21 +
 block/bitmap-populate.c       |  207 ++
 blockdev.c                    |  104 +-
 blockjob.c                    |    3 +-
 MAINTAINERS                   |    1 +
 block/Makefile.objs           |    1 +
 tests/qemu-iotests/257        |  110 +-
 tests/qemu-iotests/298        |  232 ++
 tests/qemu-iotests/298.out    | 4544 +++++++++++++++++++++++++++++++++
 tests/qemu-iotests/bitmaps.py |  131 +
 tests/qemu-iotests/group      |    1 +
 14 files changed, 5300 insertions(+), 129 deletions(-)
 create mode 100644 block/bitmap-populate.c
 create mode 100755 tests/qemu-iotests/298
 create mode 100644 tests/qemu-iotests/298.out
 create mode 100644 tests/qemu-iotests/bitmaps.py
Signature · 1 line
2.27.0
author
#1 lore raw

PATCH v3 1/6 block: add bitmap-populate job

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
 qapi/block-core.json      |  48 +++++++++
 qapi/job.json             |   6 +-
 include/block/block_int.h |  21 ++++
 block/bitmap-populate.c   | 207 ++++++++++++++++++++++++++++++++++++++
 blockjob.c                |   3 +-
 MAINTAINERS               |   1 +
 block/Makefile.objs       |   1 +
 7 files changed, 285 insertions(+), 2 deletions(-)
 create mode 100644 block/bitmap-populate.c
qapi/block-core.json +48 -0
index 0e1c6a59f228..a1bcdba04423 100644
@@ -2211,6 +2211,54 @@
       { 'command': 'block-dirty-bitmap-merge',
         'data': 'BlockDirtyBitmapMerge' }
 
+##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.1
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device supports
+#            io-status (see BlockInfo).
+#
+# @auto-finalize: When false, this job will wait in a PENDING state after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query
+#                list without user intervention.
+#                Defaults to true.
+#
+# Since: 5.1
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/job.json +5 -1
index 5e658281f5c4..33ff3500f794 100644
@@ -19,10 +19,14 @@
 #
 # @create: image creation job type, see "blockdev-create" (since 3.0)
 #
+# @bitmap-populate: drive bitmap population job type, see
+#                   "block-dirty-bitmap-populate" (since 5.1)
+#
 # Since: 1.7
 ##
 { 'enum': 'JobType',
-  'data': ['commit', 'stream', 'mirror', 'backup', 'create'] }
+  'data': ['commit', 'stream', 'mirror', 'backup', 'create',
+           'bitmap-populate'] }
 
 ##
 # @JobStatus:
include/block/block_int.h +21 -0
index 791de6a59c2c..93fb886e7e97 100644
@@ -1231,6 +1231,27 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
                             BlockCompletionFunc *cb, void *opaque,
                             JobTxn *txn, Error **errp);
 
+/*
+ * bitpop_job_create: Create a new bitmap population job.
+ *
+ * @job_id: The id of the newly-created job.
+ * @bs: Block device associated with the @target_bitmap.
+ * @target_bitmap: The bitmap to populate.
+ * @on_error: What to do if an error on @bs is encountered.
+ * @creation_flags: Flags that control the behavior of the Job lifetime.
+ *                  See @BlockJobCreateFlags
+ * @cb: Completion function for the job.
+ * @opaque: Opaque pointer value passed to @cb.
+ * @txn: Transaction that this job is part of (may be NULL).
+ */
+BlockJob *bitpop_job_create(const char *job_id, BlockDriverState *bs,
+                            BdrvDirtyBitmap *target_bitmap,
+                            BitmapPattern pattern,
+                            BlockdevOnError on_error,
+                            int creation_flags,
+                            BlockCompletionFunc *cb, void *opaque,
+                            JobTxn *txn, Error **errp);
+
 BdrvChild *bdrv_root_attach_child(BlockDriverState *child_bs,
                                   const char *child_name,
                                   const BdrvChildClass *child_class,
block/bitmap-populate.c new file +207 -0

207 lines not on this page · raw message

blockjob.c +2 -1
index 470facfd47a0..e6ee8376645d 100644
@@ -56,7 +56,8 @@ static bool is_block_job(Job *job)
     return job_type(job) == JOB_TYPE_BACKUP ||
            job_type(job) == JOB_TYPE_COMMIT ||
            job_type(job) == JOB_TYPE_MIRROR ||
-           job_type(job) == JOB_TYPE_STREAM;
+           job_type(job) == JOB_TYPE_STREAM ||
+           job_type(job) == JOB_TYPE_BITMAP_POPULATE;
 }
 
 BlockJob *block_job_next(BlockJob *bjob)
MAINTAINERS +1 -0
index 955cc8dd5cd0..99651abcac4a 100644
@@ -2050,6 +2050,7 @@ S: Supported
 F: include/qemu/hbitmap.h
 F: include/block/dirty-bitmap.h
 F: block/monitor/bitmap-qmp-cmds.c
+F: block/bitmap-populate.c
 F: block/dirty-bitmap.c
 F: block/qcow2-bitmap.c
 F: migration/block-dirty-bitmap.c
block/Makefile.objs +1 -0
index 96028eedcef7..55f5970f7625 100644
@@ -36,6 +36,7 @@ block-obj-$(CONFIG_LIBSSH) += ssh.o
 block-obj-y += accounting.o dirty-bitmap.o
 block-obj-y += write-threshold.o
 block-obj-y += backup.o
+block-obj-y += bitmap-populate.o
 block-obj-$(CONFIG_REPLICATION) += replication.o
 block-obj-y += throttle.o copy-on-read.o
 block-obj-y += block-copy.o
Signature · 1 line
2.27.0
#2 lore raw

Re: [PATCH v3 1/6] block: add bitmap-populate job

Quoted 68 lines · Eric Blake

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---
  qapi/block-core.json      |  48 +++++++++
  qapi/job.json             |   6 +-
  include/block/block_int.h |  21 ++++
  block/bitmap-populate.c   | 207 ++++++++++++++++++++++++++++++++++++++
  blockjob.c                |   3 +-
  MAINTAINERS               |   1 +
  block/Makefile.objs       |   1 +
  7 files changed, 285 insertions(+), 2 deletions(-)

create mode 100644 block/bitmap-populate.c

qapi/block-core.json +38 -0
index 0e1c6a59f228..a1bcdba04423 100644
@@ -2211,6 +2211,54 @@
        { 'command': 'block-dirty-bitmap-merge',
          'data': 'BlockDirtyBitmapMerge' }
 
+##
+# @BitmapPattern:
+#
+# An enumeration of possible patterns that can be written into a bitmap.
+#
+# @allocation-top: The allocation status of the top layer
+#                  of the attached storage node.
+#
+# Since: 5.1
+##
+{ 'enum': 'BitmapPattern',
+  'data': ['allocation-top'] }
+
+##
+# @BlockDirtyBitmapPopulate:
+#
+# @job-id: identifier for the newly-created block job.
+#
+# @pattern: What pattern should be written into the bitmap?
+#
+# @on-error: the action to take if an error is encountered on a bitmap's
+#            attached node, default 'report'.
+#            'stop' and 'enospc' can only be used if the block device supports
+#            io-status (see BlockInfo).
+#
+# @auto-finalize: When false, this job will wait in a PENDING state after it has
+#                 finished its work, waiting for @block-job-finalize before
+#                 making any block graph changes.
+#                 When true, this job will automatically
+#                 perform its abort or commit actions.
+#                 Defaults to true.
+#
+# @auto-dismiss: When false, this job will wait in a CONCLUDED state after it
+#                has completely ceased all work, and awaits @block-job-dismiss.
+#                When true, this job will automatically disappear from the query
+#                list without user intervention.
+#                Defaults to true.
+#

qapi/block-core.json line 2261

Quoted 10 lines
qapi/block-core.json +10 -0
+# Since: 5.1
+##
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+

Peter said about a possibility of populating several target bitmaps simultaneously.

What about such a generalized semantics:

Merge all sources to each target

@targets: list of bitmaps to be populated by the job { 'struct': 'BlockDirtyBitmapPopulate',
'data': { <common job fields>,

            'targets': ['BlockDirtyBitmap'],
            'sources': ['BitmapPopulateSource'] } }

@bitmap: specify dirty bitmap to be merged to target bitamp(s)
@node: specify a node name, which allocation-map is to be merged to target bitmap(s) { 'alternate': 'BitmapPopulateSource',
'data': { 'bitmap': 'BlockDirtyBitmap',

            'node': 'str' } }

- so, we can merge several bitmaps together with several allocation maps into several target bitmaps. (I remember, we also said about a possibility of starting several populating jobs, populating into same bitmap, I think it may be substituted by one job with several sources. Still, it's not hard to allow to use target bitmaps in a several jobs simultaneously and this is not about the QAPI interface)

Will this simplify things in libvirt?

Signature · 2 lines
Best regards,
Vladimir
author
#3 lore raw

Re: [PATCH v3 1/6] block: add bitmap-populate job

Quoted 27 lines · Vladimir Sementsov-Ogievskiy
Quoted 13 lines · Eric Blake

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---

qapi/block-core.json line 2261

Quoted 8 lines
qapi/block-core.json +8 -0
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+

Peter said about a possibility of populating several target bitmaps simultaneously.

What about such a generalized semantics:

Quoted 7 lines

Merge all sources to each target

@targets: list of bitmaps to be populated by the job { 'struct': 'BlockDirtyBitmapPopulate',
  'data': { <common job fields>,
            'targets': ['BlockDirtyBitmap'], 'sources': ['BitmapPopulateSource'] } }

We still need the 'pattern' argument (the idea being that if we have: Base <- Active, we want to be able to merge in the allocation map of Active into bitmaps stored in Base as part of a commit operation, whether that is active commit of a live guest or offline commit while the guest is offline). Having an array for 'targets' to merge into is fine, but for 'sources', it's less a concern about selecting from multiple sources, and more a concern about selecting the allocation pattern to be merged in (libvirt wants to merge the same allocation pattern into each bitmap in Base). Generalizing things to allow the merge of more than one source at once might not hurt, but I'm not sure we need it yet.

But there are other patterns that we may want to support: an all-ones pattern, or maybe a pattern that tracks known-zeros instead of allocation.

Quoted 8 lines

@bitmap: specify dirty bitmap to be merged to target bitamp(s) @node: specify a node name, which allocation-map is to be merged to target bitmap(s)
{ 'alternate': 'BitmapPopulateSource',
  'data': { 'bitmap': 'BlockDirtyBitmap',
            'node': 'str' } }

This design is clever in that it lets us merge in both existing bitmaps and using a node-name for merging in an allocation map instead of a bitmap; but it limits us to only one pattern. Better might be something where we supply a union (hmm, we've had proposals in the past for a default value to the discriminator to allow it to be optional, so I'll proceed as if we will finally implement that):

{ 'enum': 'BitmapPattern', 'data': [ 'bitmap', 'allocation-top' ] } { 'union': 'BitmapPopulateSource',
'base': { '*pattern': 'BitmapPattern' }, 'discriminator': { 'name': 'pattern', 'default': 'bitmap' }, 'data': { 'bitmap': 'BitmapPopulateSource',

            'allocation-top': { 'node': 'str' } } }

so that you can then do:

{ "execute": "block-dirty-bitmap-populate",
"arguments": { "targets": [ { "node": "base", "name": "b1" },

                              { "node": "base", "name": "b2" } ],
        "sources": [ { "pattern": "allocation-top", "node": "top" } ]

} }

to merge in the allocation information of top into multiple bitmaps of base at once, or conversely, do:

{ "execute": "block-dirty-bitmap-populate",
"arguments": { "targets": [ { "node": "base", "name": "b1" } ],

        "sources": [ { "pattern": "bitmap",
                       "node": "top", "name": "b1" } ]
  } }
{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b2" } ],
        "sources": [ { "node": "top", "name": "b2" } ]

} }

and of course, wrap this in a "transaction" to ensure that it all succeeds or fails as a unit, rather than messing up one bitmap if another fails, while also allowing future extension for additional patterns.

Quoted 11 lines

- so, we can merge several bitmaps together with several allocation maps
into several target bitmaps.
(I remember, we also said about a possibility of starting several populating jobs, populating into
 same bitmap, I think it may be substituted by one job with several
sources. Still, it's not hard to
 allow to use target bitmaps in a several jobs simultaneously and this
is not about the QAPI interface)

Will this simplify things in libvirt?

Peter, in your preliminary experiments with block-dirty-bitmap-populate, did you ever need to start more than one job to a single bitmap destination, or was it merely starting multiple jobs because you had multiple destinations but always just a single source?

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#4 lore raw

Re: [PATCH v3 1/6] block: add bitmap-populate job

Quoted 42 lines · Eric Blake
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
Quoted 13 lines · Eric Blake

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

---

qapi/block-core.json line 2261

Quoted 8 lines
qapi/block-core.json +8 -0
+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',
+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+

Peter said about a possibility of populating several target bitmaps simultaneously.

What about such a generalized semantics:

Quoted 7 lines

Merge all sources to each target

@targets: list of bitmaps to be populated by the job { 'struct': 'BlockDirtyBitmapPopulate', 'data': { <common job fields>,
             'targets': ['BlockDirtyBitmap'], 'sources': ['BitmapPopulateSource'] } }

We still need the 'pattern' argument (the idea being that if we have: Base <- Active, we want to be able to merge in the allocation map of Active into bitmaps stored in Base as part of a commit operation, whether that is active commit of a live guest or offline commit while the guest is offline).  Having an array for 'targets' to merge into is fine, but for 'sources', it's less a concern about selecting from multiple sources, and more a concern about selecting the allocation pattern to be merged in (libvirt wants to merge the same allocation pattern into each bitmap in Base).  Generalizing things to allow the merge of more than one source at once might not hurt, but I'm not sure we need it yet.

But there are other patterns that we may want to support: an all-ones pattern, or maybe a pattern that tracks known-zeros instead of allocation.

Quoted 2 lines
Quoted 7 lines
Quoted 5 lines

@bitmap: specify dirty bitmap to be merged to target bitamp(s)
@node: specify a node name, which allocation-map is to be merged to target bitmap(s) { 'alternate': 'BitmapPopulateSource',
   'data': { 'bitmap': 'BlockDirtyBitmap',
             'node': 'str' } }

This design is clever in that it lets us merge in both existing bitmaps and using a node-name for merging in an allocation map instead of a bitmap; but it limits us to only one pattern.

Ah, yes, we can't discriminate by type node-name from 'all-ones' or something like this.

Quoted 8 lines

Better might be something where we supply a union (hmm, we've had proposals in the past for a default value to the discriminator to allow it to be optional, so I'll proceed as if we will finally implement that):

{ 'enum': 'BitmapPattern', 'data': [ 'bitmap', 'allocation-top' ] } { 'union': 'BitmapPopulateSource',
  'base': { '*pattern': 'BitmapPattern' }, 'discriminator': { 'name': 'pattern', 'default': 'bitmap' }, 'data': { 'bitmap': 'BitmapPopulateSource', 'allocation-top': { 'node': 'str' } } }

Yes, this is better, of course.

Quoted 35 lines

so that you can then do:

{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b1" }, { "node": "base", "name": "b2" } ], "sources": [ { "pattern": "allocation-top", "node": "top" } ] } }

to merge in the allocation information of top into multiple bitmaps of base at once, or conversely, do:

{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b1" } ], "sources": [ { "pattern": "bitmap",
                       "node": "top", "name": "b1" } ] } }
{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b2" } ], "sources": [ { "node": "top", "name": "b2" } ] } }

and of course, wrap this in a "transaction" to ensure that it all succeeds or fails as a unit, rather than messing up one bitmap if another fails, while also allowing future extension for additional patterns.

Quoted 8 lines

- so, we can merge several bitmaps together with several allocation maps into several target bitmaps. (I remember, we also said about a possibility of starting several populating jobs, populating into same bitmap, I think it may be substituted by one job with several sources. Still, it's not hard to allow to use target bitmaps in a several jobs simultaneously and this is not about the QAPI interface)

Will this simplify things in libvirt?

Peter, in your preliminary experiments with block-dirty-bitmap-populate, did you ever need to start more than one job to a single bitmap destination, or was it merely starting multiple jobs because you had multiple destinations but always just a single source?

Signature · 2 lines
Best regards,
Vladimir
author
#5 lore raw

Re: [PATCH v3 1/6] block: add bitmap-populate job

In reply to Eric Blake

[reviving an old thread]

Quoted 11 lines · Eric Blake
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Eric Blake

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

---

+{ 'struct': 'BlockDirtyBitmapPopulate',
+  'base': 'BlockDirtyBitmap',
+  'data': { 'job-id': 'str',
+            'pattern': 'BitmapPattern',

As written, "pattern":"allocate-top" is rather limited - it can only grab allocation from the top node. Being able to grab the allocation from a specific node may indeed be more useful. Another bitmap patterns that might be useful would be an all-one pattern (create a bitmap that treats the entire disk as dirty). I also remember brainstorming with John the question of whether we want bitmap-populate to have different mask modes: does the population perform an overwrite (the bitmap now matches the source pattern exactly, even if some bits were set and others cleared), a union merge (any bits already set in the bitmap remain set, additional bits are set according to pattern), or even a difference (any bits already cleared in the bitmap remain clear, while bits in the pattern can also clear additional bits in the bitmap).

If I understand Peter's goals, the initial libvirt use is a union mode (keep bits in the bitmap that are already set, but set additional bits according to the population pattern).

Quoted 11 lines
Quoted 11 lines
Quoted 4 lines

+            '*on-error': 'BlockdevOnError',
+            '*auto-finalize': 'bool',
+            '*auto-dismiss': 'bool' } }
+

Peter said about a possibility of populating several target bitmaps simultaneously.

What about such a generalized semantics:

Merge all sources to each target

Quoted 7 lines
Quoted 5 lines

@targets: list of bitmaps to be populated by the job { 'struct': 'BlockDirtyBitmapPopulate', 'data': { <common job fields>,
             'targets': ['BlockDirtyBitmap'], 'sources': ['BitmapPopulateSource'] } }

We still need the 'pattern' argument (the idea being that if we have: Base <- Active, we want to be able to merge in the allocation map of Active into bitmaps stored in Base as part of a commit operation, whether that is active commit of a live guest or offline commit while the guest is offline).  Having an array for 'targets' to merge into is fine, but for 'sources', it's less a concern about selecting from multiple sources, and more a concern about selecting the allocation pattern to be merged in (libvirt wants to merge the same allocation pattern into each bitmap in Base).  Generalizing things to allow the merge of more than one source at once might not hurt, but I'm not sure we need it yet.

But when it comes to multiple destinations or multiple sources, while it seems like it might be a convenience factor, I also worry that it is over-engineering. See more below...

Quoted 23 lines

But there are other patterns that we may want to support: an all-ones pattern, or maybe a pattern that tracks known-zeros instead of allocation.

Quoted 8 lines

@bitmap: specify dirty bitmap to be merged to target bitamp(s) @node: specify a node name, which allocation-map is to be merged to target bitmap(s)
{ 'alternate': 'BitmapPopulateSource',
   'data': { 'bitmap': 'BlockDirtyBitmap',
             'node': 'str' } }

This design is clever in that it lets us merge in both existing bitmaps and using a node-name for merging in an allocation map instead of a bitmap; but it limits us to only one pattern.  Better might be something where we supply a union (hmm, we've had proposals in the past for a default value to the discriminator to allow it to be optional, so I'll proceed as if we will finally implement that):

{ 'enum': 'BitmapPattern', 'data': [ 'bitmap', 'allocation-top' ] } { 'union': 'BitmapPopulateSource',
  'base': { '*pattern': 'BitmapPattern' }, 'discriminator': { 'name': 'pattern', 'default': 'bitmap' }, 'data': { 'bitmap': 'BitmapPopulateSource', 'allocation-top': { 'node': 'str' } } }

so that you can then do:

Quoted 7 lines

{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b1" }, { "node": "base", "name": "b2" } ], "sources": [ { "pattern": "allocation-top", "node": "top" } ] } }

to merge in the allocation information of top into multiple bitmaps of base at once,

Hmm, I left out the mandatory "job-id" parameter here; one of the key points of the new command is that some patterns (like allocation) may involve potentially lengthy I/O, so we need a job-id (the existing block-dirty-bitmap-merge does not). But since the existing block-dirty-bitmap-merge supports multiple sources to one destination, supporting multiple patterns to one destination tracked by a single job id does have some appeal.

Quoted 12 lines

or conversely, do:

{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b1" } ], "sources": [ { "pattern": "bitmap",
                       "node": "top", "name": "b1" } ] } }
{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "targets": [ { "node": "base", "name": "b2" } ], "sources": [ { "node": "top", "name": "b2" } ] } }

Quoted 1 line

and of course, wrap this in a "transaction" to ensure that it all succeeds or fails as a unit, rather than messing up one bitmap if another fails, while also allowing future extension for additional patterns.

We already have transactions that let us perform multiple destinations as a group. So what is the difference in the end results between merging one source into two separate destinations in one command spelled this way:

# proposal with many:many bitmap populate
{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "job-id": "job0",
                 "targets": [ { "node": "base", "name": "b1" },
                              { "node": "base", "name": "b2" } ],
                 "source": { "pattern": "allocation", "node": "top" } } }
wait for job to complete

vs. spelled this way:

# patch as written with 1:1 bitmap-populate, but tweak to source
{ "execute": "block-dirty-bitmap-add",
  "arguments": { "node": "top", "name": "tmp" } }
{ "execute": "block-dirty-bitmap-populate",
  "arguments": { "job-id": "job0",
                 "node": "top", "name": "tmp",
                 "source": { "pattern": "allocation", "node": "top" } } }
wait for job to complete
{ "execute": "transaction",
  "arguments": { "actions": [
    { "type": "block-dirty-bitmap-merge", "node": "base", "name": "b1",
        "bitmaps": [ { "node": "top", "name": "tmp" } ] },
    { "type": "block-dirty-bitmap-merge", "node": "base", "name": "b2",
        "bitmaps": [ { "node": "top", "name": "tmp" } ] }
] } }
Quoted 9 lines
Quoted 9 lines

- so, we can merge several bitmaps together with several allocation
maps into several target bitmaps.
(I remember, we also said about a possibility of starting several populating jobs, populating into same bitmap, I think it may be substituted by one job with several sources. Still, it's not hard to allow to use target bitmaps in a several jobs simultaneously and this is not about the QAPI interface)

Quoted 3 lines
Quoted 1 line

Will this simplify things in libvirt?

Peter, in your preliminary experiments with block-dirty-bitmap-populate, did you ever need to start more than one job to a single bitmap destination, or was it merely starting multiple jobs because you had multiple destinations but always just a single source?

I guess I'm struggling in posting a v4 in part because I don't have a good answer to what is easiest for Peter to use.

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#6 lore raw

Re: [PATCH v3 1/6] block: add bitmap-populate job

In reply to Eric Blake

Am 19.06.2020 um 21:56 hat Eric Blake geschrieben:

Quoted 31 lines

From: John Snow <[email protected]>

This job copies the allocation map into a bitmap. It's a job because there's no guarantee that allocation interrogation will be quick (or won't hang), so it cannot be retrofitted into block-dirty-bitmap-merge.

It was designed with different possible population patterns in mind, but only top layer allocation was implemented for now.

block/bitmap-populate.c new file +19 -0
+BlockJob *bitpop_job_create(
+    const char *job_id,
+    BlockDriverState *bs,
+    BdrvDirtyBitmap *target_bitmap,
+    BitmapPattern pattern,
+    BlockdevOnError on_error,
+    int creation_flags,
+    BlockCompletionFunc *cb,
+    void *opaque,
+    JobTxn *txn,
+    Error **errp)
+{
+    int64_t len;
+    BitpopBlockJob *job = NULL;
+    int64_t cluster_size;
+    BdrvDirtyBitmap *new_bitmap = NULL;
+
+    assert(bs);
+    assert(target_bitmap);

block/bitmap-populate.c line 160

Quoted 10 lines
block/bitmap-populate.c new file +10 -0
+
+    if (!bdrv_is_inserted(bs)) {
+        error_setg(errp, "Device is not inserted: %s",
+                   bdrv_get_device_name(bs));
+        return NULL;
+    }
+
+    if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_BACKUP_SOURCE, errp)) {
+        return NULL;
+    }

So did we decide that we'll keep the legacy op blocker with the type of another block job even though nobody could tell what it's good for?

Kevin

author
#7 lore raw

PATCH v3 2/6 blockdev: combine DriveBackupState and BlockdevBackupState

In reply to Eric Blake

From: John Snow <[email protected]>

They have the same fields -- rename it BlockJobActionState.

---
 blockdev.c | 30 ++++++++++++------------------
 1 file changed, 12 insertions(+), 18 deletions(-)
blockdev.c +12 -18
index 72df193ca73b..6d80af903c55 100644
@@ -1655,11 +1655,11 @@ static void external_snapshot_clean(BlkActionState *common)
     aio_context_release(aio_context);
 }
 
-typedef struct DriveBackupState {
+typedef struct BlockJobActionState {
     BlkActionState common;
     BlockDriverState *bs;
     BlockJob *job;
-} DriveBackupState;
+} BlockJobActionState;
 
 static BlockJob *do_backup_common(BackupCommon *backup,
                                   BlockDriverState *bs,
@@ -1669,7 +1669,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,
 
 static void drive_backup_prepare(BlkActionState *common, Error **errp)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     DriveBackup *backup;
     BlockDriverState *bs;
     BlockDriverState *target_bs;
@@ -1806,7 +1806,7 @@ out:
 
 static void drive_backup_commit(BlkActionState *common)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     aio_context = bdrv_get_aio_context(state->bs);
@@ -1820,7 +1820,7 @@ static void drive_backup_commit(BlkActionState *common)
 
 static void drive_backup_abort(BlkActionState *common)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
 
     if (state->job) {
         AioContext *aio_context;
@@ -1836,7 +1836,7 @@ static void drive_backup_abort(BlkActionState *common)
 
 static void drive_backup_clean(BlkActionState *common)
 {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     if (!state->bs) {
@@ -1851,15 +1851,9 @@ static void drive_backup_clean(BlkActionState *common)
     aio_context_release(aio_context);
 }
 
-typedef struct BlockdevBackupState {
-    BlkActionState common;
-    BlockDriverState *bs;
-    BlockJob *job;
-} BlockdevBackupState;
-
 static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     BlockdevBackup *backup;
     BlockDriverState *bs;
     BlockDriverState *target_bs;
@@ -1907,7 +1901,7 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
 
 static void blockdev_backup_commit(BlkActionState *common)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     aio_context = bdrv_get_aio_context(state->bs);
@@ -1921,7 +1915,7 @@ static void blockdev_backup_commit(BlkActionState *common)
 
 static void blockdev_backup_abort(BlkActionState *common)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
 
     if (state->job) {
         AioContext *aio_context;
@@ -1937,7 +1931,7 @@ static void blockdev_backup_abort(BlkActionState *common)
 
 static void blockdev_backup_clean(BlkActionState *common)
 {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
     AioContext *aio_context;
 
     if (!state->bs) {
@@ -2209,14 +2203,14 @@ static const BlkActionOps actions[] = {
         .clean = external_snapshot_clean,
     },
     [TRANSACTION_ACTION_KIND_DRIVE_BACKUP] = {
-        .instance_size = sizeof(DriveBackupState),
+        .instance_size = sizeof(BlockJobActionState),
         .prepare = drive_backup_prepare,
         .commit = drive_backup_commit,
         .abort = drive_backup_abort,
         .clean = drive_backup_clean,
     },
     [TRANSACTION_ACTION_KIND_BLOCKDEV_BACKUP] = {
-        .instance_size = sizeof(BlockdevBackupState),
+        .instance_size = sizeof(BlockJobActionState),
         .prepare = blockdev_backup_prepare,
         .commit = blockdev_backup_commit,
         .abort = blockdev_backup_abort,
Signature · 1 line
2.27.0
#8 lore raw

Re: [PATCH v3 2/6] blockdev: combine DriveBackupState and BlockdevBackupState

Quoted 3 lines · Eric Blake

From: John Snow <[email protected]>

They have the same fields -- rename it BlockJobActionState.

commit/abort/clean functions are identical after it. I think better to combine them as well here

Quoted 123 lines
---
  blockdev.c | 30 ++++++++++++------------------
  1 file changed, 12 insertions(+), 18 deletions(-)
blockdev.c +12 -18
index 72df193ca73b..6d80af903c55 100644
@@ -1655,11 +1655,11 @@ static void external_snapshot_clean(BlkActionState *common)
      aio_context_release(aio_context);
  }
 
-typedef struct DriveBackupState {
+typedef struct BlockJobActionState {
      BlkActionState common;
      BlockDriverState *bs;
      BlockJob *job;
-} DriveBackupState;
+} BlockJobActionState;
 
  static BlockJob *do_backup_common(BackupCommon *backup,
                                    BlockDriverState *bs,
@@ -1669,7 +1669,7 @@ static BlockJob *do_backup_common(BackupCommon *backup,
 
  static void drive_backup_prepare(BlkActionState *common, Error **errp)
  {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
      DriveBackup *backup;
      BlockDriverState *bs;
      BlockDriverState *target_bs;
@@ -1806,7 +1806,7 @@ out:
 
  static void drive_backup_commit(BlkActionState *common)
  {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
      AioContext *aio_context;
 
      aio_context = bdrv_get_aio_context(state->bs);
@@ -1820,7 +1820,7 @@ static void drive_backup_commit(BlkActionState *common)
 
  static void drive_backup_abort(BlkActionState *common)
  {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
 
      if (state->job) {
          AioContext *aio_context;
@@ -1836,7 +1836,7 @@ static void drive_backup_abort(BlkActionState *common)
 
  static void drive_backup_clean(BlkActionState *common)
  {
-    DriveBackupState *state = DO_UPCAST(DriveBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
      AioContext *aio_context;
 
      if (!state->bs) {
@@ -1851,15 +1851,9 @@ static void drive_backup_clean(BlkActionState *common)
      aio_context_release(aio_context);
  }
 
-typedef struct BlockdevBackupState {
-    BlkActionState common;
-    BlockDriverState *bs;
-    BlockJob *job;
-} BlockdevBackupState;
-
  static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
  {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
      BlockdevBackup *backup;
      BlockDriverState *bs;
      BlockDriverState *target_bs;
@@ -1907,7 +1901,7 @@ static void blockdev_backup_prepare(BlkActionState *common, Error **errp)
 
  static void blockdev_backup_commit(BlkActionState *common)
  {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
      AioContext *aio_context;
 
      aio_context = bdrv_get_aio_context(state->bs);
@@ -1921,7 +1915,7 @@ static void blockdev_backup_commit(BlkActionState *common)
 
  static void blockdev_backup_abort(BlkActionState *common)
  {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
 
      if (state->job) {
          AioContext *aio_context;
@@ -1937,7 +1931,7 @@ static void blockdev_backup_abort(BlkActionState *common)
 
  static void blockdev_backup_clean(BlkActionState *common)
  {
-    BlockdevBackupState *state = DO_UPCAST(BlockdevBackupState, common, common);
+    BlockJobActionState *state = DO_UPCAST(BlockJobActionState, common, common);
      AioContext *aio_context;
 
      if (!state->bs) {
@@ -2209,14 +2203,14 @@ static const BlkActionOps actions[] = {
          .clean = external_snapshot_clean,
      },
      [TRANSACTION_ACTION_KIND_DRIVE_BACKUP] = {
-        .instance_size = sizeof(DriveBackupState),
+        .instance_size = sizeof(BlockJobActionState),
          .prepare = drive_backup_prepare,
          .commit = drive_backup_commit,
          .abort = drive_backup_abort,
          .clean = drive_backup_clean,
      },
      [TRANSACTION_ACTION_KIND_BLOCKDEV_BACKUP] = {
-        .instance_size = sizeof(BlockdevBackupState),
+        .instance_size = sizeof(BlockJobActionState),
          .prepare = blockdev_backup_prepare,
          .commit = blockdev_backup_commit,
          .abort = blockdev_backup_abort,
Signature · 2 lines
Best regards,
Vladimir
author
#9 lore raw

Re: [PATCH v3 2/6] blockdev: combine DriveBackupState and BlockdevBackupState

Quoted 6 lines · Vladimir Sementsov-Ogievskiy
Quoted 3 lines · Eric Blake

From: John Snow <[email protected]>

They have the same fields -- rename it BlockJobActionState.

commit/abort/clean functions are identical after it. I think better to combine them as well here

I'll give it a shot in v4 (may be done as a separate patch for ease of review, though).

Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#10 lore raw

PATCH v3 3/6 qmp: expose block-dirty-bitmap-populate

In reply to Eric Blake

From: John Snow <[email protected]>

This is a new job-creating command.

---
 qapi/block-core.json  | 18 +++++++++++
 qapi/transaction.json |  2 ++
 blockdev.c            | 74 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 94 insertions(+)
qapi/block-core.json +18 -0
index a1bcdba04423..313583b47c16 100644
@@ -2259,6 +2259,24 @@
             '*auto-finalize': 'bool',
             '*auto-dismiss': 'bool' } }
 
+##
+# @block-dirty-bitmap-populate:
+#
+# Creates a new job that writes a pattern into a dirty bitmap.
+#
+# Since: 5.1
+#
+# Example:
+#
+# -> { "execute": "block-dirty-bitmap-populate",
+#      "arguments": { "node": "drive0", "target": "bitmap0",
+#                     "job-id": "job0", "pattern": "allocate-top" } }
+# <- { "return": {} }
+#
+##
+{ 'command': 'block-dirty-bitmap-populate', 'boxed': true,
+  'data': 'BlockDirtyBitmapPopulate' }
+
 ##
 # @BlockDirtyBitmapSha256:
 #
qapi/transaction.json +2 -0
index b6c11158f0b6..21be59faae56 100644
@@ -50,6 +50,7 @@
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0
 # - @block-dirty-bitmap-merge: since 4.0
+# - @block-dirty-bitmap-populate: since 5.1
 # - @blockdev-backup: since 2.3
 # - @blockdev-snapshot: since 2.5
 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -67,6 +68,7 @@
        'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge',
+       'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
        'blockdev-backup': 'BlockdevBackup',
        'blockdev-snapshot': 'BlockdevSnapshot',
        'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +74 -0

92 lines not on this page · raw message

Signature · 1 line
2.27.0
author
#11 lore raw

PATCH v3 4/6 iotests: move bitmap helpers into their own file

In reply to Eric Blake

From: John Snow <[email protected]>

Message-Id: <[email protected]>

---
 tests/qemu-iotests/257        | 110 +---------------------------
 tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++
 2 files changed, 132 insertions(+), 109 deletions(-)
 create mode 100644 tests/qemu-iotests/bitmaps.py
tests/qemu-iotests/257 +1 -109
index 004a433b8be2..2a81f9e30c56 100755
@@ -24,120 +24,12 @@ import os
 
 import iotests
 from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS
 
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
 
 
-class Pattern:
-    def __init__(self, byte, offset, size=GRANULARITY):
-        self.byte = byte
-        self.offset = offset
-        self.size = size
-
-    def bits(self, granularity):
-        lower = self.offset // granularity
-        upper = (self.offset + self.size - 1) // granularity
-        return set(range(lower, upper + 1))
-
-
-class PatternGroup:
-    """Grouping of Pattern objects. Initialize with an iterable of Patterns."""
-    def __init__(self, patterns):
-        self.patterns = patterns
-
-    def bits(self, granularity):
-        """Calculate the unique bits dirtied by this pattern grouping"""
-        res = set()
-        for pattern in self.patterns:
-            res |= pattern.bits(granularity)
-        return res
-
-
-GROUPS = [
-    PatternGroup([
-        # Batch 0: 4 clusters
-        Pattern('0x49', 0x0000000),
-        Pattern('0x6c', 0x0100000),   # 1M
-        Pattern('0x6f', 0x2000000),   # 32M
-        Pattern('0x76', 0x3ff0000)]), # 64M - 64K
-    PatternGroup([
-        # Batch 1: 6 clusters (3 new)
-        Pattern('0x65', 0x0000000),   # Full overwrite
-        Pattern('0x77', 0x00f8000),   # Partial-left (1M-32K)
-        Pattern('0x72', 0x2008000),   # Partial-right (32M+32K)
-        Pattern('0x69', 0x3fe0000)]), # Adjacent-left (64M - 128K)
-    PatternGroup([
-        # Batch 2: 7 clusters (3 new)
-        Pattern('0x74', 0x0010000),   # Adjacent-right
-        Pattern('0x69', 0x00e8000),   # Partial-left  (1M-96K)
-        Pattern('0x6e', 0x2018000),   # Partial-right (32M+96K)
-        Pattern('0x67', 0x3fe0000,
-                2*GRANULARITY)]),     # Overwrite [(64M-128K)-64M)
-    PatternGroup([
-        # Batch 3: 8 clusters (5 new)
-        # Carefully chosen such that nothing re-dirties the one cluster
-        # that copies out successfully before failure in Group #1.
-        Pattern('0xaa', 0x0010000,
-                3*GRANULARITY),       # Overwrite and 2x Adjacent-right
-        Pattern('0xbb', 0x00d8000),   # Partial-left (1M-160K)
-        Pattern('0xcc', 0x2028000),   # Partial-right (32M+160K)
-        Pattern('0xdd', 0x3fc0000)]), # New; leaving a gap to the right
-]
-
-
-class EmulatedBitmap:
-    def __init__(self, granularity=GRANULARITY):
-        self._bits = set()
-        self.granularity = granularity
-
-    def dirty_bits(self, bits):
-        self._bits |= set(bits)
-
-    def dirty_group(self, n):
-        self.dirty_bits(GROUPS[n].bits(self.granularity))
-
-    def clear(self):
-        self._bits = set()
-
-    def clear_bits(self, bits):
-        self._bits -= set(bits)
-
-    def clear_bit(self, bit):
-        self.clear_bits({bit})
-
-    def clear_group(self, n):
-        self.clear_bits(GROUPS[n].bits(self.granularity))
-
-    @property
-    def first_bit(self):
-        return sorted(self.bits)[0]
-
-    @property
-    def bits(self):
-        return self._bits
-
-    @property
-    def count(self):
-        return len(self.bits)
-
-    def compare(self, qmp_bitmap):
-        """
-        Print a nice human-readable message checking that a bitmap as reported
-        by the QMP interface has as many bits set as we expect it to.
-        """
-
-        name = qmp_bitmap.get('name', '(anonymous)')
-        log("= Checking Bitmap {:s} =".format(name))
-
-        want = self.count
-        have = qmp_bitmap['count'] // qmp_bitmap['granularity']
-
-        log("expecting {:d} dirty sectors; have {:d}. {:s}".format(
-            want, have, "OK!" if want == have else "ERROR!"))
-        log('')
-
-
 class Drive:
     """Represents, vaguely, a drive attached to a VM.
     Includes format, graph, and device information."""
tests/qemu-iotests/bitmaps.py new file +131 -0

131 lines not on this page · raw message

Signature · 1 line
2.27.0
Reviewed-by on v3 4/6
#12 lore raw

Re: [PATCH v3 4/6] iotests: move bitmap helpers into their own file

Quoted 16 lines · Eric Blake

From: John Snow <[email protected]>

Message-Id: <[email protected]>

---
  tests/qemu-iotests/257        | 110 +---------------------------
  tests/qemu-iotests/bitmaps.py | 131 ++++++++++++++++++++++++++++++++++
  2 files changed, 132 insertions(+), 109 deletions(-)

create mode 100644 tests/qemu-iotests/bitmaps.py

tests/qemu-iotests/257 +0 -0
index 004a433b8be2..2a81f9e30c56 100755

tests/qemu-iotests/257 import os · line 27

Quoted 4 lines
tests/qemu-iotests/257 +1 -0
@@ -24,120 +24,12 @@ import os
 
  import iotests
  from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS

Exporting global variable of some unknown to this module structure doesn't seem to be good module design. One day we may want to refactor this.. For now:

Signature · 2 lines
Best regards,
Vladimir
author
#13 lore raw

PATCH v3 5/6 iotests: add 298 for block-dirty-bitmap-populate

In reply to Eric Blake

From: John Snow <[email protected]>

Give block-dirty-bitmap-populate a workout.

---
 tests/qemu-iotests/298     |  232 ++
 tests/qemu-iotests/298.out | 4544 ++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/group   |    1 +
 3 files changed, 4777 insertions(+)
 create mode 100755 tests/qemu-iotests/298
 create mode 100644 tests/qemu-iotests/298.out
tests/qemu-iotests/298 new file +232 -0
new file mode 100755
index 000000000000..4bfcecd3bc88
@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+#
+# Test block-dirty-bitmap-populate
+#
+# Copyright (c) 2020 John Snow for Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+# [email protected]
+
+from collections import namedtuple
+import itertools
+import math
+import os
+
+import iotests
+from iotests import log, qemu_img
+from bitmaps import EmulatedBitmap, GROUPS
+
+SIZE = 64 * 1024 * 1024
+GRANULARITY = 64 * 1024
+
+
+class Drive:
+    def __init__(self, path, vm):
+        self.path = path
+        self.vm = vm
+        self.fmt = None
+        self.size = None
+        self.node = None
+
+    def img_create(self, fmt, size):
+        self.fmt = fmt
+        self.size = size
+        iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size))
+
+
+def block_dirty_bitmap_populate(vm, node, bitmap, job_id, pattern, **kwargs):
+    # Strip any arguments explicitly nulled by the caller:
+    kwargs = {key: val for key, val in kwargs.items() if val is not None}
+    result = vm.qmp_log('block-dirty-bitmap-populate',
+                        node=node,
+                        name=bitmap,
+                        job_id=job_id,
+                        pattern=pattern,
+                        **kwargs)
+    return result
+
+
+def populate(drive, bitmap, job_id, pattern='allocation-top', **kwargs):
+    kwargs.setdefault('pattern', pattern)
+    kwargs.setdefault('auto-finalize', False)
+    kwargs.setdefault('auto-dismiss', False)
+    ret = block_dirty_bitmap_populate(drive.vm, drive.node,
+                                      bitmap, job_id, **kwargs)
+    return {
+        'id': job_id,
+        'auto-finalize': kwargs['auto-finalize'],
+        'auto-dismiss': kwargs['auto-dismiss'],
+        'return': ret,
+    }
+
+
+def perform_writes(drive, n, filter_node_name=None):
+    log("-- Write #{:d}:".format(n))
+    node_name = filter_node_name or drive.node
+    for pattern in GROUPS[n].patterns:
+        cmd = "write -P{:s} 0x{:07x} 0x{:x}".format(
+            pattern.byte,
+            pattern.offset,
+            pattern.size)
+        drive.vm.hmp_qemu_io(node_name, cmd, use_log=True)
+    log('')
+
+
+TestConfig = namedtuple('TestConfig', [
+    'base_pattern',
+    'disabled',
+    'pre_writes',
+    'mid_writes',
+    'cancel',
+    'post_writes',
+])
+
+
+def test_bitmap_populate(config):
+    """
+    Test bitmap populate.
+
+    :param base_pattern: Write a base pattern?
+    :param disabled:     Disable the target bitmap?
+    :param pre_writes:   Write a pattern after bitmap creation, but before job?
+    :param mid_writes:   Write a pattern before job finalizes?
+    :param cancel:       Cancel the job instead of finalizing it?
+    :param post_writes:  Write a pattern after the job?
+    """
+    with iotests.FilePaths(['img']) as (img_path,), iotests.VM() as vm:
+        log("\n=== Bitmap Populate {:s} ===\n".format(str(config)))
+
+        log('-- Prepare image & VM:')
+        drive0 = Drive(img_path, vm=vm)
+        drive0.img_create(iotests.imgfmt, SIZE)
+        vm.add_device("{},id=scsi0".format(iotests.get_virtio_scsi_device()))
+        vm.launch()
+
+        file_config = {
+            'driver': 'file',
+            'filename': drive0.path
+        }
+
+        drive0.node = 'drive0'
+        vm.qmp_log('blockdev-add',
+                   filters=[iotests.filter_qmp_testfiles],
+                   node_name=drive0.node,
+                   driver=drive0.fmt,
+                   file=file_config)
+        log('')
+
+
+        # Step 0: Prepare & Base Allocation Pattern
+
+        if config.base_pattern:
+            perform_writes(drive0, 0)
+
+
+        # Step 1: Add test bitmap
+
+        log('-- Add Bitmap:')
+        vm.qmp_log('block-dirty-bitmap-add',
+                   node=drive0.node,
+                   name="target",
+                   granularity=GRANULARITY,
+                   disabled=config.disabled)
+        ebitmap = EmulatedBitmap()
+        log('')
+
+        # Step 2: Pre-Writes
+
+        if config.pre_writes:
+            perform_writes(drive0, 1)
+            if not config.disabled:
+                ebitmap.dirty_group(1)
+            bitmap = vm.get_bitmap(drive0.node, 'target')
+            ebitmap.compare(bitmap)
+
+
+        # Step 3: Launch job & Mid-Writes
+
+        log('-- Test block-dirty-bitmap-populate (bitpop0):')
+        def pre_finalize():
+            # Writes issued prior to job finalization:
+            if config.mid_writes:
+                perform_writes(drive0, 2)
+                if not config.disabled:
+                    ebitmap.dirty_group(2)
+
+        job = populate(drive0, 'target', 'bitpop0')
+        assert job['return'] == {'return': {}}
+        vm.run_job(job['id'], auto_dismiss=job['auto-dismiss'],
+                   auto_finalize=job['auto-finalize'],
+                   pre_finalize=pre_finalize,
+                   cancel=config.cancel)
+        log('')
+
+
+        # Step 4: Post-job verification
+
+        if not config.cancel:
+            # Any writes made prior to the job finishing should now be visible.
+            if config.base_pattern:
+                ebitmap.dirty_group(0)
+            if config.pre_writes:
+                ebitmap.dirty_group(1)
+            if config.mid_writes:
+                ebitmap.dirty_group(2)
+
+        bitmap = vm.get_bitmap(drive0.node, 'target')
+        ebitmap.compare(bitmap)
+
+
+        # Step 5: Post-Writes
+
+        if config.post_writes:
+            perform_writes(drive0, 3)
+            if not config.disabled:
+                ebitmap.dirty_group(3)
+
+
+        # Step 6: Final Verification
+
+        log('-- Verification:')
+        bitmaps = vm.query_bitmaps()
+        log({'bitmaps': bitmaps}, indent=2)
+        log('')
+        bitmap = vm.get_bitmap(drive0.node, 'target', bitmaps=bitmaps)
+        ebitmap.compare(bitmap)
+
+
+        log('-- Cleanup:')
+        vm.qmp_log("block-dirty-bitmap-remove",
+                   node=drive0.node, name="target")
+
+        bitmaps = vm.query_bitmaps()
+        if bitmaps:
+            log("ERROR: bitmaps unaccounted for:")
+            log(bitmaps)
+        else:
+            log('OK: All bitmaps removed')
+        vm.shutdown()
+        log('')
+
+
+def main():
+    for args in itertools.product((True, False), repeat=6):
+        cfg = TestConfig(*args)
+        test_bitmap_populate(cfg)
+
+
+if __name__ == '__main__':
+    iotests.script_main(main, supported_fmts=['qcow2'],
+                        supported_protocols=['file'])
tests/qemu-iotests/298.out new file +4544 -0

2,768 lines not on this page · raw message

tests/qemu-iotests/group +1 -0
index d886fa0cb355..9146aae84081 100644

Lines of this file are not shown. View the raw message.

The rest of this diff is not shown. View the raw message.

Signature · 1 line
2.27.0

Part of the diff is not shown here. View the raw message.

author
#14 lore raw

PATCH v3 6/6 bitmaps: Use x- prefix for block-dirty-bitmap-popluate

In reply to Eric Blake

Give ourselves an out if we need to tweak the interface, in order to gain more experience with what works when libvirt experiments with using it.

---
 qapi/block-core.json       |   6 +-
 qapi/transaction.json      |   4 +-
 blockdev.c                 |  14 ++--
 tests/qemu-iotests/298     |   2 +-
 tests/qemu-iotests/298.out | 128 ++++++++++++++++++-------------------
 5 files changed, 77 insertions(+), 77 deletions(-)
qapi/block-core.json +3 -3
index 313583b47c16..dcf6b907e45c 100644
@@ -2260,7 +2260,7 @@
             '*auto-dismiss': 'bool' } }
 
 ##
-# @block-dirty-bitmap-populate:
+# @x-block-dirty-bitmap-populate:
 #
 # Creates a new job that writes a pattern into a dirty bitmap.
 #
@@ -2268,13 +2268,13 @@
 #
 # Example:
 #
-# -> { "execute": "block-dirty-bitmap-populate",
+# -> { "execute": "x-block-dirty-bitmap-populate",
 #      "arguments": { "node": "drive0", "target": "bitmap0",
 #                     "job-id": "job0", "pattern": "allocate-top" } }
 # <- { "return": {} }
 #
 ##
-{ 'command': 'block-dirty-bitmap-populate', 'boxed': true,
+{ 'command': 'x-block-dirty-bitmap-populate', 'boxed': true,
   'data': 'BlockDirtyBitmapPopulate' }
 
 ##
qapi/transaction.json +2 -2
index 21be59faae56..3277e948f321 100644
@@ -50,7 +50,7 @@
 # - @block-dirty-bitmap-enable: since 4.0
 # - @block-dirty-bitmap-disable: since 4.0
 # - @block-dirty-bitmap-merge: since 4.0
-# - @block-dirty-bitmap-populate: since 5.1
+# - @x-block-dirty-bitmap-populate: since 5.1
 # - @blockdev-backup: since 2.3
 # - @blockdev-snapshot: since 2.5
 # - @blockdev-snapshot-internal-sync: since 1.7
@@ -68,7 +68,7 @@
        'block-dirty-bitmap-enable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-disable': 'BlockDirtyBitmap',
        'block-dirty-bitmap-merge': 'BlockDirtyBitmapMerge',
-       'block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
+       'x-block-dirty-bitmap-populate': 'BlockDirtyBitmapPopulate',
        'blockdev-backup': 'BlockdevBackup',
        'blockdev-snapshot': 'BlockdevSnapshot',
        'blockdev-snapshot-internal-sync': 'BlockdevSnapshotInternal',
blockdev.c +7 -7
index d072519e7b91..b86ef5b7f281 100644
@@ -2188,8 +2188,8 @@ static void block_dirty_bitmap_populate_prepare(BlkActionState *common,
     int job_flags = JOB_DEFAULT;
 
     assert(common->action->type ==
-           TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE);
-    bitpop = common->action->u.block_dirty_bitmap_populate.data;
+           TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_POPULATE);
+    bitpop = common->action->u.x_block_dirty_bitmap_populate.data;
 
     bmap = block_dirty_bitmap_lookup(bitpop->node, bitpop->name, &bs, errp);
     if (!bmap) {
@@ -2317,7 +2317,7 @@ static const BlkActionOps actions[] = {
         .commit = block_dirty_bitmap_remove_commit,
         .abort = block_dirty_bitmap_remove_abort,
     },
-    [TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE] = {
+    [TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_POPULATE] = {
         .instance_size = sizeof(BlockJobActionState),
         .prepare = block_dirty_bitmap_populate_prepare,
         .commit = blockdev_backup_commit,
@@ -2443,12 +2443,12 @@ void qmp_block_passwd(bool has_device, const char *device,
                "Setting block passwords directly is no longer supported");
 }
 
-void qmp_block_dirty_bitmap_populate(BlockDirtyBitmapPopulate *bitpop,
-                                     Error **errp)
+void qmp_x_block_dirty_bitmap_populate(BlockDirtyBitmapPopulate *bitpop,
+                                       Error **errp)
 {
     TransactionAction action = {
-        .type = TRANSACTION_ACTION_KIND_BLOCK_DIRTY_BITMAP_POPULATE,
-        .u.block_dirty_bitmap_populate.data = bitpop,
+        .type = TRANSACTION_ACTION_KIND_X_BLOCK_DIRTY_BITMAP_POPULATE,
+        .u.x_block_dirty_bitmap_populate.data = bitpop,
     };
     blockdev_do_action(&action, errp);
 }
tests/qemu-iotests/298 +1 -1
index 4bfcecd3bc88..2a3df2de85db 100755
@@ -49,7 +49,7 @@ class Drive:
 def block_dirty_bitmap_populate(vm, node, bitmap, job_id, pattern, **kwargs):
     # Strip any arguments explicitly nulled by the caller:
     kwargs = {key: val for key, val in kwargs.items() if val is not None}
-    result = vm.qmp_log('block-dirty-bitmap-populate',
+    result = vm.qmp_log('x-block-dirty-bitmap-populate',
                         node=node,
                         name=bitmap,
                         job_id=job_id,
tests/qemu-iotests/298.out +64 -64

512 lines not on this page · raw message

Signature · 1 line
2.27.0
#15 lore raw

Re: [PATCH v3 6/6] bitmaps: Use x- prefix for block-dirty-bitmap-popluate

Am 19.06.2020 um 21:56 hat Eric Blake geschrieben:

Quoted 5 lines

Give ourselves an out if we need to tweak the interface, in order to gain more experience with what works when libvirt experiments with using it.

I thought libvirt wants to consume this as soon as possible? With an x- prefix, we'll prevent it from using the interface.

Kevin

#16 lore raw
In reply to Eric Blake

Am 19.06.2020 um 21:56 hat Eric Blake geschrieben:

Quoted 19 lines

[From John's original cover letter:]
This is a new (very small) block job that writes a pattern into a bitmap. The only pattern implemented is the top allocation information.

This can be used to "recover" an incremental bitmap chain if an external snapshot was taken without creating a new bitmap first: any writes made to the image will be reflected by the allocation status and can be written back into a bitmap.

This is useful for e.g. libvirt managing backup chains if a user creates an external snapshot outside of libvirt.

v3:
- Addressed a bit more feedback
- Make it easier to decide if we want an x- prefix if we think there are more tweaks to be made to the interface
- Drop dependency on John's JobRunner iotest series
- Renumber the new iotest

Quoted 6 lines

I know there was a lot of discussion about whether there are optimizations to be made with populating directly into the target bitmap rather than into a temporary that then gets merged in at the completion of the job, but the QMP aspect seems fairly stable. Even so, we may still want to consider using an x- prefix until we know for sure whether libvirt can make decent use of the interface.

Fair enough, as long as everyone agrees that the QMP aspect includes that merging the bits only at the end of the job is an implementation detail and not something that client may rely on.

Kevin