Skip to contentSkip to conversation

block: Deal with filters

PATCH Superseded Max Reitz 6 participants 173 messages Jun 25, 2020 – Aug 24, 2020 Block layer Page 1 of 2

v7 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v7 cover block: Deal with filters

v6: https://lists.nongnu.org/archive/html/qemu-devel/2019-08/msg01715.html

Branch: https://github.com/XanClic/qemu.git child-access-functions-v7
Branch: https://git.xanclic.moe/XanClic/qemu.git child-access-functions-v7

Hello!

This is v7. Conceptually, not much has changed, so please follow the above link to v6’s cover letter if you’re looking for an introduction to this series.

I did say that conceptually, not much has changed, but from a diff standpoint, a lot has changed all over this series.

Changes from v6:
- Patch 1:
- More elaborate explanation of .is_filter
- Changed function names
- Dropped bdrv_storage_child() and bdrv_metadata_child()
- Some checking of the BdrvChildRole

- Patch 2:
- Mostly changes resulting from the different naming scheme

- Patch 3: New

- Patch 5:
- Don’t rename those functions
- Don’t drop a comment that shouldn’t be dropped

- Patch 7:
- Use block_driver_can_compress()
- Move setting @filtered down where it’s needed

- Patches 10 and 11: New (extension of 8 and 9)

- Patch 12:
- Function name changes
- More cases:
- bdrv_recurse_can_replace()
- init_dirty_bitmap_migration()
- bdrv_co_truncate() has changed

- Patch 13:
- Function name changes

- Patch 14:
- Variables renamed to be more consistent with the rest of this series
- Function name changes
- The freeze backing chain functions haven’t been renamed
- STREAM_BUFFER_SIZE is STREAM_CHUNK as of some point last year
- Fix overlay finding (e.g. handle when @base is not in the device’s backing chain)

- Patch 15:
- Added note to the commit message that bdrv_find_overlay()’s behavior changes a bit
- Function name changes
- Restructured bdrv_find_backing_image() loop a bit

- Patch 16: New (became necessary because of truncate having to look at

            the backing file as of 955c7d6687fefcd903900)

- Patch 17:
- Function name changes
- The freeze backing chain functions haven’t been renamed

- Patch 18:
- Only flush children for which the parent has taken the WRITE permission
- Mention that this is a bug fix for qcow2

- Patch 19: New

- Patch 20: New, replaces “block: Use CAFs in bdrv_refresh_limits()”

- Patch 23:
- We can only really fall back to bs->file or bs->backing, so stop pretending otherwise

- Patch 24:
- Rebase conflicts

- Patches 25, 26, 27, and 28: New, they replace “block: Fix bdrv_get_allocated_file_size's fallback”

- Patch 29:
- Function name changes

- Patch 30: New, split out from the next patch

- Patch 31:
- Function name changes
- bdrv_skip_implicit_filters() can deal with NULL arguments, so don’t wrap it in an “if (bs) {}” block
- For bdrv_query_bds_stats(), the bs->file part has been split into the preceding patch (patch 30)
- Addional actual-size line in the iotest output thanks to patch 28

- Patch 32: New

- Patch 33:
- Additional note in the QAPI documentation concerning @replaces (that by default, the first non-implicit node on @device is replaced)
- Move that skippage of implicit nodes to blockdev_mirror_common() (from qmp_drive_mirror() and qmp_blockdev_mirror())
- Function name changes
- Rename s/source/target_backing_bs/ in qmp_drive_mirror(), because that’s better
- Don’t disallow mirroring through filters with sync=top

- Patch 34:
- Function name changes
- There is backup-top.c to care about, too, now

- Patch 35:
- Function name changes
- Call bdrv_commit() even for nodes that do not have backing files so we get an error
- s/above_base/base_overlay/

- Patch 36:
- Function name changes

- Patch 37:
- Function name changes
- In img_convert(), when inquiring target_backing_sectors, use bdrv_backing_chain_next() instead of bdrv_cow_bs() (because @out_bs may be a filter)
- Forgot to use the backing file of @unfiltered_bs for bdrv_is_allocated_above() in img_rebase() (instead of @unfiltered_bs itself), fixed

- Patch 39:
- Function name changes

- Patch 40:
- Function name changes
- There are backup-top and filter-compress now

- Patch 41:
- Make bdrv_backing_overridden() globally available, so block/qapi.c can use it to determine whether we can inquire bs->backing’s format to get the backing_format
- Function name changes

- Patch 42: New

- Patch 43:
- Rebase conflicts

- Patch 44:
- Create a dedicated do_test_io() function
- Don’t unnecessarily clear and pass has_quit
- Drop assert_no_active_block_jobs() that does very little
- Additional graph constraint check
- Rebase conflict in the reference output

- Patch 45:
- Rebase conflict in the reference output

- Patch 46:
- Use _rm_test_img rather than rm -f
- Skip one of the test cases when IMGOPTS asks for a data_file

- Patch 47:
- Rebase conflict in the reference output

Patches removed:
- The whole check_to_replace_node() stuff for mirror (was its own series)

- Making bdrv_get_cumulative_perm() public, because it already was

- bdrv_storage_child() (was replaced by child roles)

git-backport-diff against v6:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/47:[0126] [FC] 'block: Add child access functions' 002/47:[0032] [FC] 'block: Add chain helper functions' 003/47:[down] 'block: bdrv_cow_child() for bdrv_has_zero_init()' 004/47:[----] [-C] 'block: bdrv_set_backing_hd() is about bs->backing' 005/47:[0067] [FC] 'block: Include filters when freezing backing chain' 006/47:[----] [--] 'block: Drop bdrv_is_encrypted()' 007/47:[0005] [FC] 'block: Add bdrv_supports_compressed_writes()' 008/47:[----] [-C] 'throttle: Support compressed writes' 009/47:[----] [--] 'copy-on-read: Support compressed writes' 010/47:[down] 'mirror-top: Support compressed writes' 011/47:[down] 'backup-top: Support compressed writes' 012/47:[0036] [FC] 'block: Use bdrv_filtered_rw* where obvious' 013/47:[0010] [FC] 'block: Use CAFs in block status functions' 014/47:[0079] [FC] 'stream: Deal with filters'
015/47:[0015] [FC] 'block: Use CAFs when working with backing chains' 016/47:[down] 'block: Use bdrv_cow_child() in bdrv_co_truncate()' 017/47:[0019] [FC] 'block: Re-evaluate backing file handling in reopen' 018/47:[0010] [FC] 'block: Flush all children in generic code' 019/47:[down] 'vmdk: Drop vmdk_co_flush()'
020/47:[down] 'block: Iterate over children in refresh_limits' 021/47:[----] [--] 'block: Use CAFs in bdrv_refresh_filename()' 022/47:[----] [--] 'block: Use CAF in bdrv_co_rw_vmstate()' 023/47:[0094] [FC] 'block/snapshot: Fix fallback' 024/47:[0014] [FC] 'block: Use CAFs for debug breakpoints' 025/47:[down] 'block: Def. impl.s for get_allocated_file_size' 026/47:[down] 'block: Improve get_allocated_file_size's default' 027/47:[down] 'blkverify: Use bdrv_sum_allocated_file_size()' 028/47:[down] 'block/null: Implement bdrv_get_allocated_file_size' 029/47:[0002] [FC] 'blockdev: Use CAF in external_snapshot_prepare()' 030/47:[down] 'block: Report data child for query-blockstats' 031/47:[0031] [FC] 'block: Use child access functions for QAPI queries' 032/47:[down] 'block-copy: Use CAF to find sync=top base' 033/47:[0086] [FC] 'mirror: Deal with filters'
034/47:[0006] [FC] 'backup: Deal with filters'
035/47:[0035] [FC] 'commit: Deal with filters'
036/47:[0002] [FC] 'nbd: Use CAF when looking for dirty bitmap' 037/47:[0017] [FC] 'qemu-img: Use child access functions' 038/47:[----] [--] 'block: Drop backing_bs()'
039/47:[0002] [FC] 'blockdev: Fix active commit choice' 040/47:[0009] [FC] 'block: Inline bdrv_co_block_status_from_*()' 041/47:[0019] [FC] 'block: Leave BDS.backing_file constant' 042/47:[down] 'iotests: Test that qcow2's data-file is flushed' 043/47:[0016] [FC] 'iotests: Let complete_and_wait() work with commit' 044/47:[0042] [FC] 'iotests: Add filter commit test cases' 045/47:[0008] [FC] 'iotests: Add filter mirror test cases' 046/47:[0018] [FC] 'iotests: Add test for commit in sub directory' 047/47:[0008] [FC] 'iotests: Test committing to overridden backing'

Max Reitz (47):
block: Add child access functions
block: Add chain helper functions
block: bdrv_cow_child() for bdrv_has_zero_init()
block: bdrv_set_backing_hd() is about bs->backing
block: Include filters when freezing backing chain
block: Drop bdrv_is_encrypted()
block: Add bdrv_supports_compressed_writes()
throttle: Support compressed writes
copy-on-read: Support compressed writes
mirror-top: Support compressed writes
backup-top: Support compressed writes
block: Use bdrv_filter_(bs|child) where obvious
block: Use CAFs in block status functions
stream: Deal with filters
block: Use CAFs when working with backing chains
block: Use bdrv_cow_child() in bdrv_co_truncate()
block: Re-evaluate backing file handling in reopen
block: Flush all children in generic code
vmdk: Drop vmdk_co_flush()
block: Iterate over children in refresh_limits
block: Use CAFs in bdrv_refresh_filename()
block: Use CAF in bdrv_co_rw_vmstate() block/snapshot: Fix fallback
block: Use CAFs for debug breakpoints
block: Def. impl.s for get_allocated_file_size
block: Improve get_allocated_file_size's default
blkverify: Use bdrv_sum_allocated_file_size() block/null: Implement bdrv_get_allocated_file_size
blockdev: Use CAF in external_snapshot_prepare()
block: Report data child for query-blockstats
block: Use child access functions for QAPI queries
block-copy: Use CAF to find sync=top base
mirror: Deal with filters
backup: Deal with filters
commit: Deal with filters
nbd: Use CAF when looking for dirty bitmap
qemu-img: Use child access functions
block: Drop backing_bs()
blockdev: Fix active commit choice
block: Inline bdrv_co_block_status_from_*()
block: Leave BDS.backing_file constant
iotests: Test that qcow2's data-file is flushed
iotests: Let complete_and_wait() work with commit
iotests: Add filter commit test cases
iotests: Add filter mirror test cases
iotests: Add test for commit in sub directory
iotests: Test committing to overridden backing

 qapi/block-core.json           |  10 +-
 include/block/block.h          |   2 +-
 include/block/block_int.h      |  99 ++++---
 block.c                        | 500 +++++++++++++++++++++++++++------
 block/backup-top.c             |  14 +-
 block/backup.c                 |   9 +-
 block/blkdebug.c               |   7 +-
 block/blklogwrites.c           |   1 -
 block/blkverify.c              |   1 +
 block/block-backend.c          |   9 +-
 block/block-copy.c             |   4 +-
 block/commit.c                 |  97 +++++--
 block/copy-on-read.c           |  13 +-
 block/filter-compress.c        |   2 -
 block/io.c                     | 142 +++++-----
 block/mirror.c                 | 129 +++++++--
 block/monitor/block-hmp-cmds.c |   2 +-
 block/null.c                   |   7 +
 block/qapi.c                   |  83 ++++--
 block/snapshot.c               | 104 +++++--
 block/stream.c                 |  63 +++--
 block/throttle.c               |  11 +-
 block/vmdk.c                   |  16 --
 blockdev.c                     |  94 +++++--
 migration/block-dirty-bitmap.c |   8 +-
 nbd/server.c                   |   6 +-
 qemu-img.c                     |  36 ++-
 tests/qemu-iotests/020         |  44 +++
 tests/qemu-iotests/020.out     |  10 +
 tests/qemu-iotests/040         | 238 ++++++++++++++++
 tests/qemu-iotests/040.out     |   4 +-
 tests/qemu-iotests/041         | 146 +++++++++-
 tests/qemu-iotests/041.out     |   4 +-
 tests/qemu-iotests/153.out     |   2 +-
 tests/qemu-iotests/184.out     |  14 +-
 tests/qemu-iotests/204.out     |   1 +
 tests/qemu-iotests/228         |   6 +-
 tests/qemu-iotests/228.out     |   6 +-
 tests/qemu-iotests/244         |  49 ++++
 tests/qemu-iotests/244.out     |   7 +
 tests/qemu-iotests/245         |   4 +-
 tests/qemu-iotests/iotests.py  |  10 +-
 42 files changed, 1602 insertions(+), 412 deletions(-)
Signature · 1 line
2.26.2
author
#1 lore raw

PATCH v7 1/47 block: Add child access functions

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.

---
 include/block/block_int.h | 44 +++++++++++++++++--
 block.c                   | 90 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 131 insertions(+), 3 deletions(-)
include/block/block_int.h +41 -3
index 1b86b59af1..bb3457c5e8 100644
@@ -90,9 +90,17 @@ struct BlockDriver {
     int instance_size;
 
     /* set to true if the BlockDriver is a block filter. Block filters pass
-     * certain callbacks that refer to data (see block.c) to their bs->file if
-     * the driver doesn't implement them. Drivers that do not wish to forward
-     * must implement them and return -ENOTSUP.
+     * certain callbacks that refer to data (see block.c) to their bs->file
+     * or bs->backing (whichever one exists) if the driver doesn't implement
+     * them. Drivers that do not wish to forward must implement them and return
+     * -ENOTSUP.
+     * Note that filters are not allowed to modify data.
+     *
+     * Filters generally cannot have more than a single filtered child,
+     * because the data they present must at all times be the same as
+     * that on their filtered child.  That would be impossible to
+     * achieve for multiple filtered children.
+     * (And this filtered child must then be bs->file or bs->backing.)
      */
     bool is_filter;
     /*
@@ -1370,4 +1378,34 @@ BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name,
                                            BlockDriverState **bitmap_bs,
                                            Error **errp);
 
+BdrvChild *bdrv_cow_child(BlockDriverState *bs);
+BdrvChild *bdrv_filter_child(BlockDriverState *bs);
+BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs);
+BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+
+static inline BlockDriverState *child_bs(BdrvChild *child)
+{
+    return child ? child->bs : NULL;
+}
+
+static inline BlockDriverState *bdrv_cow_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_cow_child(bs));
+}
+
+static inline BlockDriverState *bdrv_filter_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_filter_child(bs));
+}
+
+static inline BlockDriverState *bdrv_filter_or_cow_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_filter_or_cow_child(bs));
+}
+
+static inline BlockDriverState *bdrv_primary_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_primary_child(bs));
+}
+
 #endif /* BLOCK_INT_H */
block.c +90 -0

93 lines not on this page · raw message

Signature · 1 line
2.26.2
Reviewed-by on v7 1/47
#2 lore raw

Re: [PATCH v7 01/47] block: Add child access functions

Quoted 23 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.

---
  include/block/block_int.h | 44 +++++++++++++++++--
  block.c                   | 90 +++++++++++++++++++++++++++++++++++++++
  2 files changed, 131 insertions(+), 3 deletions(-)
include/block/block_int.h +0 -0
index 1b86b59af1..bb3457c5e8 100644

include/block/block_int.h struct BlockDriver { · line 91

Quoted 3 lines
include/block/block_int.h +0 -0
@@ -90,9 +90,17 @@ struct BlockDriver {
      int instance_size;
 
...
Reviewed-by on v7 1/47
#3 lore raw

Re: [PATCH v7 01/47] block: Add child access functions

In reply to Max Reitz
Quoted 14 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.

---

[..]

Quoted 13 lines
(quoted diff) +12 -0
+/*
+ * Return the primary child of this node: For filters, that is the
+ * filtered child.  For other nodes, that is usually the child storing
+ * metadata.
+ * (A generally more helpful description is that this is (usually) the
+ * child that has the same filename as @bs.)
+ *
+ * Drivers do not necessarily have a primary child; for example quorum
+ * does not.
+ */
+BdrvChild *bdrv_primary_child(BlockDriverState *bs)
+{

Review

Quoted 10 lines
(quoted diff) +10 -0
+    BdrvChild *c;
+
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & BDRV_CHILD_PRIMARY) {
+            return c;
+        }
+    }
+
+    return NULL;
+}

Suggest squash-in to also assert that not more than one primary child:

block.c +4 -3
@@ -6998,13 +6998,14 @@ BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs)
  */
 BdrvChild *bdrv_primary_child(BlockDriverState *bs)
 {
-    BdrvChild *c;
+    BdrvChild *c, *found = NULL;
 
     QLIST_FOREACH(c, &bs->children, next) {
         if (c->role & BDRV_CHILD_PRIMARY) {
-            return c;
+            assert(!found);
+            found = c;
         }
     }
 
-    return NULL;
+    return c;
 }

with or without:

Signature · 2 lines
Best regards,
Vladimir
author
#4 lore raw

Re: [PATCH v7 01/47] block: Add child access functions

Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing.  Since the introduction of filters and external data files, their meaning is not quite clear.  bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful.  This patch adds functions that retrieve the correct child for each exact purpose.  Later patches in this series will make use of them.  Doing so will allow us to handle filter nodes in a meaningful way.

---

[..]

Quoted 13 lines
(quoted diff) +12 -0
+/*
+ * Return the primary child of this node: For filters, that is the
+ * filtered child.  For other nodes, that is usually the child storing
+ * metadata.
+ * (A generally more helpful description is that this is (usually) the
+ * child that has the same filename as @bs.)
+ *
+ * Drivers do not necessarily have a primary child; for example quorum
+ * does not.
+ */
+BdrvChild *bdrv_primary_child(BlockDriverState *bs)
+{

Review

Quoted 10 lines
(quoted diff) +10 -0
+    BdrvChild *c;
+
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & BDRV_CHILD_PRIMARY) {
+            return c;
+        }
+    }
+
+    return NULL;
+}

Suggest squash-in to also assert that not more than one primary child:

block.c +1 -1
@@ -6998,13 +6998,14 @@ BdrvChild
 *bdrv_filter_or_cow_child(BlockDriverState *bs)
   */
  BdrvChild *bdrv_primary_child(BlockDriverState *bs)
  {
-    BdrvChild *c;
+    BdrvChild *c, *found = NULL;
Quoted 5 lines


     QLIST_FOREACH(c, &bs->children, next) {
         if (c->role & BDRV_CHILD_PRIMARY) {
-            return c;
+            assert(!found);

Hm, why not.

Quoted 11 lines

+            found = c;
         }
     }

-    return NULL;
+    return c;
 }

with or without:

In any case, thanks for reviewing!

Max

#5 lore raw

Re: [PATCH v7 01/47] block: Add child access functions

In reply to Vladimir Sementsov-Ogievskiy

On 2020-07-13 at 11:06 CEST, Vladimir Sementsov-Ogievskiy wrote...

Quoted 52 lines
Quoted 14 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.

---

[..]

Quoted 13 lines
(quoted diff) +12 -0
+/*
+ * Return the primary child of this node: For filters, that is the
+ * filtered child.  For other nodes, that is usually the child storing
+ * metadata.
+ * (A generally more helpful description is that this is (usually) the
+ * child that has the same filename as @bs.)
+ *
+ * Drivers do not necessarily have a primary child; for example quorum
+ * does not.
+ */
+BdrvChild *bdrv_primary_child(BlockDriverState *bs)
+{

Review

Quoted 10 lines
(quoted diff) +10 -0
+    BdrvChild *c;
+
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & BDRV_CHILD_PRIMARY) {
+            return c;
+        }
+    }
+
+    return NULL;
+}

Suggest squash-in to also assert that not more than one primary child:

block.c +1 -1
@@ -6998,13 +6998,14 @@ BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs)
   */
  BdrvChild *bdrv_primary_child(BlockDriverState *bs)
  {
-    BdrvChild *c;
+    BdrvChild *c, *found = NULL;
 

block.c BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs) · line 7010

Quoted 10 lines
block.c +3 -2
      QLIST_FOREACH(c, &bs->children, next) {
          if (c->role & BDRV_CHILD_PRIMARY) {
-            return c;
+            assert(!found);
+            found = c;
          }
      }
 
-    return NULL;
+    return c;

Shouldn't that be "return found"?

Quoted 5 lines
block.c +0 -0
  }

with or without:

--
Cheers,
Christophe de Dinechin (IRC c3d)

#6 lore raw

Re: [PATCH v7 01/47] block: Add child access functions

Quoted 62 lines · Christophe de Dinechin

On 2020-07-13 at 11:06 CEST, Vladimir Sementsov-Ogievskiy wrote...

Quoted 60 lines
Quoted 14 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.

---

[..]

Quoted 13 lines
(quoted diff) +12 -0
+/*
+ * Return the primary child of this node: For filters, that is the
+ * filtered child.  For other nodes, that is usually the child storing
+ * metadata.
+ * (A generally more helpful description is that this is (usually) the
+ * child that has the same filename as @bs.)
+ *
+ * Drivers do not necessarily have a primary child; for example quorum
+ * does not.
+ */
+BdrvChild *bdrv_primary_child(BlockDriverState *bs)
+{

Review

Quoted 10 lines
(quoted diff) +10 -0
+    BdrvChild *c;
+
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & BDRV_CHILD_PRIMARY) {
+            return c;
+        }
+    }
+
+    return NULL;
+}

Suggest squash-in to also assert that not more than one primary child:

block.c +3 -2
@@ -6998,13 +6998,14 @@ BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs)
    */
   BdrvChild *bdrv_primary_child(BlockDriverState *bs)
   {
-    BdrvChild *c;
+    BdrvChild *c, *found = NULL;
 
       QLIST_FOREACH(c, &bs->children, next) {
           if (c->role & BDRV_CHILD_PRIMARY) {
-            return c;
+            assert(!found);
+            found = c;
           }
       }
 
Quoted 4 lines

block.c BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs) · line 7010

Quoted 2 lines
block.c +1 -1
-    return NULL;
+    return c;

Shouldn't that be "return found"?

Oops, you are right of course!

Quoted 11 lines
Quoted 5 lines
block.c +0 -0
   }

with or without:

--
Cheers,
Christophe de Dinechin (IRC c3d)

Signature · 2 lines
Best regards,
Vladimir
#7 lore raw

Re: [PATCH v7 01/47] block: Add child access functions

In reply to Max Reitz
Quoted 22 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.

---
  include/block/block_int.h | 44 +++++++++++++++++--
  block.c                   | 90 +++++++++++++++++++++++++++++++++++++++
  2 files changed, 131 insertions(+), 3 deletions(-)
include/block/block_int.h +0 -0
index 1b86b59af1..bb3457c5e8 100644

[..]

include/block/block_int.h

Quoted 5 lines
include/block/block_int.h +5 -0
+/*
+ * If @bs acts as a filter for exactly one of its children, return
+ * that child.
+ */
+BdrvChild *bdrv_filter_child(BlockDriverState *bs)

Hmm you called it filter_child instead of filterED_child..

Quoted 10 lines
include/block/block_int.h +10 -0
+{
+    BdrvChild *c;
+
+    if (!bs || !bs->drv) {
+        return NULL;
+    }
+
+    if (!bs->drv->is_filter) {
+        return NULL;
+    }

include/block/block_int.h

Quoted 10 lines
include/block/block_int.h +10 -0
+
+    /* Only one of @backing or @file may be used */
+    assert(!(bs->backing && bs->file));
+
+    c = bs->backing ?: bs->file;
+    if (!c) {
+        return NULL;
+    }
+
+    assert(c->role & BDRV_CHILD_FILTERED);

But the role is still called CHILD_FILTERED

include/block/block_int.h

Quoted 2 lines
include/block/block_int.h +2 -0
+    return c;
+}

(just note that it's a bit inconsistent, keep my r-b anyway)

Signature · 2 lines
Best regards,
Vladimir
author
#8 lore raw

PATCH v7 2/47 block: Add chain helper functions

In reply to Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
 include/block/block_int.h |  3 +++
 block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
index bb3457c5e8..5da793bfc3 100644
@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs);
 BdrvChild *bdrv_filter_child(BlockDriverState *bs);
 BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs);
 BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_implicit_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_backing_chain_next(BlockDriverState *bs);
 
 static inline BlockDriverState *child_bs(BdrvChild *child)
 {
block.c +55 -0

58 lines not on this page · raw message

Signature · 1 line
2.26.2
Reviewed-by on v7 2/47
#9 lore raw

Re: [PATCH v7 02/47] block: Add chain helper functions

Quoted 37 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
  include/block/block_int.h |  3 +++
  block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
  2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
index bb3457c5e8..5da793bfc3 100644
@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs);
  BdrvChild *bdrv_filter_child(BlockDriverState *bs);
  BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs);
  BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_implicit_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_backing_chain_next(BlockDriverState *bs);
 
  static inline BlockDriverState *child_bs(BdrvChild *child)
  {
block.c +6 -0
index 5a42ef49fd..0a0b855261 100644
@@ -7008,3 +7008,58 @@ BdrvChild *bdrv_primary_child(BlockDriverState *bs)
 
      return NULL;
  }
+
+static BlockDriverState *bdrv_do_skip_filters(BlockDriverState *bs,
+                                              bool stop_on_explicit_filter)
+{
+    BdrvChild *c;
+

block.c BdrvChild *bdrv_primary_child(BlockDriverState *bs) · line 7026

Quoted 10 lines
block.c +10 -0
+    if (!bs) {
+        return NULL;
+    }
+
+    while (!(stop_on_explicit_filter && !bs->implicit)) {
+        c = bdrv_filter_child(bs);
+        if (!c) {
+            break;
+        }
+        bs = c->bs;

Could it be child_bs(bs) ?

Andrey

block.c BdrvChild *bdrv_primary_child(BlockDriverState *bs) · line 7027

Quoted 1 line
block.c +1 -0
+    }
author
#10 lore raw

Re: [PATCH v7 02/47] block: Add chain helper functions

Quoted 42 lines · Andrey Shinkevich
Quoted 41 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
  include/block/block_int.h |  3 +++
  block.c                   | 55 +++++++++++++++++++++++++++++++++++++++

  2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
index bb3457c5e8..5da793bfc3 100644
@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs);
   BdrvChild *bdrv_filter_child(BlockDriverState *bs);
   BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs);
   BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_implicit_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_backing_chain_next(BlockDriverState *bs);
     static inline BlockDriverState *child_bs(BdrvChild *child)
   {
block.c +3 -0
index 5a42ef49fd..0a0b855261 100644
@@ -7008,3 +7008,58 @@ BdrvChild *bdrv_primary_child(BlockDriverState
 *bs)
         return NULL;
   }
+
+static BlockDriverState *bdrv_do_skip_filters(BlockDriverState *bs,
+                                              bool

stop_on_explicit_filter)

block.c +7 -0
+{
+    BdrvChild *c;
+
+    if (!bs) {
+        return NULL;
+    }
+
Quoted 8 lines
Quoted 6 lines

+    while (!(stop_on_explicit_filter && !bs->implicit)) { +        c = bdrv_filter_child(bs);
+        if (!c) {
+            break;
+        }
+        bs = c->bs;

Could it be child_bs(bs) ?

Well, in a sense, but not really. We need to check whether there is a child before overwriting @bs (because @bs must stay a non-NULL pointer), so we wouldn’t have fewer lines of code if we replaced “BdrvChild *c” by “BlockDriverState *child_bs”, and then used bdrv_child() to set child_bs.

(And because we have to check whether @c is NULL anyway, there is no real reason to use child_bs(c) instead of c->bs afterwards.)

Quoted 2 lines
Quoted 1 line

+    }

Thanks a lot for reviewing!

#11 lore raw

Re: [PATCH v7 02/47] block: Add chain helper functions

Quoted 50 lines · Max Reitz
Quoted 41 lines · Andrey Shinkevich
Quoted 40 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
   include/block/block_int.h |  3 +++
   block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
   2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
index bb3457c5e8..5da793bfc3 100644
@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs);
   BdrvChild *bdrv_filter_child(BlockDriverState *bs);
   BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs);
   BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_implicit_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_backing_chain_next(BlockDriverState *bs);
     static inline BlockDriverState *child_bs(BdrvChild *child)
   {
block.c +3 -0
index 5a42ef49fd..0a0b855261 100644
@@ -7008,3 +7008,58 @@ BdrvChild *bdrv_primary_child(BlockDriverState
 *bs)
         return NULL;
   }
+
+static BlockDriverState *bdrv_do_skip_filters(BlockDriverState *bs,
+                                              bool

stop_on_explicit_filter)

block.c +6 -0
+{
+    BdrvChild *c;
+
+    if (!bs) {
+        return NULL;
+    }
Quoted 8 lines
Quoted 7 lines

+
+    while (!(stop_on_explicit_filter && !bs->implicit)) { +        c = bdrv_filter_child(bs);
+        if (!c) {
+            break;
+        }
+        bs = c->bs;

Could it be child_bs(bs) ?

Quoted 7 lines

Well, in a sense, but not really. We need to check whether there is a child before overwriting @bs (because @bs must stay a non-NULL pointer), so we wouldn’t have fewer lines of code if we replaced “BdrvChild *c” by “BlockDriverState *child_bs”, and then used bdrv_child() to set child_bs.

(And because we have to check whether @c is NULL anyway, there is no real reason to use child_bs(c) instead of c->bs afterwards.)

Got it, thanks.

Andrey

Quoted 3 lines
Quoted 2 lines
Quoted 1 line

+    }

Thanks a lot for reviewing!

Pleasure!

Andrey

Reviewed-by on v7 2/47
#12 lore raw

Re: [PATCH v7 02/47] block: Add chain helper functions

In reply to Max Reitz

include/block/block_int.h

Quoted 11 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
  include/block/block_int.h |  3 +++
  block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
  2 files changed, 58 insertions(+)
include/block/block_int.h +0 -0
index bb3457c5e8..5da793bfc3 100644

This patch raises two questions:

1. How to treat filters at the end of the backing chain?

- child-access function will return no filter child for such nodes, it's correct of course
- filer skipping functions will return this filter.. How much is it correct - I don't know.

Consider a chain

top --- backing ---> filter-with-no-child

if bdrv_backing_chain_next(top) returns NULL, it's incorrect, because top actually have backing, and on read it will read from it for unallocated clusters (and this should crash). So, probably, returning filter as a backing-chain-next is a valid thing to do. Or we should assert that we are not in such situation (which may crash more often than trying to really read from nonexistent child).

so, returning NULL, may even less correct than returning a filter..

2. How to tread nodes with drv=NULL, but with filter child (with BDRV_CHILD_FILTERED role).
- child-access functions returns no filtered child for such nodes
- filter skipping functions will stop on it..

=======

Isn't it better to drop drv->is_filter at all? And call filter nodes with a bs->file or bs->backing child in BDRV_CHILD_FILTERED role? This automatically closes the two questions:

- node without a child in BDRV_CHILD_FILTERED is automatically non-filter. So, filter driver is responsible for having such child.
- node without a drv may still be a filter if it have BDRV_CHILD_FILTERED.. Still, not very useful.

Anyway, is_filter and BDRV_CHILD_FILTERED are in contradiction, and it seems good to get rid of is_filter. But I may miss something.

[..]

Quoted 13 lines
include/block/block_int.h +13 -0
+
+static BlockDriverState *bdrv_do_skip_filters(BlockDriverState *bs,
+                                              bool stop_on_explicit_filter)
+{
+    BdrvChild *c;
+
+    if (!bs) {
+        return NULL;
+    }
+
+    while (!(stop_on_explicit_filter && !bs->implicit)) {
+        c = bdrv_filter_child(bs);
+        if (!c) {

include/block/block_int.h

Quoted 10 lines
include/block/block_int.h +10 -0
+            break;
+        }
+        bs = c->bs;
+    }
+    /*
+     * Note that this treats nodes with bs->drv == NULL as not being
+     * filters (bs->drv == NULL should be replaced by something else
+     * anyway).
+     * The advantage of this behavior is that this function will thus
+     * always return a non-NULL value (given a non-NULL @bs).

I don't see, how it is follows from first sentence? We can skip nodes with a child of BDRV_CHILD_FILTERED and drv=NULL as well, and still return non-NULL bs at the end...

Didn't you mean "treat nodes without filter child as not being filters, even if they have drv->is_filter == true"? This is a real reason for the second sentence.

... and the disadvantage is that we may return filter node, which may be not expected by caller ...

include/block/block_int.h

Quoted 4 lines
include/block/block_int.h +4 -0
+     */
+
+    return bs;
+}

I think, I can live with it as is for now. If we don't drop is_filter, I think it worth documenting these corner cases somewhere (may be near .is_filter definition).

Signature · 2 lines
Best regards,
Vladimir
author
#13 lore raw

Re: [PATCH v7 02/47] block: Add chain helper functions

Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
  include/block/block_int.h |  3 +++
  block.c                   | 55 +++++++++++++++++++++++++++++++++++++++

  2 files changed, 58 insertions(+)
Quoted 7 lines

include/block/block_int.h

Quoted 2 lines
include/block/block_int.h +0 -0
index bb3457c5e8..5da793bfc3 100644

This patch raises two questions:

1. How to treat filters at the end of the backing chain?

It was my understanding that this configuration would be impossible.

Quoted 9 lines

- child-access function will return no filter child for such nodes, it's correct of course
- filer skipping functions will return this filter.. How much is it correct - I don't know.

Consider a chain

top --- backing ---> filter-with-no-child

How would it be possible to have filter without a child?

Quoted 7 lines

if bdrv_backing_chain_next(top) returns NULL, it's incorrect, because top actually have backing, and on read it will read from it for unallocated clusters (and this should crash). So, probably, returning filter as a backing-chain-next is a valid thing to do. Or we should assert that we are not in such situation (which may crash more often than trying to really read from nonexistent child).

Quoted 5 lines

so, returning NULL, may even less correct than returning a filter..

2. How to tread nodes with drv=NULL, but with filter child (with BDRV_CHILD_FILTERED role).

drv=NULL is a bug on its own that should be fixed... (The idea we had at some point was to introduce a special driver that just always returns -EIO on everything, and to replace corrupt qcow2 nodes by that. Or, well, we might just return -EIO from the qcow2 driver, actually, if we never use drv=NULL anywhere else.)

In any case, drv=NULL is an edge case that I think never has anything to do with filters.

Quoted 40 lines

- child-access functions returns no filtered child for such nodes
- filter skipping functions will stop on it..

=======

Isn't it better to drop drv->is_filter at all? And call filter nodes with a bs->file or bs->backing
child in BDRV_CHILD_FILTERED role? This automatically closes the two questions:

- node without a child in BDRV_CHILD_FILTERED is automatically non-filter. So, filter driver is responsible for having such child.
- node without a drv may still be a filter if it have BDRV_CHILD_FILTERED.. Still, not very useful.

Anyway, is_filter and BDRV_CHILD_FILTERED are in contradiction, and it seems good to get rid of is_filter. But I may miss something.

[..]

Quoted 20 lines
include/block/block_int.h +3 -0
+
+static BlockDriverState *bdrv_do_skip_filters(BlockDriverState *bs,
+                                              bool

stop_on_explicit_filter)

include/block/block_int.h +16 -0
+{
+    BdrvChild *c;
+
+    if (!bs) {
+        return NULL;
+    }
+
+    while (!(stop_on_explicit_filter && !bs->implicit)) {
+        c = bdrv_filter_child(bs);
+        if (!c) {
+            break;
+        }
+        bs = c->bs;
+    }
+    /*
+     * Note that this treats nodes with bs->drv == NULL as not being
Quoted 8 lines
Quoted 4 lines

+     * filters (bs->drv == NULL should be replaced by something else +     * anyway).
+     * The advantage of this behavior is that this function will thus +     * always return a non-NULL value (given a non-NULL @bs).

I don't see, how it is follows from first sentence? We can skip nodes with a child of BDRV_CHILD_FILTERED and drv=NULL as well, and still return non-NULL bs at the end...

My idea was that nodes with bs->drv == NULL might not even have children. If we treated them like filters and skipped through them, we would have to return NULL if there is no child.

Quoted 3 lines

Didn't you mean "treat nodes without filter child as not being filters, even if they have drv->is_filter == true"? This is a real reason for the second sentence.

Hm. I implicitly always assume that filters always have a filter child, so I tend to not even question that part.

Max

#14 lore raw

Re: [PATCH v7 02/47] block: Add chain helper functions

Quoted 17 lines · Max Reitz
Quoted 16 lines · Vladimir Sementsov-Ogievskiy

include/block/block_int.h

Quoted 11 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
   include/block/block_int.h |  3 +++
   block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
   2 files changed, 58 insertions(+)
include/block/block_int.h +0 -0
index bb3457c5e8..5da793bfc3 100644

This patch raises two questions:

Quoted 3 lines
Quoted 1 line

1. How to treat filters at the end of the backing chain?

It was my understanding that this configuration would be impossible.

OK for me, but I'd prefer to have a comment and assertions.

Quoted 26 lines
Quoted 10 lines

- child-access function will return no filter child for such nodes, it's correct of course
- filer skipping functions will return this filter.. How much is it correct - I don't know.

Consider a chain

top --- backing ---> filter-with-no-child

How would it be possible to have filter without a child?

Quoted 7 lines

if bdrv_backing_chain_next(top) returns NULL, it's incorrect, because top actually have backing, and on read it will read from it for unallocated clusters (and this should crash). So, probably, returning filter as a backing-chain-next is a valid thing to do. Or we should assert that we are not in such situation (which may crash more often than trying to really read from nonexistent child).

Quoted 5 lines

so, returning NULL, may even less correct than returning a filter..

2. How to tread nodes with drv=NULL, but with filter child (with BDRV_CHILD_FILTERED role).

Quoted 8 lines

drv=NULL is a bug on its own that should be fixed... (The idea we had at some point was to introduce a special driver that just always returns -EIO on everything, and to replace corrupt qcow2 nodes by that. Or, well, we might just return -EIO from the qcow2 driver, actually, if we never use drv=NULL anywhere else.)

In any case, drv=NULL is an edge case that I think never has anything to do with filters.

So, again some good comment and assertion won't hurt.

Quoted 54 lines
Quoted 41 lines

- child-access functions returns no filtered child for such nodes
- filter skipping functions will stop on it..

=======

Isn't it better to drop drv->is_filter at all? And call filter nodes with a bs->file or bs->backing
child in BDRV_CHILD_FILTERED role? This automatically closes the two questions:

- node without a child in BDRV_CHILD_FILTERED is automatically non-filter. So, filter driver is responsible for having such child.
- node without a drv may still be a filter if it have BDRV_CHILD_FILTERED.. Still, not very useful.

Anyway, is_filter and BDRV_CHILD_FILTERED are in contradiction, and it seems good to get rid of is_filter. But I may miss something.

[..]

Quoted 20 lines
include/block/block_int.h +3 -0
+
+static BlockDriverState *bdrv_do_skip_filters(BlockDriverState *bs,
+                                              bool

stop_on_explicit_filter)

include/block/block_int.h +16 -0
+{
+    BdrvChild *c;
+
+    if (!bs) {
+        return NULL;
+    }
+
+    while (!(stop_on_explicit_filter && !bs->implicit)) {
+        c = bdrv_filter_child(bs);
+        if (!c) {
+            break;
+        }
+        bs = c->bs;
+    }
+    /*
+     * Note that this treats nodes with bs->drv == NULL as not being
Quoted 8 lines
Quoted 4 lines

+     * filters (bs->drv == NULL should be replaced by something else +     * anyway).
+     * The advantage of this behavior is that this function will thus +     * always return a non-NULL value (given a non-NULL @bs).

I don't see, how it is follows from first sentence? We can skip nodes with a child of BDRV_CHILD_FILTERED and drv=NULL as well, and still return non-NULL bs at the end...

My idea was that nodes with bs->drv == NULL might not even have children. If we treated them like filters and skipped through them, we would have to return NULL if there is no child.

Quoted 7 lines
Quoted 3 lines

Didn't you mean "treat nodes without filter child as not being filters, even if they have drv->is_filter == true"? This is a real reason for the second sentence.

Hm. I implicitly always assume that filters always have a filter child, so I tend to not even question that part.

Hmm. Still, the relationship in the comment seems unclear to me, the code itself is simpler :)

Okay, I'm actually OK with this all. I'd prefer to have assertions and comments on corner-cases I mentioned, but patch is OK as is:

Signature · 2 lines
Best regards,
Vladimir
author
#15 lore raw

PATCH v7 3/47 block: bdrv_cow_child() for bdrv_has_zero_init()

In reply to Max Reitz

bdrv_has_zero_init() and the related bdrv_unallocated_blocks_are_zero() should use bdrv_cow_child() if they want to check whether the given BDS has a COW backing file.

---
 block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
index 0a0b855261..f3e2aae49c 100644
@@ -5394,7 +5394,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)
 
     /* If BS is a copy on write image, it is initialized to
        the contents of the base image, which may not be zeroes.  */
-    if (bs->backing) {
+    if (bdrv_cow_child(bs)) {
         return 0;
     }
     if (bs->drv->bdrv_has_zero_init) {
@@ -5412,7 +5412,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)
 {
     BlockDriverInfo bdi;
 
-    if (bs->backing) {
+    if (bdrv_cow_child(bs)) {
         return false;
     }
 
Signature · 1 line
2.26.2
Reviewed-by on v7 3/47
#16 lore raw

Re: [PATCH v7 03/47] block: bdrv_cow_child() for bdrv_has_zero_init()

Quoted 23 lines · Max Reitz

bdrv_has_zero_init() and the related bdrv_unallocated_blocks_are_zero() should use bdrv_cow_child() if they want to check whether the given BDS has a COW backing file.

---
  block.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
block.c +1 -1
index 0a0b855261..f3e2aae49c 100644
@@ -5394,7 +5394,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)
 
      /* If BS is a copy on write image, it is initialized to
         the contents of the base image, which may not be zeroes.  */
-    if (bs->backing) {
+    if (bdrv_cow_child(bs)) {
          return 0;
      }
      if (bs->drv->bdrv_has_zero_init) {

block.c bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs) · line 5418

Quoted 8 lines
block.c +1 -1
@@ -5412,7 +5412,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)
  {
      BlockDriverInfo bdi;
 
-    if (bs->backing) {
+    if (bdrv_cow_child(bs)) {
          return false;
      }
 
#17 lore raw

Re: [PATCH v7 03/47] block: bdrv_cow_child() for bdrv_has_zero_init()

In reply to Max Reitz

block.c bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)

Quoted 23 lines · Max Reitz

bdrv_has_zero_init() and the related bdrv_unallocated_blocks_are_zero() should use bdrv_cow_child() if they want to check whether the given BDS has a COW backing file.

---
  block.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
block.c +1 -1
index 0a0b855261..f3e2aae49c 100644
@@ -5394,7 +5394,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)
 
      /* If BS is a copy on write image, it is initialized to
         the contents of the base image, which may not be zeroes.  */
-    if (bs->backing) {
+    if (bdrv_cow_child(bs)) {
          return 0;
      }
      if (bs->drv->bdrv_has_zero_init) {
@@ -5412,7 +5412,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)

the function is removed, so, this chunk should be dropped or rebased..

Quoted 9 lines
block.c +1 -1
  {
      BlockDriverInfo bdi;
 
-    if (bs->backing) {
+    if (bdrv_cow_child(bs)) {
          return false;
      }
 
Signature · 2 lines
Best regards,
Vladimir
author
#18 lore raw

PATCH v7 4/47 block: bdrv_set_backing_hd() is about bs->backing

In reply to Max Reitz

bdrv_set_backing_hd() is a function that explicitly cares about the bs->backing child. Highlight that in its description and use child_bs(bs->backing) instead of backing_bs(bs) to make it more obvious.

---
 block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
index f3e2aae49c..d139ffb57d 100644
@@ -2846,7 +2846,7 @@ static BdrvChildRole bdrv_backing_role(BlockDriverState *bs)
 }
 
 /*
- * Sets the backing file link of a BDS. A new reference is created; callers
+ * Sets the bs->backing link of a BDS. A new reference is created; callers
  * which don't need their own reference any more must call bdrv_unref().
  */
 void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
@@ -2855,7 +2855,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
     bool update_inherits_from = bdrv_chain_contains(bs, backing_hd) &&
         bdrv_inherits_from_recursive(backing_hd, bs);
 
-    if (bdrv_is_backing_chain_frozen(bs, backing_bs(bs), errp)) {
+    if (bdrv_is_backing_chain_frozen(bs, child_bs(bs->backing), errp)) {
         return;
     }
 
Signature · 1 line
2.26.2
Reviewed-by on v7 4/47
#19 lore raw

Re: [PATCH v7 04/47] block: bdrv_set_backing_hd() is about bs->backing

Quoted 24 lines · Max Reitz

bdrv_set_backing_hd() is a function that explicitly cares about the bs->backing child. Highlight that in its description and use child_bs(bs->backing) instead of backing_bs(bs) to make it more obvious.

---
  block.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
block.c +1 -1
index f3e2aae49c..d139ffb57d 100644
@@ -2846,7 +2846,7 @@ static BdrvChildRole bdrv_backing_role(BlockDriverState *bs)
  }
 
  /*
- * Sets the backing file link of a BDS. A new reference is created; callers
+ * Sets the bs->backing link of a BDS. A new reference is created; callers
   * which don't need their own reference any more must call bdrv_unref().
   */
  void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,

block.c void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd, · line 2861

Quoted 8 lines
block.c +1 -1
@@ -2855,7 +2855,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
      bool update_inherits_from = bdrv_chain_contains(bs, backing_hd) &&
          bdrv_inherits_from_recursive(backing_hd, bs);
 
-    if (bdrv_is_backing_chain_frozen(bs, backing_bs(bs), errp)) {
+    if (bdrv_is_backing_chain_frozen(bs, child_bs(bs->backing), errp)) {
          return;
      }
 
author
#20 lore raw

PATCH v7 5/47 block: Include filters when freezing backing chain

In reply to Max Reitz

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze both COW and filter child links.

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

---
 block.c | 60 +++++++++++++++++++++++++++++++++++++--------------------
 1 file changed, 39 insertions(+), 21 deletions(-)
block.c +39 -21
index d139ffb57d..b59bd776cd 100644
@@ -2595,12 +2595,15 @@ static void bdrv_replace_child_noperm(BdrvChild *child,
  * If @new_bs is not NULL, bdrv_check_perm() must be called beforehand, as this
  * function uses bdrv_set_perm() to update the permissions according to the new
  * reference that @new_bs gets.
+ *
+ * Callers must ensure that child->frozen is false.
  */
 static void bdrv_replace_child(BdrvChild *child, BlockDriverState *new_bs)
 {
     BlockDriverState *old_bs = child->bs;
     uint64_t perm, shared_perm;
 
+    /* Asserts that child->frozen == false */
     bdrv_replace_child_noperm(child, new_bs);
 
     /*
@@ -2761,6 +2764,7 @@ static void bdrv_detach_child(BdrvChild *child)
     g_free(child);
 }
 
+/* Callers must ensure that child->frozen is false. */
 void bdrv_root_unref_child(BdrvChild *child)
 {
     BlockDriverState *child_bs;
@@ -2798,6 +2802,7 @@ static void bdrv_unset_inherits_from(BlockDriverState *root, BdrvChild *child)
     }
 }
 
+/* Callers must ensure that child->frozen is false. */
 void bdrv_unref_child(BlockDriverState *parent, BdrvChild *child)
 {
     if (child == NULL) {
@@ -2864,6 +2869,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
     }
 
     if (bs->backing) {
+        /* Cannot be frozen, we checked that above */
         bdrv_unref_child(bs, bs->backing);
         bs->backing = NULL;
     }
@@ -4372,6 +4378,7 @@ static void bdrv_close(BlockDriverState *bs)
 
     if (bs->drv) {
         if (bs->drv->bdrv_close) {
+            /* Must unfreeze all children, so bdrv_unref_child() works */
             bs->drv->bdrv_close(bs);
         }
         bs->drv = NULL;
@@ -4741,20 +4748,22 @@ BlockDriverState *bdrv_find_base(BlockDriverState *bs)
 }
 
 /*
- * Return true if at least one of the backing links between @bs and
- * @base is frozen. @errp is set if that's the case.
+ * Return true if at least one of the COW (backing) and filter links
+ * between @bs and @base is frozen. @errp is set if that's the case.
  * @base must be reachable from @bs, or NULL.
  */
 bool bdrv_is_backing_chain_frozen(BlockDriverState *bs, BlockDriverState *base,
                                   Error **errp)
 {
     BlockDriverState *i;
+    BdrvChild *child;
 
-    for (i = bs; i != base; i = backing_bs(i)) {
-        if (i->backing && i->backing->frozen) {
+    for (i = bs; i != base; i = child_bs(child)) {
+        child = bdrv_filter_or_cow_child(i);
+
+        if (child && child->frozen) {
             error_setg(errp, "Cannot change '%s' link from '%s' to '%s'",
-                       i->backing->name, i->node_name,
-                       backing_bs(i)->node_name);
+                       child->name, i->node_name, child->bs->node_name);
             return true;
         }
     }
@@ -4763,7 +4772,7 @@ bool bdrv_is_backing_chain_frozen(BlockDriverState *bs, BlockDriverState *base,
 }
 
 /*
- * Freeze all backing links between @bs and @base.
+ * Freeze all COW (backing) and filter links between @bs and @base.
  * If any of the links is already frozen the operation is aborted and
  * none of the links are modified.
  * @base must be reachable from @bs, or NULL.
@@ -4773,22 +4782,25 @@ int bdrv_freeze_backing_chain(BlockDriverState *bs, BlockDriverState *base,
                               Error **errp)
 {
     BlockDriverState *i;
+    BdrvChild *child;
 
     if (bdrv_is_backing_chain_frozen(bs, base, errp)) {
         return -EPERM;
     }
 
-    for (i = bs; i != base; i = backing_bs(i)) {
-        if (i->backing && backing_bs(i)->never_freeze) {
+    for (i = bs; i != base; i = child_bs(child)) {
+        child = bdrv_filter_or_cow_child(i);
+        if (child && child->bs->never_freeze) {
             error_setg(errp, "Cannot freeze '%s' link to '%s'",
-                       i->backing->name, backing_bs(i)->node_name);
+                       child->name, child->bs->node_name);
             return -EPERM;
         }
     }
 
-    for (i = bs; i != base; i = backing_bs(i)) {
-        if (i->backing) {
-            i->backing->frozen = true;
+    for (i = bs; i != base; i = child_bs(child)) {
+        child = bdrv_filter_or_cow_child(i);
+        if (child) {
+            child->frozen = true;
         }
     }
 
@@ -4796,18 +4808,21 @@ int bdrv_freeze_backing_chain(BlockDriverState *bs, BlockDriverState *base,
 }
 
 /*
- * Unfreeze all backing links between @bs and @base. The caller must
- * ensure that all links are frozen before using this function.
+ * Unfreeze all COW (backing) and filter links between @bs and @base.
+ * The caller must ensure that all links are frozen before using this
+ * function.
  * @base must be reachable from @bs, or NULL.
  */
 void bdrv_unfreeze_backing_chain(BlockDriverState *bs, BlockDriverState *base)
 {
     BlockDriverState *i;
+    BdrvChild *child;
 
-    for (i = bs; i != base; i = backing_bs(i)) {
-        if (i->backing) {
-            assert(i->backing->frozen);
-            i->backing->frozen = false;
+    for (i = bs; i != base; i = child_bs(child)) {
+        child = bdrv_filter_or_cow_child(i);
+        if (child) {
+            assert(child->frozen);
+            child->frozen = false;
         }
     }
 }
@@ -4910,8 +4925,11 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
             }
         }
 
-        /* Do the actual switch in the in-memory graph.
-         * Completes bdrv_check_update_perm() transaction internally. */
+        /*
+         * Do the actual switch in the in-memory graph.
+         * Completes bdrv_check_update_perm() transaction internally.
+         * c->frozen is false, we have checked that above.
+         */
         bdrv_ref(base);
         bdrv_replace_child(c, base);
         bdrv_unref(top);
Signature · 1 line
2.26.2
Reviewed-by on v7 5/47
#21 lore raw

Re: [PATCH v7 05/47] block: Include filters when freezing backing chain

block.c

Quoted 17 lines · Max Reitz

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze both COW and filter child links.

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

---
  block.c | 60 +++++++++++++++++++++++++++++++++++++--------------------
  1 file changed, 39 insertions(+), 21 deletions(-)
block.c +0 -0
index d139ffb57d..b59bd776cd 100644
author
#22 lore raw

PATCH v7 6/47 block: Drop bdrv_is_encrypted()

In reply to Max Reitz

The original purpose of bdrv_is_encrypted() was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.

Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.

So now we have to decide in which direction we want to fix bdrv_is_encrypted(): Recursively query all children, or drop it and just use bs->encrypted to get the current node's status?

Nowadays, its only purpose is to report through bdrv_query_image_info() whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).

---
 include/block/block.h | 1 -
 block.c               | 8 --------
 block/qapi.c          | 2 +-
 3 files changed, 1 insertion(+), 10 deletions(-)
include/block/block.h +0 -1
index 86f9728f00..0080fe1311 100644
@@ -538,7 +538,6 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);
 void bdrv_next_cleanup(BdrvNextIterator *it);
 
 BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
-bool bdrv_is_encrypted(BlockDriverState *bs);
 void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
                          void *opaque, bool read_only);
 const char *bdrv_get_node_name(const BlockDriverState *bs);
block.c +0 -8
index b59bd776cd..76277ea4e0 100644
@@ -5044,14 +5044,6 @@ bool bdrv_is_sg(BlockDriverState *bs)
     return bs->sg;
 }
 
-bool bdrv_is_encrypted(BlockDriverState *bs)
-{
-    if (bs->backing && bs->backing->bs->encrypted) {
-        return true;
-    }
-    return bs->encrypted;
-}
-
 const char *bdrv_get_format_name(BlockDriverState *bs)
 {
     return bs->drv ? bs->drv->format_name : NULL;
block/qapi.c +1 -1
index afd9f3b4a7..4807a2b344 100644
@@ -288,7 +288,7 @@ void bdrv_query_image_info(BlockDriverState *bs,
     info->virtual_size    = size;
     info->actual_size     = bdrv_get_allocated_file_size(bs);
     info->has_actual_size = info->actual_size >= 0;
-    if (bdrv_is_encrypted(bs)) {
+    if (bs->encrypted) {
         info->encrypted = true;
         info->has_encrypted = true;
     }
Signature · 1 line
2.26.2
Reviewed-by on v7 6/47
#23 lore raw

Re: [PATCH v7 06/47] block: Drop bdrv_is_encrypted()

Quoted 63 lines · Max Reitz

The original purpose of bdrv_is_encrypted() was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.

Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.

So now we have to decide in which direction we want to fix bdrv_is_encrypted(): Recursively query all children, or drop it and just use bs->encrypted to get the current node's status?

Nowadays, its only purpose is to report through bdrv_query_image_info() whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).

---
  include/block/block.h | 1 -
  block.c               | 8 --------
  block/qapi.c          | 2 +-
  3 files changed, 1 insertion(+), 10 deletions(-)
include/block/block.h +0 -1
index 86f9728f00..0080fe1311 100644
@@ -538,7 +538,6 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);
  void bdrv_next_cleanup(BdrvNextIterator *it);
 
  BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
-bool bdrv_is_encrypted(BlockDriverState *bs);
  void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
                           void *opaque, bool read_only);
  const char *bdrv_get_node_name(const BlockDriverState *bs);
block.c +0 -8
index b59bd776cd..76277ea4e0 100644
@@ -5044,14 +5044,6 @@ bool bdrv_is_sg(BlockDriverState *bs)
      return bs->sg;
  }
 
-bool bdrv_is_encrypted(BlockDriverState *bs)
-{
-    if (bs->backing && bs->backing->bs->encrypted) {
-        return true;
-    }
-    return bs->encrypted;
-}
-
  const char *bdrv_get_format_name(BlockDriverState *bs)
  {
      return bs->drv ? bs->drv->format_name : NULL;
block/qapi.c +0 -0
index afd9f3b4a7..4807a2b344 100644

block/qapi.c void bdrv_query_image_info(BlockDriverState *bs, · line 294

Quoted 8 lines
block/qapi.c +1 -1
@@ -288,7 +288,7 @@ void bdrv_query_image_info(BlockDriverState *bs,
      info->virtual_size    = size;
      info->actual_size     = bdrv_get_allocated_file_size(bs);
      info->has_actual_size = info->actual_size >= 0;
-    if (bdrv_is_encrypted(bs)) {
+    if (bs->encrypted) {
          info->encrypted = true;
          info->has_encrypted = true;
      }
author
#24 lore raw

PATCH v7 7/47 block: Add bdrv_supports_compressed_writes()

In reply to Max Reitz

Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).

---
 include/block/block.h |  1 +
 block.c               | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)
include/block/block.h +1 -0
index 0080fe1311..a905a5ec05 100644
@@ -538,6 +538,7 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);
 void bdrv_next_cleanup(BdrvNextIterator *it);
 
 BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
+bool bdrv_supports_compressed_writes(BlockDriverState *bs);
 void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
                          void *opaque, bool read_only);
 const char *bdrv_get_node_name(const BlockDriverState *bs);
block.c +23 -0
index 76277ea4e0..6449f3a11d 100644
@@ -5044,6 +5044,29 @@ bool bdrv_is_sg(BlockDriverState *bs)
     return bs->sg;
 }
 
+/**
+ * Return whether the given node supports compressed writes.
+ */
+bool bdrv_supports_compressed_writes(BlockDriverState *bs)
+{
+    BlockDriverState *filtered;
+
+    if (!bs->drv || !block_driver_can_compress(bs->drv)) {
+        return false;
+    }
+
+    filtered = bdrv_filter_bs(bs);
+    if (filtered) {
+        /*
+         * Filters can only forward compressed writes, so we have to
+         * check the child.
+         */
+        return bdrv_supports_compressed_writes(filtered);
+    }
+
+    return true;
+}
+
 const char *bdrv_get_format_name(BlockDriverState *bs)
 {
     return bs->drv ? bs->drv->format_name : NULL;
Signature · 1 line
2.26.2
Reviewed-by on v7 7/47
#25 lore raw

Re: [PATCH v7 07/47] block: Add bdrv_supports_compressed_writes()

Quoted 51 lines · Max Reitz

Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).

---
  include/block/block.h |  1 +
  block.c               | 23 +++++++++++++++++++++++
  2 files changed, 24 insertions(+)
include/block/block.h +1 -0
index 0080fe1311..a905a5ec05 100644
@@ -538,6 +538,7 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);
  void bdrv_next_cleanup(BdrvNextIterator *it);
 
  BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
+bool bdrv_supports_compressed_writes(BlockDriverState *bs);
  void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
                           void *opaque, bool read_only);
  const char *bdrv_get_node_name(const BlockDriverState *bs);
block.c +16 -0
index 76277ea4e0..6449f3a11d 100644
@@ -5044,6 +5044,29 @@ bool bdrv_is_sg(BlockDriverState *bs)
      return bs->sg;
  }
 
+/**
+ * Return whether the given node supports compressed writes.
+ */
+bool bdrv_supports_compressed_writes(BlockDriverState *bs)
+{
+    BlockDriverState *filtered;
+
+    if (!bs->drv || !block_driver_can_compress(bs->drv)) {
+        return false;
+    }
+
+    filtered = bdrv_filter_bs(bs);
+    if (filtered) {
+        /*
+         * Filters can only forward compressed writes, so we have to
+         * check the child.

block.c bool bdrv_is_sg(BlockDriverState *bs) · line 5072

Quoted 10 lines
block.c +7 -0
+         */
+        return bdrv_supports_compressed_writes(filtered);
+    }
+
+    return true;
+}
+
  const char *bdrv_get_format_name(BlockDriverState *bs)
  {
      return bs->drv ? bs->drv->format_name : NULL;
author
#26 lore raw

PATCH v7 8/47 throttle: Support compressed writes

In reply to Max Reitz
---
 block/throttle.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/throttle.c +10 -0
index 0ebbad0743..f6e619aca2 100644
@@ -154,6 +154,15 @@ static int coroutine_fn throttle_co_pdiscard(BlockDriverState *bs,
     return bdrv_co_pdiscard(bs->file, offset, bytes);
 }
 
+static int coroutine_fn throttle_co_pwritev_compressed(BlockDriverState *bs,
+                                                       uint64_t offset,
+                                                       uint64_t bytes,
+                                                       QEMUIOVector *qiov)
+{
+    return throttle_co_pwritev(bs, offset, bytes, qiov,
+                               BDRV_REQ_WRITE_COMPRESSED);
+}
+
 static int throttle_co_flush(BlockDriverState *bs)
 {
     return bdrv_co_flush(bs->file->bs);
@@ -246,6 +255,7 @@ static BlockDriver bdrv_throttle = {
 
     .bdrv_co_pwrite_zeroes              =   throttle_co_pwrite_zeroes,
     .bdrv_co_pdiscard                   =   throttle_co_pdiscard,
+    .bdrv_co_pwritev_compressed         =   throttle_co_pwritev_compressed,
 
     .bdrv_attach_aio_context            =   throttle_attach_aio_context,
     .bdrv_detach_aio_context            =   throttle_detach_aio_context,
Signature · 1 line
2.26.2
Reviewed-by on v7 8/47
#27 lore raw

Re: [PATCH v7 08/47] throttle: Support compressed writes

Quoted 27 lines · Max Reitz
---
  block/throttle.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
block/throttle.c +9 -0
index 0ebbad0743..f6e619aca2 100644
@@ -154,6 +154,15 @@ static int coroutine_fn throttle_co_pdiscard(BlockDriverState *bs,
      return bdrv_co_pdiscard(bs->file, offset, bytes);
  }
 
+static int coroutine_fn throttle_co_pwritev_compressed(BlockDriverState *bs,
+                                                       uint64_t offset,
+                                                       uint64_t bytes,
+                                                       QEMUIOVector *qiov)
+{
+    return throttle_co_pwritev(bs, offset, bytes, qiov,
+                               BDRV_REQ_WRITE_COMPRESSED);
+}
+
  static int throttle_co_flush(BlockDriverState *bs)
  {
      return bdrv_co_flush(bs->file->bs);

block/throttle.c static BlockDriver bdrv_throttle = { · line 261

Quoted 7 lines
block/throttle.c +1 -0
@@ -246,6 +255,7 @@ static BlockDriver bdrv_throttle = {
 
      .bdrv_co_pwrite_zeroes              =   throttle_co_pwrite_zeroes,
      .bdrv_co_pdiscard                   =   throttle_co_pdiscard,
+    .bdrv_co_pwritev_compressed         =   throttle_co_pwritev_compressed,
 
      .bdrv_attach_aio_context            =   throttle_attach_aio_context,
      .bdrv_detach_aio_context            =   throttle_detach_aio_context,
author
#28 lore raw

PATCH v7 9/47 copy-on-read: Support compressed writes

In reply to Max Reitz
---
 block/copy-on-read.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
block/copy-on-read.c +11 -0
index a6e3c74a68..a6a864f147 100644
@@ -107,6 +107,16 @@ static int coroutine_fn cor_co_pdiscard(BlockDriverState *bs,
 }
 
 
+static int coroutine_fn cor_co_pwritev_compressed(BlockDriverState *bs,
+                                                  uint64_t offset,
+                                                  uint64_t bytes,
+                                                  QEMUIOVector *qiov)
+{
+    return bdrv_co_pwritev(bs->file, offset, bytes, qiov,
+                           BDRV_REQ_WRITE_COMPRESSED);
+}
+
+
 static void cor_eject(BlockDriverState *bs, bool eject_flag)
 {
     bdrv_eject(bs->file->bs, eject_flag);
@@ -131,6 +141,7 @@ static BlockDriver bdrv_copy_on_read = {
     .bdrv_co_pwritev                    = cor_co_pwritev,
     .bdrv_co_pwrite_zeroes              = cor_co_pwrite_zeroes,
     .bdrv_co_pdiscard                   = cor_co_pdiscard,
+    .bdrv_co_pwritev_compressed         = cor_co_pwritev_compressed,
 
     .bdrv_eject                         = cor_eject,
     .bdrv_lock_medium                   = cor_lock_medium,
Signature · 1 line
2.26.2
Reviewed-by on v7 9/47
#29 lore raw

Re: [PATCH v7 09/47] copy-on-read: Support compressed writes

Quoted 27 lines · Max Reitz
---
  block/copy-on-read.c | 11 +++++++++++
  1 file changed, 11 insertions(+)
block/copy-on-read.c +10 -0
index a6e3c74a68..a6a864f147 100644
@@ -107,6 +107,16 @@ static int coroutine_fn cor_co_pdiscard(BlockDriverState *bs,
  }
 
 
+static int coroutine_fn cor_co_pwritev_compressed(BlockDriverState *bs,
+                                                  uint64_t offset,
+                                                  uint64_t bytes,
+                                                  QEMUIOVector *qiov)
+{
+    return bdrv_co_pwritev(bs->file, offset, bytes, qiov,
+                           BDRV_REQ_WRITE_COMPRESSED);
+}
+
+
  static void cor_eject(BlockDriverState *bs, bool eject_flag)
  {
      bdrv_eject(bs->file->bs, eject_flag);

block/copy-on-read.c static BlockDriver bdrv_copy_on_read = { · line 147

Quoted 7 lines
block/copy-on-read.c +1 -0
@@ -131,6 +141,7 @@ static BlockDriver bdrv_copy_on_read = {
      .bdrv_co_pwritev                    = cor_co_pwritev,
      .bdrv_co_pwrite_zeroes              = cor_co_pwrite_zeroes,
      .bdrv_co_pdiscard                   = cor_co_pdiscard,
+    .bdrv_co_pwritev_compressed         = cor_co_pwritev_compressed,
 
      .bdrv_eject                         = cor_eject,
      .bdrv_lock_medium                   = cor_lock_medium,
author
#30 lore raw

PATCH v7 10/47 mirror-top: Support compressed writes

In reply to Max Reitz
---
 block/mirror.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/mirror.c +10 -0
index e8e8844afc..469acf4600 100644
@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs,
                                     NULL, 0);
 }
 
+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs,
+                                                           uint64_t offset,
+                                                           uint64_t bytes,
+                                                           QEMUIOVector *qiov)
+{
+    return bdrv_mirror_top_pwritev(bs, offset, bytes, qiov,
+                                   BDRV_REQ_WRITE_COMPRESSED);
+}
+
 static void bdrv_mirror_top_refresh_filename(BlockDriverState *bs)
 {
     if (bs->backing == NULL) {
@@ -1526,6 +1535,7 @@ static BlockDriver bdrv_mirror_top = {
     .bdrv_co_pwritev            = bdrv_mirror_top_pwritev,
     .bdrv_co_pwrite_zeroes      = bdrv_mirror_top_pwrite_zeroes,
     .bdrv_co_pdiscard           = bdrv_mirror_top_pdiscard,
+    .bdrv_co_pwritev_compressed = bdrv_mirror_top_pwritev_compressed,
     .bdrv_co_flush              = bdrv_mirror_top_flush,
     .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
     .bdrv_refresh_filename      = bdrv_mirror_top_refresh_filename,
Signature · 1 line
2.26.2
Reviewed-by on v7 10/47
#31 lore raw

Re: [PATCH v7 10/47] mirror-top: Support compressed writes

Quoted 26 lines · Max Reitz
---
  block/mirror.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
block/mirror.c +9 -0
index e8e8844afc..469acf4600 100644
@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs,
                                      NULL, 0);
  }
 
+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs,
+                                                           uint64_t offset,
+                                                           uint64_t bytes,
+                                                           QEMUIOVector *qiov)
+{
+    return bdrv_mirror_top_pwritev(bs, offset, bytes, qiov,
+                                   BDRV_REQ_WRITE_COMPRESSED);
+}
+
  static void bdrv_mirror_top_refresh_filename(BlockDriverState *bs)
  {
      if (bs->backing == NULL) {

block/mirror.c static BlockDriver bdrv_mirror_top = { · line 1541

Quoted 7 lines
block/mirror.c +1 -0
@@ -1526,6 +1535,7 @@ static BlockDriver bdrv_mirror_top = {
      .bdrv_co_pwritev            = bdrv_mirror_top_pwritev,
      .bdrv_co_pwrite_zeroes      = bdrv_mirror_top_pwrite_zeroes,
      .bdrv_co_pdiscard           = bdrv_mirror_top_pdiscard,
+    .bdrv_co_pwritev_compressed = bdrv_mirror_top_pwritev_compressed,
      .bdrv_co_flush              = bdrv_mirror_top_flush,
      .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
      .bdrv_refresh_filename      = bdrv_mirror_top_refresh_filename,
#32 lore raw

Re: [PATCH v7 10/47] mirror-top: Support compressed writes

In reply to Max Reitz

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 11 lines
---
 block/mirror.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/mirror.c +0 -0
index e8e8844afc..469acf4600 100644
@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs,
                                     NULL, 0);

block/mirror.c static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, · line 1490

Quoted 10 lines
block/mirror.c +8 -0
 }
 
+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs,
+                                                           uint64_t offset,
+                                                           uint64_t bytes,
+                                                           QEMUIOVector *qiov)
+{
+    return bdrv_mirror_top_pwritev(bs, offset, bytes, qiov,
+                                   BDRV_REQ_WRITE_COMPRESSED);
+}

Hm, not sure if it's a problem, but bdrv_supports_compressed_writes() will now return true for mirror-top. However, with an active mirror to a target that doesn't support compression, trying to actually do a compressed write will always return -ENOTSUP.

So I guess the set of nodes patch 7 looks at still isn't quite complete. However, it's not obvious how to make it more complete without delegating to the driver.

Maybe we need to use bs->supported_write_flags, which is set by the driver, instead of looking at the presence of callbacks.

Of course, in the general case, we also should make sure that graph changes will be reflected in bs->supported_write_flags, but we already fail to do this in raw-format, so I guess ignoring it for now is good enough here, too...

Kevin

author
#33 lore raw

Re: [PATCH v7 10/47] mirror-top: Support compressed writes

Quoted 14 lines · Kevin Wolf

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 13 lines
---
 block/mirror.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/mirror.c +0 -0
index e8e8844afc..469acf4600 100644
@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs,
                                     NULL, 0);
 }
 
Quoted 13 lines

block/mirror.c static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, · line 1490

Quoted 8 lines
block/mirror.c +8 -0
+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs,
+                                                           uint64_t offset,
+                                                           uint64_t bytes,
+                                                           QEMUIOVector *qiov)
+{
+    return bdrv_mirror_top_pwritev(bs, offset, bytes, qiov,
+                                   BDRV_REQ_WRITE_COMPRESSED);
+}

Hm, not sure if it's a problem, but bdrv_supports_compressed_writes() will now return true for mirror-top. However, with an active mirror to a target that doesn't support compression, trying to actually do a compressed write will always return -ENOTSUP.

Right.

Quoted 6 lines

So I guess the set of nodes patch 7 looks at still isn't quite complete. However, it's not obvious how to make it more complete without delegating to the driver.

Maybe we need to use bs->supported_write_flags, which is set by the driver, instead of looking at the presence of callbacks.

Hm, yes, that would work better. Not sure if it’s worth it for this series.

The only problem we’d have is late failure when trying to do a compressed backup to a target that’s running an active mirror. (Late as in “first write fails without explanation”, as opposed to “job fails during set-up”.)

Which I hope is not a case anyone would ever encounter, and even if they do, the failure doesn’t seem catastrophic to me. So I don’t think it’s really a problem.

Max

#34 lore raw

Re: [PATCH v7 10/47] mirror-top: Support compressed writes

Am 19.08.2020 um 17:35 hat Max Reitz geschrieben:

Quoted 35 lines · Kevin Wolf
Quoted 14 lines

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 13 lines
---
 block/mirror.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/mirror.c +0 -0
index e8e8844afc..469acf4600 100644
@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs,
                                     NULL, 0);
 }
 
Quoted 13 lines

block/mirror.c static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, · line 1490

Quoted 8 lines
block/mirror.c +8 -0
+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs,
+                                                           uint64_t offset,
+                                                           uint64_t bytes,
+                                                           QEMUIOVector *qiov)
+{
+    return bdrv_mirror_top_pwritev(bs, offset, bytes, qiov,
+                                   BDRV_REQ_WRITE_COMPRESSED);
+}

Hm, not sure if it's a problem, but bdrv_supports_compressed_writes() will now return true for mirror-top. However, with an active mirror to a target that doesn't support compression, trying to actually do a compressed write will always return -ENOTSUP.

Right.

Quoted 4 lines

So I guess the set of nodes patch 7 looks at still isn't quite complete. However, it's not obvious how to make it more complete without delegating to the driver.

Quoted 5 lines
Quoted 2 lines

Maybe we need to use bs->supported_write_flags, which is set by the driver, instead of looking at the presence of callbacks.

Hm, yes, that would work better. Not sure if it’s worth it for this series.

This patch looks like a feature addition that is only marginally related to the goal of the series anyway. Maybe it should be a separate small series on top?

The other compression related patches in the series don't seem to have this problem, so they could stay there anyway.

Quoted 8 lines

The only problem we’d have is late failure when trying to do a compressed backup to a target that’s running an active mirror. (Late as in “first write fails without explanation”, as opposed to “job fails during set-up”.)

Which I hope is not a case anyone would ever encounter, and even if they do, the failure doesn’t seem catastrophic to me. So I don’t think it’s really a problem.

Yeah, it's just a bit unfortunate to add a new function that we know doesn't do what it promises.

Kevin

author
#35 lore raw

PATCH v7 11/47 backup-top: Support compressed writes

In reply to Max Reitz
---
 block/backup-top.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/backup-top.c +10 -0
index af2f20f346..f304df8f26 100644
@@ -99,6 +99,15 @@ static coroutine_fn int backup_top_co_pwritev(BlockDriverState *bs,
     return bdrv_co_pwritev(bs->backing, offset, bytes, qiov, flags);
 }
 
+static coroutine_fn int backup_top_co_pwritev_compressed(BlockDriverState *bs,
+                                                         uint64_t offset,
+                                                         uint64_t bytes,
+                                                         QEMUIOVector *qiov)
+{
+    return backup_top_co_pwritev(bs, offset, bytes, qiov,
+                                 BDRV_REQ_WRITE_COMPRESSED);
+}
+
 static int coroutine_fn backup_top_co_flush(BlockDriverState *bs)
 {
     if (!bs->backing) {
@@ -173,6 +182,7 @@ BlockDriver bdrv_backup_top_filter = {
     .bdrv_co_pwritev            = backup_top_co_pwritev,
     .bdrv_co_pwrite_zeroes      = backup_top_co_pwrite_zeroes,
     .bdrv_co_pdiscard           = backup_top_co_pdiscard,
+    .bdrv_co_pwritev_compressed = backup_top_co_pwritev_compressed,
     .bdrv_co_flush              = backup_top_co_flush,
 
     .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
Signature · 1 line
2.26.2
Reviewed-by on v7 11/47
#36 lore raw

Re: [PATCH v7 11/47] backup-top: Support compressed writes

Quoted 26 lines · Max Reitz
---
  block/backup-top.c | 10 ++++++++++
  1 file changed, 10 insertions(+)
block/backup-top.c +9 -0
index af2f20f346..f304df8f26 100644
@@ -99,6 +99,15 @@ static coroutine_fn int backup_top_co_pwritev(BlockDriverState *bs,
      return bdrv_co_pwritev(bs->backing, offset, bytes, qiov, flags);
  }
 
+static coroutine_fn int backup_top_co_pwritev_compressed(BlockDriverState *bs,
+                                                         uint64_t offset,
+                                                         uint64_t bytes,
+                                                         QEMUIOVector *qiov)
+{
+    return backup_top_co_pwritev(bs, offset, bytes, qiov,
+                                 BDRV_REQ_WRITE_COMPRESSED);
+}
+
  static int coroutine_fn backup_top_co_flush(BlockDriverState *bs)
  {
      if (!bs->backing) {

block/backup-top.c BlockDriver bdrv_backup_top_filter = { · line 188

Quoted 7 lines
block/backup-top.c +1 -0
@@ -173,6 +182,7 @@ BlockDriver bdrv_backup_top_filter = {
      .bdrv_co_pwritev            = backup_top_co_pwritev,
      .bdrv_co_pwrite_zeroes      = backup_top_co_pwrite_zeroes,
      .bdrv_co_pdiscard           = backup_top_co_pdiscard,
+    .bdrv_co_pwritev_compressed = backup_top_co_pwritev_compressed,
      .bdrv_co_flush              = backup_top_co_flush,
 
      .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
author
#37 lore raw

PATCH v7 12/47 block: Use bdrv_filter_(bs|child) where obvious

In reply to Max Reitz

Places that use patterns like

    if (bs->drv->is_filter && bs->file) {
        ... something about bs->file->bs ...
    }

should be

    BlockDriverState *filtered = bdrv_filter_bs(bs);
    if (filtered) {
        ... something about @filtered ...
    }

instead.

---
 block.c                        | 31 ++++++++++++++++++++-----------
 block/io.c                     |  7 +++++--
 migration/block-dirty-bitmap.c |  8 +-------
 3 files changed, 26 insertions(+), 20 deletions(-)
block.c +20 -11
index 6449f3a11d..a44af9c3c1 100644
@@ -710,11 +710,12 @@ int coroutine_fn bdrv_co_delete_file(BlockDriverState *bs, Error **errp)
 int bdrv_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *filtered = bdrv_filter_bs(bs);
 
     if (drv && drv->bdrv_probe_blocksizes) {
         return drv->bdrv_probe_blocksizes(bs, bsz);
-    } else if (drv && drv->is_filter && bs->file) {
-        return bdrv_probe_blocksizes(bs->file->bs, bsz);
+    } else if (filtered) {
+        return bdrv_probe_blocksizes(filtered, bsz);
     }
 
     return -ENOTSUP;
@@ -729,11 +730,12 @@ int bdrv_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 int bdrv_probe_geometry(BlockDriverState *bs, HDGeometry *geo)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *filtered = bdrv_filter_bs(bs);
 
     if (drv && drv->bdrv_probe_geometry) {
         return drv->bdrv_probe_geometry(bs, geo);
-    } else if (drv && drv->is_filter && bs->file) {
-        return bdrv_probe_geometry(bs->file->bs, geo);
+    } else if (filtered) {
+        return bdrv_probe_geometry(filtered, geo);
     }
 
     return -ENOTSUP;
@@ -5421,6 +5423,8 @@ int bdrv_has_zero_init_1(BlockDriverState *bs)
 
 int bdrv_has_zero_init(BlockDriverState *bs)
 {
+    BlockDriverState *filtered;
+
     if (!bs->drv) {
         return 0;
     }
@@ -5433,8 +5437,10 @@ int bdrv_has_zero_init(BlockDriverState *bs)
     if (bs->drv->bdrv_has_zero_init) {
         return bs->drv->bdrv_has_zero_init(bs);
     }
-    if (bs->file && bs->drv->is_filter) {
-        return bdrv_has_zero_init(bs->file->bs);
+
+    filtered = bdrv_filter_bs(bs);
+    if (filtered) {
+        return bdrv_has_zero_init(filtered);
     }
 
     /* safe default */
@@ -5479,8 +5485,9 @@ int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
         return -ENOMEDIUM;
     }
     if (!drv->bdrv_get_info) {
-        if (bs->file && drv->is_filter) {
-            return bdrv_get_info(bs->file->bs, bdi);
+        BlockDriverState *filtered = bdrv_filter_bs(bs);
+        if (filtered) {
+            return bdrv_get_info(filtered, bdi);
         }
         return -ENOTSUP;
     }
@@ -6546,6 +6553,8 @@ int bdrv_amend_options(BlockDriverState *bs, QemuOpts *opts,
 bool bdrv_recurse_can_replace(BlockDriverState *bs,
                               BlockDriverState *to_replace)
 {
+    BlockDriverState *filtered;
+
     if (!bs || !bs->drv) {
         return false;
     }
@@ -6560,9 +6569,9 @@ bool bdrv_recurse_can_replace(BlockDriverState *bs,
     }
 
     /* For filters without an own implementation, we can recurse on our own */
-    if (bs->drv->is_filter) {
-        BdrvChild *child = bs->file ?: bs->backing;
-        return bdrv_recurse_can_replace(child->bs, to_replace);
+    filtered = bdrv_filter_bs(bs);
+    if (filtered) {
+        return bdrv_recurse_can_replace(filtered, to_replace);
     }
 
     /* Safe default */
block/io.c +5 -2
index df8f2a98d4..385176b331 100644
@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
                                   Error **errp)
 {
     BlockDriverState *bs = child->bs;
+    BdrvChild *filtered;
     BlockDriver *drv = bs->drv;
     BdrvTrackedRequest req;
     int64_t old_size, new_bytes;
@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
         goto out;
     }
 
+    filtered = bdrv_filter_child(bs);
+
     /*
      * If the image has a backing file that is large enough that it would
      * provide data for the new area, we cannot leave it unallocated because
@@ -3390,8 +3393,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
             goto out;
         }
         ret = drv->bdrv_co_truncate(bs, offset, exact, prealloc, flags, errp);
-    } else if (bs->file && drv->is_filter) {
-        ret = bdrv_co_truncate(bs->file, offset, exact, prealloc, flags, errp);
+    } else if (filtered) {
+        ret = bdrv_co_truncate(filtered, offset, exact, prealloc, flags, errp);
     } else {
         error_setg(errp, "Image format driver does not support resize");
         ret = -ENOTSUP;
migration/block-dirty-bitmap.c +1 -7
index 47bc0f650c..dec656c074 100644
@@ -356,13 +356,7 @@ static int init_dirty_bitmap_migration(void)
         while (bs && bs->drv && bs->drv->is_filter &&
                !bdrv_has_named_bitmaps(bs))
         {
-            if (bs->backing) {
-                bs = bs->backing->bs;
-            } else if (bs->file) {
-                bs = bs->file->bs;
-            } else {
-                bs = NULL;
-            }
+            bs = bdrv_filter_bs(bs);
         }
 
         if (bs && bs->drv && !bs->drv->is_filter) {
Signature · 1 line
2.26.2
Reviewed-by on v7 12/47
#38 lore raw

Re: [PATCH v7 12/47] block: Use bdrv_filter_(bs|child) where obvious

Quoted 22 lines · Max Reitz

Places that use patterns like

     if (bs->drv->is_filter && bs->file) {
         ... something about bs->file->bs ...
     }

should be

     BlockDriverState *filtered = bdrv_filter_bs(bs);
     if (filtered) {
         ... something about @filtered ...
     }

instead.

---
  block.c                        | 31 ++++++++++++++++++++-----------
  block/io.c                     |  7 +++++--
  migration/block-dirty-bitmap.c |  8 +-------
  3 files changed, 26 insertions(+), 20 deletions(-)

...

Quoted 13 lines
block/io.c +1 -0
index df8f2a98d4..385176b331 100644
@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
                                    Error **errp)
  {
      BlockDriverState *bs = child->bs;
+    BdrvChild *filtered;
      BlockDriver *drv = bs->drv;
      BdrvTrackedRequest req;
      int64_t old_size, new_bytes;

block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3363

Quoted 5 lines
block/io.c +2 -0
@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
          goto out;
      }
 
+    filtered = bdrv_filter_child(bs);
+

Isn't better to have this initialization right before the relevant if/else block?

Andrey

Quoted 14 lines
block/io.c +2 -2
      /*
       * If the image has a backing file that is large enough that it would
       * provide data for the new area, we cannot leave it unallocated because
@@ -3390,8 +3393,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
              goto out;
          }
          ret = drv->bdrv_co_truncate(bs, offset, exact, prealloc, flags, errp);
-    } else if (bs->file && drv->is_filter) {
-        ret = bdrv_co_truncate(bs->file, offset, exact, prealloc, flags, errp);
+    } else if (filtered) {
+        ret = bdrv_co_truncate(filtered, offset, exact, prealloc, flags, errp);
      } else {
          error_setg(errp, "Image format driver does not support resize");
          ret = -ENOTSUP;

...

author
#39 lore raw

Re: [PATCH v7 12/47] block: Use bdrv_filter_(bs|child) where obvious

Quoted 38 lines · Andrey Shinkevich
Quoted 22 lines · Max Reitz

Places that use patterns like

     if (bs->drv->is_filter && bs->file) {
         ... something about bs->file->bs ...
     }

should be

     BlockDriverState *filtered = bdrv_filter_bs(bs); if (filtered) {
         ... something about @filtered ...
     }

instead.

---
  block.c                        | 31 ++++++++++++++++++++----------- block/io.c                     |  7 +++++-- migration/block-dirty-bitmap.c |  8 +-------

  3 files changed, 26 insertions(+), 20 deletions(-)

...

Quoted 14 lines
block/io.c +1 -0
index df8f2a98d4..385176b331 100644
@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild
 *child, int64_t offset, bool exact,
                                     Error **errp)
   {
       BlockDriverState *bs = child->bs;
+    BdrvChild *filtered;
       BlockDriver *drv = bs->drv;
       BdrvTrackedRequest req;

      int64_t old_size, new_bytes;

block/io.c +0 -0
@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild
Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3363

Quoted 5 lines
block/io.c +1 -0
 *child, int64_t offset, bool exact,
           goto out;
       }
   +    filtered = bdrv_filter_child(bs);
+

Isn't better to have this initialization right before the relevant if/else block?

Hm, well, yes. In this case, though, maybe not. Patch 16 will add another BdrvChild to be initialized here (@backing), and we need to initialize that one here. So I felt it made sense to group them together.

They got split up when I decided to put @filtered into this patch and @backing into its own. So now it may look a bit weird, but I feel like after patch 16 it makes sense.

(I’m indifferent, basically.)

Max

#40 lore raw

Re: [PATCH v7 12/47] block: Use bdrv_filter_(bs|child) where obvious

Quoted 50 lines · Max Reitz
Quoted 37 lines · Andrey Shinkevich
Quoted 22 lines · Max Reitz

Places that use patterns like

     if (bs->drv->is_filter && bs->file) {
         ... something about bs->file->bs ...
     }

should be

     BlockDriverState *filtered = bdrv_filter_bs(bs); if (filtered) {
         ... something about @filtered ...
     }

instead.

---
   block.c                        | 31 ++++++++++++++++++++-----------
   block/io.c                     |  7 +++++--
   migration/block-dirty-bitmap.c |  8 +-------
   3 files changed, 26 insertions(+), 20 deletions(-)

...

Quoted 13 lines
block/io.c +1 -0
index df8f2a98d4..385176b331 100644
@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild
 *child, int64_t offset, bool exact,
                                     Error **errp)
   {
       BlockDriverState *bs = child->bs;
+    BdrvChild *filtered;
       BlockDriver *drv = bs->drv;
       BdrvTrackedRequest req;

      int64_t old_size, new_bytes;

Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild · line 3363

Quoted 6 lines
block/io.c +1 -0
@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild
 *child, int64_t offset, bool exact,
           goto out;
       }
   +    filtered = bdrv_filter_child(bs);
+

Isn't better to have this initialization right before the relevant if/else block?

Hm, well, yes. In this case, though, maybe not. Patch 16 will add another BdrvChild to be initialized here (@backing), and we need to initialize that one here. So I felt it made sense to group them together.

Quoted 7 lines

They got split up when I decided to put @filtered into this patch and @backing into its own. So now it may look a bit weird, but I feel like after patch 16 it makes sense.

(I’m indifferent, basically.)

Max

Yes, it makes a sence. I am on the way to reviewing further and have not reached the 16th yet.

It is a minor thing anyway )) Thank you for your response.

Andrey

author
#41 lore raw

PATCH v7 13/47 block: Use CAFs in block status functions

In reply to Max Reitz

Use the child access functions in the block status inquiry functions as appropriate.

---
 block/io.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
block/io.c +10 -9
index 385176b331..dc9891d6ce 100644
@@ -2407,11 +2407,12 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,
     if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) {
         ret |= BDRV_BLOCK_ALLOCATED;
     } else if (want_zero) {
+        BlockDriverState *cow_bs = bdrv_cow_bs(bs);
+
         if (bdrv_unallocated_blocks_are_zero(bs)) {
             ret |= BDRV_BLOCK_ZERO;
-        } else if (bs->backing) {
-            BlockDriverState *bs2 = bs->backing->bs;
-            int64_t size2 = bdrv_getlength(bs2);
+        } else if (cow_bs) {
+            int64_t size2 = bdrv_getlength(cow_bs);
 
             if (size2 >= 0 && offset >= size2) {
                 ret |= BDRV_BLOCK_ZERO;
@@ -2477,7 +2478,7 @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,
     bool first = true;
 
     assert(bs != base);
-    for (p = bs; p != base; p = backing_bs(p)) {
+    for (p = bs; p != base; p = bdrv_filter_or_cow_bs(p)) {
         ret = bdrv_co_block_status(p, want_zero, offset, bytes, pnum, map,
                                    file);
         if (ret < 0) {
@@ -2551,7 +2552,7 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,
 int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes,
                       int64_t *pnum, int64_t *map, BlockDriverState **file)
 {
-    return bdrv_block_status_above(bs, backing_bs(bs),
+    return bdrv_block_status_above(bs, bdrv_filter_or_cow_bs(bs),
                                    offset, bytes, pnum, map, file);
 }
 
@@ -2561,9 +2562,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
     int ret;
     int64_t dummy;
 
-    ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, offset,
-                                         bytes, pnum ? pnum : &dummy, NULL,
-                                         NULL);
+    ret = bdrv_common_block_status_above(bs, bdrv_filter_or_cow_bs(bs), false,
+                                         offset, bytes, pnum ? pnum : &dummy,
+                                         NULL, NULL);
     if (ret < 0) {
         return ret;
     }
@@ -2626,7 +2627,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,
             break;
         }
 
-        intermediate = backing_bs(intermediate);
+        intermediate = bdrv_filter_or_cow_bs(intermediate);
     }
 
     *pnum = n;
Signature · 1 line
2.26.2
Reviewed-by on v7 13/47
#42 lore raw

Re: [PATCH v7 13/47] block: Use CAFs in block status functions

Quoted 60 lines · Max Reitz

Use the child access functions in the block status inquiry functions as appropriate.

---
  block/io.c | 19 ++++++++++---------
  1 file changed, 10 insertions(+), 9 deletions(-)
block/io.c +9 -8
index 385176b331..dc9891d6ce 100644
@@ -2407,11 +2407,12 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,
      if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) {
          ret |= BDRV_BLOCK_ALLOCATED;
      } else if (want_zero) {
+        BlockDriverState *cow_bs = bdrv_cow_bs(bs);
+
          if (bdrv_unallocated_blocks_are_zero(bs)) {
              ret |= BDRV_BLOCK_ZERO;
-        } else if (bs->backing) {
-            BlockDriverState *bs2 = bs->backing->bs;
-            int64_t size2 = bdrv_getlength(bs2);
+        } else if (cow_bs) {
+            int64_t size2 = bdrv_getlength(cow_bs);
 
              if (size2 >= 0 && offset >= size2) {
                  ret |= BDRV_BLOCK_ZERO;
@@ -2477,7 +2478,7 @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,
      bool first = true;
 
      assert(bs != base);
-    for (p = bs; p != base; p = backing_bs(p)) {
+    for (p = bs; p != base; p = bdrv_filter_or_cow_bs(p)) {
          ret = bdrv_co_block_status(p, want_zero, offset, bytes, pnum, map,
                                     file);
          if (ret < 0) {
@@ -2551,7 +2552,7 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,
  int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes,
                        int64_t *pnum, int64_t *map, BlockDriverState **file)
  {
-    return bdrv_block_status_above(bs, backing_bs(bs),
+    return bdrv_block_status_above(bs, bdrv_filter_or_cow_bs(bs),
                                     offset, bytes, pnum, map, file);
  }
 
@@ -2561,9 +2562,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
      int ret;
      int64_t dummy;
 
-    ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, offset,
-                                         bytes, pnum ? pnum : &dummy, NULL,
-                                         NULL);
+    ret = bdrv_common_block_status_above(bs, bdrv_filter_or_cow_bs(bs), false,
+                                         offset, bytes, pnum ? pnum : &dummy,
+                                         NULL, NULL);
      if (ret < 0) {
          return ret;
      }

block/io.c int bdrv_is_allocated_above(BlockDriverState *top, · line 2633

Quoted 8 lines
block/io.c +1 -1
@@ -2626,7 +2627,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,
              break;
          }
 
-        intermediate = backing_bs(intermediate);
+        intermediate = bdrv_filter_or_cow_bs(intermediate);
      }
 
      *pnum = n;
author
#43 lore raw

PATCH v7 14/47 stream: Deal with filters

In reply to Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 +++
 block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
 blockdev.c           |  4 ++-
 3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +4 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.
+#
 # @job-id: identifier for the newly-created block job. If
 #          omitted, the device name will be used. (Since 2.7)
 #
block/stream.c +46 -17

137 lines not on this page · raw message

blockdev.c +3 -1
index 72df193ca7..1eb0fcdea2 100644
@@ -2515,7 +2515,9 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
     }
 
     /* Check for op blockers in the whole chain between bs and base */
-    for (iter = bs; iter && iter != base_bs; iter = backing_bs(iter)) {
+    for (iter = bs; iter && iter != base_bs;
+         iter = bdrv_filter_or_cow_bs(iter))
+    {
         if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_STREAM, errp)) {
             goto out;
         }
Signature · 1 line
2.26.2
#44 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 33 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
  qapi/block-core.json |  4 +++
  block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
  blockdev.c           |  4 ++-
  3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +4 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
  # On successful completion the image file is updated to drop the backing file
  # and the BLOCK_JOB_COMPLETED event is emitted.
  #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.
+#
  # @job-id: identifier for the newly-created block job. If
  #          omitted, the device name will be used. (Since 2.7)
  #
block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644

block/stream.c enum { · line 35

Quoted 6 lines
block/stream.c +2 -1
@@ -31,7 +31,8 @@ enum {
 
  typedef struct StreamBlockJob {
      BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

The above_base may disappear during the job and we can't rely on it.

Quoted 13 lines
block/stream.c +1 -1
      BlockdevOnError on_error;
      char *backing_file_str;
      bool bs_read_only;
@@ -53,7 +54,7 @@ static void stream_abort(Job *job)
 
      if (s->chain_frozen) {
          BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
      }
  }
 

block/stream.c static int stream_prepare(Job *job) · line 67

Quoted 6 lines
block/stream.c +2 -1
@@ -62,14 +63,15 @@ static int stream_prepare(Job *job)
      StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
      BlockJob *bjob = &s->common;
      BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+    BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);

The initial base node may be a top node for a concurrent commit job and

may disappear. It is true for the above_base as well.

base = bdrv_filter_or_cow_bs(s->base_overlay) is more reliable.

Quoted 60 lines
block/stream.c +10 -8
      Error *local_err = NULL;
      int ret = 0;
 
-    bdrv_unfreeze_backing_chain(bs, s->bottom);
+    bdrv_unfreeze_backing_chain(bs, s->above_base);
      s->chain_frozen = false;
 
-    if (bs->backing) {
+    if (bdrv_cow_child(unfiltered_bs)) {
          const char *base_id = NULL, *base_fmt = NULL;
          if (base) {
              base_id = s->backing_file_str;
@@ -77,8 +79,8 @@ static int stream_prepare(Job *job)
                  base_fmt = base->drv->format_name;
              }
          }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt);
          if (local_err) {
              error_report_err(local_err);
              return -EPERM;
@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
      StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
      BlockBackend *blk = s->common.blk;
      BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+    bool enable_cor = !bdrv_cow_child(s->base_overlay);
      int64_t len;
      int64_t offset = 0;
      uint64_t delay_ns = 0;
      int error = 0;
      int64_t n = 0; /* bytes */
 
-    if (bs == s->bottom) {
+    if (unfiltered_bs == s->base_overlay) {
          /* Nothing to stream */
          return 0;
      }
@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 
          copy = false;
 
-        ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK, &n);
          if (ret == 1) {
              /* Allocated in the top, no need to copy.  */
          } else if (ret >= 0) {
              /* Copy if allocated in the intermediate images.  Limit to the
               * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs),
+                                          s->base_overlay, true,
                                            offset, n, &n);
              /* Finish early if end of backing file has been reached */
              if (ret == 0 && n == 0) {
@@ -223,9 +227,29 @@ void stream_start(const char *job_id, BlockDriverState *bs,
      BlockDriverState *iter;

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 235

Quoted 10 lines
block/stream.c +5 -2
      bool bs_read_only;
      int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base);
+    BlockDriverState *base_overlay = bdrv_find_overlay(bs, base);
+    BlockDriverState *above_base;
 
-    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) {
+    if (!base_overlay) {
+        error_setg(errp, "'%s' is not in the backing chain of '%s'",
+                   base->node_name, bs->node_name);

Sorry, I am not clear with the error message.

In this case, there is no an intermediate COW node but the base, if not NULL, is

in the backing chain of bs, isn't it?

Quoted 7 lines
block/stream.c +7 -0
+        return;
+    }
+
+    /*
+     * Find the node directly above @base.  @base_overlay is a COW overlay, so
+     * it must have a bdrv_cow_child(), but it is the immediate overlay of
+     * @base, so between the two there can only be filters.

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 252

Quoted 10 lines
block/stream.c +10 -0
+     */
+    above_base = base_overlay;
+    if (bdrv_cow_bs(above_base) != base) {
+        above_base = bdrv_cow_bs(above_base);
+        while (bdrv_filter_bs(above_base) != base) {
+            above_base = bdrv_filter_bs(above_base);
+        }
+    }
+
+    if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) {

When a concurrent stream job tries to freeze or remove the above_base node,

we will encounter the frozen node error. The above_base node is a part of the

concurrent job frozen chain.

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 282

Quoted 8 lines
block/stream.c +1 -0
          return;
      }
 
@@ -255,14 +279,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,
       * and resizes. Reassign the base node pointer because the backing BS of the
       * bottom node might change after the call to bdrv_reopen_set_read_only()
       * due to parallel block jobs running.
+     * above_base node might change after the call to

Yes, if not frozen.

Quoted 5 lines
block/stream.c +2 -2
+     * bdrv_reopen_set_read_only() due to parallel block jobs running.
       */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filter_or_cow_bs(above_base);

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 294

Quoted 10 lines
block/stream.c +5 -1
+    for (iter = bdrv_filter_or_cow_bs(bs); iter != base;
+         iter = bdrv_filter_or_cow_bs(iter))
+    {
          block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
                             basic_flags, &error_abort);
      }
 
-    s->bottom = bottom;
+    s->base_overlay = base_overlay;
+    s->above_base = above_base;

Generally, being the filter for a concurrent job, the above_base node may be deleted any time

and we will keep the dangling pointer. It may happen even earlier if above_base is not frozen.

If it is, as it here, we may get the frozen link error then.

Andrey

Quoted 25 lines
block/stream.c +1 -1
      s->backing_file_str = g_strdup(backing_file_str);
      s->bs_read_only = bs_read_only;
      s->chain_frozen = true;
@@ -276,5 +305,5 @@ fail:
      if (bs_read_only) {
          bdrv_reopen_set_read_only(bs, true, NULL);
      }
-    bdrv_unfreeze_backing_chain(bs, bottom);
+    bdrv_unfreeze_backing_chain(bs, above_base);
  }
blockdev.c +3 -1
index 72df193ca7..1eb0fcdea2 100644
@@ -2515,7 +2515,9 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
      }
 
      /* Check for op blockers in the whole chain between bs and base */
-    for (iter = bs; iter && iter != base_bs; iter = backing_bs(iter)) {
+    for (iter = bs; iter && iter != base_bs;
+         iter = bdrv_filter_or_cow_bs(iter))
+    {
          if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_STREAM, errp)) {
              goto out;
          }
#45 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 14 lines · Andrey Shinkevich
Quoted 13 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
  qapi/block-core.json |  4 +++
  block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c           |  4 ++-

  3 files changed, 53 insertions(+), 18 deletions(-)

...

Quoted 7 lines
Quoted 6 lines

+    BlockDriverState *base_overlay = bdrv_find_overlay(bs, base); +    BlockDriverState *above_base;
  -    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) { +    if (!base_overlay) {
+        error_setg(errp, "'%s' is not in the backing chain of '%s'", +                   base->node_name, bs->node_name);

Quoted 6 lines

Sorry, I am not clear with the error message.

In this case, there is no an intermediate COW node but the base, if not NULL, is

in the backing chain of bs, isn't it?

I am discarding this question. No need to answer.

Andrey

Quoted 3 lines
Quoted 3 lines

+        return;
+    }
+

author
#46 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

In reply to Andrey Shinkevich
Quoted 36 lines · Andrey Shinkevich
Quoted 35 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
  qapi/block-core.json |  4 +++
  block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c           |  4 ++-

  3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
   # On successful completion the image file is updated to drop the

backing file

  # and the BLOCK_JOB_COMPLETED event is emitted.
  #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#

  # @job-id: identifier for the newly-created block job. If #          omitted, the device name will be used. (Since 2.7) #

block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
Quoted 8 lines

block/stream.c enum { · line 34

Quoted 6 lines
block/stream.c +2 -1
@@ -31,7 +31,8 @@ enum {
     typedef struct StreamBlockJob {
       BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Depends on the definition. If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
So the above_base cannot disappear.

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we keep above_base and use its immediate child.

If we don’t keep above_base, then we’re basically left guessing as to what should be the backing file after the stream job.

Quoted 20 lines
Quoted 12 lines
block/stream.c +1 -1
       BlockdevOnError on_error;
       char *backing_file_str;
       bool bs_read_only;
@@ -53,7 +54,7 @@ static void stream_abort(Job *job)
         if (s->chain_frozen) {
           BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
       }
   }
   @@ -62,14 +63,15 @@ static int stream_prepare(Job *job)
       StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
Quoted 5 lines

      BlockJob *bjob = &s->common;
      BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); +    BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);

The initial base node may be a top node for a concurrent commit job and

Quoted 1 line

may disappear.

Then it would just be replaced by another node, though, so above_base keeps a child. The @base here is not necessarily the initial @base, and that’s intentional.

Quoted 1 line

base = bdrv_filter_or_cow_bs(s->base_overlay) is more reliable.

But also wrong. The point of keeping above_base around is to get its child here to use that child as the new backing child of the top node.

Quoted 103 lines
Quoted 65 lines

      Error *local_err = NULL;
      int ret = 0;
  -    bdrv_unfreeze_backing_chain(bs, s->bottom); +    bdrv_unfreeze_backing_chain(bs, s->above_base); s->chain_frozen = false;
  -    if (bs->backing) {
+    if (bdrv_cow_child(unfiltered_bs)) { const char *base_id = NULL, *base_fmt = NULL; if (base) {
              base_id = s->backing_file_str;

block/stream.c +2 -2
@@ -77,8 +79,8 @@ static int stream_prepare(Job *job)
                   base_fmt = base->drv->format_name;
               }
           }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id,
 base_fmt);
           if (local_err) {
               error_report_err(local_err);

              return -EPERM;

block/stream.c +2 -1
@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
       StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
       BlockBackend *blk = s->common.blk;
       BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+    bool enable_cor = !bdrv_cow_child(s->base_overlay);
       int64_t len;
       int64_t offset = 0;
       uint64_t delay_ns = 0;
       int error = 0;

      int64_t n = 0; /* bytes */
  -    if (bs == s->bottom) {
+    if (unfiltered_bs == s->base_overlay) {
          /* Nothing to stream */
          return 0;
      }

block/stream.c +1 -0
@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
             copy = false;
   -        ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK,
 &n);
           if (ret == 1) {
               /* Allocated in the top, no need to copy.  */
           } else if (ret >= 0) {
               /* Copy if allocated in the intermediate images.  Limit

to the
               * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +                                          s->base_overlay, true, offset, n, &n); /* Finish early if end of backing file has been reached */ if (ret == 0 && n == 0) {

block/stream.c +0 -0
@@ -223,9 +227,29 @@ void stream_start(const char *job_id,
 BlockDriverState *bs,
       BlockDriverState *iter;
       bool bs_read_only;

      int basic_flags = BLK_PERM_CONSISTENT_READ |

Quoted 8 lines

BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base); +    BlockDriverState *base_overlay = bdrv_find_overlay(bs, base); +    BlockDriverState *above_base;
  -    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) { +    if (!base_overlay) {
+        error_setg(errp, "'%s' is not in the backing chain of '%s'", +                   base->node_name, bs->node_name);

Sorry, I am not clear with the error message.

In this case, there is no an intermediate COW node but the base, if not NULL, is

in the backing chain of bs, isn't it?

Quoted 11 lines

+        return;
+    }
+
+    /*
+     * Find the node directly above @base.  @base_overlay is a COW overlay, so
+     * it must have a bdrv_cow_child(), but it is the immediate overlay of
+     * @base, so between the two there can only be filters. +     */
+    above_base = base_overlay;

Quoted 8 lines

+    if (bdrv_cow_bs(above_base) != base) {
+        above_base = bdrv_cow_bs(above_base);
+        while (bdrv_filter_bs(above_base) != base) { +            above_base = bdrv_filter_bs(above_base); +        }
+    }
+
+    if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) {

When a concurrent stream job tries to freeze or remove the above_base node,

Quoted 4 lines

we will encounter the frozen node error. The above_base node is a part of the

concurrent job frozen chain.

Correct.

Quoted 27 lines
Quoted 10 lines

          return;
      }
  @@ -255,14 +279,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,
       * and resizes. Reassign the base node pointer because the backing BS of the
       * bottom node might change after the call to bdrv_reopen_set_read_only()
       * due to parallel block jobs running.
+     * above_base node might change after the call to

Yes, if not frozen.

Quoted 7 lines

+     * bdrv_reopen_set_read_only() due to parallel block jobs running. */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filter_or_cow_bs(above_base);
+    for (iter = bdrv_filter_or_cow_bs(bs); iter != base;

Quoted 8 lines

+         iter = bdrv_filter_or_cow_bs(iter))
+    {
          block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
      }
  -    s->bottom = bottom;
+    s->base_overlay = base_overlay;
+    s->above_base = above_base;

Quoted 7 lines

Generally, being the filter for a concurrent job, the above_base node may be deleted any time

and we will keep the dangling pointer. It may happen even earlier if above_base is not frozen.

If it is, as it here, we may get the frozen link error then.

I’m not sure what you mean here. Freezing it was absolutely intentional. A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.

Max

#47 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 44 lines · Max Reitz
Quoted 35 lines · Andrey Shinkevich
Quoted 34 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
   qapi/block-core.json |  4 +++
   block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
   blockdev.c           |  4 ++-
   3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
   # On successful completion the image file is updated to drop the

backing file

   # and the BLOCK_JOB_COMPLETED event is emitted.
   #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#
   # @job-id: identifier for the newly-created block job. If
   #          omitted, the device name will be used. (Since 2.7)
   #
block/stream.c +0 -1
index aa2e7af98e..b9c1141656 100644
--- a/block/stream.c
Quoted 8 lines

block/stream.c enum { · line 34

Quoted 7 lines

+++ b/block/stream.c

block/stream.c +2 -1
@@ -31,7 +31,8 @@ enum {
     typedef struct StreamBlockJob {
       BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 6 lines

Depends on the definition. If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
So the above_base cannot disappear.

Once we insert a filter above the top bs of the stream job, the parallel jobs in

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

Quoted 6 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we keep above_base and use its immediate child.

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

Quoted 23 lines

If we don’t keep above_base, then we’re basically left guessing as to what should be the backing file after the stream job.

Quoted 19 lines
Quoted 12 lines
block/stream.c +1 -1
       BlockdevOnError on_error;
       char *backing_file_str;
       bool bs_read_only;
@@ -53,7 +54,7 @@ static void stream_abort(Job *job)
         if (s->chain_frozen) {
           BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
       }
   }
   @@ -62,14 +63,15 @@ static int stream_prepare(Job *job)
       StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
Quoted 5 lines

      BlockJob *bjob = &s->common;
      BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); +    BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);

The initial base node may be a top node for a concurrent commit job and

Quoted 4 lines
Quoted 1 line

may disappear.

Then it would just be replaced by another node, though, so above_base keeps a child. The @base here is not necessarily the initial @base, and that’s intentional.

Not really. In my example, above_base becomes a dangling

pointer because after the commit job finishes, its filter that should belong to the

commit job frozen chain will be deleted. If we freeze the link to the above_base

for this job, the iotests #30 will not pass.

Quoted 143 lines
Quoted 1 line

base = bdrv_filter_or_cow_bs(s->base_overlay) is more reliable.

But also wrong. The point of keeping above_base around is to get its child here to use that child as the new backing child of the top node.

Quoted 101 lines
Quoted 63 lines

      Error *local_err = NULL;
      int ret = 0;
  -    bdrv_unfreeze_backing_chain(bs, s->bottom);
+    bdrv_unfreeze_backing_chain(bs, s->above_base);
      s->chain_frozen = false;
  -    if (bs->backing) {
+    if (bdrv_cow_child(unfiltered_bs)) {
          const char *base_id = NULL, *base_fmt = NULL; if (base) {
              base_id = s->backing_file_str;

block/stream.c +2 -2
@@ -77,8 +79,8 @@ static int stream_prepare(Job *job)
                   base_fmt = base->drv->format_name;
               }
           }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id,
 base_fmt);
           if (local_err) {
               error_report_err(local_err);

              return -EPERM;

block/stream.c +4 -1
@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
       StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
       BlockBackend *blk = s->common.blk;
       BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+    bool enable_cor = !bdrv_cow_child(s->base_overlay);
       int64_t len;
       int64_t offset = 0;
       uint64_t delay_ns = 0;
       int error = 0;
       int64_t n = 0; /* bytes */
   -    if (bs == s->bottom) {
+    if (unfiltered_bs == s->base_overlay) {
           /* Nothing to stream */
           return 0;
       }
@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
             copy = false;
   -        ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK,
 &n);
           if (ret == 1) {
               /* Allocated in the top, no need to copy.  */
           } else if (ret >= 0) {
               /* Copy if allocated in the intermediate images.  Limit

to the
               * known-unallocated area [offset,
offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +                                          s->base_overlay, true,
                                            offset, n, &n); /* Finish early if end of backing file has been reached */ if (ret == 0 && n == 0) {

block/stream.c +0 -0
@@ -223,9 +227,29 @@ void stream_start(const char *job_id,
 BlockDriverState *bs,
       BlockDriverState *iter;

block/stream.c void stream_start(const char *job_id, · line 237

Quoted 10 lines
block/stream.c +5 -1
       bool bs_read_only;
       int basic_flags = BLK_PERM_CONSISTENT_READ |
 BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base);
+    BlockDriverState *base_overlay = bdrv_find_overlay(bs, base);
+    BlockDriverState *above_base;
   -    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) {
+    if (!base_overlay) {
+        error_setg(errp, "'%s' is not in the backing chain of '%s'",
+                   base->node_name, bs->node_name);

Sorry, I am not clear with the error message.

In this case, there is no an intermediate COW node but the base, if not NULL, is

in the backing chain of bs, isn't it?

Quoted 11 lines
block/stream.c +5 -0
+        return;
+    }
+
+    /*
+     * Find the node directly above @base.  @base_overlay is a COW

overlay, so
+     * it must have a bdrv_cow_child(), but it is the immediate overlay of
+     * @base, so between the two there can only be filters. +     */
+    above_base = base_overlay;

Quoted 8 lines

+    if (bdrv_cow_bs(above_base) != base) {
+        above_base = bdrv_cow_bs(above_base);
+        while (bdrv_filter_bs(above_base) != base) { +            above_base = bdrv_filter_bs(above_base); +        }
+    }
+
+    if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) {

When a concurrent stream job tries to freeze or remove the above_base node,

Quoted 4 lines

we will encounter the frozen node error. The above_base node is a part of the

concurrent job frozen chain.

Correct.

Quoted 32 lines
Quoted 10 lines

          return;
      }
  @@ -255,14 +279,19 @@ void stream_start(const char *job_id,
BlockDriverState *bs,
       * and resizes. Reassign the base node pointer because the
backing BS of the
       * bottom node might change after the call to
bdrv_reopen_set_read_only()
       * due to parallel block jobs running.
+     * above_base node might change after the call to

Yes, if not frozen.

Quoted 7 lines

+     * bdrv_reopen_set_read_only() due to parallel block jobs running.
       */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filter_or_cow_bs(above_base);
+    for (iter = bdrv_filter_or_cow_bs(bs); iter != base;

Quoted 8 lines

+         iter = bdrv_filter_or_cow_bs(iter))
+    {
          block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
      }
  -    s->bottom = bottom;
+    s->base_overlay = base_overlay;
+    s->above_base = above_base;

Generally, being the filter for a concurrent job, the above_base node may be deleted any time

and we will keep the dangling pointer. It may happen even earlier if above_base is not frozen.

Quoted 6 lines
Quoted 1 line

If it is, as it here, we may get the frozen link error then.

I’m not sure what you mean here. Freezing it was absolutely intentional. A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.

Max

The nodes we freeze should be in one context of the relevant job:

filter->top_node->intermediate_node(s)

We would not include the base or any filter above it to the frozen chain

because they are of a different job context.

Once 'this' job is completed, we set the current backing child of the base_overlay

and may not care of its character. If that is another job filter, it will be replaced

with the proper node afterwards.

Andrey

author
#48 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 54 lines · Andrey Shinkevich
Quoted 46 lines · Max Reitz
Quoted 37 lines · Andrey Shinkevich
Quoted 36 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
   qapi/block-core.json |  4 +++
   block/stream.c       | 63
++++++++++++++++++++++++++++++++------------
   blockdev.c           |  4 ++-

   3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
    # On successful completion the image file is updated to drop the

backing file

   # and the BLOCK_JOB_COMPLETED event is emitted.
   #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#

   # @job-id: identifier for the newly-created block job. If #          omitted, the device name will be used. (Since 2.7) #

block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
Quoted 8 lines
Quoted 7 lines
block/stream.c +1 -1
@@ -31,7 +31,8 @@ enum {
      typedef struct StreamBlockJob {
        BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from

this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 6 lines

Depends on the definition.  If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen. So the above_base cannot disappear.

Quoted 8 lines

Once we insert a filter above the top bs of the stream job, the parallel jobs in

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

I’m not sure whether that’s a problem with this series specifically.

Quoted 11 lines
Quoted 5 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming.  How do we figure out what that should be?  My proposal is we keep above_base and use its immediate child.

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?

Quoted 30 lines
Quoted 23 lines

If we don’t keep above_base, then we’re basically left guessing as to what should be the backing file after the stream job.

Quoted 20 lines
Quoted 12 lines

       BlockdevOnError on_error;
       char *backing_file_str;
       bool bs_read_only;

block/stream.c +1 -1
@@ -53,7 +54,7 @@ static void stream_abort(Job *job)
          if (s->chain_frozen) {
            BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
        }
    }
    @@ -62,14 +63,15 @@ static int stream_prepare(Job *job)
        StreamBlockJob *s = container_of(job, StreamBlockJob,
Quoted 6 lines

common.job);
       BlockJob *bjob = &s->common;
       BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); +    BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);

The initial base node may be a top node for a concurrent commit job and

Quoted 4 lines
Quoted 1 line

may disappear.

Then it would just be replaced by another node, though, so above_base keeps a child.  The @base here is not necessarily the initial @base, and that’s intentional.

Not really. In my example, above_base becomes a dangling

Quoted 7 lines

pointer because after the commit job finishes, its filter that should belong to the

commit job frozen chain will be deleted. If we freeze the link to the above_base

for this job, the iotests #30 will not pass.

So it doesn’t become a dangling pointer, because it’s frozen.

030 passes after this series, so I’m not sure whether I can consider that problem part of this series.

I think if adding a filter node becomes a problem, we have to consider relaxing the restrictions when we do that, not now.

Quoted 153 lines
Quoted 146 lines
Quoted 1 line

base = bdrv_filter_or_cow_bs(s->base_overlay) is more reliable.

But also wrong.  The point of keeping above_base around is to get its child here to use that child as the new backing child of the top node.

Quoted 104 lines
Quoted 67 lines

       Error *local_err = NULL;
       int ret = 0;
   -    bdrv_unfreeze_backing_chain(bs, s->bottom); +    bdrv_unfreeze_backing_chain(bs, s->above_base); s->chain_frozen = false;
   -    if (bs->backing) {
+    if (bdrv_cow_child(unfiltered_bs)) {
           const char *base_id = NULL, *base_fmt = NULL; if (base) {
               base_id = s->backing_file_str;

block/stream.c +2 -2
@@ -77,8 +79,8 @@ static int stream_prepare(Job *job)
                    base_fmt = base->drv->format_name;
                }
            }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id,
 base_fmt);
            if (local_err) {
                error_report_err(local_err);

               return -EPERM;

block/stream.c +2 -1
@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
        StreamBlockJob *s = container_of(job, StreamBlockJob,
 common.job);
        BlockBackend *blk = s->common.blk;
        BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+    bool enable_cor = !bdrv_cow_child(s->base_overlay);
        int64_t len;
        int64_t offset = 0;
        uint64_t delay_ns = 0;
        int error = 0;

       int64_t n = 0; /* bytes */
   -    if (bs == s->bottom) {
+    if (unfiltered_bs == s->base_overlay) {
           /* Nothing to stream */
           return 0;
       }

block/stream.c +1 -0
@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
              copy = false;
    -        ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK,
 &n);
            if (ret == 1) {
                /* Allocated in the top, no need to copy.  */
            } else if (ret >= 0) {
                /* Copy if allocated in the intermediate images.  Limit

to the
                * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +                                          s->base_overlay, true, offset, n, &n); /* Finish early if end of backing file has been reached */
               if (ret == 0 && n == 0) {

block/stream.c +0 -0
@@ -223,9 +227,29 @@ void stream_start(const char *job_id,
 BlockDriverState *bs,
        BlockDriverState *iter;
        bool bs_read_only;

       int basic_flags = BLK_PERM_CONSISTENT_READ |

Quoted 8 lines

BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base); +    BlockDriverState *base_overlay = bdrv_find_overlay(bs, base); +    BlockDriverState *above_base;
   -    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) { +    if (!base_overlay) {
+        error_setg(errp, "'%s' is not in the backing chain of '%s'", +                   base->node_name, bs->node_name);

Sorry, I am not clear with the error message.

In this case, there is no an intermediate COW node but the base, if not NULL, is

in the backing chain of bs, isn't it?

Quoted 11 lines

+        return;
+    }
+
+    /*
+     * Find the node directly above @base.  @base_overlay is a COW overlay, so
+     * it must have a bdrv_cow_child(), but it is the immediate overlay of
+     * @base, so between the two there can only be filters. +     */
+    above_base = base_overlay;

Quoted 8 lines

+    if (bdrv_cow_bs(above_base) != base) {
+        above_base = bdrv_cow_bs(above_base);
+        while (bdrv_filter_bs(above_base) != base) { +            above_base = bdrv_filter_bs(above_base); +        }
+    }
+
+    if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) {

When a concurrent stream job tries to freeze or remove the above_base node,

Quoted 4 lines

we will encounter the frozen node error. The above_base node is a part of the

concurrent job frozen chain.

Correct.

Quoted 32 lines
Quoted 10 lines

           return;
       }
   @@ -255,14 +279,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,
        * and resizes. Reassign the base node pointer because the backing BS of the
        * bottom node might change after the call to bdrv_reopen_set_read_only()
        * due to parallel block jobs running.
+     * above_base node might change after the call to

Yes, if not frozen.

Quoted 7 lines

+     * bdrv_reopen_set_read_only() due to parallel block jobs running. */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filter_or_cow_bs(above_base);
+    for (iter = bdrv_filter_or_cow_bs(bs); iter != base;

Quoted 8 lines

+         iter = bdrv_filter_or_cow_bs(iter))
+    {
           block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
       }
   -    s->bottom = bottom;
+    s->base_overlay = base_overlay;
+    s->above_base = above_base;

Generally, being the filter for a concurrent job, the above_base node may be deleted any time

and we will keep the dangling pointer. It may happen even earlier if above_base is not frozen.

Quoted 6 lines
Quoted 1 line

If it is, as it here, we may get the frozen link error then.

I’m not sure what you mean here.  Freezing it was absolutely intentional.  A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.

Max

Quoted 7 lines

The nodes we freeze should be in one context of the relevant job:

filter->top_node->intermediate_node(s)

We would not include the base or any filter above it to the frozen chain

because they are of a different job context.

They aren’t really, because we need to know the backing node of @device after the job.

Quoted 7 lines

Once 'this' job is completed, we set the current backing child of the base_overlay

and may not care of its character. If that is another job filter, it will be replaced

with the proper node afterwards.

But what if there are filters above the base that the user wants to keep after the job?

Max

#49 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 72 lines · Max Reitz
Quoted 54 lines · Andrey Shinkevich
Quoted 46 lines · Max Reitz
Quoted 37 lines · Andrey Shinkevich
Quoted 36 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
    qapi/block-core.json |  4 +++
    block/stream.c       | 63
++++++++++++++++++++++++++++++++------------
    blockdev.c           |  4 ++-
    3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
    # On successful completion the image file is updated to drop the

backing file

    # and the BLOCK_JOB_COMPLETED event is emitted.
    #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#
    # @job-id: identifier for the newly-created block job. If
    #          omitted, the device name will be used. (Since 2.7)
    #
block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
Quoted 8 lines
Quoted 7 lines
block/stream.c +1 -1
@@ -31,7 +31,8 @@ enum {
      typedef struct StreamBlockJob {
        BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from

this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 6 lines

Depends on the definition.  If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
  So the above_base cannot disappear.

Quoted 8 lines

Once we insert a filter above the top bs of the stream job, the parallel jobs in

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

I’m not sure whether that’s a problem with this series specifically.

Quoted 6 lines
Quoted 5 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming.  How do we figure out what that should be?  My proposal is we keep above_base and use its immediate child.

Quoted 8 lines
Quoted 5 lines

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?

It's automatically kept, if we use base_overlay->backing->bs as final backing node.

You mean, that they want it to be dropped?

so, assuming the following:

top -(backing)-> manually-inserted-filter -(file)-> base

and user do stream with base=base, and expects filter to be removed by stream job?

Hmm, yes, such use-case is broken with our proposed way...

====

Let me now clarify the problem we'll have with your way.

When stream don't have any filter, we can easily imagine two parallel stream jobs:

top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base

stream1: top=top, base=mid2
stream2: top=mid2, base=NULL

final picture is obvious:

top (merged with mid1) -(backing)-> mid2 (merged with base)

But we want stream job has own filter, like mirror. So the picture becomes more complex.

Assume stream2 starts first.

top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base

Now, when we run stream1, with your solution, stream1 will freeze stream2-filter (wrong thing, stream2 will fail to remove it if it finished first), and stream1 will remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to removing of its filter)..

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

====

So, these are two mutually exclusive cases.. I vote for freezing up to base_overlay, and use backing(base_overlay) as final backing, because:

1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)

2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:

- Stream job is not intended to remove filters, but to stream data. Filters between base_overlay and base don't contain any data and unrelated to stream process
- I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable. On the contrary, removing all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.

Next, with your way, what about filters, inserted above base during stream job? They will be between above_base and base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.

Quoted 224 lines
Quoted 31 lines
Quoted 24 lines

If we don’t keep above_base, then we’re basically left guessing as to what should be the backing file after the stream job.

Quoted 20 lines
Quoted 12 lines

       BlockdevOnError on_error;
       char *backing_file_str;
       bool bs_read_only;

block/stream.c +1 -1
@@ -53,7 +54,7 @@ static void stream_abort(Job *job)
          if (s->chain_frozen) {
            BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base);
        }
    }
    @@ -62,14 +63,15 @@ static int stream_prepare(Job *job)
        StreamBlockJob *s = container_of(job, StreamBlockJob,
Quoted 6 lines

common.job);
       BlockJob *bjob = &s->common;
       BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); +    BlockDriverState *base = bdrv_filter_or_cow_bs(s->above_base);

The initial base node may be a top node for a concurrent commit job and

Quoted 4 lines
Quoted 1 line

may disappear.

Then it would just be replaced by another node, though, so above_base keeps a child.  The @base here is not necessarily the initial @base, and that’s intentional.

Not really. In my example, above_base becomes a dangling

Quoted 7 lines

pointer because after the commit job finishes, its filter that should belong to the

commit job frozen chain will be deleted. If we freeze the link to the above_base

for this job, the iotests #30 will not pass.

So it doesn’t become a dangling pointer, because it’s frozen.

030 passes after this series, so I’m not sure whether I can consider that problem part of this series.

I think if adding a filter node becomes a problem, we have to consider relaxing the restrictions when we do that, not now.

Quoted 153 lines
Quoted 146 lines
Quoted 1 line

base = bdrv_filter_or_cow_bs(s->base_overlay) is more reliable.

But also wrong.  The point of keeping above_base around is to get its child here to use that child as the new backing child of the top node.

Quoted 104 lines
Quoted 65 lines

       Error *local_err = NULL;
       int ret = 0;
   -    bdrv_unfreeze_backing_chain(bs, s->bottom);
+    bdrv_unfreeze_backing_chain(bs, s->above_base);
       s->chain_frozen = false;
   -    if (bs->backing) {
+    if (bdrv_cow_child(unfiltered_bs)) {
           const char *base_id = NULL, *base_fmt = NULL; if (base) {
               base_id = s->backing_file_str;

block/stream.c +2 -2
@@ -77,8 +79,8 @@ static int stream_prepare(Job *job)
                    base_fmt = base->drv->format_name;
                }
            }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id,
 base_fmt);
            if (local_err) {
                error_report_err(local_err);

               return -EPERM;

block/stream.c +4 -1
@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
        StreamBlockJob *s = container_of(job, StreamBlockJob,
 common.job);
        BlockBackend *blk = s->common.blk;
        BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs);
+    bool enable_cor = !bdrv_cow_child(s->base_overlay);
        int64_t len;
        int64_t offset = 0;
        uint64_t delay_ns = 0;
        int error = 0;
        int64_t n = 0; /* bytes */
    -    if (bs == s->bottom) {
+    if (unfiltered_bs == s->base_overlay) {
            /* Nothing to stream */
            return 0;
        }
@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job,
 Error **errp)
              copy = false;
    -        ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK,
 &n);
            if (ret == 1) {
                /* Allocated in the top, no need to copy.  */
            } else if (ret >= 0) {
                /* Copy if allocated in the intermediate images.  Limit

to the
                * known-unallocated area [offset,
offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +                                          s->base_overlay, true,
                                             offset, n, &n); /* Finish early if end of backing file has been
reached */
               if (ret == 0 && n == 0) {

block/stream.c +0 -0
@@ -223,9 +227,29 @@ void stream_start(const char *job_id,
 BlockDriverState *bs,
        BlockDriverState *iter;

block/stream.c void stream_start(const char *job_id, · line 237

Quoted 10 lines
block/stream.c +5 -1
        bool bs_read_only;
        int basic_flags = BLK_PERM_CONSISTENT_READ |
 BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base);
+    BlockDriverState *base_overlay = bdrv_find_overlay(bs, base);
+    BlockDriverState *above_base;
    -    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) {
+    if (!base_overlay) {
+        error_setg(errp, "'%s' is not in the backing chain of '%s'",
+                   base->node_name, bs->node_name);

Sorry, I am not clear with the error message.

In this case, there is no an intermediate COW node but the base, if not NULL, is

in the backing chain of bs, isn't it?

Quoted 11 lines
block/stream.c +5 -0
+        return;
+    }
+
+    /*
+     * Find the node directly above @base.  @base_overlay is a COW

overlay, so
+     * it must have a bdrv_cow_child(), but it is the immediate overlay of
+     * @base, so between the two there can only be filters. +     */
+    above_base = base_overlay;

Quoted 8 lines

+    if (bdrv_cow_bs(above_base) != base) {
+        above_base = bdrv_cow_bs(above_base);
+        while (bdrv_filter_bs(above_base) != base) { +            above_base = bdrv_filter_bs(above_base); +        }
+    }
+
+    if (bdrv_freeze_backing_chain(bs, above_base, errp) < 0) {

When a concurrent stream job tries to freeze or remove the above_base node,

Quoted 4 lines

we will encounter the frozen node error. The above_base node is a part of the

concurrent job frozen chain.

Correct.

Quoted 32 lines
Quoted 10 lines

           return;
       }
   @@ -255,14 +279,19 @@ void stream_start(const char *job_id,
BlockDriverState *bs,
        * and resizes. Reassign the base node pointer because the
backing BS of the
        * bottom node might change after the call to
bdrv_reopen_set_read_only()
        * due to parallel block jobs running.
+     * above_base node might change after the call to

Yes, if not frozen.

Quoted 7 lines

+     * bdrv_reopen_set_read_only() due to parallel block jobs running.
        */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filter_or_cow_bs(above_base);
+    for (iter = bdrv_filter_or_cow_bs(bs); iter != base;

Quoted 8 lines

+         iter = bdrv_filter_or_cow_bs(iter))
+    {
           block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
       }
   -    s->bottom = bottom;
+    s->base_overlay = base_overlay;
+    s->above_base = above_base;

Generally, being the filter for a concurrent job, the above_base node may be deleted any time

and we will keep the dangling pointer. It may happen even earlier if above_base is not frozen.

Quoted 6 lines
Quoted 1 line

If it is, as it here, we may get the frozen link error then.

I’m not sure what you mean here.  Freezing it was absolutely intentional.  A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.

Max

Quoted 7 lines

The nodes we freeze should be in one context of the relevant job:

filter->top_node->intermediate_node(s)

We would not include the base or any filter above it to the frozen chain

because they are of a different job context.

They aren’t really, because we need to know the backing node of @device after the job.

Quoted 7 lines

Once 'this' job is completed, we set the current backing child of the base_overlay

and may not care of its character. If that is another job filter, it will be replaced

with the proper node afterwards.

But what if there are filters above the base that the user wants to keep after the job?

Max

Signature · 2 lines
Best regards,
Vladimir
author
#50 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 60 lines · Andrey Shinkevich
Quoted 48 lines · Max Reitz
Quoted 39 lines · Andrey Shinkevich
Quoted 38 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
    qapi/block-core.json |  4 +++
    block/stream.c       | 63
++++++++++++++++++++++++++++++++------------
    blockdev.c           |  4 ++-

    3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
     # On successful completion the image file is updated to drop the

backing file

    # and the BLOCK_JOB_COMPLETED event is emitted.
    #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#

    # @job-id: identifier for the newly-created block job. If #          omitted, the device name will be used. (Since 2.7) #

block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
@@ -31,7 +31,8 @@ enum {
Quoted 8 lines
Quoted 7 lines
block/stream.c +1 -1
       typedef struct StreamBlockJob {
         BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from

this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 7 lines

Depends on the definition.  If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
   So the above_base cannot disappear.

Once we insert a filter above the top bs of the stream job, the parallel jobs in

Quoted 7 lines

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

I’m not sure whether that’s a problem with this series specifically.

Quoted 14 lines
Quoted 7 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming.  How do we figure out what that should be?  My proposal is we
keep above_base and use its immediate child.

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

Quoted 8 lines
Quoted 2 lines

And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?

It's automatically kept, if we use base_overlay->backing->bs as final backing node.

You mean, that they want it to be dropped?

Er, yes. Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.

Quoted 16 lines

so, assuming the following:

top -(backing)-> manually-inserted-filter -(file)-> base

and user do stream with base=base, and expects filter to be removed by stream job?

Hmm, yes, such use-case is broken with our proposed way...

====

Let me now clarify the problem we'll have with your way.

When stream don't have any filter, we can easily imagine two parallel stream jobs:

Quoted 8 lines

top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base

stream1: top=top, base=mid2
stream2: top=mid2, base=NULL

final picture is obvious:

top (merged with mid1) -(backing)-> mid2 (merged with base)

Yes, and I don’t think this currently working case is broken by this series.

Quoted 1 line

But we want stream job has own filter, like mirror.

Which it does not have yet, right? Which is why I was saying that I don’t think this is a problem with this series. We could try to address it later.

Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?

Quoted 6 lines

So the picture becomes more complex.

Assume stream2 starts first.

top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base

stream2-filter would be on top of mid2, right?

Quoted 7 lines

Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..

Note that the user first needs to pass “mid2” as the base to the stream job stream1. Why don’t they just pass “stream2-filter”? In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.

I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.

If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.

Quoted 3 lines

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.

I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?

Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.

Quoted 9 lines

====

So, these are two mutually exclusive cases.. I vote for freezing up to base_overlay, and use backing(base_overlay) as final backing, because:

1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)

Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.

You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.

(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)

Quoted 6 lines

2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:

 - Stream job is not intended to remove filters, but to stream data. Filters between base_overlay and base don't contain any data and unrelated to stream process

Well, it is intended to remove nodes. You can only remove data-bearing nodes by moving data around. I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.

Quoted 4 lines

 - I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.

Yes.

Although it could be argued that it is a handy way to remove filters, in a backing chain at least. (Thanks to bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)

*shrug*

Quoted 3 lines

On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.

I don’t think so, actually. Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain? Or a COR node, shouldn’t that go above the chain after streaming?

I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here. The only reason to drop them is because that’s what I expect users to expect of the stream job.

Quoted 7 lines

Next, with your way, what about filters, inserted above base during stream job? They will be between above_base and base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.

Hm. The documentation seems the same to me. Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.

The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.

OK, so after all this text, maybe some more original problem searching.
I think it the root of the problem is that the stream job takes a @base
parameter, but as of c624b015bf14fe01, it doesn’t really matter anymore.
Maybe c624b015bf14fe01 should have introduced a new parameter for users
to specify the bottom node instead of @base.

Well, that would have made everything a parameter mess, but it would have saved us the trouble now.

In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it. So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.

I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming. (If you do graph modifications during the job, that’s on you, because since c624b015bf14fe01 we can’t keep the base frozen.)

Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change. The problem I have with it is that you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.

I think it would be nice if we could have something that remains truthful to the current documentation. If just ignoring implicit filters above @base would work, then I’d find that nice. If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.

Max

#51 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 129 lines · Max Reitz
Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 60 lines · Andrey Shinkevich
Quoted 48 lines · Max Reitz
Quoted 39 lines · Andrey Shinkevich
Quoted 38 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
     qapi/block-core.json |  4 +++
     block/stream.c       | 63
++++++++++++++++++++++++++++++++------------
     blockdev.c           |  4 ++-
     3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
     # On successful completion the image file is updated to drop the

backing file

     # and the BLOCK_JOB_COMPLETED event is emitted.
     #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#
     # @job-id: identifier for the newly-created block job. If
     #          omitted, the device name will be used. (Since 2.7)
     #
block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
@@ -31,7 +31,8 @@ enum {
Quoted 8 lines
Quoted 7 lines
block/stream.c +1 -1
       typedef struct StreamBlockJob {
         BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from

this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 7 lines

Depends on the definition.  If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
   So the above_base cannot disappear.

Once we insert a filter above the top bs of the stream job, the parallel jobs in

Quoted 7 lines

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

I’m not sure whether that’s a problem with this series specifically.

Quoted 14 lines
Quoted 7 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming.  How do we figure out what that should be?  My proposal is we
keep above_base and use its immediate child.

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

Quoted 8 lines
Quoted 2 lines

And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?

It's automatically kept, if we use base_overlay->backing->bs as final backing node.

You mean, that they want it to be dropped?

Er, yes. Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.

Quoted 16 lines

so, assuming the following:

top -(backing)-> manually-inserted-filter -(file)-> base

and user do stream with base=base, and expects filter to be removed by stream job?

Hmm, yes, such use-case is broken with our proposed way...

====

Let me now clarify the problem we'll have with your way.

When stream don't have any filter, we can easily imagine two parallel stream jobs:

Quoted 8 lines

top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base

stream1: top=top, base=mid2
stream2: top=mid2, base=NULL

final picture is obvious:

top (merged with mid1) -(backing)-> mid2 (merged with base)

Yes, and I don’t think this currently working case is broken by this series.

Quoted 1 line

But we want stream job has own filter, like mirror.

Quoted 7 lines

Which it does not have yet, right? Which is why I was saying that I don’t think this is a problem with this series. We could try to address it later.

Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?

I think, we can address it later, but it would be good to fit into one release cycle with these series, to not make incompatible behavior changes later.

Quoted 9 lines
Quoted 7 lines

So the picture becomes more complex.

Assume stream2 starts first.

top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base

stream2-filter would be on top of mid2, right?

Right. In my picture, "-(backing)->" means backing link. Hmm, most probably stream-filter is COR, which actually have file child. It doesn't matter here.

Quoted 9 lines
Quoted 8 lines

Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..

Quoted 4 lines

Note that the user first needs to pass “mid2” as the base to the stream job stream1. Why don’t they just pass “stream2-filter”? In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.

Hmm. I'm sure we already tried/discussed this.. But I don't remember the results. Seems, such logic should work as well. Andrey don't you remember? I think, the only difficulty is to update iotests 30 (in our stream-filter series) to use node-names, not filename.. Can we go this way?

Quoted 4 lines

I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.

Agree. I think stream-filter should be explicit, like recently introduced backup-top.

Quoted 6 lines

If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.

Quoted 8 lines
Quoted 3 lines

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.

But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.

Quoted 3 lines

I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?

I don't, but I like the idea :)

Quoted 4 lines

Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.

And again, this is impossible even with your way. I have an idea:

What about making the whole thing explicit?

We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.

Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)). Benefit: user have guarantee of what would be final backing node.

Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.

But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..

Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying exact node-names in it?..

Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.

What do you think of it? Could we break this scenario in one release without deprecation and don't care? Than I think my idea about base vs bottom-node arguments for stream job may be applied. Or what to do?

If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.

Quoted 16 lines
Quoted 10 lines

====

So, these are two mutually exclusive cases.. I vote for freezing up to base_overlay, and use backing(base_overlay) as final backing, because:

1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)

Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.

Quoted 5 lines

You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.

(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)

We can break existing user scenarios.. Seems unlikely. I don't know.

Quoted 61 lines
Quoted 7 lines

2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:

 - Stream job is not intended to remove filters, but to stream data.
Filters between base_overlay and base don't contain any data and unrelated to stream process

Well, it is intended to remove nodes. You can only remove data-bearing nodes by moving data around. I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.

Quoted 4 lines

 - I think, that filters are "more related" to their children than to
their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.

Yes.

Although it could be argued that it is a handy way to remove filters, in a backing chain at least. (Thanks to bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)

*shrug*

Quoted 3 lines

On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.

I don’t think so, actually. Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain? Or a COR node, shouldn’t that go above the chain after streaming?

I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here. The only reason to drop them is because that’s what I expect users to expect of the stream job.

Quoted 7 lines

Next, with your way, what about filters, inserted above base during stream job? They will be between above_base and base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.

Hm. The documentation seems the same to me. Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.

The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.

Quoted 5 lines

OK, so after all this text, maybe some more original problem searching.
I think it the root of the problem is that the stream job takes a @base
parameter, but as of c624b015bf14fe01, it doesn’t really matter anymore.
Maybe c624b015bf14fe01 should have introduced a new parameter for users
to specify the bottom node instead of @base.

Yes, absolutely agree. If we do it now, would it be incompatible change or not?

Quoted 16 lines

Well, that would have made everything a parameter mess, but it would have saved us the trouble now.

In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it. So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.

I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming. (If you do graph modifications during the job, that’s on you, because since c624b015bf14fe01 we can’t keep the base frozen.)

Quoted 5 lines

Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change.

I also try to think about future introduction of stream filter. It shouldn't be a breaking change as well. But now I think it can be done the hard-way if needed: just work without a filter if filter-node-name is not given. but probably I care to much. Finally we can just drop the test-case from 030 or insert additional intermediate node into it..

Quoted 8 lines

The problem I have with it is that
you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.

I think it would be nice if we could have something that remains truthful to the current documentation.

Then, we should freeze base node again, so just revert c624b015bf14fe01

I go and see it cover-letter of the series introduced c624b015bf14fe01:

This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job and can change before the stream job is completed. We avoid that dependency by introducing the bottom node.

Hmm. Bad that we didn't add an iotest, but the series solved existing problem:
parallel stream and commit, as commit already has filter. Commit has filter with "backing" child, so, I assume that the case worked prior to introducing frozen chains, broken then by frozen chains and then fixed by c624b015bf14fe01. Still I don't know are there some real users of such parallel jobs..

Quoted 5 lines

If just ignoring implicit
filters above @base would work, then I’d find that nice. If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.

OK, at this point, I think, I'm OK with your patch in context of these series.

But I fill that something more should be done. Could we just revert c624b015bf14fe01 as "non corresponding to specification"?

Signature · 2 lines
Best regards,
Vladimir
#52 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 211 lines · Vladimir Sementsov-Ogievskiy
Quoted 131 lines · Max Reitz
Quoted 89 lines · Vladimir Sementsov-Ogievskiy
Quoted 88 lines · Max Reitz
Quoted 62 lines · Andrey Shinkevich
Quoted 50 lines · Max Reitz
Quoted 41 lines · Andrey Shinkevich
Quoted 40 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
     qapi/block-core.json |  4 +++
     block/stream.c       | 63
++++++++++++++++++++++++++++++++------------
     blockdev.c           |  4 ++-

     3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
      # On successful completion the image file is updated to drop the

backing file

     # and the BLOCK_JOB_COMPLETED event is emitted.
     #
+# In case @device is a filter node, block-stream modifies the 
first
non-filter
+# overlay node below it to point to base's backing node (or 
NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#

     # @job-id: identifier for the newly-created block job. If #          omitted, the device name will be used. (Since 2.7) #

block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
@@ -31,7 +31,8 @@ enum {
Quoted 8 lines
Quoted 7 lines
block/stream.c +1 -1
        typedef struct StreamBlockJob {
          BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from

this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 7 lines

Depends on the definition.  If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
    So the above_base cannot disappear.

Once we insert a filter above the top bs of the stream job, the parallel jobs in

Quoted 7 lines

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

I’m not sure whether that’s a problem with this series specifically.

Quoted 14 lines
Quoted 7 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming.  How do we figure out what that should be? My proposal is we
keep above_base and use its immediate child.

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

Quoted 8 lines
Quoted 2 lines

And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?

It's automatically kept, if we use base_overlay->backing->bs as final backing node.

You mean, that they want it to be dropped?

Er, yes.  Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.

Quoted 16 lines

so, assuming the following:

top -(backing)-> manually-inserted-filter -(file)-> base

and user do stream with base=base, and expects filter to be removed by stream job?

Hmm, yes, such use-case is broken with our proposed way...

====

Let me now clarify the problem we'll have with your way.

When stream don't have any filter, we can easily imagine two parallel stream jobs:

Quoted 8 lines

top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base

stream1: top=top, base=mid2
stream2: top=mid2, base=NULL

final picture is obvious:

top (merged with mid1) -(backing)-> mid2 (merged with base)

Yes, and I don’t think this currently working case is broken by this series.

Quoted 1 line

But we want stream job has own filter, like mirror.

Quoted 7 lines

Which it does not have yet, right?  Which is why I was saying that I don’t think this is a problem with this series.  We could try to address it later.

Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?

I think, we can address it later, but it would be good to fit into one release cycle with these series, to not make incompatible behavior changes later.

Quoted 9 lines
Quoted 7 lines

So the picture becomes more complex.

Assume stream2 starts first.

top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base

stream2-filter would be on top of mid2, right?

Right. In my picture, "-(backing)->" means backing link. Hmm, most probably stream-filter is COR, which actually have file child. It doesn't matter here.

Quoted 9 lines
Quoted 8 lines

Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..

Quoted 4 lines

Note that the user first needs to pass “mid2” as the base to the stream job stream1.  Why don’t they just pass “stream2-filter”?  In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.

Hmm. I'm sure we already tried/discussed this.. But I don't remember the results. Seems, such logic should work as well. Andrey don't you remember? I think, the only difficulty is to update iotests 30 (in our stream-filter series) to use node-names, not filename.. Can we go this way?

Quoted 4 lines

I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.

Agree. I think stream-filter should be explicit, like recently introduced backup-top.

Quoted 6 lines

If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.

Quoted 8 lines
Quoted 3 lines

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming.  And the QAPI schema says that base-node is to become the backing file of the top node after streaming.

But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.

Quoted 3 lines

I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?

I don't, but I like the idea :)

Quoted 4 lines

Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.

And again, this is impossible even with your way. I have an idea:

What about making the whole thing explicit?

We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.

Quoted 8 lines

Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)). Benefit: user have guarantee of what would be final backing node.

Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.

But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..

Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying  exact node-names in it?..

That is what I am doing now but have not completed yet. Actually, when we insert the COR-filter for a stream job, the cases with concurrent jobs fail in the iotests:030. I am trying to cope with the issues and still in progress. The problem I am forseeing is that if a user specifies the filter node as a base for a job, the name of the filter will be written down to the QCOW2 header on the disk. It can happen if bdrv_refresh_filename() returns the json-name of the filter. Max separated the backing file of QCOW2 header from the one, let's say, in the running program with this series. Imagine, If the user doesn't specify the backing file in a QAPI command, the program will use the name of the filter read from the QCOW2 header. Let me please finish the experiment and I will back to you with results or issues...

Andrey

Quoted 162 lines

Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.

What do you think of it? Could we break this scenario in one release without deprecation and don't care? Than I think my idea about base vs bottom-node arguments for stream job may be applied. Or what to do?

If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.

Quoted 16 lines
Quoted 10 lines

====

So, these are two mutually exclusive cases.. I vote for freezing up to base_overlay, and use backing(base_overlay) as final backing, because:

1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)

Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.

Quoted 5 lines

You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.

(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)

We can break existing user scenarios.. Seems unlikely. I don't know.

Quoted 61 lines
Quoted 7 lines

2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:

  - Stream job is not intended to remove filters, but to stream data. Filters between base_overlay and base don't contain any data and unrelated to stream process

Well, it is intended to remove nodes.  You can only remove data-bearing nodes by moving data around.  I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.

Quoted 4 lines

  - I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.

Yes.

Although it could be argued that it is a handy way to remove filters, in a backing chain at least.  (Thanks to bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)

*shrug*

Quoted 3 lines

On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.

I don’t think so, actually.  Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain?  Or a COR node, shouldn’t that go above the chain after streaming?

I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here.  The only reason to drop them is because that’s what I expect users to expect of the stream job.

Quoted 7 lines

Next, with your way, what about filters, inserted above base during stream job? They will be between above_base and base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.

Hm.  The documentation seems the same to me.  Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.

The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.

Quoted 6 lines

OK, so after all this text, maybe some more original problem searching. I think it the root of the problem is that the stream job takes a @base
parameter, but as of c624b015bf14fe01, it doesn’t really matter anymore. Maybe c624b015bf14fe01 should have introduced a new parameter for users to specify the bottom node instead of @base.

Yes, absolutely agree. If we do it now, would it be incompatible change or not?

Quoted 16 lines

Well, that would have made everything a parameter mess, but it would have saved us the trouble now.

In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it.  So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.

I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming.  (If you do graph modifications during the job, that’s on you, because since c624b015bf14fe01 we can’t keep the base frozen.)

Quoted 5 lines

Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change.

I also try to think about future introduction of stream filter. It shouldn't be a breaking change as well. But now I think it can be done the hard-way if needed: just work without a filter if filter-node-name is not given. but probably I care to much. Finally we can just drop the test-case from 030 or insert additional intermediate node into it..

Quoted 8 lines

 The problem I have with it is that
you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.

I think it would be nice if we could have something that remains truthful to the current documentation.

Then, we should freeze base node again, so just revert c624b015bf14fe01

I go and see it cover-letter of the series introduced c624b015bf14fe01:

  This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job and can change before the stream job is completed. We avoid that dependency by introducing the bottom node.

Hmm. Bad that we didn't add an iotest, but the series solved existing problem:
parallel stream and commit, as commit already has filter. Commit has filter with "backing" child, so, I assume that the case worked prior to introducing frozen chains, broken then by frozen chains and then fixed by c624b015bf14fe01. Still I don't know are there some real users of such parallel jobs..

Quoted 5 lines

 If just ignoring implicit
filters above @base would work, then I’d find that nice.  If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.

OK, at this point, I think, I'm OK with your patch in context of these series.

But I fill that something more should be done. Could we just revert c624b015bf14fe01 as "non corresponding to specification"?

Reviewed-by
#53 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

In reply to Vladimir Sementsov-Ogievskiy
Quoted 381 lines · Vladimir Sementsov-Ogievskiy
Quoted 131 lines · Max Reitz
Quoted 89 lines · Vladimir Sementsov-Ogievskiy
Quoted 88 lines · Max Reitz
Quoted 62 lines · Andrey Shinkevich
Quoted 50 lines · Max Reitz
Quoted 41 lines · Andrey Shinkevich
Quoted 40 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
     qapi/block-core.json |  4 +++
     block/stream.c       | 63
++++++++++++++++++++++++++++++++------------
     blockdev.c           |  4 ++-

     3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
      # On successful completion the image file is updated to drop the

backing file

     # and the BLOCK_JOB_COMPLETED event is emitted.
     #
+# In case @device is a filter node, block-stream modifies the 
first
non-filter
+# overlay node below it to point to base's backing node (or 
NULL if
@base was
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#

     # @job-id: identifier for the newly-created block job. If #          omitted, the device name will be used. (Since 2.7) #

block/stream.c +0 -0
index aa2e7af98e..b9c1141656 100644
@@ -31,7 +31,8 @@ enum {
Quoted 8 lines
Quoted 7 lines
block/stream.c +1 -1
        typedef struct StreamBlockJob {
          BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *base_overlay; /* COW overlay (stream from

this) */
+    BlockDriverState *above_base;   /* Node directly above the base */

Keeping the base_overlay is enough to complete the stream job.

Quoted 7 lines

Depends on the definition.  If we decide it isn’t enough, then it isn’t enough.

Quoted 1 line

The above_base may disappear during the job and we can't rely on it.

In this version of this series, it may not, because the chain is frozen.
    So the above_base cannot disappear.

Once we insert a filter above the top bs of the stream job, the parallel jobs in

Quoted 7 lines

the iotests #030 will fail with 'frozen link error'. It is because of the

independent parallel stream or commit jobs that insert/remove their filters

asynchroniously.

I’m not sure whether that’s a problem with this series specifically.

Quoted 14 lines
Quoted 7 lines

We can discuss whether we should allow it to disappear, but I think not.

The problem is, we need something to set as the backing file after streaming.  How do we figure out what that should be? My proposal is we
keep above_base and use its immediate child.

We can do the same with the base_overlay.

If the backing node turns out to be a filter, the proper backing child will

be set after the filter is removed. So, we shouldn't care.

Quoted 8 lines
Quoted 2 lines

And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?

It's automatically kept, if we use base_overlay->backing->bs as final backing node.

You mean, that they want it to be dropped?

Er, yes.  Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.

Quoted 16 lines

so, assuming the following:

top -(backing)-> manually-inserted-filter -(file)-> base

and user do stream with base=base, and expects filter to be removed by stream job?

Hmm, yes, such use-case is broken with our proposed way...

====

Let me now clarify the problem we'll have with your way.

When stream don't have any filter, we can easily imagine two parallel stream jobs:

Quoted 8 lines

top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base

stream1: top=top, base=mid2
stream2: top=mid2, base=NULL

final picture is obvious:

top (merged with mid1) -(backing)-> mid2 (merged with base)

Yes, and I don’t think this currently working case is broken by this series.

Quoted 1 line

But we want stream job has own filter, like mirror.

Quoted 7 lines

Which it does not have yet, right?  Which is why I was saying that I don’t think this is a problem with this series.  We could try to address it later.

Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?

I think, we can address it later, but it would be good to fit into one release cycle with these series, to not make incompatible behavior changes later.

Quoted 9 lines
Quoted 7 lines

So the picture becomes more complex.

Assume stream2 starts first.

top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base

stream2-filter would be on top of mid2, right?

Right. In my picture, "-(backing)->" means backing link. Hmm, most probably stream-filter is COR, which actually have file child. It doesn't matter here.

Quoted 9 lines
Quoted 8 lines

Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..

Quoted 4 lines

Note that the user first needs to pass “mid2” as the base to the stream job stream1.  Why don’t they just pass “stream2-filter”?  In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.

Hmm. I'm sure we already tried/discussed this.. But I don't remember the results. Seems, such logic should work as well. Andrey don't you remember? I think, the only difficulty is to update iotests 30 (in our stream-filter series) to use node-names, not filename.. Can we go this way?

Quoted 4 lines

I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.

Agree. I think stream-filter should be explicit, like recently introduced backup-top.

Quoted 6 lines

If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.

Quoted 8 lines
Quoted 3 lines

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming.  And the QAPI schema says that base-node is to become the backing file of the top node after streaming.

But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.

Quoted 3 lines

I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?

I don't, but I like the idea :)

Quoted 4 lines

Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.

And again, this is impossible even with your way. I have an idea:

What about making the whole thing explicit?

We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.

Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)). Benefit: user have guarantee of what would be final backing node.

Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.

But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..

Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying  exact node-names in it?..

Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.

What do you think of it? Could we break this scenario in one release without deprecation and don't care? Than I think my idea about base vs bottom-node arguments for stream job may be applied. Or what to do?

If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.

Quoted 16 lines
Quoted 10 lines

====

So, these are two mutually exclusive cases.. I vote for freezing up to base_overlay, and use backing(base_overlay) as final backing, because:

1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)

Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.

Quoted 5 lines

You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.

(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)

We can break existing user scenarios.. Seems unlikely. I don't know.

Quoted 61 lines
Quoted 7 lines

2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:

  - Stream job is not intended to remove filters, but to stream data. Filters between base_overlay and base don't contain any data and unrelated to stream process

Well, it is intended to remove nodes.  You can only remove data-bearing nodes by moving data around.  I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.

Quoted 4 lines

  - I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.

Yes.

Although it could be argued that it is a handy way to remove filters, in a backing chain at least.  (Thanks to bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)

*shrug*

Quoted 3 lines

On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.

I don’t think so, actually.  Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain?  Or a COR node, shouldn’t that go above the chain after streaming?

I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here.  The only reason to drop them is because that’s what I expect users to expect of the stream job.

Quoted 7 lines

Next, with your way, what about filters, inserted above base during stream job? They will be between above_base and base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.

Hm.  The documentation seems the same to me.  Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.

The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.

Quoted 6 lines

OK, so after all this text, maybe some more original problem searching. I think it the root of the problem is that the stream job takes a @base
parameter, but as of c624b015bf14fe01, it doesn’t really matter anymore. Maybe c624b015bf14fe01 should have introduced a new parameter for users to specify the bottom node instead of @base.

Yes, absolutely agree. If we do it now, would it be incompatible change or not?

Quoted 16 lines

Well, that would have made everything a parameter mess, but it would have saved us the trouble now.

In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it.  So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.

I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming.  (If you do graph modifications during the job, that’s on you, because since c624b015bf14fe01 we can’t keep the base frozen.)

Quoted 5 lines

Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change.

I also try to think about future introduction of stream filter. It shouldn't be a breaking change as well. But now I think it can be done the hard-way if needed: just work without a filter if filter-node-name is not given. but probably I care to much. Finally we can just drop the test-case from 030 or insert additional intermediate node into it..

Quoted 8 lines

 The problem I have with it is that
you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.

I think it would be nice if we could have something that remains truthful to the current documentation.

Then, we should freeze base node again, so just revert c624b015bf14fe01

I go and see it cover-letter of the series introduced c624b015bf14fe01:

  This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job and can change before the stream job is completed. We avoid that dependency by introducing the bottom node.

Hmm. Bad that we didn't add an iotest, but the series solved existing problem:
parallel stream and commit, as commit already has filter. Commit has filter with "backing" child, so, I assume that the case worked prior to introducing frozen chains, broken then by frozen chains and then fixed by c624b015bf14fe01. Still I don't know are there some real users of such parallel jobs..

Quoted 5 lines

 If just ignoring implicit
filters above @base would work, then I’d find that nice.  If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.

OK, at this point, I think, I'm OK with your patch in context of these series.

But I fill that something more should be done. Could we just revert c624b015bf14fe01 as "non corresponding to specification"?

author
#54 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

In reply to Vladimir Sementsov-Ogievskiy
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Quoted 1 line · Max Reitz

On 07.08.20 12:29, Vladimir Sementsov-Ogievskiy wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

Quoted 8 lines
Quoted 4 lines

I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming.  And the QAPI schema says that base-node is to become the backing file of the top node after streaming.

But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.

Yeah, but after c624b015bf, that’s just how it is. I thought the best would be an approach where if there are no graph changes during the job, you would indeed end up with @base as the backing file afterwards.

[...]

Quoted 8 lines
Quoted 3 lines

Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.

And again, this is impossible even with your way. I have an idea:

What about making the whole thing explicit?

Quoted 8 lines

We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.

Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)).
Benefit: user have guarantee of what would be final backing node.

Sounds very nice to me, but...

Quoted 7 lines

Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.

But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..

...yeah. Hm. What a pain, but right, we can just deprecate it.

Unfortunately, I don’t think there’s any way we could issue a warning (we’d want to deprecate using the @base node in something outside of the stream job, and we can’t really detect this case to issue a warning). So it would be a deprecation found only in the deprecation notes and the QAPI spec...

Quoted 6 lines

Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying  exact node-names in it?..

Quoted 8 lines

Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.

What do you think of it? Could we break this scenario in one release without deprecation and don't care?

I don’t know, but I’m afraid I don’t think we can.

Quoted 11 lines

Than I think my idea about base vs
bottom-node arguments for stream job may be applied. Or what to do?

If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.

If that works...

OK. So what I think we can do is first just take this patch as part of this series. Then, we add @bottom-node separately and deprecate @base not being frozen.

If it’s feasible to not add a stream filter node until the deprecation period is over, then that should work.

Max

#55 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 52 lines · Max Reitz
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Quoted 1 line · Max Reitz

On 07.08.20 12:29, Vladimir Sementsov-Ogievskiy wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..

Quoted 8 lines
Quoted 4 lines

I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming.  And the QAPI schema says that base-node is to become the backing file of the top node after streaming.

But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.

Yeah, but after c624b015bf, that’s just how it is. I thought the best would be an approach where if there are no graph changes during the job, you would indeed end up with @base as the backing file afterwards.

[...]

Quoted 8 lines
Quoted 3 lines

Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.

And again, this is impossible even with your way. I have an idea:

What about making the whole thing explicit?

Quoted 8 lines

We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.

Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)).
Benefit: user have guarantee of what would be final backing node.

Sounds very nice to me, but...

Quoted 7 lines

Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.

But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..

Quoted 7 lines

...yeah. Hm. What a pain, but right, we can just deprecate it.

Unfortunately, I don’t think there’s any way we could issue a warning (we’d want to deprecate using the @base node in something outside of the stream job, and we can’t really detect this case to issue a warning). So it would be a deprecation found only in the deprecation notes and the QAPI spec...

Hmm.. add "bool frozen_only_warn" field to BdrvChild and print warning instead of fail where c->frozen cause failure?

I can make a patch, if you don't think that its too much.

Quoted 13 lines
Quoted 13 lines

Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying  exact node-names in it?..

Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.

Quoted 4 lines
Quoted 2 lines

What do you think of it? Could we break this scenario in one release without deprecation and don't care?

I don’t know, but I’m afraid I don’t think we can.

Actually, we already done so once:

Freezing of base was introduced in 4.0, when c624b015bf comes in 4.1 (there were no real user bugs or feature requests, it was just my [bad] idea, related to the problem around our stream-filter and parallel jobs in iotest 30).
So, at least for 4.0 we had frozen base.. Were there any bugs? But now we live with not-frozen base for several releases..

Quoted 13 lines
Quoted 4 lines

Than I think my idea about base vs
bottom-node arguments for stream job may be applied. Or what to do?

Quoted 8 lines

If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.

Quoted 5 lines

If that works...

OK. So what I think we can do is first just take this patch as part of this series.

Yes, let's start with it.

Quoted 2 lines

Then, we add @bottom-node separately and deprecate @base
not being frozen.

I think we can just deprecate, and not add the bottom-node. If someone will come during deprecation period and say that he have such use-case, we'll add @bottom-node. Otherwise we'll just start to freeze base node again.

Quoted 6 lines

If it’s feasible to not add a stream filter node until the deprecation period is over, then that should work.

Max

Signature · 2 lines
Best regards,
Vladimir
#56 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

In reply to Max Reitz
Quoted 18 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
  qapi/block-core.json |  4 +++
  block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
  blockdev.c           |  4 ++-
  3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644

qapi/block-core.json line 2490

Quoted 5 lines
qapi/block-core.json +2 -0
@@ -2486,6 +2486,10 @@
  # On successful completion the image file is updated to drop the backing file
  # and the BLOCK_JOB_COMPLETED event is emitted.
  #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was

Forgot one thing. To me, it would be more understandable to read

"...to point to the base as backing node..." because it may be thought as a backing

node of the base.

Andrey

Quoted 5 lines
qapi/block-core.json +2 -0
+# not specified) instead of modifying @device itself.
+#
  # @job-id: identifier for the newly-created block job. If
  #          omitted, the device name will be used. (Since 2.7)
  #
author
#57 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 30 lines · Andrey Shinkevich
Quoted 20 lines · Max Reitz

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
  qapi/block-core.json |  4 +++
  block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c           |  4 ++-

  3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
   # On successful completion the image file is updated to drop the

backing file

Quoted 6 lines
  # and the BLOCK_JOB_COMPLETED event is emitted.
  #
+# In case @device is a filter node, block-stream modifies the first
non-filter
+# overlay node below it to point to base's backing node (or NULL if
@base was

Forgot one thing. To me, it would be more understandable to read

Quoted 4 lines

"...to point to the base as backing node..." because it may be thought as a backing

node of the base.

This doesn’t sound like it’s about understandability; “point to the base as backing node” and “point to base’s backing node [as backing node]” are semantically different.

Was my phrasing just wrong? @base should be the backing node, so yours seems correct.

Max

#58 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

In reply to Max Reitz

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 18 lines

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 +++
 block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
 blockdev.c           |  4 ++-
 3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +0 -0
index b20332e592..df87855429 100644

qapi/block-core.json line 2491

Quoted 6 lines
qapi/block-core.json +3 -0
@@ -2486,6 +2486,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.

Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).

Should we also document what using a filter node for @base means?

The code changes look good.

Kevin

author
#59 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 14 lines · Kevin Wolf

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 13 lines

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 +++
 block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
 blockdev.c           |  4 ++-
 3 files changed, 53 insertions(+), 18 deletions(-)
Quoted 15 lines

qapi/block-core.json line 2491

Quoted 11 lines
qapi/block-core.json +3 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.

Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).

Oh, yes.

(I thought I had noticed that already at some point and fixed it locally... But apparently not.)

Quoted 1 line

Should we also document what using a filter node for @base means?

Hm. What does it mean? I think the more interesting case is what it means if above_base is a filter, right?

Maybe we can put in somewhere in the “If a base file is specified then sectors are not copied from that base file and its backing chain.” But the more I think about it, the less I know what we could add to it. What happens if there are filters above @base is that their data isn’t copied, because that’s exactly the data in @base.

Max

#60 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Am 19.08.2020 um 16:47 hat Max Reitz geschrieben:

Quoted 38 lines · Kevin Wolf
Quoted 14 lines

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 13 lines

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 +++
 block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
 blockdev.c           |  4 ++-
 3 files changed, 53 insertions(+), 18 deletions(-)
Quoted 15 lines

qapi/block-core.json line 2491

Quoted 11 lines
qapi/block-core.json +3 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.

Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).

Oh, yes.

(I thought I had noticed that already at some point and fixed it locally... But apparently not.)

Quoted 1 line

Should we also document what using a filter node for @base means?

Quoted 8 lines

Hm. What does it mean? I think the more interesting case is what it means if above_base is a filter, right?

Maybe we can put in somewhere in the “If a base file is specified then sectors are not copied from that base file and its backing chain.” But the more I think about it, the less I know what we could add to it. What happens if there are filters above @base is that their data isn’t copied, because that’s exactly the data in @base.

The interesting part is probably the graph reconfiguration at the end of the job. Which is actually already documented:

# When streaming completes the image file will have the base

# file as its backing file.

Of course, this is not entirely correct any more (because the base may have changed).

If @base is a filter, what backing file path do we write into the top layer? A json: filename including the filter? Is this worth mentioning or do you consider it obvious?

Kevin

author
#61 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 51 lines · Kevin Wolf

Am 19.08.2020 um 16:47 hat Max Reitz geschrieben:

Quoted 38 lines · Kevin Wolf
Quoted 14 lines

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 13 lines

Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 +++
 block/stream.c       | 63 ++++++++++++++++++++++++++++++++------------
 blockdev.c           |  4 ++-
 3 files changed, 53 insertions(+), 18 deletions(-)
Quoted 15 lines

qapi/block-core.json line 2491

Quoted 11 lines
qapi/block-core.json +3 -0
index b20332e592..df87855429 100644
@@ -2486,6 +2486,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.

Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).

Oh, yes.

(I thought I had noticed that already at some point and fixed it locally... But apparently not.)

Quoted 1 line

Should we also document what using a filter node for @base means?

Quoted 8 lines

Hm. What does it mean? I think the more interesting case is what it means if above_base is a filter, right?

Maybe we can put in somewhere in the “If a base file is specified then sectors are not copied from that base file and its backing chain.” But the more I think about it, the less I know what we could add to it. What happens if there are filters above @base is that their data isn’t copied, because that’s exactly the data in @base.

The interesting part is probably the graph reconfiguration at the end of the job. Which is actually already documented:

Quoted 8 lines

# When streaming completes the image file will have the base

# file as its backing file.

Of course, this is not entirely correct any more (because the base may have changed).

If @base is a filter, what backing file path do we write into the top layer? A json: filename including the filter?

Yes.

Or, actually. Now that I read the code... It takes @base’s filename before the stream job and then uses that. So if @base has changed during the job, then it still uses the old filename.

But that’s not really due to this series.

Quoted 2 lines

Is this worth mentioning
or do you consider it obvious?

Hm. I consider it obvious, yes. @base becomes @top’s backing file (at least without any graph changes while the job is running), so naturally what’s written into the image header is @base’s filename – which is a json:{} filename.

On second thought, @backing-file’s description mysteriously states that “QEMU will automatically determine the backing file string to use”. Which makes sense because it would clearly not make sense to describe what’s actually happening, which is to use @base’s filename at job start regardless of whether it’s still there at the end of the job.

So I suppose I have the choice of either documenting exactly what’s happening, even though it doesn’t make much sense, or just not, keeping it mysterious.

So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)

The more I think about it, the more I think there are problems beyond the scope of this series here. :/

Max

author
#62 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

On 20.08.20 10:31, Max Reitz wrote:

[...]

Quoted 4 lines

So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)

(I tried triggering that, but, oh, it’s strdup’ed() in stream_start(). I’m a bit daft.)

#63 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 4 lines · Max Reitz

On 20.08.20 10:31, Max Reitz wrote:

[...]

Quoted 8 lines
Quoted 4 lines

So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)

(I tried triggering that, but, oh, it’s strdup’ed() in stream_start(). I’m a bit daft.)

If it's broken anyway, probably we can just revert c624b015bf and start to freeze base again?

Signature · 2 lines
Best regards,
Vladimir
author
#64 lore raw

Re: [PATCH v7 14/47] stream: Deal with filters

Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Max Reitz

On 20.08.20 10:31, Max Reitz wrote:

[...]

Quoted 4 lines

So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore?  (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)

Quoted 7 lines
Quoted 3 lines

(I tried triggering that, but, oh, it’s strdup’ed() in stream_start(). I’m a bit daft.)

If it's broken anyway, probably we can just revert c624b015bf and start to freeze base again?

Well, it’s only broken if you care about the backing filename string that’s written to @top. So it isn’t broken altogether.

Though, well. If we all agree to just revert it and maybe add a @bottom parameter instead, then I suppose we could do it.

(Maybe in a follow-up, though.)

Max

author
#65 lore raw

PATCH v7 15/47 block: Use CAFs when working with backing chains

In reply to Max Reitz

Use child access functions when iterating through backing chains so filters do not break the chain.

In addition, bdrv_find_overlay() will now always return the actual overlay; that is, it will never return a filter node but only one with a COW backing file (there may be filter nodes between that node and @bs).

---
 block.c | 41 +++++++++++++++++++++++++++++------------
 1 file changed, 29 insertions(+), 12 deletions(-)
block.c +29 -12
index a44af9c3c1..712230ef5c 100644
@@ -4724,7 +4724,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 }
 
 /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.
  *
  * active is the current topmost image.
  *
@@ -4736,11 +4737,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs)
 {
-    while (active && bs != backing_bs(active)) {
-        active = backing_bs(active);
+    bs = bdrv_skip_filters(bs);
+    active = bdrv_skip_filters(active);
+
+    while (active) {
+        BlockDriverState *next = bdrv_backing_chain_next(active);
+        if (bs == next) {
+            return active;
+        }
+        active = next;
     }
 
-    return active;
+    return NULL;
 }
 
 /* Given a BDS, searches for the base layer. */
@@ -4892,9 +4900,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
      * other intermediate nodes have been dropped.
      * If 'top' is an implicit node (e.g. "commit_top") we should skip
      * it because no one inherits from it. We use explicit_top for that. */
-    while (explicit_top && explicit_top->implicit) {
-        explicit_top = backing_bs(explicit_top);
-    }
+    explicit_top = bdrv_skip_implicit_filters(explicit_top);
     update_inherits_from = bdrv_inherits_from_recursive(base, explicit_top);
 
     /* success - we can delete the intermediate states, and link top->base */
@@ -5351,7 +5357,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,
 bool bdrv_chain_contains(BlockDriverState *top, BlockDriverState *base)
 {
     while (top && top != base) {
-        top = backing_bs(top);
+        top = bdrv_filter_or_cow_bs(top);
     }
 
     return top != NULL;
@@ -5607,6 +5613,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
     int is_protocol = 0;
     BlockDriverState *curr_bs = NULL;
     BlockDriverState *retval = NULL;
+    BlockDriverState *bs_below;
 
     if (!bs || !bs->drv || !backing_file) {
         return NULL;
@@ -5617,7 +5624,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     is_protocol = path_has_protocol(backing_file);
 
-    for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) {
+    /*
+     * Being largely a legacy function, skip any filters here
+     * (because filters do not have normal filenames, so they cannot
+     * match anyway; and allowing json:{} filenames is a bit out of
+     * scope).
+     */
+    for (curr_bs = bdrv_skip_filters(bs);
+         bdrv_cow_child(curr_bs) != NULL;
+         curr_bs = bs_below)
+    {
+        bs_below = bdrv_backing_chain_next(curr_bs);
 
         /* If either of the filename paths is actually a protocol, then
          * compare unmodified paths; otherwise make paths relative */
@@ -5625,7 +5642,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
             char *backing_file_full_ret;
 
             if (strcmp(backing_file, curr_bs->backing_file) == 0) {
-                retval = curr_bs->backing->bs;
+                retval = bs_below;
                 break;
             }
             /* Also check against the full backing filename for the image */
@@ -5635,7 +5652,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
                 bool equal = strcmp(backing_file, backing_file_full_ret) == 0;
                 g_free(backing_file_full_ret);
                 if (equal) {
-                    retval = curr_bs->backing->bs;
+                    retval = bs_below;
                     break;
                 }
             }
@@ -5661,7 +5678,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
             g_free(filename_tmp);
 
             if (strcmp(backing_file_full, filename_full) == 0) {
-                retval = curr_bs->backing->bs;
+                retval = bs_below;
                 break;
             }
         }
Signature · 1 line
2.26.2
Reviewed-by on v7 15/47
#66 lore raw

Re: [PATCH v7 15/47] block: Use CAFs when working with backing chains

Quoted 17 lines · Max Reitz

Use child access functions when iterating through backing chains so filters do not break the chain.

In addition, bdrv_find_overlay() will now always return the actual overlay; that is, it will never return a filter node but only one with a COW backing file (there may be filter nodes between that node and @bs).

---
  block.c | 41 +++++++++++++++++++++++++++++------------
  1 file changed, 29 insertions(+), 12 deletions(-)
block.c +0 -0
index a44af9c3c1..712230ef5c 100644

block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4728

Quoted 6 lines
block.c +2 -1
@@ -4724,7 +4724,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
  }
 
  /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.

One can optionally say "Finds the first non-filter parent of bs in the chain".

Quoted 1 line

...

author
#67 lore raw

PATCH v7 16/47 block: Use bdrv_cow_child() in bdrv_co_truncate()

In reply to Max Reitz

The condition modified here is not about potentially filtered children, but only about COW sources (i.e. traditional backing files).

---
 block/io.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
block/io.c +4 -3
index dc9891d6ce..097a3861d8 100644
@@ -3308,7 +3308,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
                                   Error **errp)
 {
     BlockDriverState *bs = child->bs;
-    BdrvChild *filtered;
+    BdrvChild *filtered, *backing;
     BlockDriver *drv = bs->drv;
     BdrvTrackedRequest req;
     int64_t old_size, new_bytes;
@@ -3361,6 +3361,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
     }
 
     filtered = bdrv_filter_child(bs);
+    backing = bdrv_cow_child(bs);
 
     /*
      * If the image has a backing file that is large enough that it would
@@ -3372,10 +3373,10 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
      * backing file, taking care of keeping things consistent with that backing
      * file is the user's responsibility.
      */
-    if (new_bytes && bs->backing) {
+    if (new_bytes && backing) {
         int64_t backing_len;
 
-        backing_len = bdrv_getlength(backing_bs(bs));
+        backing_len = bdrv_getlength(backing->bs);
         if (backing_len < 0) {
             ret = backing_len;
             error_setg_errno(errp, -ret, "Could not get backing file size");
Signature · 1 line
2.26.2
Reviewed-by on v7 16/47
#68 lore raw

Re: [PATCH v7 16/47] block: Use bdrv_cow_child() in bdrv_co_truncate()

Quoted 32 lines · Max Reitz

The condition modified here is not about potentially filtered children, but only about COW sources (i.e. traditional backing files).

---
  block/io.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
block/io.c +2 -1
index dc9891d6ce..097a3861d8 100644
@@ -3308,7 +3308,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
                                    Error **errp)
  {
      BlockDriverState *bs = child->bs;
-    BdrvChild *filtered;
+    BdrvChild *filtered, *backing;
      BlockDriver *drv = bs->drv;
      BdrvTrackedRequest req;
      int64_t old_size, new_bytes;
@@ -3361,6 +3361,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
      }
 
      filtered = bdrv_filter_child(bs);
+    backing = bdrv_cow_child(bs);
 
      /*
       * If the image has a backing file that is large enough that it would
@@ -3372,10 +3373,10 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,
       * backing file, taking care of keeping things consistent with that backing
       * file is the user's responsibility.

block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3382

Quoted 10 lines
block/io.c +2 -2
       */
-    if (new_bytes && bs->backing) {
+    if (new_bytes && backing) {
          int64_t backing_len;
 
-        backing_len = bdrv_getlength(backing_bs(bs));
+        backing_len = bdrv_getlength(backing->bs);
          if (backing_len < 0) {
              ret = backing_len;
              error_setg_errno(errp, -ret, "Could not get backing file size");
author
#69 lore raw

PATCH v7 17/47 block: Re-evaluate backing file handling in reopen

In reply to Max Reitz

Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).

---
 block.c | 46 ++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 38 insertions(+), 8 deletions(-)
block.c +38 -8
index 712230ef5c..8131d0b5eb 100644
@@ -4026,26 +4026,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state,
         }
     }
 
+    /*
+     * Ensure that @bs can really handle backing files, because we are
+     * about to give it one (or swap the existing one)
+     */
+    if (bs->drv->is_filter) {
+        /* Filters always have a file or a backing child */
+        if (!bs->backing) {
+            error_setg(errp, "'%s' is a %s filter node that does not support a "
+                       "backing child", bs->node_name, bs->drv->format_name);
+            return -EINVAL;
+        }
+    } else if (!bs->drv->supports_backing) {
+        error_setg(errp, "Driver '%s' of node '%s' does not support backing "
+                   "files", bs->drv->format_name, bs->node_name);
+        return -EINVAL;
+    }
+
     /*
      * Find the "actual" backing file by skipping all links that point
      * to an implicit node, if any (e.g. a commit filter node).
+     * We cannot use any of the bdrv_skip_*() functions here because
+     * those return the first explicit node, while we are looking for
+     * its overlay here.
      */
     overlay_bs = bs;
-    while (backing_bs(overlay_bs) && backing_bs(overlay_bs)->implicit) {
-        overlay_bs = backing_bs(overlay_bs);
+    while (bdrv_filter_or_cow_bs(overlay_bs) &&
+           bdrv_filter_or_cow_bs(overlay_bs)->implicit)
+    {
+        overlay_bs = bdrv_filter_or_cow_bs(overlay_bs);
     }
 
     /* If we want to replace the backing file we need some extra checks */
-    if (new_backing_bs != backing_bs(overlay_bs)) {
+    if (new_backing_bs != bdrv_filter_or_cow_bs(overlay_bs)) {
         /* Check for implicit nodes between bs and its backing file */
         if (bs != overlay_bs) {
             error_setg(errp, "Cannot change backing link if '%s' has "
                        "an implicit backing file", bs->node_name);
             return -EPERM;
         }
-        /* Check if the backing link that we want to replace is frozen */
-        if (bdrv_is_backing_chain_frozen(overlay_bs, backing_bs(overlay_bs),
-                                         errp)) {
+        /*
+         * Check if the backing link that we want to replace is frozen.
+         * Note that
+         * bdrv_filter_or_cow_child(overlay_bs) == overlay_bs->backing,
+         * because we know that overlay_bs == bs, and that @bs
+         * either is a filter that uses ->backing or a COW format BDS
+         * with bs->drv->supports_backing == true.
+         */
+        if (bdrv_is_backing_chain_frozen(overlay_bs,
+                                         child_bs(overlay_bs->backing), errp))
+        {
             return -EPERM;
         }
         reopen_state->replace_backing_bs = true;
@@ -4196,7 +4226,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
      * its metadata. Otherwise the 'backing' option can be omitted.
      */
     if (drv->supports_backing && reopen_state->backing_missing &&
-        (backing_bs(reopen_state->bs) || reopen_state->bs->backing_file[0])) {
+        (reopen_state->bs->backing || reopen_state->bs->backing_file[0])) {
         error_setg(errp, "backing is missing for '%s'",
                    reopen_state->bs->node_name);
         ret = -EINVAL;
@@ -4337,7 +4367,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)
      * from bdrv_set_backing_hd()) has the new values.
      */
     if (reopen_state->replace_backing_bs) {
-        BlockDriverState *old_backing_bs = backing_bs(bs);
+        BlockDriverState *old_backing_bs = child_bs(bs->backing);
         assert(!old_backing_bs || !old_backing_bs->implicit);
         /* Abort the permission update on the backing bs we're detaching */
         if (old_backing_bs) {
Signature · 1 line
2.26.2
#70 lore raw

Re: [PATCH v7 17/47] block: Re-evaluate backing file handling in reopen

Quoted 42 lines · Max Reitz

Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).

---
  block.c | 46 ++++++++++++++++++++++++++++++++++++++--------
  1 file changed, 38 insertions(+), 8 deletions(-)
block.c +19 -0
index 712230ef5c..8131d0b5eb 100644
@@ -4026,26 +4026,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state,
          }
      }
 
+    /*
+     * Ensure that @bs can really handle backing files, because we are
+     * about to give it one (or swap the existing one)
+     */
+    if (bs->drv->is_filter) {
+        /* Filters always have a file or a backing child */
+        if (!bs->backing) {
+            error_setg(errp, "'%s' is a %s filter node that does not support a "
+                       "backing child", bs->node_name, bs->drv->format_name);
+            return -EINVAL;
+        }
+    } else if (!bs->drv->supports_backing) {
+        error_setg(errp, "Driver '%s' of node '%s' does not support backing "
+                   "files", bs->drv->format_name, bs->node_name);
+        return -EINVAL;
+    }
+
      /*
       * Find the "actual" backing file by skipping all links that point
       * to an implicit node, if any (e.g. a commit filter node).
+     * We cannot use any of the bdrv_skip_*() functions here because
+     * those return the first explicit node, while we are looking for

block.c static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, · line 4058

Quoted 10 lines
block.c +5 -2
+     * its overlay here.
       */
      overlay_bs = bs;
-    while (backing_bs(overlay_bs) && backing_bs(overlay_bs)->implicit) {
-        overlay_bs = backing_bs(overlay_bs);
+    while (bdrv_filter_or_cow_bs(overlay_bs) &&
+           bdrv_filter_or_cow_bs(overlay_bs)->implicit)
+    {
+        overlay_bs = bdrv_filter_or_cow_bs(overlay_bs);
      }

I believe that little optimization would work properly:

for (BlockDriverState *below_bs = bdrv_filter_or_cow_bs(overlay_bs); below_bs && below_bs->implicit;
       below_bs = bdrv_filter_or_cow_bs(overlay_bs)) { overlay_bs = below_bs;
}

Quoted 13 lines
block.c +1 -4
      /* If we want to replace the backing file we need some extra checks */
-    if (new_backing_bs != backing_bs(overlay_bs)) {
+    if (new_backing_bs != bdrv_filter_or_cow_bs(overlay_bs)) {
          /* Check for implicit nodes between bs and its backing file */
          if (bs != overlay_bs) {
              error_setg(errp, "Cannot change backing link if '%s' has "
                         "an implicit backing file", bs->node_name);
              return -EPERM;
          }
-        /* Check if the backing link that we want to replace is frozen */
-        if (bdrv_is_backing_chain_frozen(overlay_bs, backing_bs(overlay_bs),
-                                         errp)) {

block.c static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, · line 4077

Quoted 10 lines
block.c +10 -0
+        /*
+         * Check if the backing link that we want to replace is frozen.
+         * Note that
+         * bdrv_filter_or_cow_child(overlay_bs) == overlay_bs->backing,
+         * because we know that overlay_bs == bs, and that @bs
+         * either is a filter that uses ->backing or a COW format BDS
+         * with bs->drv->supports_backing == true.
+         */
+        if (bdrv_is_backing_chain_frozen(overlay_bs,
+                                         child_bs(overlay_bs->backing), errp))

What would be wrong with bdrv_filter_or_cow_bs(overlay_bs) here?

block.c int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, · line 4199

Quoted 9 lines
block.c +1 -1
+        {
              return -EPERM;
          }
          reopen_state->replace_backing_bs = true;
@@ -4196,7 +4226,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
       * its metadata. Otherwise the 'backing' option can be omitted.
       */
      if (drv->supports_backing && reopen_state->backing_missing &&
-        (backing_bs(reopen_state->bs) || reopen_state->bs->backing_file[0])) {

= BlockDriverState*

block.c int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, · line 4229

Quoted 1 line
block.c +1 -0
+        (reopen_state->bs->backing || reopen_state->bs->backing_file[0])) {

= BdrvChild*

Are we OK with that?

Quoted 12 lines
block.c +1 -1
          error_setg(errp, "backing is missing for '%s'",
                     reopen_state->bs->node_name);
          ret = -EINVAL;
@@ -4337,7 +4367,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)
       * from bdrv_set_backing_hd()) has the new values.
       */
      if (reopen_state->replace_backing_bs) {
-        BlockDriverState *old_backing_bs = backing_bs(bs);
+        BlockDriverState *old_backing_bs = child_bs(bs->backing);
          assert(!old_backing_bs || !old_backing_bs->implicit);
          /* Abort the permission update on the backing bs we're detaching */
          if (old_backing_bs) {
author
#71 lore raw

Re: [PATCH v7 17/47] block: Re-evaluate backing file handling in reopen

Quoted 60 lines · Andrey Shinkevich
Quoted 47 lines · Max Reitz

Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things).  Adding filter support here is a bit more difficult than just using the child access functions.  In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).

---
  block.c | 46 ++++++++++++++++++++++++++++++++++++++--------

  1 file changed, 38 insertions(+), 8 deletions(-)
block.c +0 -0
index 712230ef5c..8131d0b5eb 100644
@@ -4026,26 +4026,56 @@ static int
 bdrv_reopen_parse_backing(BDRVReopenState *reopen_state,
           }
       }

  +    /*
+     * Ensure that @bs can really handle backing files, because we are +     * about to give it one (or swap the existing one)
+     */
+    if (bs->drv->is_filter) {
+        /* Filters always have a file or a backing child */ +        if (!bs->backing) {
+            error_setg(errp, "'%s' is a %s filter node that does not support a "
+                       "backing child", bs->node_name, bs->drv->format_name);
+            return -EINVAL;
+        }
+    } else if (!bs->drv->supports_backing) {
+        error_setg(errp, "Driver '%s' of node '%s' does not support backing "
+                   "files", bs->drv->format_name, bs->node_name); +        return -EINVAL;
+    }
+
      /*
       * Find the "actual" backing file by skipping all links that point
       * to an implicit node, if any (e.g. a commit filter node). +     * We cannot use any of the bdrv_skip_*() functions here because +     * those return the first explicit node, while we are looking for +     * its overlay here.
       */

Quoted 8 lines

      overlay_bs = bs;
-    while (backing_bs(overlay_bs) && backing_bs(overlay_bs)->implicit) { -        overlay_bs = backing_bs(overlay_bs);
+    while (bdrv_filter_or_cow_bs(overlay_bs) &&
+           bdrv_filter_or_cow_bs(overlay_bs)->implicit)
+    {
+        overlay_bs = bdrv_filter_or_cow_bs(overlay_bs);
      }

I believe that little optimization would work properly:

Quoted 5 lines

for (BlockDriverState *below_bs = bdrv_filter_or_cow_bs(overlay_bs); below_bs && below_bs->implicit;
       below_bs = bdrv_filter_or_cow_bs(overlay_bs)) { overlay_bs = below_bs;
}

Looks good, thanks.

Quoted 26 lines
Quoted 18 lines

        /* If we want to replace the backing file we need some extra checks */
-    if (new_backing_bs != backing_bs(overlay_bs)) {
+    if (new_backing_bs != bdrv_filter_or_cow_bs(overlay_bs)) { /* Check for implicit nodes between bs and its backing file */ if (bs != overlay_bs) {
              error_setg(errp, "Cannot change backing link if '%s' has " "an implicit backing file", bs->node_name); return -EPERM;
          }
-        /* Check if the backing link that we want to replace is frozen */
-        if (bdrv_is_backing_chain_frozen(overlay_bs, backing_bs(overlay_bs),
-                                         errp)) {
+        /*
+         * Check if the backing link that we want to replace is frozen. +         * Note that

Quoted 8 lines

+         * bdrv_filter_or_cow_child(overlay_bs) == overlay_bs->backing, +         * because we know that overlay_bs == bs, and that @bs +         * either is a filter that uses ->backing or a COW format BDS +         * with bs->drv->supports_backing == true.
+         */
+        if (bdrv_is_backing_chain_frozen(overlay_bs,
+
child_bs(overlay_bs->backing), errp))

Quoted 1 line

What would be wrong with bdrv_filter_or_cow_bs(overlay_bs) here?

As the comment says, it’s the same thing.

I prefer ->backing here, because this function is about reopening the ->backing child.

Quoted 12 lines
Quoted 11 lines

+        {
              return -EPERM;
          } reopen_state->replace_backing_bs = true;

block.c +0 -0
@@ -4196,7 +4226,7 @@ int bdrv_reopen_prepare(BDRVReopenState
 *reopen_state, BlockReopenQueue *queue,
        * its metadata. Otherwise the 'backing' option can be omitted.
        */

      if (drv->supports_backing && reopen_state->backing_missing && -        (backing_bs(reopen_state->bs) || reopen_state->bs->backing_file[0])) {

= BlockDriverState*

Quoted 6 lines
Quoted 2 lines

+        (reopen_state->bs->backing ||
reopen_state->bs->backing_file[0])) {

= BdrvChild*

Are we OK with that?

Sure, the question is whether it’s non-NULL, and BdrvChild.bs is always non-NULL.

Max

author
#72 lore raw

PATCH v7 18/47 block: Flush all children in generic code

In reply to Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children that might have been written to (judging from the permissions taken on them).

This is a bug fix for qcow2 images with an external data file, as they so far did not flush that data_file node.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
 block/io.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +17 -6
index 097a3861d8..c2af7711d6 100644
@@ -2769,6 +2769,8 @@ static int coroutine_fn bdrv_flush_co_entry(void *opaque)
 
 int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
     int current_gen;
     int ret = 0;
 
@@ -2798,7 +2800,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     }
 
     /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
     if (bs->drv->bdrv_co_flush_to_os) {
         ret = bs->drv->bdrv_co_flush_to_os(bs);
         if (ret < 0) {
@@ -2808,15 +2810,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
     /* But don't actually force it to the disk with cache=unsafe */
     if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
     }
 
     /* Check if we really need to flush anything */
     if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
     }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
     if (!bs->drv) {
         /* bs->drv->bdrv_co_flush() might have ejected the BDS
          * (even in case of apparent success) */
@@ -2860,8 +2862,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
      * in the case of cache=unsafe, so there are no useless flushes.
      */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0;
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->perm & (BLK_PERM_WRITE | BLK_PERM_WRITE_UNCHANGED)) {
+            int this_child_ret = bdrv_co_flush(child->bs);
+            if (!ret) {
+                ret = this_child_ret;
+            }
+        }
+    }
+
 out:
     /* Notify any pending flushes that we have completed */
     if (ret == 0) {
Signature · 1 line
2.26.2
Reviewed-by on v7 18/47
#73 lore raw

Re: [PATCH v7 18/47] block: Flush all children in generic code

Quoted 16 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children that might have been written to (judging from the permissions taken on them).

This is a bug fix for qcow2 images with an external data file, as they so far did not flush that data_file node.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)

...

Quoted 10 lines

@@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)

      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
block/io.c +4 -2
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0;
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->perm & (BLK_PERM_WRITE | BLK_PERM_WRITE_UNCHANGED)) {

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2878

Quoted 10 lines
block/io.c +7 -0
+            int this_child_ret = bdrv_co_flush(child->bs);
+            if (!ret) {
+                ret = this_child_ret;
+            }
+        }
+    }
+
  out:
      /* Notify any pending flushes that we have completed */
      if (ret == 0) {

I have not tested if the running application do reaches the flush_parent point to flush the data_file but with the code it looks OK.

author
#74 lore raw

PATCH v7 19/47 vmdk: Drop vmdk_co_flush()

In reply to Max Reitz

Before HEAD^, we needed this because bdrv_co_flush() by itself would only flush bs->file. With HEAD^, bdrv_co_flush() will flush all children on which a WRITE or WRITE_UNCHANGED permission has been taken. Thus, vmdk no longer needs to do it itself.

---
 block/vmdk.c | 16 ----------------
 1 file changed, 16 deletions(-)
block/vmdk.c +0 -16
index 62da465126..a23890e6ec 100644
@@ -2802,21 +2802,6 @@ static void vmdk_close(BlockDriverState *bs)
     error_free(s->migration_blocker);
 }
 
-static coroutine_fn int vmdk_co_flush(BlockDriverState *bs)
-{
-    BDRVVmdkState *s = bs->opaque;
-    int i, err;
-    int ret = 0;
-
-    for (i = 0; i < s->num_extents; i++) {
-        err = bdrv_co_flush(s->extents[i].file->bs);
-        if (err < 0) {
-            ret = err;
-        }
-    }
-    return ret;
-}
-
 static int64_t vmdk_get_allocated_file_size(BlockDriverState *bs)
 {
     int i;
@@ -3075,7 +3060,6 @@ static BlockDriver bdrv_vmdk = {
     .bdrv_close                   = vmdk_close,
     .bdrv_co_create_opts          = vmdk_co_create_opts,
     .bdrv_co_create               = vmdk_co_create,
-    .bdrv_co_flush_to_disk        = vmdk_co_flush,
     .bdrv_co_block_status         = vmdk_co_block_status,
     .bdrv_get_allocated_file_size = vmdk_get_allocated_file_size,
     .bdrv_has_zero_init           = vmdk_has_zero_init,
Signature · 1 line
2.26.2
#75 lore raw

Re: [PATCH v7 19/47] vmdk: Drop vmdk_co_flush()

Quoted 37 lines · Max Reitz

Before HEAD^, we needed this because bdrv_co_flush() by itself would only flush bs->file. With HEAD^, bdrv_co_flush() will flush all children on which a WRITE or WRITE_UNCHANGED permission has been taken. Thus, vmdk no longer needs to do it itself.

---
  block/vmdk.c | 16 ----------------
  1 file changed, 16 deletions(-)
block/vmdk.c +0 -15
index 62da465126..a23890e6ec 100644
@@ -2802,21 +2802,6 @@ static void vmdk_close(BlockDriverState *bs)
      error_free(s->migration_blocker);
  }
 
-static coroutine_fn int vmdk_co_flush(BlockDriverState *bs)
-{
-    BDRVVmdkState *s = bs->opaque;
-    int i, err;
-    int ret = 0;
-
-    for (i = 0; i < s->num_extents; i++) {
-        err = bdrv_co_flush(s->extents[i].file->bs);
-        if (err < 0) {
-            ret = err;
-        }
-    }
-    return ret;
-}
-
  static int64_t vmdk_get_allocated_file_size(BlockDriverState *bs)
  {
      int i;

block/vmdk.c static BlockDriver bdrv_vmdk = { · line 3078

Quoted 4 lines
block/vmdk.c +0 -1
@@ -3075,7 +3060,6 @@ static BlockDriver bdrv_vmdk = {
      .bdrv_close                   = vmdk_close,
      .bdrv_co_create_opts          = vmdk_co_create_opts,
      .bdrv_co_create               = vmdk_co_create,
-    .bdrv_co_flush_to_disk        = vmdk_co_flush,

After HEAD^ applied, wouldn't we get an endless recursion in bdrv_co_flush() if the HEAD (this patch) had not been merged into HEAD^?

Andrey

Quoted 3 lines
block/vmdk.c +0 -0
      .bdrv_co_block_status         = vmdk_co_block_status,
      .bdrv_get_allocated_file_size = vmdk_get_allocated_file_size,
      .bdrv_has_zero_init           = vmdk_has_zero_init,
author
#76 lore raw

Re: [PATCH v7 19/47] vmdk: Drop vmdk_co_flush()

Quoted 37 lines · Andrey Shinkevich
Quoted 36 lines · Max Reitz

Before HEAD^, we needed this because bdrv_co_flush() by itself would only flush bs->file.  With HEAD^, bdrv_co_flush() will flush all children on which a WRITE or WRITE_UNCHANGED permission has been taken. Thus, vmdk no longer needs to do it itself.

---
  block/vmdk.c | 16 ----------------

  1 file changed, 16 deletions(-)
block/vmdk.c +0 -14
index 62da465126..a23890e6ec 100644
@@ -2802,21 +2802,6 @@ static void vmdk_close(BlockDriverState *bs)
       error_free(s->migration_blocker);
   }
   -static coroutine_fn int vmdk_co_flush(BlockDriverState *bs)
-{
-    BDRVVmdkState *s = bs->opaque;
-    int i, err;
-    int ret = 0;
-
-    for (i = 0; i < s->num_extents; i++) {
-        err = bdrv_co_flush(s->extents[i].file->bs);
-        if (err < 0) {
-            ret = err;
-        }
-    }
-    return ret;
-}
-
   static int64_t vmdk_get_allocated_file_size(BlockDriverState *bs)
   {
       int i;
@@ -3075,7 +3060,6 @@ static BlockDriver bdrv_vmdk = {
Quoted 8 lines

block/vmdk.c static BlockDriver bdrv_vmdk = { · line 3078

Quoted 4 lines
block/vmdk.c +0 -1
       .bdrv_close                   = vmdk_close,
       .bdrv_co_create_opts          = vmdk_co_create_opts,
       .bdrv_co_create               = vmdk_co_create,
-    .bdrv_co_flush_to_disk        = vmdk_co_flush,

After HEAD^ applied, wouldn't we get an endless recursion in bdrv_co_flush() if the HEAD (this patch) had not been merged into HEAD^?

Hm, how so? HEAD^ just flushes all children, just like vmdk_co_flush() does. So it seems to me just like double the work. (Which is unfortunate but shouldn’t be a problem.)

Max

author
#77 lore raw

PATCH v7 20/47 block: Iterate over children in refresh_limits

In reply to Max Reitz

Instead of looking at just bs->file and bs->backing, we should look at all children that could end up receiving forwarded requests.

---
 block/io.c | 32 ++++++++++++++++----------------
 1 file changed, 16 insertions(+), 16 deletions(-)
block/io.c +16 -16
index c2af7711d6..37057f13e0 100644
@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)
 void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BdrvChild *c;
+    bool have_limits;
     Error *local_err = NULL;
 
     memset(&bs->bl, 0, sizeof(bs->bl));
@@ -149,14 +151,21 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
                                 drv->bdrv_co_preadv_part) ? 1 : 512;
 
     /* Take some limits from the children as a default */
-    if (bs->file) {
-        bdrv_refresh_limits(bs->file->bs, &local_err);
-        if (local_err) {
-            error_propagate(errp, local_err);
-            return;
+    have_limits = false;
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & (BDRV_CHILD_DATA | BDRV_CHILD_FILTERED | BDRV_CHILD_COW))
+        {
+            bdrv_refresh_limits(c->bs, &local_err);
+            if (local_err) {
+                error_propagate(errp, local_err);
+                return;
+            }
+            bdrv_merge_limits(&bs->bl, &c->bs->bl);
+            have_limits = true;
         }
-        bdrv_merge_limits(&bs->bl, &bs->file->bs->bl);
-    } else {
+    }
+
+    if (!have_limits) {
         bs->bl.min_mem_alignment = 512;
         bs->bl.opt_mem_alignment = qemu_real_host_page_size;
 
@@ -164,15 +173,6 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
         bs->bl.max_iov = IOV_MAX;
     }
 
-    if (bs->backing) {
-        bdrv_refresh_limits(bs->backing->bs, &local_err);
-        if (local_err) {
-            error_propagate(errp, local_err);
-            return;
-        }
-        bdrv_merge_limits(&bs->bl, &bs->backing->bs->bl);
-    }
-
     /* Then let the driver override it */
     if (drv->bdrv_refresh_limits) {
         drv->bdrv_refresh_limits(bs, errp);
Signature · 1 line
2.26.2
#78 lore raw

Re: [PATCH v7 20/47] block: Iterate over children in refresh_limits

Quoted 37 lines · Max Reitz

Instead of looking at just bs->file and bs->backing, we should look at all children that could end up receiving forwarded requests.

---
  block/io.c | 32 ++++++++++++++++----------------
  1 file changed, 16 insertions(+), 16 deletions(-)
block/io.c +9 -5
index c2af7711d6..37057f13e0 100644
@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)
  void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
  {
      BlockDriver *drv = bs->drv;
+    BdrvChild *c;
+    bool have_limits;
      Error *local_err = NULL;
 
      memset(&bs->bl, 0, sizeof(bs->bl));
@@ -149,14 +151,21 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
                                  drv->bdrv_co_preadv_part) ? 1 : 512;
 
      /* Take some limits from the children as a default */
-    if (bs->file) {
-        bdrv_refresh_limits(bs->file->bs, &local_err);
-        if (local_err) {
-            error_propagate(errp, local_err);
-            return;
+    have_limits = false;
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & (BDRV_CHILD_DATA | BDRV_CHILD_FILTERED | BDRV_CHILD_COW))
+        {
+            bdrv_refresh_limits(c->bs, &local_err);
+            if (local_err) {
+                error_propagate(errp, local_err);

block/io.c void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) · line 168

Quoted 10 lines
block/io.c +7 -2
+                return;
+            }
+            bdrv_merge_limits(&bs->bl, &c->bs->bl);
+            have_limits = true;
          }
-        bdrv_merge_limits(&bs->bl, &bs->file->bs->bl);
-    } else {
+    }
+
+    if (!have_limits) {

This conditioned piece of code worked with (bs->file == NULL) only.

Now, it works only if there are neither bs->file, nor bs->backing, nor else filtered children.

Is it OK and doesn't break the logic for all cases?

Andrey

Quoted 19 lines
block/io.c +0 -9
          bs->bl.min_mem_alignment = 512;
          bs->bl.opt_mem_alignment = qemu_real_host_page_size;
 
@@ -164,15 +173,6 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
          bs->bl.max_iov = IOV_MAX;
      }
 
-    if (bs->backing) {
-        bdrv_refresh_limits(bs->backing->bs, &local_err);
-        if (local_err) {
-            error_propagate(errp, local_err);
-            return;
-        }
-        bdrv_merge_limits(&bs->bl, &bs->backing->bs->bl);
-    }
-
      /* Then let the driver override it */
      if (drv->bdrv_refresh_limits) {
          drv->bdrv_refresh_limits(bs, errp);
author
#79 lore raw

Re: [PATCH v7 20/47] block: Iterate over children in refresh_limits

Quoted 53 lines · Andrey Shinkevich
Quoted 38 lines · Max Reitz

Instead of looking at just bs->file and bs->backing, we should look at all children that could end up receiving forwarded requests.

---
  block/io.c | 32 ++++++++++++++++----------------

  1 file changed, 16 insertions(+), 16 deletions(-)
block/io.c +9 -5
index c2af7711d6..37057f13e0 100644
@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst,
 const BlockLimits *src)
   void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
   {
       BlockDriver *drv = bs->drv;
+    BdrvChild *c;
+    bool have_limits;
       Error *local_err = NULL;
         memset(&bs->bl, 0, sizeof(bs->bl));
@@ -149,14 +151,21 @@ void bdrv_refresh_limits(BlockDriverState *bs,
 Error **errp)
                                   drv->bdrv_co_preadv_part) ? 1 : 512;
         /* Take some limits from the children as a default */
-    if (bs->file) {
-        bdrv_refresh_limits(bs->file->bs, &local_err);
-        if (local_err) {
-            error_propagate(errp, local_err);
-            return;
+    have_limits = false;
+    QLIST_FOREACH(c, &bs->children, next) {
+        if (c->role & (BDRV_CHILD_DATA | BDRV_CHILD_FILTERED |
 BDRV_CHILD_COW))
+        {
+            bdrv_refresh_limits(c->bs, &local_err);
+            if (local_err) {
+                error_propagate(errp, local_err);

block/io.c void bdrv_refresh_limits(BlockDriverState *bs, · line 169

Quoted 10 lines
block/io.c +7 -2
+                return;
+            }
+            bdrv_merge_limits(&bs->bl, &c->bs->bl);
+            have_limits = true;
           }
-        bdrv_merge_limits(&bs->bl, &bs->file->bs->bl);
-    } else {
+    }
+
+    if (!have_limits) {

This conditioned piece of code worked with (bs->file == NULL) only.

Quoted 4 lines

Now, it works only if there are neither bs->file, nor bs->backing, nor else filtered children.

Is it OK and doesn't break the logic for all cases?

Hm. Good question.

I think the answer is it’s OK.

For DATA and FILTERED, it makes absolute sense to just use their alignments. For COW, maybe not so much? But if there’s a COW child, there has to be a DATA child as well (in practice). So we’ll always consider its alignment, too.

(And hypothetically speaking, if there was a COW child but no DATA child, then the only alignment we need to observe is in fact the one of the COW child.)

Max

author
#80 lore raw

PATCH v7 21/47 block: Use CAFs in bdrv_refresh_filename()

In reply to Max Reitz

bdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.

---
 block.c | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)
block.c +21 -8
index 8131d0b5eb..7c827fefa0 100644
@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
     BdrvChild *child;
+    BlockDriverState *primary_child_bs;
     QDict *opts;
     bool backing_overridden;
     bool generate_json_filename; /* Whether our default implementation should
@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     qobject_unref(bs->full_open_options);
     bs->full_open_options = opts;
 
+    primary_child_bs = bdrv_primary_bs(bs);
+
     if (drv->bdrv_refresh_filename) {
         /* Obsolete information is of no use here, so drop the old file name
          * information before refreshing it */
         bs->exact_filename[0] = '\0';
 
         drv->bdrv_refresh_filename(bs);
-    } else if (bs->file) {
-        /* Try to reconstruct valid information from the underlying file */
+    } else if (primary_child_bs) {
+        /*
+         * Try to reconstruct valid information from the underlying
+         * file -- this only works for format nodes (filter nodes
+         * cannot be probed and as such must be selected by the user
+         * either through an options dict, or through a special
+         * filename which the filter driver must construct in its
+         * .bdrv_refresh_filename() implementation).
+         */
 
         bs->exact_filename[0] = '\0';
 
         /*
          * We can use the underlying file's filename if:
          * - it has a filename,
+         * - the current BDS is not a filter,
          * - the file is a protocol BDS, and
          * - opening that file (as this BDS's format) will automatically create
          *   the BDS tree we have right now, that is:
@@ -6888,11 +6899,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)
          *   - no non-file child of this BDS has been overridden by the user
          *   Both of these conditions are represented by generate_json_filename.
          */
-        if (bs->file->bs->exact_filename[0] &&
-            bs->file->bs->drv->bdrv_file_open &&
-            !generate_json_filename)
+        if (primary_child_bs->exact_filename[0] &&
+            primary_child_bs->drv->bdrv_file_open &&
+            !drv->is_filter && !generate_json_filename)
         {
-            strcpy(bs->exact_filename, bs->file->bs->exact_filename);
+            strcpy(bs->exact_filename, primary_child_bs->exact_filename);
         }
     }
 
@@ -6912,6 +6923,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
 char *bdrv_dirname(BlockDriverState *bs, Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs;
 
     if (!drv) {
         error_setg(errp, "Node '%s' is ejected", bs->node_name);
@@ -6922,8 +6934,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp)
         return drv->bdrv_dirname(bs, errp);
     }
 
-    if (bs->file) {
-        return bdrv_dirname(bs->file->bs, errp);
+    child_bs = bdrv_primary_bs(bs);
+    if (child_bs) {
+        return bdrv_dirname(child_bs, errp);
     }
 
     bdrv_refresh_filename(bs);
Signature · 1 line
2.26.2
#81 lore raw

Re: [PATCH v7 21/47] block: Use CAFs in bdrv_refresh_filename()

Quoted 34 lines · Max Reitz

bdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.

---
  block.c | 29 +++++++++++++++++++++--------
  1 file changed, 21 insertions(+), 8 deletions(-)
block.c +3 -1
index 8131d0b5eb..7c827fefa0 100644
@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
  {
      BlockDriver *drv = bs->drv;
      BdrvChild *child;
+    BlockDriverState *primary_child_bs;
      QDict *opts;
      bool backing_overridden;
      bool generate_json_filename; /* Whether our default implementation should
@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)
      qobject_unref(bs->full_open_options);
      bs->full_open_options = opts;
 
+    primary_child_bs = bdrv_primary_bs(bs);
+
      if (drv->bdrv_refresh_filename) {
          /* Obsolete information is of no use here, so drop the old file name
           * information before refreshing it */
          bs->exact_filename[0] = '\0';
 
          drv->bdrv_refresh_filename(bs);
-    } else if (bs->file) {

block.c void bdrv_refresh_filename(BlockDriverState *bs) · line 6886

Quoted 10 lines
block.c +9 -1
-        /* Try to reconstruct valid information from the underlying file */
+    } else if (primary_child_bs) {
+        /*
+         * Try to reconstruct valid information from the underlying
+         * file -- this only works for format nodes (filter nodes
+         * cannot be probed and as such must be selected by the user
+         * either through an options dict, or through a special
+         * filename which the filter driver must construct in its
+         * .bdrv_refresh_filename() implementation).
+         */

The caller may not be aware of a filter node and intend to refresh the name of underlying format node.

In that case, the filter driver should redirect the call to the format node.

What are situations the name of the filter itself should be refreshed in?

If there are any, should we do both actions or choose either?

Andrey

block.c void bdrv_refresh_filename(BlockDriverState *bs) · line 6893

Quoted 7 lines
block.c +1 -0
          bs->exact_filename[0] = '\0';
 
          /*
           * We can use the underlying file's filename if:
           * - it has a filename,
+         * - the current BDS is not a filter,

Should we check the function input parameter for being a filter's BS here, in this function, and handle the case here or let the filter driver function do that or else the caller should check it?

Andrey

Quoted 39 lines
block.c +8 -6
           * - the file is a protocol BDS, and
           * - opening that file (as this BDS's format) will automatically create
           *   the BDS tree we have right now, that is:
@@ -6888,11 +6899,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)
           *   - no non-file child of this BDS has been overridden by the user
           *   Both of these conditions are represented by generate_json_filename.
           */
-        if (bs->file->bs->exact_filename[0] &&
-            bs->file->bs->drv->bdrv_file_open &&
-            !generate_json_filename)
+        if (primary_child_bs->exact_filename[0] &&
+            primary_child_bs->drv->bdrv_file_open &&
+            !drv->is_filter && !generate_json_filename)
          {
-            strcpy(bs->exact_filename, bs->file->bs->exact_filename);
+            strcpy(bs->exact_filename, primary_child_bs->exact_filename);
          }
      }
 
@@ -6912,6 +6923,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
  char *bdrv_dirname(BlockDriverState *bs, Error **errp)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs;
 
      if (!drv) {
          error_setg(errp, "Node '%s' is ejected", bs->node_name);
@@ -6922,8 +6934,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp)
          return drv->bdrv_dirname(bs, errp);
      }
 
-    if (bs->file) {
-        return bdrv_dirname(bs->file->bs, errp);
+    child_bs = bdrv_primary_bs(bs);
+    if (child_bs) {
+        return bdrv_dirname(child_bs, errp);
      }
 
      bdrv_refresh_filename(bs);
#82 lore raw

Re: [PATCH v7 21/47] block: Use CAFs in bdrv_refresh_filename()

Quoted 48 lines · Andrey Shinkevich
Quoted 35 lines · Max Reitz

bdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.

---
  block.c | 29 +++++++++++++++++++++--------

  1 file changed, 21 insertions(+), 8 deletions(-)
block.c +1 -0
index 8131d0b5eb..7c827fefa0 100644
@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
   {
       BlockDriver *drv = bs->drv;
       BdrvChild *child;
+    BlockDriverState *primary_child_bs;
       QDict *opts;
       bool backing_overridden;

      bool generate_json_filename; /* Whether our default implementation should

block.c +1 -2
@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)
       qobject_unref(bs->full_open_options);
       bs->full_open_options = opts;
   +    primary_child_bs = bdrv_primary_bs(bs);
+
       if (drv->bdrv_refresh_filename) {
           /* Obsolete information is of no use here, so drop the old file name
            * information before refreshing it */
           bs->exact_filename[0] = '\0';
             drv->bdrv_refresh_filename(bs);
-    } else if (bs->file) {
-        /* Try to reconstruct valid information from the underlying 

file */
+    } else if (primary_child_bs) {

Quoted 8 lines

+        /*
+         * Try to reconstruct valid information from the underlying +         * file -- this only works for format nodes (filter nodes +         * cannot be probed and as such must be selected by the user +         * either through an options dict, or through a special +         * filename which the filter driver must construct in its +         * .bdrv_refresh_filename() implementation).
+         */

The caller may not be aware of a filter node and intend to refresh the name of underlying format node.

Quoted 8 lines

In that case, the filter driver should redirect the call to the format node.

What are situations the name of the filter itself should be refreshed in?

If there are any, should we do both actions or choose either?

Andrey

I ment the node FILE name.

Andrey

Quoted 51 lines
Quoted 6 lines

            bs->exact_filename[0] = '\0';
            /*
           * We can use the underlying file's filename if:
           * - it has a filename,
+         * - the current BDS is not a filter,

Should we check the function input parameter for being a filter's BS here, in this function, and handle the case here or let the filter driver function do that or else the caller should check it?

Andrey

Quoted 37 lines

           * - the file is a protocol BDS, and
           * - opening that file (as this BDS's format) will automatically create *   the BDS tree we have right now, that is:

block.c +4 -4
@@ -6888,11 +6899,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)
            *   - no non-file child of this BDS has been overridden by the user
            *   Both of these conditions are represented by generate_json_filename.
            */
-        if (bs->file->bs->exact_filename[0] &&
-            bs->file->bs->drv->bdrv_file_open &&
-            !generate_json_filename)
+        if (primary_child_bs->exact_filename[0] &&
+            primary_child_bs->drv->bdrv_file_open &&
+            !drv->is_filter && !generate_json_filename)
           {
-            strcpy(bs->exact_filename, bs->file->bs->exact_filename);
+            strcpy(bs->exact_filename, 
 primary_child_bs->exact_filename);
           }
       }

  @@ -6912,6 +6923,7 @@ void bdrv_refresh_filename(BlockDriverState *bs) char *bdrv_dirname(BlockDriverState *bs, Error **errp)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs;
        if (!drv) {
          error_setg(errp, "Node '%s' is ejected", bs->node_name);

block.c +3 -1
@@ -6922,8 +6934,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error 
 **errp)
           return drv->bdrv_dirname(bs, errp);
       }
   -    if (bs->file) {
-        return bdrv_dirname(bs->file->bs, errp);
+    child_bs = bdrv_primary_bs(bs);
+    if (child_bs) {
+        return bdrv_dirname(child_bs, errp);
       }
         bdrv_refresh_filename(bs);
author
#83 lore raw

Re: [PATCH v7 21/47] block: Use CAFs in bdrv_refresh_filename()

In reply to Andrey Shinkevich
Quoted 51 lines · Andrey Shinkevich
Quoted 37 lines · Max Reitz

bdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.

---
  block.c | 29 +++++++++++++++++++++--------

  1 file changed, 21 insertions(+), 8 deletions(-)
block.c +1 -0
index 8131d0b5eb..7c827fefa0 100644
@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
   {
       BlockDriver *drv = bs->drv;
       BdrvChild *child;
+    BlockDriverState *primary_child_bs;
       QDict *opts;
       bool backing_overridden;

      bool generate_json_filename; /* Whether our default implementation should

block.c +1 -0
@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)
       qobject_unref(bs->full_open_options);
       bs->full_open_options = opts;
   +    primary_child_bs = bdrv_primary_bs(bs);
+
       if (drv->bdrv_refresh_filename) {
           /* Obsolete information is of no use here, so drop the old

file name
           * information before refreshing it */ bs->exact_filename[0] = '\0'; drv->bdrv_refresh_filename(bs);
-    } else if (bs->file) {
-        /* Try to reconstruct valid information from the underlying file */
+    } else if (primary_child_bs) {

Quoted 8 lines

+        /*
+         * Try to reconstruct valid information from the underlying +         * file -- this only works for format nodes (filter nodes +         * cannot be probed and as such must be selected by the user +         * either through an options dict, or through a special +         * filename which the filter driver must construct in its +         * .bdrv_refresh_filename() implementation).
+         */

The caller may not be aware of a filter node and intend to refresh the name of underlying format node.

Quoted 2 lines

In that case, the filter driver should redirect the call to the format node.

It shouldn’t. We can only return a plain filename if passing that filename to qemu (e.g. to -drive) will result in the same block graph configuration.

This is what the comment means by “filter nodes cannot be probed”: If there is a filter node, we must generate a json:{} filename, because otherwise reopening the block device with -drive by passing the filename generated here would result in a configuration where the filter is missing.

Quoted 1 line

What are situations the name of the filter itself should be refreshed in?

Hypothetically, if a filename could specify a filter. For example, say the filename “filter[copy-on-read]:foo.qcow2” would result in qemu creating a COR filter on top of a qcow2 node, then we could generate such a filename.

In practice, filters cannot be configured through plain filenames (apart from blkverify, but that’s a debugging feature, so it doesn’t really matter), so there is no such situation. All filter nodes should have an empty exact_filename and thus get a json:{} pseudo-filename.

Quoted 12 lines

If there are any, should we do both actions or choose either?

Andrey

Quoted 5 lines

            bs->exact_filename[0] = '\0';
            /*
           * We can use the underlying file's filename if:
           * - it has a filename,
+         * - the current BDS is not a filter,

Quoted 3 lines

Should we check the function input parameter for being a filter's BS here, in this function, and handle the case here or let the filter driver function do that or else the caller should check it?

bdrv_refresh_filename() is called whenever some node in the block graph has changed, just to refresh its filename (after that change). The caller generally doesn’t really care about the result, so it doesn’t matter whether the node is a filter or not (i.e., whether it gets a plain filename or not).

I don’t think the caller should check it, and in this implementation we simply have to handle filter nodes correctly: That is, not give them a plain filename.

Max

author
#84 lore raw

PATCH v7 22/47 block: Use CAF in bdrv_co_rw_vmstate()

In reply to Max Reitz

If a node whose driver does not provide VM state functions has a metadata child, the VM state should probably go there; if it is a filter, the VM state should probably go there. It follows that we should generally go down to the primary child.

---
 block/io.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
block/io.c +3 -2
index 37057f13e0..9e802804bb 100644
@@ -2646,6 +2646,7 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
                    bool is_read)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs = bdrv_primary_bs(bs);
     int ret = -ENOTSUP;
 
     bdrv_inc_in_flight(bs);
@@ -2658,8 +2659,8 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
         } else {
             ret = drv->bdrv_save_vmstate(bs, qiov, pos);
         }
-    } else if (bs->file) {
-        ret = bdrv_co_rw_vmstate(bs->file->bs, qiov, pos, is_read);
+    } else if (child_bs) {
+        ret = bdrv_co_rw_vmstate(child_bs, qiov, pos, is_read);
     }
 
     bdrv_dec_in_flight(bs);
Signature · 1 line
2.26.2
Reviewed-by on v7 22/47
#85 lore raw

Re: [PATCH v7 22/47] block: Use CAF in bdrv_co_rw_vmstate()

Quoted 24 lines · Max Reitz

If a node whose driver does not provide VM state functions has a metadata child, the VM state should probably go there; if it is a filter, the VM state should probably go there. It follows that we should generally go down to the primary child.

---
  block/io.c | 5 +++--
  1 file changed, 3 insertions(+), 2 deletions(-)
block/io.c +1 -0
index 37057f13e0..9e802804bb 100644
@@ -2646,6 +2646,7 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
                     bool is_read)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs = bdrv_primary_bs(bs);
      int ret = -ENOTSUP;
 
      bdrv_inc_in_flight(bs);

block/io.c bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, · line 2666

Quoted 10 lines
block/io.c +2 -2
@@ -2658,8 +2659,8 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
          } else {
              ret = drv->bdrv_save_vmstate(bs, qiov, pos);
          }
-    } else if (bs->file) {
-        ret = bdrv_co_rw_vmstate(bs->file->bs, qiov, pos, is_read);
+    } else if (child_bs) {
+        ret = bdrv_co_rw_vmstate(child_bs, qiov, pos, is_read);
      }
 
      bdrv_dec_in_flight(bs);
author
#86 lore raw

PATCH v7 23/47 block/snapshot: Fix fallback

In reply to Max Reitz

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child. Furthermore, we really only can fall back to bs->file and bs->backing, because bdrv_snapshot_goto() has to modify the child link (notably, set it to NULL).

---
 block/snapshot.c | 104 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 83 insertions(+), 21 deletions(-)
block/snapshot.c +83 -21
index bd9fb01817..a2bf3a54eb 100644
@@ -147,6 +147,56 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
     return ret;
 }
 
+/**
+ * Return a pointer to the child BDS pointer to which we can fall
+ * back if the given BDS does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ *
+ * We need to return an indirect pointer because bdrv_snapshot_goto()
+ * has to modify the BdrvChild pointer.
+ */
+static BdrvChild **bdrv_snapshot_fallback_ptr(BlockDriverState *bs)
+{
+    BdrvChild **fallback;
+    BdrvChild *child;
+
+    /*
+     * The only BdrvChild pointers that are safe to modify (and which
+     * we can thus return a reference to) are bs->file and
+     * bs->backing.
+     */
+    fallback = &bs->file;
+    if (!*fallback && bs->drv && bs->drv->is_filter) {
+        fallback = &bs->backing;
+    }
+
+    if (!*fallback) {
+        return NULL;
+    }
+
+    /*
+     * Check that there are no other children that would need to be
+     * snapshotted.  If there are, it is not safe to fall back to
+     * *fallback.
+     */
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->role & (BDRV_CHILD_DATA | BDRV_CHILD_METADATA |
+                           BDRV_CHILD_FILTERED) &&
+            child != *fallback)
+        {
+            return NULL;
+        }
+    }
+
+    return fallback;
+}
+
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BdrvChild **child_ptr = bdrv_snapshot_fallback_ptr(bs);
+    return child_ptr ? (*child_ptr)->bs : NULL;
+}
+
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
@@ -155,8 +205,9 @@ int bdrv_can_snapshot(BlockDriverState *bs)
     }
 
     if (!drv->bdrv_snapshot_create) {
-        if (bs->file != NULL) {
-            return bdrv_can_snapshot(bs->file->bs);
+        BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
+        if (fallback_bs) {
+            return bdrv_can_snapshot(fallback_bs);
         }
         return 0;
     }
@@ -168,14 +219,15 @@ int bdrv_snapshot_create(BlockDriverState *bs,
                          QEMUSnapshotInfo *sn_info)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
     if (!drv) {
         return -ENOMEDIUM;
     }
     if (drv->bdrv_snapshot_create) {
         return drv->bdrv_snapshot_create(bs, sn_info);
     }
-    if (bs->file) {
-        return bdrv_snapshot_create(bs->file->bs, sn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_create(fallback_bs, sn_info);
     }
     return -ENOTSUP;
 }
@@ -185,6 +237,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
                        Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BdrvChild **fallback_ptr;
     int ret, open_ret;
 
     if (!drv) {
@@ -205,39 +258,46 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
         return ret;
     }
 
-    if (bs->file) {
-        BlockDriverState *file;
-        QDict *options = qdict_clone_shallow(bs->options);
+    fallback_ptr = bdrv_snapshot_fallback_ptr(bs);
+    if (fallback_ptr) {
+        QDict *options;
         QDict *file_options;
         Error *local_err = NULL;
+        BlockDriverState *fallback_bs = (*fallback_ptr)->bs;
+        char *subqdict_prefix = g_strdup_printf("%s.", (*fallback_ptr)->name);
+
+        options = qdict_clone_shallow(bs->options);
 
-        file = bs->file->bs;
         /* Prevent it from getting deleted when detached from bs */
-        bdrv_ref(file);
+        bdrv_ref(fallback_bs);
 
-        qdict_extract_subqdict(options, &file_options, "file.");
+        qdict_extract_subqdict(options, &file_options, subqdict_prefix);
         qobject_unref(file_options);
-        qdict_put_str(options, "file", bdrv_get_node_name(file));
+        g_free(subqdict_prefix);
+
+        qdict_put_str(options, (*fallback_ptr)->name,
+                      bdrv_get_node_name(fallback_bs));
 
         if (drv->bdrv_close) {
             drv->bdrv_close(bs);
         }
-        bdrv_unref_child(bs, bs->file);
-        bs->file = NULL;
 
-        ret = bdrv_snapshot_goto(file, snapshot_id, errp);
+        bdrv_unref_child(bs, *fallback_ptr);
+        *fallback_ptr = NULL;
+
+        ret = bdrv_snapshot_goto(fallback_bs, snapshot_id, errp);
         open_ret = drv->bdrv_open(bs, options, bs->open_flags, &local_err);
         qobject_unref(options);
         if (open_ret < 0) {
-            bdrv_unref(file);
+            bdrv_unref(fallback_bs);
             bs->drv = NULL;
             /* A bdrv_snapshot_goto() error takes precedence */
             error_propagate(errp, local_err);
             return ret < 0 ? ret : open_ret;
         }
 
-        assert(bs->file->bs == file);
-        bdrv_unref(file);
+        assert(fallback_bs == (*fallback_ptr)->bs);
+        bdrv_unref(fallback_bs);
         return ret;
     }
 
@@ -273,6 +333,7 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
                          Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
     int ret;
 
     if (!drv) {
@@ -289,8 +350,8 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
 
     if (drv->bdrv_snapshot_delete) {
         ret = drv->bdrv_snapshot_delete(bs, snapshot_id, name, errp);
-    } else if (bs->file) {
-        ret = bdrv_snapshot_delete(bs->file->bs, snapshot_id, name, errp);
+    } else if (fallback_bs) {
+        ret = bdrv_snapshot_delete(fallback_bs, snapshot_id, name, errp);
     } else {
         error_setg(errp, "Block format '%s' used by device '%s' "
                    "does not support internal snapshot deletion",
@@ -306,14 +367,15 @@ int bdrv_snapshot_list(BlockDriverState *bs,
                        QEMUSnapshotInfo **psn_info)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
     if (!drv) {
         return -ENOMEDIUM;
     }
     if (drv->bdrv_snapshot_list) {
         return drv->bdrv_snapshot_list(bs, psn_info);
     }
-    if (bs->file) {
-        return bdrv_snapshot_list(bs->file->bs, psn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_list(fallback_bs, psn_info);
     }
     return -ENOTSUP;
 }
Signature · 1 line
2.26.2
Reviewed-by on v7 23/47
#87 lore raw

Re: [PATCH v7 23/47] block/snapshot: Fix fallback

Quoted 3 lines · Max Reitz

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there

I guess it comes to COW children like BDRV_CHILD_DATA | BDRV_CHILD_METADATA  rather than non-COW ones, does it?

Andrey

Quoted 11 lines

is more than one such child. Furthermore, we really only can fall back to bs->file and bs->backing, because bdrv_snapshot_goto() has to modify the child link (notably, set it to NULL).

---
  block/snapshot.c | 104 +++++++++++++++++++++++++++++++++++++----------
  1 file changed, 83 insertions(+), 21 deletions(-)
block/snapshot.c +0 -0

...

Quoted 16 lines
block/snapshot.c +16 -0
+    /*
+     * Check that there are no other children that would need to be
+     * snapshotted.  If there are, it is not safe to fall back to
+     * *fallback.
+     */
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->role & (BDRV_CHILD_DATA | BDRV_CHILD_METADATA |
+                           BDRV_CHILD_FILTERED) &&
+            child != *fallback)
+        {
+            return NULL;
+        }
+    }
+
+    return fallback;
+}

...

#88 lore raw

Re: [PATCH v7 23/47] block/snapshot: Fix fallback

Quoted 7 lines · Andrey Shinkevich
Quoted 3 lines · Max Reitz

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children.  For simplicity's sake, just do not fall back if there

I guess it comes to COW children like BDRV_CHILD_DATA | BDRV_CHILD_METADATA  rather than non-COW ones, does it?

The BDRV_CHILD_COW is mutually exclusive with DATA, METADATA and FILTERED per definition.

Sorry about the question.

Quoted 37 lines

Andrey

Quoted 11 lines

is more than one such child.  Furthermore, we really only can fall back to bs->file and bs->backing, because bdrv_snapshot_goto() has to modify the child link (notably, set it to NULL).

---
  block/snapshot.c | 104 +++++++++++++++++++++++++++++++++++++----------

  1 file changed, 83 insertions(+), 21 deletions(-)
block/snapshot.c +0 -0

...

Quoted 16 lines

+    /*
+     * Check that there are no other children that would need to be +     * snapshotted.  If there are, it is not safe to fall back to +     * *fallback.
+     */
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->role & (BDRV_CHILD_DATA | BDRV_CHILD_METADATA | +                           BDRV_CHILD_FILTERED) &&
+            child != *fallback)
+        {
+            return NULL;
+        }
+    }

block/snapshot.c +3 -0
+
+    return fallback;
+}

...

author
#89 lore raw

PATCH v7 24/47 block: Use CAFs for debug breakpoints

In reply to Max Reitz

When looking for a blkdebug node (which implements debug breakpoints), use bdrv_primary_bs() to iterate through the graph, because that is where a blkdebug node would be.

---
 block.c | 16 +++-------------
 1 file changed, 3 insertions(+), 13 deletions(-)
block.c +3 -13
index 7c827fefa0..1c71ecab7c 100644
@@ -5562,17 +5562,7 @@ void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event)
 static BlockDriverState *bdrv_find_debug_node(BlockDriverState *bs)
 {
     while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) {
-        if (bs->file) {
-            bs = bs->file->bs;
-            continue;
-        }
-
-        if (bs->drv->is_filter && bs->backing) {
-            bs = bs->backing->bs;
-            continue;
-        }
-
-        break;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {
@@ -5607,7 +5597,7 @@ int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)
 int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
 {
     while (bs && (!bs->drv || !bs->drv->bdrv_debug_resume)) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_resume) {
@@ -5620,7 +5610,7 @@ int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
 bool bdrv_debug_is_suspended(BlockDriverState *bs, const char *tag)
 {
     while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
Signature · 1 line
2.26.2
Reviewed-by on v7 24/47
#90 lore raw

Re: [PATCH v7 24/47] block: Use CAFs for debug breakpoints

Quoted 42 lines · Max Reitz

When looking for a blkdebug node (which implements debug breakpoints), use bdrv_primary_bs() to iterate through the graph, because that is where a blkdebug node would be.

---
  block.c | 16 +++-------------
  1 file changed, 3 insertions(+), 13 deletions(-)
block.c +2 -12
index 7c827fefa0..1c71ecab7c 100644
@@ -5562,17 +5562,7 @@ void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event)
  static BlockDriverState *bdrv_find_debug_node(BlockDriverState *bs)
  {
      while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) {
-        if (bs->file) {
-            bs = bs->file->bs;
-            continue;
-        }
-
-        if (bs->drv->is_filter && bs->backing) {
-            bs = bs->backing->bs;
-            continue;
-        }
-
-        break;
+        bs = bdrv_primary_bs(bs);
      }
 
      if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {
@@ -5607,7 +5597,7 @@ int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)
  int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
  {
      while (bs && (!bs->drv || !bs->drv->bdrv_debug_resume)) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
      }
 
      if (bs && bs->drv && bs->drv->bdrv_debug_resume) {

block.c int bdrv_debug_resume(BlockDriverState *bs, const char *tag) · line 5616

Quoted 8 lines
block.c +1 -1
@@ -5620,7 +5610,7 @@ int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
  bool bdrv_debug_is_suspended(BlockDriverState *bs, const char *tag)
  {
      while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
      }
 
      if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
author
#91 lore raw

PATCH v7 25/47 block: Def. impl.s for get_allocated_file_size

In reply to Max Reitz

If every BlockDriver were to implement bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return -ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizes

For the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child

(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.

---
 include/block/block_int.h |  5 ++++
 block.c                   | 51 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/block/block_int.h +5 -0
index 5da793bfc3..c963ee9f28 100644
@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,
                                                    int64_t *pnum,
                                                    int64_t *map,
                                                    BlockDriverState **file);
+
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs);
+
 const char *bdrv_get_parent_name(const BlockDriverState *bs);
 void blk_dev_change_media_cb(BlockBackend *blk, bool load, Error **errp);
 bool blk_dev_has_removable_media(BlockBackend *blk);
block.c +51 -0

57 lines not on this page · raw message

Signature · 1 line
2.26.2
Reviewed-by on v7 25/47
#92 lore raw

Re: [PATCH v7 25/47] block: Def. impl.s for get_allocated_file_size

Quoted 90 lines · Max Reitz

If every BlockDriver were to implement bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return -ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizes

For the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child

(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.

---
  include/block/block_int.h |  5 ++++
  block.c                   | 51 +++++++++++++++++++++++++++++++++++++++
  2 files changed, 56 insertions(+)
include/block/block_int.h +5 -0
index 5da793bfc3..c963ee9f28 100644
@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,
                                                     int64_t *pnum,
                                                     int64_t *map,
                                                     BlockDriverState **file);
+
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs);
+
  const char *bdrv_get_parent_name(const BlockDriverState *bs);
  void blk_dev_change_media_cb(BlockBackend *blk, bool load, Error **errp);
  bool blk_dev_has_removable_media(BlockBackend *blk);
block.c +44 -0
index 1c71ecab7c..fc01ce90b3 100644
@@ -5003,6 +5003,57 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
      return -ENOTSUP;
  }
 
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * block drivers that want it to sum all children they store data on.
+ * (This excludes backing children.)
+ */
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs)
+{
+    BdrvChild *child;
+    int64_t child_size, sum = 0;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->role & (BDRV_CHILD_DATA | BDRV_CHILD_METADATA |
+                           BDRV_CHILD_FILTERED))
+        {
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;
+            }
+            sum += child_size;
+        }
+    }
+
+    return sum;
+}
+
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * block drivers that want it to return only the size of a node's
+ * primary child.
+ */
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs)
+{
+    BlockDriverState *primary_bs;
+
+    primary_bs = bdrv_primary_bs(bs);
+    if (!primary_bs) {
+        return -ENOTSUP;
+    }
+
+    return bdrv_get_allocated_file_size(primary_bs);
+}
+
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5059

Quoted 10 lines
block.c +7 -0
+ * protocol block drivers that just do not support it.
+ */
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs)
+{
+    return -ENOTSUP;
+}
+
  /*
   * bdrv_measure:
   * @drv: Format driver
#93 lore raw

Re: [PATCH v7 25/47] block: Def. impl.s for get_allocated_file_size

In reply to Max Reitz

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 60 lines

If every BlockDriver were to implement bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return -ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizes

For the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child

(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.

---
 include/block/block_int.h |  5 ++++
 block.c                   | 51 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/block/block_int.h +5 -0
index 5da793bfc3..c963ee9f28 100644
@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,
                                                    int64_t *pnum,
                                                    int64_t *map,
                                                    BlockDriverState **file);
+
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs);
+
 const char *bdrv_get_parent_name(const BlockDriverState *bs);
 void blk_dev_change_media_cb(BlockBackend *blk, bool load, Error **errp);
 bool blk_dev_has_removable_media(BlockBackend *blk);
block.c +14 -0
index 1c71ecab7c..fc01ce90b3 100644
@@ -5003,6 +5003,57 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
     return -ENOTSUP;
 }
 
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * block drivers that want it to sum all children they store data on.
+ * (This excludes backing children.)
+ */
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs)
+{
+    BdrvChild *child;
+    int64_t child_size, sum = 0;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->role & (BDRV_CHILD_DATA | BDRV_CHILD_METADATA |
+                           BDRV_CHILD_FILTERED))
+        {

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5029

Quoted 10 lines
block.c +10 -0
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;
+            }
+            sum += child_size;
+        }
+    }
+
+    return sum;
+}

The only user apart from bdrv_get_allocated_file_size() is blkverify. As I argued that blkverify shouldn't use it, this can become static.

Quoted 6 lines
block.c +6 -0
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * block drivers that want it to return only the size of a node's
+ * primary child.
+ */
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs)

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5046

Quoted 10 lines
block.c +10 -0
+{
+    BlockDriverState *primary_bs;
+
+    primary_bs = bdrv_primary_bs(bs);
+    if (!primary_bs) {
+        return -ENOTSUP;
+    }
+
+    return bdrv_get_allocated_file_size(primary_bs);
+}

This can become static, too (never used as a callback), and possibly even be inlined.

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5055

Quoted 8 lines
block.c +8 -0
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * protocol block drivers that just do not support it.
+ */
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs)
+{
+    return -ENOTSUP;
+}

Also never used as a callback. I think inlining it would almost certainly make more sense.

Kevin

author
#94 lore raw

Re: [PATCH v7 25/47] block: Def. impl.s for get_allocated_file_size

Quoted 95 lines · Kevin Wolf

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 60 lines

If every BlockDriver were to implement bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return -ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizes

For the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child

(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.

---
 include/block/block_int.h |  5 ++++
 block.c                   | 51 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)
include/block/block_int.h +5 -0
index 5da793bfc3..c963ee9f28 100644
@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,
                                                    int64_t *pnum,
                                                    int64_t *map,
                                                    BlockDriverState **file);
+
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs);
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs);
+
 const char *bdrv_get_parent_name(const BlockDriverState *bs);
 void blk_dev_change_media_cb(BlockBackend *blk, bool load, Error **errp);
 bool blk_dev_has_removable_media(BlockBackend *blk);
block.c +14 -0
index 1c71ecab7c..fc01ce90b3 100644
@@ -5003,6 +5003,57 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
     return -ENOTSUP;
 }
 
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * block drivers that want it to sum all children they store data on.
+ * (This excludes backing children.)
+ */
+int64_t bdrv_sum_allocated_file_size(BlockDriverState *bs)
+{
+    BdrvChild *child;
+    int64_t child_size, sum = 0;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child->role & (BDRV_CHILD_DATA | BDRV_CHILD_METADATA |
+                           BDRV_CHILD_FILTERED))
+        {

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5029

Quoted 10 lines
block.c +10 -0
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;
+            }
+            sum += child_size;
+        }
+    }
+
+    return sum;
+}

The only user apart from bdrv_get_allocated_file_size() is blkverify. As I argued that blkverify shouldn't use it, this can become static.

Quoted 6 lines
block.c +6 -0
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * block drivers that want it to return only the size of a node's
+ * primary child.
+ */
+int64_t bdrv_primary_allocated_file_size(BlockDriverState *bs)

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5046

Quoted 10 lines
block.c +10 -0
+{
+    BlockDriverState *primary_bs;
+
+    primary_bs = bdrv_primary_bs(bs);
+    if (!primary_bs) {
+        return -ENOTSUP;
+    }
+
+    return bdrv_get_allocated_file_size(primary_bs);
+}

This can become static, too (never used as a callback), and possibly even be inlined.

Quoted 11 lines

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5055

Quoted 8 lines
block.c +8 -0
+/**
+ * Implementation of BlockDriver.bdrv_get_allocated_file_size() for
+ * protocol block drivers that just do not support it.
+ */
+int64_t bdrv_notsup_allocated_file_size(BlockDriverState *bs)
+{
+    return -ENOTSUP;
+}

Also never used as a callback. I think inlining it would almost certainly make more sense.

I think they’re all artifacts from the development process, yeah.

Originally, I wanted to make .bdrv_get_allocated_file_size() mandatory, but then I saw that led nowhere and could be done well generically.

Max

author
#95 lore raw

PATCH v7 26/47 block: Improve get_allocated_file_size's default

In reply to Max Reitz

There are two practical problems with bdrv_get_allocated_file_size()'s default right now:
(1) For drivers with children, we should generally sum all their sizes instead of just passing the request through to bs->file. The latter is good for filters, but not so much for format drivers.

(2) Filters need not have bs->file, so we should actually go to the

    filtered child instead of hard-coding bs->file.

And we can make the whole default implementation more idiomatic by using the three generic functions added by the previous patch.

---
 block.c | 17 ++++++++++++++---
 1 file changed, 14 insertions(+), 3 deletions(-)
block.c +14 -3
index fc01ce90b3..a19f243997 100644
@@ -4997,10 +4997,21 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
     if (drv->bdrv_get_allocated_file_size) {
         return drv->bdrv_get_allocated_file_size(bs);
     }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (drv->bdrv_file_open) {
+        /*
+         * Protocol drivers default to -ENOTSUP (most of their data is
+         * not stored in any of their children (if they even have any),
+         * so there is no generic way to figure it out).
+         */
+        return bdrv_notsup_allocated_file_size(bs);
+    } else if (drv->is_filter) {
+        /* Filter drivers default to the size of their primary child */
+        return bdrv_primary_allocated_file_size(bs);
+    } else {
+        /* Other drivers default to summing their children's sizes */
+        return bdrv_sum_allocated_file_size(bs);
     }
-    return -ENOTSUP;
 }
 
 /**
Signature · 1 line
2.26.2
Reviewed-by on v7 26/47
#96 lore raw

Re: [PATCH v7 26/47] block: Improve get_allocated_file_size's default

Quoted 36 lines · Max Reitz

There are two practical problems with bdrv_get_allocated_file_size()'s default right now:
(1) For drivers with children, we should generally sum all their sizes instead of just passing the request through to bs->file. The latter is good for filters, but not so much for format drivers.

(2) Filters need not have bs->file, so we should actually go to the

     filtered child instead of hard-coding bs->file.

And we can make the whole default implementation more idiomatic by using the three generic functions added by the previous patch.

---
  block.c | 17 ++++++++++++++---
  1 file changed, 14 insertions(+), 3 deletions(-)
block.c +9 -2
index fc01ce90b3..a19f243997 100644
@@ -4997,10 +4997,21 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
      if (drv->bdrv_get_allocated_file_size) {
          return drv->bdrv_get_allocated_file_size(bs);
      }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (drv->bdrv_file_open) {
+        /*
+         * Protocol drivers default to -ENOTSUP (most of their data is
+         * not stored in any of their children (if they even have any),
+         * so there is no generic way to figure it out).
+         */
+        return bdrv_notsup_allocated_file_size(bs);
+    } else if (drv->is_filter) {

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5017

Quoted 10 lines
block.c +5 -1
+        /* Filter drivers default to the size of their primary child */
+        return bdrv_primary_allocated_file_size(bs);
+    } else {
+        /* Other drivers default to summing their children's sizes */
+        return bdrv_sum_allocated_file_size(bs);
      }
-    return -ENOTSUP;
  }
 
  /**
author
#97 lore raw

PATCH v7 27/47 blkverify: Use bdrv_sum_allocated_file_size()

In reply to Max Reitz

blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.

---
 block/blkverify.c | 1 +
 1 file changed, 1 insertion(+)
block/blkverify.c +1 -0
index 2f261de24b..64858c8df0 100644
@@ -323,6 +323,7 @@ static BlockDriver bdrv_blkverify = {
     .bdrv_getlength                   = blkverify_getlength,
     .bdrv_refresh_filename            = blkverify_refresh_filename,
     .bdrv_dirname                     = blkverify_dirname,
+    .bdrv_get_allocated_file_size     = bdrv_sum_allocated_file_size,
 
     .bdrv_co_preadv                   = blkverify_co_preadv,
     .bdrv_co_pwritev                  = blkverify_co_pwritev,
Signature · 1 line
2.26.2
Reviewed-by on v7 27/47
#98 lore raw

Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()

Quoted 15 lines · Max Reitz

blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.

---
  block/blkverify.c | 1 +
  1 file changed, 1 insertion(+)
block/blkverify.c +0 -0
index 2f261de24b..64858c8df0 100644

block/blkverify.c static BlockDriver bdrv_blkverify = { · line 329

Quoted 7 lines
block/blkverify.c +1 -0
@@ -323,6 +323,7 @@ static BlockDriver bdrv_blkverify = {
      .bdrv_getlength                   = blkverify_getlength,
      .bdrv_refresh_filename            = blkverify_refresh_filename,
      .bdrv_dirname                     = blkverify_dirname,
+    .bdrv_get_allocated_file_size     = bdrv_sum_allocated_file_size,
 
      .bdrv_co_preadv                   = blkverify_co_preadv,
      .bdrv_co_pwritev                  = blkverify_co_pwritev,
#99 lore raw

Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()

In reply to Max Reitz

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 4 lines

blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.

Hm, but so are the children of, say, backup-top. I don't think you're suggesting that backup-top should add the sizes of both images, even though the backup job is actively increasing the allocated size of the non-primary node, much like blkverify.

So I believe returning only the allocated size of the primary child in blkverify would be more consistent with what we do elsewhere.

Kevin

author
#100 lore raw

Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()

Quoted 8 lines · Kevin Wolf

Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:

Quoted 4 lines

blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.

Hm, but so are the children of, say, backup-top. I don't think you're suggesting that backup-top should add the sizes of both images,

Can be argued either way.

I lean on the side of that it should not, because: The backup is external. The job is copying data off. So it isn’t really directly data that serves qemu, which can be seen from the fact that it’s never read.

Which is not the case for quorum and blkverify. Though one can argue that blkverify is different from quorum here in that it doesn’t read data from the non-filtered child to serve a guest device, but just to verify it node-internally.

Quoted 6 lines

even
though the backup job is actively increasing the allocated size of the non-primary node, much like blkverify.

So I believe returning only the allocated size of the primary child in blkverify would be more consistent with what we do elsewhere.

For me, blkverify is basically an archaic mode of quorum, and for quorum it’s clear that we should sum the sizes. Which is why I thought summing the sizes would be more consistent.

But honestly, I just don’t care about blkverify whatsoever. I don’t believe anyone actually cares about whether what blkverify returns for .bdrv_get_allocated_file_size() is consistent. I believe we could return 42 and nobody would bat an eyelash.

(But that’s the curse of this series. I have to touch stuff that nobody cares about, and then we have discussions on stuff nobody cares about.)

So from that POV I’m happy to drop this patch if it means there’s just one less opportunity to have a discussion on blkverify.

Max

author
#101 lore raw

PATCH v7 28/47 block/null: Implement bdrv_get_allocated_file_size

In reply to Max Reitz

It is trivial, so we might as well do it.

---
 block/null.c               | 7 +++++++
 tests/qemu-iotests/153.out | 2 +-
 tests/qemu-iotests/184.out | 6 ++++--
 3 files changed, 12 insertions(+), 3 deletions(-)
block/null.c +7 -0
index 15e1d56746..cc9b1d4ea7 100644
@@ -262,6 +262,11 @@ static void null_refresh_filename(BlockDriverState *bs)
              bs->drv->format_name);
 }
 
+static int64_t null_allocated_file_size(BlockDriverState *bs)
+{
+    return 0;
+}
+
 static const char *const null_strong_runtime_opts[] = {
     BLOCK_OPT_SIZE,
     NULL_OPT_ZEROES,
@@ -277,6 +282,7 @@ static BlockDriver bdrv_null_co = {
     .bdrv_file_open         = null_file_open,
     .bdrv_parse_filename    = null_co_parse_filename,
     .bdrv_getlength         = null_getlength,
+    .bdrv_get_allocated_file_size = null_allocated_file_size,
 
     .bdrv_co_preadv         = null_co_preadv,
     .bdrv_co_pwritev        = null_co_pwritev,
@@ -297,6 +303,7 @@ static BlockDriver bdrv_null_aio = {
     .bdrv_file_open         = null_file_open,
     .bdrv_parse_filename    = null_aio_parse_filename,
     .bdrv_getlength         = null_getlength,
+    .bdrv_get_allocated_file_size = null_allocated_file_size,
 
     .bdrv_aio_preadv        = null_aio_preadv,
     .bdrv_aio_pwritev       = null_aio_pwritev,
tests/qemu-iotests/153.out +1 -1
index b2a90caa6b..8659e6463b 100644
@@ -461,7 +461,7 @@ No conflict:
 image: null-co://
 file format: null-co
 virtual size: 1 GiB (1073741824 bytes)
-disk size: unavailable
+disk size: 0 B
 
 Conflict:
 qemu-img: --force-share/-U conflicts with image options
tests/qemu-iotests/184.out +4 -2
index 3deb3cfb94..28b104da89 100644
@@ -29,7 +29,8 @@ Testing:
             "image": {
                 "virtual-size": 1073741824,
                 "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
-                "format": "throttle"
+                "format": "throttle",
+                "actual-size": SIZE
             },
             "iops_wr": 0,
             "ro": false,
@@ -56,7 +57,8 @@ Testing:
             "image": {
                 "virtual-size": 1073741824,
                 "filename": "null-co://",
-                "format": "null-co"
+                "format": "null-co",
+                "actual-size": SIZE
             },
             "iops_wr": 0,
             "ro": false,
Signature · 1 line
2.26.2
Reviewed-by on v7 28/47
#102 lore raw

Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size

Quoted 59 lines · Max Reitz

It is trivial, so we might as well do it.

---
  block/null.c               | 7 +++++++
  tests/qemu-iotests/153.out | 2 +-
  tests/qemu-iotests/184.out | 6 ++++--
  3 files changed, 12 insertions(+), 3 deletions(-)
block/null.c +7 -0
index 15e1d56746..cc9b1d4ea7 100644
@@ -262,6 +262,11 @@ static void null_refresh_filename(BlockDriverState *bs)
               bs->drv->format_name);
  }
 
+static int64_t null_allocated_file_size(BlockDriverState *bs)
+{
+    return 0;
+}
+
  static const char *const null_strong_runtime_opts[] = {
      BLOCK_OPT_SIZE,
      NULL_OPT_ZEROES,
@@ -277,6 +282,7 @@ static BlockDriver bdrv_null_co = {
      .bdrv_file_open         = null_file_open,
      .bdrv_parse_filename    = null_co_parse_filename,
      .bdrv_getlength         = null_getlength,
+    .bdrv_get_allocated_file_size = null_allocated_file_size,
 
      .bdrv_co_preadv         = null_co_preadv,
      .bdrv_co_pwritev        = null_co_pwritev,
@@ -297,6 +303,7 @@ static BlockDriver bdrv_null_aio = {
      .bdrv_file_open         = null_file_open,
      .bdrv_parse_filename    = null_aio_parse_filename,
      .bdrv_getlength         = null_getlength,
+    .bdrv_get_allocated_file_size = null_allocated_file_size,
 
      .bdrv_aio_preadv        = null_aio_preadv,
      .bdrv_aio_pwritev       = null_aio_pwritev,
tests/qemu-iotests/153.out +1 -1
index b2a90caa6b..8659e6463b 100644
@@ -461,7 +461,7 @@ No conflict:
  image: null-co://
  file format: null-co
  virtual size: 1 GiB (1073741824 bytes)
-disk size: unavailable
+disk size: 0 B
 
  Conflict:
  qemu-img: --force-share/-U conflicts with image options
tests/qemu-iotests/184.out +0 -0
index 3deb3cfb94..28b104da89 100644

tests/qemu-iotests/184.out Testing: · line 33

Quoted 6 lines
tests/qemu-iotests/184.out +2 -1
@@ -29,7 +29,8 @@ Testing:
              "image": {
                  "virtual-size": 1073741824,
                  "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
-                "format": "throttle"
+                "format": "throttle",
+                "actual-size": SIZE

If we remove the _filter_generated_node_ids() in the current implementation of the test #184, we will get '"actual-size": 0'. It might be more informative when analyzing the output in 184.out.

Andrey

tests/qemu-iotests/184.out Testing: · line 64

Quoted 13 lines
tests/qemu-iotests/184.out +2 -1
              },
              "iops_wr": 0,
              "ro": false,
@@ -56,7 +57,8 @@ Testing:
              "image": {
                  "virtual-size": 1073741824,
                  "filename": "null-co://",
-                "format": "null-co"
+                "format": "null-co",
+                "actual-size": SIZE
              },
              "iops_wr": 0,
              "ro": false,
author
#103 lore raw

Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size

Quoted 61 lines · Andrey Shinkevich
Quoted 60 lines · Max Reitz

It is trivial, so we might as well do it.

---
  block/null.c               | 7 +++++++
  tests/qemu-iotests/153.out | 2 +-
  tests/qemu-iotests/184.out | 6 ++++--

  3 files changed, 12 insertions(+), 3 deletions(-)
block/null.c +4 -0
index 15e1d56746..cc9b1d4ea7 100644
@@ -262,6 +262,11 @@ static void
 null_refresh_filename(BlockDriverState *bs)
                bs->drv->format_name);
   }
   +static int64_t null_allocated_file_size(BlockDriverState *bs)
+{
+    return 0;
+}
+
   static const char *const null_strong_runtime_opts[] = {
       BLOCK_OPT_SIZE,

      NULL_OPT_ZEROES,

block/null.c +2 -0
@@ -277,6 +282,7 @@ static BlockDriver bdrv_null_co = {
       .bdrv_file_open         = null_file_open,
       .bdrv_parse_filename    = null_co_parse_filename,
       .bdrv_getlength         = null_getlength,
+    .bdrv_get_allocated_file_size = null_allocated_file_size,
         .bdrv_co_preadv         = null_co_preadv,
       .bdrv_co_pwritev        = null_co_pwritev,
@@ -297,6 +303,7 @@ static BlockDriver bdrv_null_aio = {
       .bdrv_file_open         = null_file_open,
       .bdrv_parse_filename    = null_aio_parse_filename,
       .bdrv_getlength         = null_getlength,
+    .bdrv_get_allocated_file_size = null_allocated_file_size,
         .bdrv_aio_preadv        = null_aio_preadv,
       .bdrv_aio_pwritev       = null_aio_pwritev,
tests/qemu-iotests/153.out +0 -0
index b2a90caa6b..8659e6463b 100644
@@ -461,7 +461,7 @@ No conflict:

  image: null-co://
  file format: null-co
  virtual size: 1 GiB (1073741824 bytes)

tests/qemu-iotests/153.out +1 -1
-disk size: unavailable
+disk size: 0 B

    Conflict:
  qemu-img: --force-share/-U conflicts with image options

tests/qemu-iotests/184.out +0 -0
index 3deb3cfb94..28b104da89 100644
@@ -29,7 +29,8 @@ Testing:
               "image": {
                   "virtual-size": 1073741824,
                   "filename": "json:{\"throttle-group\": \"group0\",
 \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
Quoted 8 lines

tests/qemu-iotests/184.out Testing: · line 33

Quoted 3 lines
tests/qemu-iotests/184.out +2 -1
-                "format": "throttle"
+                "format": "throttle",
+                "actual-size": SIZE

If we remove the _filter_generated_node_ids() in the current implementation of the test #184, we will get '"actual-size": 0'. It might be more informative when analyzing the output in 184.out.

You mean _filter_actual_image_size? Yeah, why not, it doesn’t seem necessary here.

Max

#104 lore raw

Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size

Quoted 10 lines · Max Reitz
Quoted 9 lines · Andrey Shinkevich
Quoted 8 lines · Max Reitz

It is trivial, so we might as well do it.

---
   block/null.c               | 7 +++++++
   tests/qemu-iotests/153.out | 2 +-
   tests/qemu-iotests/184.out | 6 ++++--
   3 files changed, 12 insertions(+), 3 deletions(-)

...

Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
tests/qemu-iotests/184.out +0 -0
index 3deb3cfb94..28b104da89 100644
@@ -29,7 +29,8 @@ Testing:
               "image": {
                   "virtual-size": 1073741824,
                   "filename": "json:{\"throttle-group\": \"group0\",
Quoted 8 lines

tests/qemu-iotests/184.out Testing: · line 33

Quoted 4 lines
tests/qemu-iotests/184.out +2 -1
 \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
-                "format": "throttle"
+                "format": "throttle",
+                "actual-size": SIZE

If we remove the _filter_generated_node_ids() in the current implementation of the test #184, we will get '"actual-size": 0'. It might be more informative when analyzing the output in 184.out.

Quoted 5 lines

You mean _filter_actual_image_size? Yeah, why not, it doesn’t seem necessary here.

Max

Yes Max, you are right, I ment the _filter_actual_image_size. It was my copy/paste mistake.

Andrey

author
#105 lore raw

PATCH v7 29/47 blockdev: Use CAF in external_snapshot_prepare()

In reply to Max Reitz

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

---
 blockdev.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +6 -1
index 1eb0fcdea2..aabe51036d 100644
@@ -1549,7 +1549,12 @@ static void external_snapshot_prepare(BlkActionState *common,
         goto out;
     }
 
-    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {
+        error_setg(errp, "Filters cannot be used as overlays");
+        goto out;
+    }
+
+    if (bdrv_cow_child(state->new_bs)) {
         error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
Signature · 1 line
2.26.2
Reviewed-by on v7 29/47
#106 lore raw

Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()

Quoted 14 lines · Max Reitz

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

---
  blockdev.c | 7 ++++++-
  1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +0 -0
index 1eb0fcdea2..aabe51036d 100644

blockdev.c static void external_snapshot_prepare(BlkActionState *common, · line 1552

Quoted 5 lines
blockdev.c +1 -1
@@ -1549,7 +1549,12 @@ static void external_snapshot_prepare(BlkActionState *common,
          goto out;
      }
 
-    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {

Is there a chance to get a filter here? If so, is that when a user specifies the file name of such a kind “filter[filter-name]:foo.qcow2” or somehow else?

Andrey

blockdev.c static void external_snapshot_prepare(BlkActionState *common, · line 1560

Quoted 8 lines
blockdev.c +5 -0
+        error_setg(errp, "Filters cannot be used as overlays");
+        goto out;
+    }
+
+    if (bdrv_cow_child(state->new_bs)) {
          error_setg(errp, "The overlay already has a backing image");
          goto out;
      }
author
#107 lore raw

Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()

Quoted 10 lines · Andrey Shinkevich
Quoted 9 lines · Max Reitz

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

---
  blockdev.c | 7 ++++++-

  1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 15 lines

blockdev.c static void · line 1553

Quoted 10 lines
blockdev.c +1 -0
index 1eb0fcdea2..aabe51036d 100644
@@ -1549,7 +1549,12 @@ static void
 external_snapshot_prepare(BlkActionState *common,
           goto out;
       }
   -    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {

Is there a chance to get a filter here? If so, is that when a user specifies the file name of such a kind “filter[filter-name]:foo.qcow2” or somehow else?

It would be with blockdev-snapshot and by specifying a filter for @overlay. Technically that’s already caught by the check whether the overlay supports backing images (whether drv->supports_backing is true), but we might as well give a nicer error message.

Example:

{"execute":"qmp_capabilities"}

{"execute":"blockdev-add","arguments":
{"node-name":"overlay","driver":"copy-on-read",
"file":{"driver":"null-co"}}}

{"execute":"blockdev-add","arguments":
{"node-name":"base","driver":"null-co"}}

{"execute":"blockdev-snapshot","arguments":
{"node":"base","overlay":"overlay"}}

Max

#108 lore raw

Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()

Quoted 38 lines · Max Reitz
Quoted 10 lines · Andrey Shinkevich
Quoted 9 lines · Max Reitz

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

---
   blockdev.c | 7 ++++++-
   1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 14 lines

blockdev.c static void · line 1553

Quoted 10 lines
blockdev.c +1 -0
index 1eb0fcdea2..aabe51036d 100644
@@ -1549,7 +1549,12 @@ static void
 external_snapshot_prepare(BlkActionState *common,
           goto out;
       }
   -    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {

Is there a chance to get a filter here? If so, is that when a user specifies the file name of such a kind “filter[filter-name]:foo.qcow2” or somehow else?

It would be with blockdev-snapshot and by specifying a filter for @overlay. Technically that’s already caught by the check whether the overlay supports backing images (whether drv->supports_backing is true), but we might as well give a nicer error message.

Example:

{"execute":"qmp_capabilities"}

{"execute":"blockdev-add","arguments":
{"node-name":"overlay","driver":"copy-on-read",
"file":{"driver":"null-co"}}}

Quoted 8 lines

{"execute":"blockdev-add","arguments":
{"node-name":"base","driver":"null-co"}}

{"execute":"blockdev-snapshot","arguments":
{"node":"base","overlay":"overlay"}}

Max

Thank you for the example.

Andrey

author
#109 lore raw

PATCH v7 30/47 block: Report data child for query-blockstats

In reply to Max Reitz

It makes no sense to report the block stats of a purely metadata-storing child in query-blockstats. So if the primary child does not have any data, try to find a unique data-storing child.

---
 block/qapi.c | 31 +++++++++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)
block/qapi.c +29 -2
index 4807a2b344..c57b42d86d 100644
@@ -526,6 +526,7 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk)
 static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
                                         bool blk_level)
 {
+    BdrvChild *parent_child;
     BlockStats *s = NULL;
 
     s = g_malloc0(sizeof(*s));
@@ -555,9 +556,35 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
         s->has_driver_specific = true;
     }
 
-    if (bs->file) {
+    parent_child = bdrv_primary_child(bs);
+    if (!parent_child ||
+        !(parent_child->role & (BDRV_CHILD_DATA | BDRV_CHILD_FILTERED)))
+    {
+        BdrvChild *c;
+
+        /*
+         * Look for a unique data-storing child.  We do not need to look for
+         * filtered children, as there would be only one and it would have been
+         * the primary child.
+         */
+        parent_child = NULL;
+        QLIST_FOREACH(c, &bs->children, next) {
+            if (c->role & BDRV_CHILD_DATA) {
+                if (parent_child) {
+                    /*
+                     * There are multiple data-storing children and we cannot
+                     * choose between them.
+                     */
+                    parent_child = NULL;
+                    break;
+                }
+                parent_child = c;
+            }
+        }
+    }
+    if (parent_child) {
         s->has_parent = true;
-        s->parent = bdrv_query_bds_stats(bs->file->bs, blk_level);
+        s->parent = bdrv_query_bds_stats(parent_child->bs, blk_level);
     }
 
     if (blk_level && bs->backing) {
Signature · 1 line
2.26.2
Reviewed-by on v7 30/47
#110 lore raw

Re: [PATCH v7 30/47] block: Report data child for query-blockstats

Quoted 49 lines · Max Reitz

It makes no sense to report the block stats of a purely metadata-storing child in query-blockstats. So if the primary child does not have any data, try to find a unique data-storing child.

---
  block/qapi.c | 31 +++++++++++++++++++++++++++++--
  1 file changed, 29 insertions(+), 2 deletions(-)
block/qapi.c +24 -1
index 4807a2b344..c57b42d86d 100644
@@ -526,6 +526,7 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk)
  static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
                                          bool blk_level)
  {
+    BdrvChild *parent_child;
      BlockStats *s = NULL;
 
      s = g_malloc0(sizeof(*s));
@@ -555,9 +556,35 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
          s->has_driver_specific = true;
      }
 
-    if (bs->file) {
+    parent_child = bdrv_primary_child(bs);
+    if (!parent_child ||
+        !(parent_child->role & (BDRV_CHILD_DATA | BDRV_CHILD_FILTERED)))
+    {
+        BdrvChild *c;
+
+        /*
+         * Look for a unique data-storing child.  We do not need to look for
+         * filtered children, as there would be only one and it would have been
+         * the primary child.
+         */
+        parent_child = NULL;
+        QLIST_FOREACH(c, &bs->children, next) {
+            if (c->role & BDRV_CHILD_DATA) {
+                if (parent_child) {
+                    /*
+                     * There are multiple data-storing children and we cannot
+                     * choose between them.
+                     */
+                    parent_child = NULL;
+                    break;
+                }
+                parent_child = c;

block/qapi.c static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, · line 590

Quoted 10 lines
block/qapi.c +5 -1
+            }
+        }
+    }
+    if (parent_child) {
          s->has_parent = true;
-        s->parent = bdrv_query_bds_stats(bs->file->bs, blk_level);
+        s->parent = bdrv_query_bds_stats(parent_child->bs, blk_level);
      }
 
      if (blk_level && bs->backing) {
author
#111 lore raw

PATCH v7 31/47 block: Use child access functions for QAPI queries

In reply to Max Reitz

query-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just bs->backing or COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.

---
 block/qapi.c               | 33 ++++++++++++++++++++-------------
 tests/qemu-iotests/184.out |  8 +++++++-
 2 files changed, 27 insertions(+), 14 deletions(-)
block/qapi.c +20 -13
index c57b42d86d..2628323b63 100644
@@ -163,9 +163,13 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
             break;
         }
 
-        if (bs0->drv && bs0->backing) {
+        if (bs0->drv && bdrv_filter_or_cow_child(bs0)) {
+            /*
+             * Put any filtered child here (for backwards compatibility to when
+             * we put bs0->backing here, which might be any filtered child).
+             */
             info->backing_file_depth++;
-            bs0 = bs0->backing->bs;
+            bs0 = bdrv_filter_or_cow_bs(bs0);
             (*p_image_info)->has_backing_image = true;
             p_image_info = &((*p_image_info)->backing_image);
         } else {
@@ -174,9 +178,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
 
         /* Skip automatically inserted nodes that the user isn't aware of for
          * query-block (blk != NULL), but not for query-named-block-nodes */
-        while (blk && bs0->drv && bs0->implicit) {
-            bs0 = backing_bs(bs0);
-            assert(bs0);
+        if (blk) {
+            bs0 = bdrv_skip_implicit_filters(bs0);
         }
     }
 
@@ -362,9 +365,7 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,
     char *qdev;
 
     /* Skip automatically inserted nodes that the user isn't aware of */
-    while (bs && bs->drv && bs->implicit) {
-        bs = backing_bs(bs);
-    }
+    bs = bdrv_skip_implicit_filters(bs);
 
     info->device = g_strdup(blk_name(blk));
     info->type = g_strdup("unknown");
@@ -527,6 +528,7 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
                                         bool blk_level)
 {
     BdrvChild *parent_child;
+    BlockDriverState *filter_or_cow_bs;
     BlockStats *s = NULL;
 
     s = g_malloc0(sizeof(*s));
@@ -539,9 +541,8 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
     /* Skip automatically inserted nodes that the user isn't aware of in
      * a BlockBackend-level command. Stay at the exact node for a node-level
      * command. */
-    while (blk_level && bs->drv && bs->implicit) {
-        bs = backing_bs(bs);
-        assert(bs);
+    if (blk_level) {
+        bs = bdrv_skip_implicit_filters(bs);
     }
 
     if (bdrv_get_node_name(bs)[0]) {
@@ -587,9 +588,15 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
         s->parent = bdrv_query_bds_stats(parent_child->bs, blk_level);
     }
 
-    if (blk_level && bs->backing) {
+    filter_or_cow_bs = bdrv_filter_or_cow_bs(bs);
+    if (blk_level && filter_or_cow_bs) {
+        /*
+         * Put any filtered or COW child here (for backwards
+         * compatibility to when we put bs0->backing here, which might
+         * be either)
+         */
         s->has_backing = true;
-        s->backing = bdrv_query_bds_stats(bs->backing->bs, blk_level);
+        s->backing = bdrv_query_bds_stats(filter_or_cow_bs, blk_level);
     }
 
     return s;
tests/qemu-iotests/184.out +7 -1
index 28b104da89..4e92fcfb51 100644
@@ -27,6 +27,12 @@ Testing:
             "iops_rd": 0,
             "detect_zeroes": "off",
             "image": {
+                "backing-image": {
+                    "virtual-size": 1073741824,
+                    "filename": "null-co://",
+                    "format": "null-co",
+                    "actual-size": SIZE
+                },
                 "virtual-size": 1073741824,
                 "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
                 "format": "throttle",
@@ -35,7 +41,7 @@ Testing:
             "iops_wr": 0,
             "ro": false,
             "node-name": "throttle0",
-            "backing_file_depth": 0,
+            "backing_file_depth": 1,
             "drv": "throttle",
             "iops": 0,
             "bps_wr": 0,
Signature · 1 line
2.26.2
Reviewed-by on v7 31/47
#112 lore raw

Re: [PATCH v7 31/47] block: Use child access functions for QAPI queries

Quoted 11 lines · Max Reitz

query-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just bs->backing or COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.

---
  block/qapi.c               | 33 ++++++++++++++++++++-------------
  tests/qemu-iotests/184.out |  8 +++++++-
  2 files changed, 27 insertions(+), 14 deletions(-)

...

author
#113 lore raw

PATCH v7 32/47 block-copy: Use CAF to find sync=top base

In reply to Max Reitz
---
 block/block-copy.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block/block-copy.c +2 -2
index f7428a7c08..5e80569bb8 100644
@@ -437,8 +437,8 @@ static int block_copy_block_status(BlockCopyState *s, int64_t offset,
     BlockDriverState *base;
     int ret;
 
-    if (s->skip_unallocated && s->source->bs->backing) {
-        base = s->source->bs->backing->bs;
+    if (s->skip_unallocated) {
+        base = bdrv_backing_chain_next(s->source->bs);
     } else {
         base = NULL;
     }
Signature · 1 line
2.26.2
Reviewed-by on v7 32/47
#114 lore raw

Re: [PATCH v7 32/47] block-copy: Use CAF to find sync=top base

Quoted 10 lines · Max Reitz
---
  block/block-copy.c | 4 ++--
  1 file changed, 2 insertions(+), 2 deletions(-)
block/block-copy.c +0 -0
index f7428a7c08..5e80569bb8 100644

block/block-copy.c static int block_copy_block_status(BlockCopyState *s, int64_t offset, · line 444

Quoted 10 lines
block/block-copy.c +2 -2
@@ -437,8 +437,8 @@ static int block_copy_block_status(BlockCopyState *s, int64_t offset,
      BlockDriverState *base;
      int ret;
 
-    if (s->skip_unallocated && s->source->bs->backing) {
-        base = s->source->bs->backing->bs;
+    if (s->skip_unallocated) {
+        base = bdrv_backing_chain_next(s->source->bs);
      } else {
          base = NULL;
      }
author
#115 lore raw

PATCH v7 33/47 mirror: Deal with filters

In reply to Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.

---
 qapi/block-core.json |   6 ++-
 block/mirror.c       | 118 +++++++++++++++++++++++++++++++++----------
 blockdev.c           |  36 +++++++++----
 3 files changed, 121 insertions(+), 39 deletions(-)
qapi/block-core.json +4 -2
index df87855429..0b8ccd30aa 100644
@@ -1943,7 +1943,8 @@
 #
 # @replaces: with sync=full graph node name to be replaced by the new
 #            image when a whole image copy is done. This can be used to repair
-#            broken Quorum files. (Since 2.1)
+#            broken Quorum files.  By default, @device is replaced, although
+#            implicitly created filters on it are kept. (Since 2.1)
 #
 # @mode: whether and how QEMU should create a new image, default is
 #        'absolute-paths'.
@@ -2254,7 +2255,8 @@
 #
 # @replaces: with sync=full graph node name to be replaced by the new
 #            image when a whole image copy is done. This can be used to repair
-#            broken Quorum files.
+#            broken Quorum files.  By default, @device is replaced, although
+#            implicitly created filters on it are kept.
 #
 # @speed:  the maximum speed, in bytes per second
 #
block/mirror.c +91 -27

206 lines not on this page · raw message

blockdev.c +26 -10
index aabe51036d..6ca9b19779 100644
@@ -2886,6 +2886,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
                                    bool has_auto_dismiss, bool auto_dismiss,
                                    Error **errp)
 {
+    BlockDriverState *unfiltered_bs;
     int job_flags = JOB_DEFAULT;
 
     if (!has_speed) {
@@ -2937,10 +2938,19 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
         return;
     }
 
-    if (!bs->backing && sync == MIRROR_SYNC_MODE_TOP) {
+    if (!bdrv_backing_chain_next(bs) && sync == MIRROR_SYNC_MODE_TOP) {
         sync = MIRROR_SYNC_MODE_FULL;
     }
 
+    if (!has_replaces) {
+        /* We want to mirror from @bs, but keep implicit filters on top */
+        unfiltered_bs = bdrv_skip_implicit_filters(bs);
+        if (unfiltered_bs != bs) {
+            replaces = unfiltered_bs->node_name;
+            has_replaces = true;
+        }
+    }
+
     if (has_replaces) {
         BlockDriverState *to_replace_bs;
         AioContext *replace_aio_context;
@@ -2987,7 +2997,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
 void qmp_drive_mirror(DriveMirror *arg, Error **errp)
 {
     BlockDriverState *bs;
-    BlockDriverState *source, *target_bs;
+    BlockDriverState *target_backing_bs, *target_bs;
     AioContext *aio_context;
     AioContext *old_context;
     BlockMirrorBackingMode backing_mode;
@@ -2996,6 +3006,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     int flags;
     int64_t size;
     const char *format = arg->format;
+    const char *replaces_node_name = NULL;
     bool zero_target;
     int ret;
 
@@ -3022,12 +3033,12 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     }
 
     flags = bs->open_flags | BDRV_O_RDWR;
-    source = backing_bs(bs);
-    if (!source && arg->sync == MIRROR_SYNC_MODE_TOP) {
+    target_backing_bs = bdrv_cow_bs(bdrv_skip_filters(bs));
+    if (!target_backing_bs && arg->sync == MIRROR_SYNC_MODE_TOP) {
         arg->sync = MIRROR_SYNC_MODE_FULL;
     }
     if (arg->sync == MIRROR_SYNC_MODE_NONE) {
-        source = bs;
+        target_backing_bs = bs;
     }
 
     size = bdrv_getlength(bs);
@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                              " named node of the graph");
             goto out;
         }
+        replaces_node_name = arg->replaces;
     }
 
     if (arg->mode == NEW_IMAGE_MODE_ABSOLUTE_PATHS) {
@@ -3053,7 +3065,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     /* Don't open backing image in create() */
     flags |= BDRV_O_NO_BACKING;
 
-    if ((arg->sync == MIRROR_SYNC_MODE_FULL || !source)
+    if ((arg->sync == MIRROR_SYNC_MODE_FULL || !target_backing_bs)
         && arg->mode != NEW_IMAGE_MODE_EXISTING)
     {
         /* create new image w/o backing file */
@@ -3061,15 +3073,19 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
         bdrv_img_create(arg->target, format,
                         NULL, NULL, NULL, size, flags, false, &local_err);
     } else {
+        /* Implicit filters should not appear in the filename */
+        BlockDriverState *explicit_backing =
+            bdrv_skip_implicit_filters(target_backing_bs);
+
         switch (arg->mode) {
         case NEW_IMAGE_MODE_EXISTING:
             break;
         case NEW_IMAGE_MODE_ABSOLUTE_PATHS:
             /* create new image with backing file */
-            bdrv_refresh_filename(source);
+            bdrv_refresh_filename(explicit_backing);
             bdrv_img_create(arg->target, format,
-                            source->filename,
-                            source->drv->format_name,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name,
                             NULL, size, flags, false, &local_err);
             break;
         default:
@@ -3119,7 +3135,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     aio_context_acquire(aio_context);
 
     blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
-                           arg->has_replaces, arg->replaces, arg->sync,
+                           !!replaces_node_name, replaces_node_name, arg->sync,
                            backing_mode, zero_target,
                            arg->has_speed, arg->speed,
                            arg->has_granularity, arg->granularity,
Signature · 1 line
2.26.2
#116 lore raw

Re: [PATCH v7 33/47] mirror: Deal with filters

Quoted 14 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.

---
  qapi/block-core.json |   6 ++-
  block/mirror.c       | 118 +++++++++++++++++++++++++++++++++----------
  blockdev.c           |  36 +++++++++----
  3 files changed, 121 insertions(+), 39 deletions(-)

...

Quoted 13 lines
block/mirror.c +1 -0
index 469acf4600..770de3b34e 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
      BlockBackend *target;
      BlockDriverState *mirror_top_bs;
      BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
      /* The name of the graph node to replace */
      char *replaces;

block/mirror.c static int mirror_exit_common(Job *job) · line 683

Quoted 8 lines
block/mirror.c +3 -2
@@ -677,8 +678,10 @@ static int mirror_exit_common(Job *job)
                               &error_abort);
      if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
          BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_filters(target_bs);
+
+        if (bdrv_cow_bs(unfiltered_target) != backing) {

I just worry about a filter node of the concurrent job right below the unfiltered_target. The filter has unfiltered_target in its parent list. Will that filter node be replaced correctly then?

Andrey

...

Quoted 11 lines
block/mirror.c +11 -0
+        /*
+         * The topmost node with
+         * bdrv_skip_filters(filtered_target) == bdrv_skip_filters(target)
+         */
+        filtered_target = bdrv_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_filters(filtered_target) ==
+               bdrv_skip_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block

block/mirror.c static BlockJob *mirror_start_job( · line 1778

Quoted 10 lines
block/mirror.c +10 -0
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filter_or_cow_bs(bs); iter != target;
+             iter = bdrv_filter_or_cow_bs(iter))
+        {
+            if (iter == filtered_target) {

For one filter node only?

Quoted 14 lines
block/mirror.c +8 -2
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
              if (ret < 0) {
                  goto fail;
              }

...

block/mirror.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3056

Quoted 5 lines
block/mirror.c +1 -0
@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                               " named node of the graph");
              goto out;
          }
+        replaces_node_name = arg->replaces;

What is the idea behind the variables substitution?

Probably, the patch might be split out.

Andrey

author
#117 lore raw

Re: [PATCH v7 33/47] mirror: Deal with filters

Quoted 33 lines · Andrey Shinkevich
Quoted 14 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.

---
  qapi/block-core.json |   6 ++-
  block/mirror.c       | 118 +++++++++++++++++++++++++++++++++---------- blockdev.c           |  36 +++++++++----

  3 files changed, 121 insertions(+), 39 deletions(-)

...

Quoted 17 lines
block/mirror.c +1 -2
index 469acf4600..770de3b34e 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
       BlockBackend *target;
       BlockDriverState *mirror_top_bs;
       BlockDriverState *base;
+    BlockDriverState *base_overlay;
         /* The name of the graph node to replace */
       char *replaces;
@@ -677,8 +678,10 @@ static int mirror_exit_common(Job *job)
                                &error_abort);
       if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
           BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
Quoted 8 lines

block/mirror.c static int mirror_exit_common(Job *job) · line 683

Quoted 4 lines
block/mirror.c +3 -0
+        BlockDriverState *unfiltered_target =
 bdrv_skip_filters(target_bs);
+
+        if (bdrv_cow_bs(unfiltered_target) != backing) {

I just worry about a filter node of the concurrent job right below the unfiltered_target.

Having a concurrent job on the target sounds extremely problematic in itself (because at least for most of the mirror job, the target isn’t in a consistent state). Is that a real use case?

Quoted 2 lines

The filter has unfiltered_target in its parent list. Will that filter node be replaced correctly then?

I’m also not quite sure what you mean. We need to attach the source’s backing chain to the target here, so we go down to the first node that might accept COW backing files (by invoking bdrv_skip_filters()). That should be correct no matter what kind of filters are on it.

Quoted 19 lines
Quoted 19 lines
block/mirror.c +3 -0
+        /*
+         * The topmost node with
+         * bdrv_skip_filters(filtered_target) ==

bdrv_skip_filters(target)
+         */
+        filtered_target = bdrv_cow_bs(bdrv_find_overlay(bs, target));

block/mirror.c +7 -0
+
+        assert(bdrv_skip_filters(filtered_target) ==
+               bdrv_skip_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node,

they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target). +         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;

Quoted 8 lines
Quoted 5 lines

+
+        for (iter = bdrv_filter_or_cow_bs(bs); iter != target; +             iter = bdrv_filter_or_cow_bs(iter))
+        {
+            if (iter == filtered_target) {

For one filter node only?

No, iter_shared_perms is never reset, so it retains the BLK_PERM_CONSISTENT_READ flag until the end of the loop.

Quoted 17 lines
Quoted 16 lines

+                /*
+                 * From here on, all nodes are filters on the base. +                 * This allows us to share BLK_PERM_CONSISTENT_READ. +                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ; +            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp); if (ret < 0) {
                  goto fail;
              }

...

Quoted 9 lines

block/mirror.c void qmp_drive_mirror(DriveMirror *arg, Error · line 3057

Quoted 6 lines
block/mirror.c +1 -0
@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error
 **errp)
                                " named node of the graph");
               goto out;
           }
+        replaces_node_name = arg->replaces;

What is the idea behind the variables substitution?

Looks like a remnant from v6, where there was an

if (arg->has_replaces) {
    ...
    replaces_node_name = arg->replaces;
} else if (unfiltered_bs != bs) {
    replaces_node_name = unfiltered_bs->node_name;
}

But I moved that logic to blockdev_mirror_common() in this version.

So it’s just useless now and replaces_node_name shouldn’t exist.

Max

#118 lore raw

Re: [PATCH v7 33/47] mirror: Deal with filters

Quoted 41 lines · Max Reitz
Quoted 32 lines · Andrey Shinkevich
Quoted 14 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.

---
   qapi/block-core.json |   6 ++-
   block/mirror.c       | 118 +++++++++++++++++++++++++++++++++----------
   blockdev.c           |  36 +++++++++----
   3 files changed, 121 insertions(+), 39 deletions(-)

...

Quoted 16 lines
block/mirror.c +1 -1
index 469acf4600..770de3b34e 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
       BlockBackend *target;
       BlockDriverState *mirror_top_bs;
       BlockDriverState *base;
+    BlockDriverState *base_overlay;
         /* The name of the graph node to replace */
       char *replaces;
@@ -677,8 +678,10 @@ static int mirror_exit_common(Job *job)
                                &error_abort);
       if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
           BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
Quoted 8 lines

block/mirror.c static int mirror_exit_common(Job *job) · line 683

Quoted 5 lines
block/mirror.c +3 -1
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target =
 bdrv_skip_filters(target_bs);
+
+        if (bdrv_cow_bs(unfiltered_target) != backing) {

I just worry about a filter node of the concurrent job right below the unfiltered_target.

Quoted 3 lines

Having a concurrent job on the target sounds extremely problematic in itself (because at least for most of the mirror job, the target isn’t in a consistent state). Is that a real use case?

It might be at the TestParallelOps of iotests #30 but I am not sure now. I am going to apply my series with copy-on-read filter for the stream job above this one and will see then.

Andrey

Quoted 7 lines
Quoted 3 lines

The filter has unfiltered_target in its parent list. Will that filter node be replaced correctly then?

I’m also not quite sure what you mean. We need to attach the source’s backing chain to the target here, so we go down to the first node that might accept COW backing files (by invoking bdrv_skip_filters()). That should be correct no matter what kind of filters are on it.

I ment when a filter is removed with the bdrv_replace_node() afterwards. As I mentioned above, I am going to test the case later.

Andrey

Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
block/mirror.c +3 -0
+        /*
+         * The topmost node with
+         * bdrv_skip_filters(filtered_target) ==

bdrv_skip_filters(target)
+         */
+        filtered_target = bdrv_cow_bs(bdrv_find_overlay(bs, target));

block/mirror.c +7 -0
+
+        assert(bdrv_skip_filters(filtered_target) ==
+               bdrv_skip_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node,

they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target). +         */

Quoted 8 lines
Quoted 6 lines

+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE; +
+        for (iter = bdrv_filter_or_cow_bs(bs); iter != target; +             iter = bdrv_filter_or_cow_bs(iter))
+        {
+            if (iter == filtered_target) {

For one filter node only?

Quoted 2 lines

No, iter_shared_perms is never reset, so it retains the BLK_PERM_CONSISTENT_READ flag until the end of the loop.

Yes, that's right. Clear.

Andrey

Quoted 41 lines
Quoted 18 lines
Quoted 17 lines

+                /*
+                 * From here on, all nodes are filters on the base. +                 * This allows us to share BLK_PERM_CONSISTENT_READ. +                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ; +            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate
node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
              if (ret < 0) {
                  goto fail;
              }

...

Quoted 8 lines

block/mirror.c void qmp_drive_mirror(DriveMirror *arg, Error · line 3057

Quoted 6 lines
block/mirror.c +1 -0
@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error
 **errp)
                                " named node of the graph");
               goto out;
           }
+        replaces_node_name = arg->replaces;

What is the idea behind the variables substitution?

Looks like a remnant from v6, where there was an

if (arg->has_replaces) {
     ...
     replaces_node_name = arg->replaces;
} else if (unfiltered_bs != bs) {
     replaces_node_name = unfiltered_bs->node_name;
}

But I moved that logic to blockdev_mirror_common() in this version.

So it’s just useless now and replaces_node_name shouldn’t exist.

Max

#119 lore raw

Re: [PATCH v7 33/47] mirror: Deal with filters

In reply to Max Reitz

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 10 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.

block/mirror.c static BlockJob *mirror_start_job( · line 1727

Quoted 7 lines
block/mirror.c +1 -0
@@ -1682,6 +1721,7 @@ static BlockJob *mirror_start_job(
     s->zero_target = zero_target;
     s->copy_mode = copy_mode;
     s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
     s->granularity = granularity;
     s->buf_size = ROUND_UP(buf_size, granularity);
     s->unmap = unmap;

Is this valid without freezing the links between base_overlay and base?

Actually, I guess we should freeze everything between bs and base (for base != NULL) and it's a preexisting problem that just happens to affect this code, too.

Or maybe freezing everything is too much. We only want to make sure that no non-filter is inserted between base and base_overlay and that base (and now base_overlay) always stay in the backing chain of bs. But what options apart from freezing do we have to achieve this?

Why is using base_overlay even better than using base? Assuming there is a good reason, maybe the commit message could spell it out.

Kevin

author
#120 lore raw

Re: [PATCH v7 33/47] mirror: Deal with filters

Quoted 10 lines · Kevin Wolf

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 8 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.

Quoted 10 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1727

Quoted 8 lines
block/mirror.c +1 -0
@@ -1682,6 +1721,7 @@ static BlockJob *mirror_start_job(
     s->zero_target = zero_target;
     s->copy_mode = copy_mode;
     s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
     s->granularity = granularity;
     s->buf_size = ROUND_UP(buf_size, granularity);
     s->unmap = unmap;

Is this valid without freezing the links between base_overlay and base?

Er...

Quoted 3 lines

Actually, I guess we should freeze everything between bs and base (for base != NULL) and it's a preexisting problem that just happens to affect this code, too.

Yes, that’s how it looks to me, too. I don’t think that has anything to do with this patch.

Quoted 4 lines

Or maybe freezing everything is too much. We only want to make sure that no non-filter is inserted between base and base_overlay and that base (and now base_overlay) always stay in the backing chain of bs. But what options apart from freezing do we have to achieve this?

I don’t know of any, and I don’t know whether anyone would actually care if we were to just freeze everything.

Quoted 2 lines

Why is using base_overlay even better than using base? Assuming there is a good reason, maybe the commit message could spell it out.

The problem is that querying the block status for a filter node falls through to the underlying data-carrying node. So if there’s a filter on top of @base, and we query for is_allocated_above above @base, then we’ll include @base, which we do not want.

Max

author
#121 lore raw

PATCH v7 34/47 backup: Deal with filters

In reply to Max Reitz
---
 block/backup-top.c |  2 +-
 block/backup.c     |  9 +++++----
 blockdev.c         | 19 +++++++++++++++----
 3 files changed, 21 insertions(+), 9 deletions(-)
block/backup-top.c +1 -1
index f304df8f26..89bd3937d0 100644
@@ -291,7 +291,7 @@ void bdrv_backup_top_drop(BlockDriverState *bs)
 
     s->active = false;
     bdrv_child_refresh_perms(bs, bs->backing, &error_abort);
-    bdrv_replace_node(bs, backing_bs(bs), &error_abort);
+    bdrv_replace_node(bs, bs->backing->bs, &error_abort);
     bdrv_set_backing_hd(bs, NULL, &error_abort);
 
     bdrv_drained_end(bs);
block/backup.c +5 -4
index 4f13bb20a5..9afa0bf3b4 100644
@@ -297,6 +297,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
 {
     int ret;
     BlockDriverInfo bdi;
+    bool target_does_cow = bdrv_backing_chain_next(target);
 
     /*
      * If there is no backing file on the target, we cannot rely on COW if our
@@ -304,7 +305,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
      * targets with a backing file, try to avoid COW if possible.
      */
     ret = bdrv_get_info(target, &bdi);
-    if (ret == -ENOTSUP && !target->backing) {
+    if (ret == -ENOTSUP && !target_does_cow) {
         /* Cluster size is not defined */
         warn_report("The target block device doesn't provide "
                     "information about the block size and it doesn't have a "
@@ -313,14 +314,14 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
                     "this default, the backup may be unusable",
                     BACKUP_CLUSTER_SIZE_DEFAULT);
         return BACKUP_CLUSTER_SIZE_DEFAULT;
-    } else if (ret < 0 && !target->backing) {
+    } else if (ret < 0 && !target_does_cow) {
         error_setg_errno(errp, -ret,
             "Couldn't determine the cluster size of the target image, "
             "which has no backing file");
         error_append_hint(errp,
             "Aborting, since this may create an unusable destination image\n");
         return ret;
-    } else if (ret < 0 && target->backing) {
+    } else if (ret < 0 && target_does_cow) {
         /* Not fatal; just trudge on ahead. */
         return BACKUP_CLUSTER_SIZE_DEFAULT;
     }
@@ -371,7 +372,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    if (compress && !block_driver_can_compress(target->drv)) {
+    if (compress && !bdrv_supports_compressed_writes(target)) {
         error_setg(errp, "Compression is not supported for this drive %s",
                    bdrv_get_device_name(target));
         return NULL;
blockdev.c +15 -4
index 6ca9b19779..9ce99b9cbc 100644
@@ -1728,7 +1728,13 @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
      * on top of.
      */
     if (backup->sync == MIRROR_SYNC_MODE_TOP) {
-        source = backing_bs(bs);
+        /*
+         * Backup will not replace the source by the target, so none
+         * of the filters skipped here will be removed (in contrast to
+         * mirror).  Therefore, we can skip all of them when looking
+         * for the first COW relationship.
+         */
+        source = bdrv_cow_bs(bdrv_skip_filters(bs));
         if (!source) {
             backup->sync = MIRROR_SYNC_MODE_FULL;
         }
@@ -1748,9 +1754,14 @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)
     if (backup->mode != NEW_IMAGE_MODE_EXISTING) {
         assert(backup->format);
         if (source) {
-            bdrv_refresh_filename(source);
-            bdrv_img_create(backup->target, backup->format, source->filename,
-                            source->drv->format_name, NULL,
+            /* Implicit filters should not appear in the filename */
+            BlockDriverState *explicit_backing =
+                bdrv_skip_implicit_filters(source);
+
+            bdrv_refresh_filename(explicit_backing);
+            bdrv_img_create(backup->target, backup->format,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name, NULL,
                             size, flags, false, &local_err);
         } else {
             bdrv_img_create(backup->target, backup->format, NULL, NULL, NULL,
Signature · 1 line
2.26.2
Reviewed-by on v7 34/47
#122 lore raw

Re: [PATCH v7 34/47] backup: Deal with filters

Quoted 13 lines · Max Reitz
---
  block/backup-top.c |  2 +-
  block/backup.c     |  9 +++++----
  blockdev.c         | 19 +++++++++++++++----
  3 files changed, 21 insertions(+), 9 deletions(-)
block/backup.c +0 -0
index 4f13bb20a5..9afa0bf3b4 100644

block/backup.c static int64_t backup_calculate_cluster_size(BlockDriverState *target, · line 301

Quoted 5 lines
block/backup.c +1 -0
@@ -297,6 +297,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
  {
      int ret;
      BlockDriverInfo bdi;
+    bool target_does_cow = bdrv_backing_chain_next(target);
 

Wouldn't it better to make the explicit type conversion or use "!!" approach?

Andrey

Quoted 9 lines
block/backup.c +1 -1
      /*
       * If there is no backing file on the target, we cannot rely on COW if our
@@ -304,7 +305,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
       * targets with a backing file, try to avoid COW if possible.
       */
      ret = bdrv_get_info(target, &bdi);
-    if (ret == -ENOTSUP && !target->backing) {
+    if (ret == -ENOTSUP && !target_does_cow) {
          /* Cluster size is not defined */

...

author
#123 lore raw

Re: [PATCH v7 34/47] backup: Deal with filters

Quoted 9 lines · Andrey Shinkevich
Quoted 8 lines · Max Reitz

---
  block/backup-top.c |  2 +-
  block/backup.c     |  9 +++++----
  blockdev.c         | 19 +++++++++++++++----

  3 files changed, 21 insertions(+), 9 deletions(-)
Quoted 15 lines
Quoted 11 lines
block/backup.c +1 -0
index 4f13bb20a5..9afa0bf3b4 100644
@@ -297,6 +297,7 @@ static int64_t
 backup_calculate_cluster_size(BlockDriverState *target,
   {
       int ret;
       BlockDriverInfo bdi;
+    bool target_does_cow = bdrv_backing_chain_next(target);

Wouldn't it better to make the explicit type conversion or use "!!" approach?

I don’t know. O:)

I personally don’t like too may exclamation marks because I feel like the code is screaming at me. So I tend to use them only where necessary.

As for doing an explicit cast... I think I’ll keep that in mind to reduce my future use of !!. But in this case, the type name is in the same line, so I feel like it’s sufficiently clear.

Max

author
#124 lore raw

PATCH v7 35/47 commit: Deal with filters

In reply to Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
 block/block-backend.c          |  9 +++-
 block/commit.c                 | 96 +++++++++++++++++++++++++---------
 block/monitor/block-hmp-cmds.c |  2 +-
 blockdev.c                     |  4 +-
 4 files changed, 81 insertions(+), 30 deletions(-)
block/block-backend.c +7 -2
index 6936b25c83..7f2c7dbccc 100644
@@ -2271,8 +2271,13 @@ int blk_commit_all(void)
         AioContext *aio_context = blk_get_aio_context(blk);
 
         aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;
+            int ret;
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            ret = bdrv_commit(non_filter);
             if (ret < 0) {
                 aio_context_release(aio_context);
                 return ret;
block/commit.c +70 -26

173 lines not on this page · raw message

block/monitor/block-hmp-cmds.c +1 -1
index 4c8c375172..4d3db5ed3c 100644
@@ -217,7 +217,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)
             return;
         }
 
-        bs = blk_bs(blk);
+        bs = bdrv_skip_implicit_filters(blk_bs(blk));
         aio_context = bdrv_get_aio_context(bs);
         aio_context_acquire(aio_context);
 
blockdev.c +3 -1
index 9ce99b9cbc..402f1d1df1 100644
@@ -2690,7 +2690,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
 
     assert(bdrv_get_aio_context(base_bs) == aio_context);
 
-    for (iter = top_bs; iter != backing_bs(base_bs); iter = backing_bs(iter)) {
+    for (iter = top_bs; iter != bdrv_filter_or_cow_bs(base_bs);
+         iter = bdrv_filter_or_cow_bs(iter))
+    {
         if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_COMMIT_TARGET, errp)) {
             goto out;
         }
Signature · 1 line
2.26.2
Reviewed-by on v7 35/47
#125 lore raw

Re: [PATCH v7 35/47] commit: Deal with filters

Quoted 11 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
  block/block-backend.c          |  9 +++-
  block/commit.c                 | 96 +++++++++++++++++++++++++---------
  block/monitor/block-hmp-cmds.c |  2 +-
  blockdev.c                     |  4 +-
  4 files changed, 81 insertions(+), 30 deletions(-)

...

Quoted 18 lines
block/commit.c +18 -0
+    /*
+     * Block all nodes between top and base, because they will
+     * disappear from the chain after this operation.
+     * Note that this assumes that the user is fine with removing all
+     * nodes (including R/W filters) between top and base.  Assuring
+     * this is the responsibility of the interface (i.e. whoever calls
+     * commit_start()).
+     */
+    s->base_overlay = bdrv_find_overlay(top, base);
+    assert(s->base_overlay);
+
+    /*
+     * The topmost node with
+     * bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base)
+     */
+    filtered_base = bdrv_cow_bs(s->base_overlay);
+    assert(bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base));
+

block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 352

Quoted 10 lines
block/commit.c +10 -0
+    /*
+     * XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
+     * at s->base (if writes are blocked for a node, they are also blocked
+     * for its backing file). The other options would be a second filter
+     * driver above s->base.
+     */
+    iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+    for (iter = top; iter != base; iter = bdrv_filter_or_cow_bs(iter)) {
+        if (iter == filtered_base) {

The question same to mirroring:

in case of multiple filters, one above another, the permission is extended for the filtered_base only.

Andrey

Quoted 14 lines
block/commit.c +8 -2
+            /*
+             * From here on, all nodes are filters on the base.  This
+             * allows us to share BLK_PERM_CONSISTENT_READ.
+             */
+            iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+        }
+
          ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                 BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                 errp);
+                                 iter_shared_perms, errp);
          if (ret < 0) {
              goto fail;
          }

...

#126 lore raw

Re: [PATCH v7 35/47] commit: Deal with filters

Quoted 50 lines · Andrey Shinkevich
Quoted 11 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
  block/block-backend.c          |  9 +++-
  block/commit.c                 | 96 +++++++++++++++++++++++++--------- block/monitor/block-hmp-cmds.c |  2 +-
  blockdev.c                     |  4 +-

  4 files changed, 81 insertions(+), 30 deletions(-)

...

Quoted 25 lines

+    /*
+     * Block all nodes between top and base, because they will +     * disappear from the chain after this operation.
+     * Note that this assumes that the user is fine with removing all +     * nodes (including R/W filters) between top and base. Assuring +     * this is the responsibility of the interface (i.e. whoever calls +     * commit_start()).
+     */
+    s->base_overlay = bdrv_find_overlay(top, base);
+    assert(s->base_overlay);
+
+    /*
+     * The topmost node with
+     * bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base) +     */
+    filtered_base = bdrv_cow_bs(s->base_overlay);
+    assert(bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base));
+
+    /*
+     * XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
+     * at s->base (if writes are blocked for a node, they are also blocked
+     * for its backing file). The other options would be a second

Quoted 8 lines

filter
+     * driver above s->base.
+     */
+    iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE; +
+    for (iter = top; iter != base; iter = bdrv_filter_or_cow_bs(iter)) {
+        if (iter == filtered_base) {

The question same to mirroring:

Quoted 4 lines

in case of multiple filters, one above another, the permission is extended for the filtered_base only.

Andrey

The question has been answered already.

Andrey

Quoted 23 lines
Quoted 16 lines

+            /*
+             * From here on, all nodes are filters on the base.  This +             * allows us to share BLK_PERM_CONSISTENT_READ.
+             */
+            iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+        }
+
          ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0, -                                 BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                 errp);
+                                 iter_shared_perms, errp);
          if (ret < 0) {
              goto fail;
          }

...

#127 lore raw

Re: [PATCH v7 35/47] commit: Deal with filters

In reply to Max Reitz

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 16 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
 block/block-backend.c          |  9 +++-
 block/commit.c                 | 96 +++++++++++++++++++++++++---------
 block/monitor/block-hmp-cmds.c |  2 +-
 blockdev.c                     |  4 +-
 4 files changed, 81 insertions(+), 30 deletions(-)
block/block-backend.c +0 -0
index 6936b25c83..7f2c7dbccc 100644

block/block-backend.c int blk_commit_all(void) · line 2275

Quoted 5 lines
block/block-backend.c +0 -2
@@ -2271,8 +2271,13 @@ int blk_commit_all(void)
         AioContext *aio_context = blk_get_aio_context(blk);
 
         aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);

The old code didn't try to commit nodes that don't have a backing file.

block/block-backend.c int blk_commit_all(void) · line 2280

Quoted 7 lines
block/block-backend.c +7 -0
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;
+            int ret;
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            ret = bdrv_commit(non_filter);

The new one tries unconditionally. For nodes without a backing file, bdrv_commit() will return -ENOTSUP, so the whole function fails.

(First real bug at patch 35. I almost thought I wouldn't find any!)

Quoted 8 lines
block/block-backend.c +0 -0
             if (ret < 0) {
                 aio_context_release(aio_context);
                 return ret;
block/commit.c +0 -0
index 7732d02dfe..4122b6736d 100644

block/commit.c typedef struct CommitBlockJob { · line 43

Quoted 7 lines
block/commit.c +1 -0
@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {
     BlockBackend *top;
     BlockBackend *base;
     BlockDriverState *base_bs;
+    BlockDriverState *base_overlay;
     BlockdevOnError on_error;
     bool base_read_only;
     bool chain_frozen;

Hm, again this mysterious base_overlay. I know that stream introduced it to avoid freezing the link to base, but commit doesn't seem to do that.

Is it to avoid using the block status of filter drivers between base_overlay and base? If so, I guess that goes back to the question I raised earlier in this series: What is the block status supposed to tell for filter nodes?

But anyway, in contrast to mirror, commit actually freezes the chain between commit_top_bs and base, so it should be safe at least.

Quoted 41 lines
block/commit.c +12 -3
@@ -89,7 +90,7 @@ static void commit_abort(Job *job)
      * XXX Can (or should) we somehow keep 'consistent read' blocked even
      * after the failed/cancelled commit job is gone? If we already wrote
      * something to base, the intermediate images aren't valid any more. */
-    bdrv_replace_node(s->commit_top_bs, backing_bs(s->commit_top_bs),
+    bdrv_replace_node(s->commit_top_bs, s->commit_top_bs->backing->bs,
                       &error_abort);
 
     bdrv_unref(s->commit_top_bs);
@@ -153,7 +154,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)
             break;
         }
         /* Copy if allocated above the base */
-        ret = bdrv_is_allocated_above(blk_bs(s->top), blk_bs(s->base), false,
+        ret = bdrv_is_allocated_above(blk_bs(s->top), s->base_overlay, true,
                                       offset, COMMIT_BUFFER_SIZE, &n);
         copy = (ret == 1);
         trace_commit_one_iteration(s, offset, n, ret);
@@ -253,15 +254,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     CommitBlockJob *s;
     BlockDriverState *iter;
     BlockDriverState *commit_top_bs = NULL;
+    BlockDriverState *filtered_base;
     Error *local_err = NULL;
+    int64_t base_size, top_size;
+    uint64_t perms, iter_shared_perms;
     int ret;
 
     assert(top != bs);
-    if (top == base) {
+    if (bdrv_skip_filters(top) == bdrv_skip_filters(base)) {
         error_setg(errp, "Invalid files for merge: top and base are the same");
         return;
     }
 
+    base_size = bdrv_getlength(base);
+    if (base_size < 0) {
+        error_setg_errno(errp, -base_size, "Could not inquire base image size");
+        return;
+    }
+

block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 284

Quoted 10 lines
block/commit.c +10 -0
+    top_size = bdrv_getlength(top);
+    if (top_size < 0) {
+        error_setg_errno(errp, -top_size, "Could not inquire top image size");
+        return;
+    }
+
+    perms = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE;
+    if (base_size < top_size) {
+        perms |= BLK_PERM_RESIZE;
+    }

base_perms would indicate which permissions these are (particularly because it's not the next thing that requires permissions, but only used further down the function).

Quoted 105 lines
block/commit.c +44 -19
     s = block_job_create(job_id, &commit_job_driver, NULL, bs, 0, BLK_PERM_ALL,
                          speed, creation_flags, NULL, NULL, errp);
     if (!s) {
@@ -301,17 +322,43 @@ void commit_start(const char *job_id, BlockDriverState *bs,
 
     s->commit_top_bs = commit_top_bs;
 
-    /* Block all nodes between top and base, because they will
-     * disappear from the chain after this operation. */
-    assert(bdrv_chain_contains(top, base));
-    for (iter = top; iter != base; iter = backing_bs(iter)) {
-        /* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
-         * at s->base (if writes are blocked for a node, they are also blocked
-         * for its backing file). The other options would be a second filter
-         * driver above s->base. */
+    /*
+     * Block all nodes between top and base, because they will
+     * disappear from the chain after this operation.
+     * Note that this assumes that the user is fine with removing all
+     * nodes (including R/W filters) between top and base.  Assuring
+     * this is the responsibility of the interface (i.e. whoever calls
+     * commit_start()).
+     */
+    s->base_overlay = bdrv_find_overlay(top, base);
+    assert(s->base_overlay);
+
+    /*
+     * The topmost node with
+     * bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base)
+     */
+    filtered_base = bdrv_cow_bs(s->base_overlay);
+    assert(bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base));
+
+    /*
+     * XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
+     * at s->base (if writes are blocked for a node, they are also blocked
+     * for its backing file). The other options would be a second filter
+     * driver above s->base.
+     */
+    iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+    for (iter = top; iter != base; iter = bdrv_filter_or_cow_bs(iter)) {
+        if (iter == filtered_base) {
+            /*
+             * From here on, all nodes are filters on the base.  This
+             * allows us to share BLK_PERM_CONSISTENT_READ.
+             */
+            iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+        }
+
         ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                 BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                 errp);
+                                 iter_shared_perms, errp);
         if (ret < 0) {
             goto fail;
         }
@@ -328,9 +375,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     }
 
     s->base = blk_new(s->common.job.aio_context,
-                      BLK_PERM_CONSISTENT_READ
-                      | BLK_PERM_WRITE
-                      | BLK_PERM_RESIZE,
+                      perms,
                       BLK_PERM_CONSISTENT_READ
                       | BLK_PERM_GRAPH_MOD
                       | BLK_PERM_WRITE_UNCHANGED);
@@ -398,19 +443,22 @@ int bdrv_commit(BlockDriverState *bs)
     if (!drv)
         return -ENOMEDIUM;
 
-    if (!bs->backing) {
+    backing_file_bs = bdrv_cow_bs(bs);
+
+    if (!backing_file_bs) {
         return -ENOTSUP;
     }
 
     if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_COMMIT_SOURCE, NULL) ||
-        bdrv_op_is_blocked(bs->backing->bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL)) {
+        bdrv_op_is_blocked(backing_file_bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL))
+    {
         return -EBUSY;
     }
 
-    ro = bs->backing->bs->read_only;
+    ro = backing_file_bs->read_only;
 
     if (ro) {
-        if (bdrv_reopen_set_read_only(bs->backing->bs, false, NULL)) {
+        if (bdrv_reopen_set_read_only(backing_file_bs, false, NULL)) {
             return -EACCES;
         }
     }
@@ -428,8 +476,6 @@ int bdrv_commit(BlockDriverState *bs)
     }
 
     /* Insert commit_top block node above backing, so we can write to it */
-    backing_file_bs = backing_bs(bs);
-
     commit_top_bs = bdrv_new_open_driver(&bdrv_commit_top, NULL, BDRV_O_RDWR,
                                          &local_err);
     if (commit_top_bs == NULL) {

block/commit.c ro_cleanup: · line 564

Quoted 7 lines
block/commit.c +1 -3
@@ -515,15 +561,13 @@ ro_cleanup:
     qemu_vfree(buf);
 
     blk_unref(backing);
-    if (backing_file_bs) {
-        bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);
-    }
+    bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);

This means that bdrv_set_backing_hd() is now called to undo a change that hasn't even been made yet. This fails (with &error_abort) if the backing chain is frozen.

On the other hand, the other bdrv_set_backing_hd() calls in the same function would fail the same way.

block/commit.c ro_cleanup: · line 573

Quoted 10 lines
block/commit.c +1 -1
     bdrv_unref(commit_top_bs);
     blk_unref(src);
 
     if (ro) {
         /* ignoring error return here */
-        bdrv_reopen_set_read_only(bs->backing->bs, true, NULL);
+        bdrv_reopen_set_read_only(backing_file_bs, true, NULL);
     }
 
     return ret;

Kevin

author
#128 lore raw

Re: [PATCH v7 35/47] commit: Deal with filters

Quoted 33 lines · Kevin Wolf

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 16 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
 block/block-backend.c          |  9 +++-
 block/commit.c                 | 96 +++++++++++++++++++++++++---------
 block/monitor/block-hmp-cmds.c |  2 +-
 blockdev.c                     |  4 +-
 4 files changed, 81 insertions(+), 30 deletions(-)
block/block-backend.c +0 -0
index 6936b25c83..7f2c7dbccc 100644

block/block-backend.c int blk_commit_all(void) · line 2275

Quoted 5 lines
block/block-backend.c +0 -2
@@ -2271,8 +2271,13 @@ int blk_commit_all(void)
         AioContext *aio_context = blk_get_aio_context(blk);
 
         aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);

The old code didn't try to commit nodes that don't have a backing file.

block/block-backend.c int blk_commit_all(void) · line 2280

Quoted 7 lines
block/block-backend.c +7 -0
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;
+            int ret;
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            ret = bdrv_commit(non_filter);
Quoted 2 lines

The new one tries unconditionally. For nodes without a backing file, bdrv_commit() will return -ENOTSUP, so the whole function fails.

:(

Hm. Should I fix it by checking for bdrv_cow_bs(bdrv_skip_implicit_filters())? Or bdrv_backing_chain_next() and change the bdrv_skip_implicit_filters() to a bdrv_skip_filters()? I feel like that would make even more sense.

Quoted 1 line

(First real bug at patch 35. I almost thought I wouldn't find any!)

:)

Quoted 19 lines
Quoted 8 lines
block/block-backend.c +0 -0
             if (ret < 0) {
                 aio_context_release(aio_context);
                 return ret;
block/commit.c +0 -0
index 7732d02dfe..4122b6736d 100644

block/commit.c typedef struct CommitBlockJob { · line 43

Quoted 7 lines
block/commit.c +1 -0
@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {
     BlockBackend *top;
     BlockBackend *base;
     BlockDriverState *base_bs;
+    BlockDriverState *base_overlay;
     BlockdevOnError on_error;
     bool base_read_only;
     bool chain_frozen;

Hm, again this mysterious base_overlay. I know that stream introduced it to avoid freezing the link to base, but commit doesn't seem to do that.

Quoted 2 lines

Is it to avoid using the block status of filter drivers between base_overlay and base?

Yes.

Quoted 3 lines

If so, I guess that goes back to the question I
raised earlier in this series: What is the block status supposed to tell for filter nodes?

Honestly, I would really like to get away without having to answer that question in this series. Intuitively, I feel like falling through to the next data-bearing layer is not something most callers want. But I’d rather investigate that question separately from this series (even though that likely means we’ll never do it), and just treat it as it is in this series.

Quoted 50 lines

But anyway, in contrast to mirror, commit actually freezes the chain between commit_top_bs and base, so it should be safe at least.

Quoted 47 lines
block/commit.c +18 -3
@@ -89,7 +90,7 @@ static void commit_abort(Job *job)
      * XXX Can (or should) we somehow keep 'consistent read' blocked even
      * after the failed/cancelled commit job is gone? If we already wrote
      * something to base, the intermediate images aren't valid any more. */
-    bdrv_replace_node(s->commit_top_bs, backing_bs(s->commit_top_bs),
+    bdrv_replace_node(s->commit_top_bs, s->commit_top_bs->backing->bs,
                       &error_abort);
 
     bdrv_unref(s->commit_top_bs);
@@ -153,7 +154,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)
             break;
         }
         /* Copy if allocated above the base */
-        ret = bdrv_is_allocated_above(blk_bs(s->top), blk_bs(s->base), false,
+        ret = bdrv_is_allocated_above(blk_bs(s->top), s->base_overlay, true,
                                       offset, COMMIT_BUFFER_SIZE, &n);
         copy = (ret == 1);
         trace_commit_one_iteration(s, offset, n, ret);
@@ -253,15 +254,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     CommitBlockJob *s;
     BlockDriverState *iter;
     BlockDriverState *commit_top_bs = NULL;
+    BlockDriverState *filtered_base;
     Error *local_err = NULL;
+    int64_t base_size, top_size;
+    uint64_t perms, iter_shared_perms;
     int ret;
 
     assert(top != bs);
-    if (top == base) {
+    if (bdrv_skip_filters(top) == bdrv_skip_filters(base)) {
         error_setg(errp, "Invalid files for merge: top and base are the same");
         return;
     }
 
+    base_size = bdrv_getlength(base);
+    if (base_size < 0) {
+        error_setg_errno(errp, -base_size, "Could not inquire base image size");
+        return;
+    }
+
+    top_size = bdrv_getlength(top);
+    if (top_size < 0) {
+        error_setg_errno(errp, -top_size, "Could not inquire top image size");
+        return;
+    }
+
Quoted 8 lines

block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 284

Quoted 4 lines
block/commit.c +4 -0
+    perms = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE;
+    if (base_size < top_size) {
+        perms |= BLK_PERM_RESIZE;
+    }

base_perms would indicate which permissions these are (particularly because it's not the next thing that requires permissions, but only used further down the function).

%s/\<perms\>/base_perms/? Sure.

Quoted 117 lines
Quoted 105 lines
block/commit.c +44 -19
     s = block_job_create(job_id, &commit_job_driver, NULL, bs, 0, BLK_PERM_ALL,
                          speed, creation_flags, NULL, NULL, errp);
     if (!s) {
@@ -301,17 +322,43 @@ void commit_start(const char *job_id, BlockDriverState *bs,
 
     s->commit_top_bs = commit_top_bs;
 
-    /* Block all nodes between top and base, because they will
-     * disappear from the chain after this operation. */
-    assert(bdrv_chain_contains(top, base));
-    for (iter = top; iter != base; iter = backing_bs(iter)) {
-        /* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
-         * at s->base (if writes are blocked for a node, they are also blocked
-         * for its backing file). The other options would be a second filter
-         * driver above s->base. */
+    /*
+     * Block all nodes between top and base, because they will
+     * disappear from the chain after this operation.
+     * Note that this assumes that the user is fine with removing all
+     * nodes (including R/W filters) between top and base.  Assuring
+     * this is the responsibility of the interface (i.e. whoever calls
+     * commit_start()).
+     */
+    s->base_overlay = bdrv_find_overlay(top, base);
+    assert(s->base_overlay);
+
+    /*
+     * The topmost node with
+     * bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base)
+     */
+    filtered_base = bdrv_cow_bs(s->base_overlay);
+    assert(bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base));
+
+    /*
+     * XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
+     * at s->base (if writes are blocked for a node, they are also blocked
+     * for its backing file). The other options would be a second filter
+     * driver above s->base.
+     */
+    iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+    for (iter = top; iter != base; iter = bdrv_filter_or_cow_bs(iter)) {
+        if (iter == filtered_base) {
+            /*
+             * From here on, all nodes are filters on the base.  This
+             * allows us to share BLK_PERM_CONSISTENT_READ.
+             */
+            iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+        }
+
         ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                 BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                 errp);
+                                 iter_shared_perms, errp);
         if (ret < 0) {
             goto fail;
         }
@@ -328,9 +375,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     }
 
     s->base = blk_new(s->common.job.aio_context,
-                      BLK_PERM_CONSISTENT_READ
-                      | BLK_PERM_WRITE
-                      | BLK_PERM_RESIZE,
+                      perms,
                       BLK_PERM_CONSISTENT_READ
                       | BLK_PERM_GRAPH_MOD
                       | BLK_PERM_WRITE_UNCHANGED);
@@ -398,19 +443,22 @@ int bdrv_commit(BlockDriverState *bs)
     if (!drv)
         return -ENOMEDIUM;
 
-    if (!bs->backing) {
+    backing_file_bs = bdrv_cow_bs(bs);
+
+    if (!backing_file_bs) {
         return -ENOTSUP;
     }
 
     if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_COMMIT_SOURCE, NULL) ||
-        bdrv_op_is_blocked(bs->backing->bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL)) {
+        bdrv_op_is_blocked(backing_file_bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL))
+    {
         return -EBUSY;
     }
 
-    ro = bs->backing->bs->read_only;
+    ro = backing_file_bs->read_only;
 
     if (ro) {
-        if (bdrv_reopen_set_read_only(bs->backing->bs, false, NULL)) {
+        if (bdrv_reopen_set_read_only(backing_file_bs, false, NULL)) {
             return -EACCES;
         }
     }
@@ -428,8 +476,6 @@ int bdrv_commit(BlockDriverState *bs)
     }
 
     /* Insert commit_top block node above backing, so we can write to it */
-    backing_file_bs = backing_bs(bs);
-
     commit_top_bs = bdrv_new_open_driver(&bdrv_commit_top, NULL, BDRV_O_RDWR,
                                          &local_err);
     if (commit_top_bs == NULL) {

block/commit.c ro_cleanup: · line 564

Quoted 7 lines
block/commit.c +1 -3
@@ -515,15 +561,13 @@ ro_cleanup:
     qemu_vfree(buf);
 
     blk_unref(backing);
-    if (backing_file_bs) {
-        bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);
-    }
+    bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);

This means that bdrv_set_backing_hd() is now called to undo a change that hasn't even been made yet. This fails (with &error_abort) if the backing chain is frozen.

Quoted 2 lines

On the other hand, the other bdrv_set_backing_hd() calls in the same function would fail the same way.

True. :)

Still, maybe there’s an op blocker from a concurrent job, so we go to the failure path and then we’d abort here. So better to guard it by checking whether bdrv_cow_bs(bs) != backing_file_bs.

Max

#129 lore raw

Re: [PATCH v7 35/47] commit: Deal with filters

Am 20.08.2020 um 13:27 hat Max Reitz geschrieben:

Quoted 37 lines · Kevin Wolf
Quoted 33 lines

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 16 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
 block/block-backend.c          |  9 +++-
 block/commit.c                 | 96 +++++++++++++++++++++++++---------
 block/monitor/block-hmp-cmds.c |  2 +-
 blockdev.c                     |  4 +-
 4 files changed, 81 insertions(+), 30 deletions(-)
block/block-backend.c +0 -0
index 6936b25c83..7f2c7dbccc 100644

block/block-backend.c int blk_commit_all(void) · line 2275

Quoted 5 lines
block/block-backend.c +0 -2
@@ -2271,8 +2271,13 @@ int blk_commit_all(void)
         AioContext *aio_context = blk_get_aio_context(blk);
 
         aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);

The old code didn't try to commit nodes that don't have a backing file.

block/block-backend.c int blk_commit_all(void) · line 2280

Quoted 7 lines
block/block-backend.c +7 -0
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;
+            int ret;
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            ret = bdrv_commit(non_filter);
Quoted 2 lines

The new one tries unconditionally. For nodes without a backing file, bdrv_commit() will return -ENOTSUP, so the whole function fails.

Quoted 6 lines

:(

Hm. Should I fix it by checking for bdrv_cow_bs(bdrv_skip_implicit_filters())? Or bdrv_backing_chain_next() and change the bdrv_skip_implicit_filters() to a bdrv_skip_filters()? I feel like that would make even more sense.

I agree that bdrv_skip_filters() makes more sense. If I have a qcow2 image and an explicit throttle filter on top, there is no reason to skip this image.

bdrv_backing_chain_next() or bdrv_cow_bs() should be the same in a boolean context, so I'd vote for bdrv_cow_bs() because it has less work to do to get the same result.

Quoted 32 lines
Quoted 1 line

(First real bug at patch 35. I almost thought I wouldn't find any!)

:)

Quoted 19 lines
Quoted 8 lines
block/block-backend.c +0 -0
             if (ret < 0) {
                 aio_context_release(aio_context);
                 return ret;
block/commit.c +0 -0
index 7732d02dfe..4122b6736d 100644

block/commit.c typedef struct CommitBlockJob { · line 43

Quoted 7 lines
block/commit.c +1 -0
@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {
     BlockBackend *top;
     BlockBackend *base;
     BlockDriverState *base_bs;
+    BlockDriverState *base_overlay;
     BlockdevOnError on_error;
     bool base_read_only;
     bool chain_frozen;

Hm, again this mysterious base_overlay. I know that stream introduced it to avoid freezing the link to base, but commit doesn't seem to do that.

Quoted 2 lines

Is it to avoid using the block status of filter drivers between base_overlay and base?

Yes.

Quoted 3 lines

If so, I guess that goes back to the question I
raised earlier in this series: What is the block status supposed to tell for filter nodes?

Quoted 6 lines

Honestly, I would really like to get away without having to answer that question in this series. Intuitively, I feel like falling through to the next data-bearing layer is not something most callers want. But I’d rather investigate that question separately from this series (even though that likely means we’ll never do it), and just treat it as it is in this series.

Well, I'm asking the question because not having the answer makes us jump through hoops in this series to accomodate a behaviour it probably shouldn't even have. (Because I agree that filters should probably keep DATA clear, i.e. they are never the layer that defines the content.)

Additional node references (i.e. references that are not edges in the graph) always make the design more complicated and require us to consider more things like what happens on graph changes. So it's a question of maintainability.

Quoted 59 lines
Quoted 50 lines

But anyway, in contrast to mirror, commit actually freezes the chain between commit_top_bs and base, so it should be safe at least.

Quoted 47 lines
block/commit.c +18 -3
@@ -89,7 +90,7 @@ static void commit_abort(Job *job)
      * XXX Can (or should) we somehow keep 'consistent read' blocked even
      * after the failed/cancelled commit job is gone? If we already wrote
      * something to base, the intermediate images aren't valid any more. */
-    bdrv_replace_node(s->commit_top_bs, backing_bs(s->commit_top_bs),
+    bdrv_replace_node(s->commit_top_bs, s->commit_top_bs->backing->bs,
                       &error_abort);
 
     bdrv_unref(s->commit_top_bs);
@@ -153,7 +154,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)
             break;
         }
         /* Copy if allocated above the base */
-        ret = bdrv_is_allocated_above(blk_bs(s->top), blk_bs(s->base), false,
+        ret = bdrv_is_allocated_above(blk_bs(s->top), s->base_overlay, true,
                                       offset, COMMIT_BUFFER_SIZE, &n);
         copy = (ret == 1);
         trace_commit_one_iteration(s, offset, n, ret);
@@ -253,15 +254,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     CommitBlockJob *s;
     BlockDriverState *iter;
     BlockDriverState *commit_top_bs = NULL;
+    BlockDriverState *filtered_base;
     Error *local_err = NULL;
+    int64_t base_size, top_size;
+    uint64_t perms, iter_shared_perms;
     int ret;
 
     assert(top != bs);
-    if (top == base) {
+    if (bdrv_skip_filters(top) == bdrv_skip_filters(base)) {
         error_setg(errp, "Invalid files for merge: top and base are the same");
         return;
     }
 
+    base_size = bdrv_getlength(base);
+    if (base_size < 0) {
+        error_setg_errno(errp, -base_size, "Could not inquire base image size");
+        return;
+    }
+
+    top_size = bdrv_getlength(top);
+    if (top_size < 0) {
+        error_setg_errno(errp, -top_size, "Could not inquire top image size");
+        return;
+    }
+
Quoted 8 lines

block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 284

Quoted 4 lines
block/commit.c +4 -0
+    perms = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE;
+    if (base_size < top_size) {
+        perms |= BLK_PERM_RESIZE;
+    }

base_perms would indicate which permissions these are (particularly because it's not the next thing that requires permissions, but only used further down the function).

Quoted 1 line

%s/\<perms\>/base_perms/? Sure.

Sorry, I admit this wasn't phrased very clearly. But yes, renaming the variable this way is what I meant.

Quoted 117 lines
Quoted 117 lines
Quoted 105 lines
block/commit.c +44 -19
     s = block_job_create(job_id, &commit_job_driver, NULL, bs, 0, BLK_PERM_ALL,
                          speed, creation_flags, NULL, NULL, errp);
     if (!s) {
@@ -301,17 +322,43 @@ void commit_start(const char *job_id, BlockDriverState *bs,
 
     s->commit_top_bs = commit_top_bs;
 
-    /* Block all nodes between top and base, because they will
-     * disappear from the chain after this operation. */
-    assert(bdrv_chain_contains(top, base));
-    for (iter = top; iter != base; iter = backing_bs(iter)) {
-        /* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
-         * at s->base (if writes are blocked for a node, they are also blocked
-         * for its backing file). The other options would be a second filter
-         * driver above s->base. */
+    /*
+     * Block all nodes between top and base, because they will
+     * disappear from the chain after this operation.
+     * Note that this assumes that the user is fine with removing all
+     * nodes (including R/W filters) between top and base.  Assuring
+     * this is the responsibility of the interface (i.e. whoever calls
+     * commit_start()).
+     */
+    s->base_overlay = bdrv_find_overlay(top, base);
+    assert(s->base_overlay);
+
+    /*
+     * The topmost node with
+     * bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base)
+     */
+    filtered_base = bdrv_cow_bs(s->base_overlay);
+    assert(bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base));
+
+    /*
+     * XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
+     * at s->base (if writes are blocked for a node, they are also blocked
+     * for its backing file). The other options would be a second filter
+     * driver above s->base.
+     */
+    iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+    for (iter = top; iter != base; iter = bdrv_filter_or_cow_bs(iter)) {
+        if (iter == filtered_base) {
+            /*
+             * From here on, all nodes are filters on the base.  This
+             * allows us to share BLK_PERM_CONSISTENT_READ.
+             */
+            iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+        }
+
         ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                 BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                 errp);
+                                 iter_shared_perms, errp);
         if (ret < 0) {
             goto fail;
         }
@@ -328,9 +375,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     }
 
     s->base = blk_new(s->common.job.aio_context,
-                      BLK_PERM_CONSISTENT_READ
-                      | BLK_PERM_WRITE
-                      | BLK_PERM_RESIZE,
+                      perms,
                       BLK_PERM_CONSISTENT_READ
                       | BLK_PERM_GRAPH_MOD
                       | BLK_PERM_WRITE_UNCHANGED);
@@ -398,19 +443,22 @@ int bdrv_commit(BlockDriverState *bs)
     if (!drv)
         return -ENOMEDIUM;
 
-    if (!bs->backing) {
+    backing_file_bs = bdrv_cow_bs(bs);
+
+    if (!backing_file_bs) {
         return -ENOTSUP;
     }
 
     if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_COMMIT_SOURCE, NULL) ||
-        bdrv_op_is_blocked(bs->backing->bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL)) {
+        bdrv_op_is_blocked(backing_file_bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL))
+    {
         return -EBUSY;
     }
 
-    ro = bs->backing->bs->read_only;
+    ro = backing_file_bs->read_only;
 
     if (ro) {
-        if (bdrv_reopen_set_read_only(bs->backing->bs, false, NULL)) {
+        if (bdrv_reopen_set_read_only(backing_file_bs, false, NULL)) {
             return -EACCES;
         }
     }
@@ -428,8 +476,6 @@ int bdrv_commit(BlockDriverState *bs)
     }
 
     /* Insert commit_top block node above backing, so we can write to it */
-    backing_file_bs = backing_bs(bs);
-
     commit_top_bs = bdrv_new_open_driver(&bdrv_commit_top, NULL, BDRV_O_RDWR,
                                          &local_err);
     if (commit_top_bs == NULL) {

block/commit.c ro_cleanup: · line 564

Quoted 7 lines
block/commit.c +1 -3
@@ -515,15 +561,13 @@ ro_cleanup:
     qemu_vfree(buf);
 
     blk_unref(backing);
-    if (backing_file_bs) {
-        bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);
-    }
+    bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);

This means that bdrv_set_backing_hd() is now called to undo a change that hasn't even been made yet. This fails (with &error_abort) if the backing chain is frozen.

Quoted 8 lines
Quoted 2 lines

On the other hand, the other bdrv_set_backing_hd() calls in the same function would fail the same way.

True. :)

Still, maybe there’s an op blocker from a concurrent job, so we go to the failure path and then we’d abort here. So better to guard it by checking whether bdrv_cow_bs(bs) != backing_file_bs.

Certainly can't hurt.

Kevin

author
#130 lore raw

PATCH v7 36/47 nbd: Use CAF when looking for dirty bitmap

In reply to Max Reitz

When looking for a dirty bitmap to share, we should handle filters by just including them in the search (so they do not break backing chains).

---
 nbd/server.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
nbd/server.c +3 -3
index 20754e9ebc..b504a79435 100644
@@ -1561,13 +1561,13 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
     if (bitmap) {
         BdrvDirtyBitmap *bm = NULL;
 
-        while (true) {
+        while (bs) {
             bm = bdrv_find_dirty_bitmap(bs, bitmap);
-            if (bm != NULL || bs->backing == NULL) {
+            if (bm != NULL) {
                 break;
             }
 
-            bs = bs->backing->bs;
+            bs = bdrv_filter_or_cow_bs(bs);
         }
 
         if (bm == NULL) {
Signature · 1 line
2.26.2
Reviewed-by on v7 36/47
#131 lore raw

Re: [PATCH v7 36/47] nbd: Use CAF when looking for dirty bitmap

Quoted 19 lines · Max Reitz

When looking for a dirty bitmap to share, we should handle filters by just including them in the search (so they do not break backing chains).

---
  nbd/server.c | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
nbd/server.c +1 -1
index 20754e9ebc..b504a79435 100644
@@ -1561,13 +1561,13 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
      if (bitmap) {
          BdrvDirtyBitmap *bm = NULL;
 
-        while (true) {
+        while (bs) {
              bm = bdrv_find_dirty_bitmap(bs, bitmap);

nbd/server.c NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset, · line 1573

Quoted 10 lines
nbd/server.c +2 -2
-            if (bm != NULL || bs->backing == NULL) {
+            if (bm != NULL) {
                  break;
              }
 
-            bs = bs->backing->bs;
+            bs = bdrv_filter_or_cow_bs(bs);
          }
 
          if (bm == NULL) {
author
#132 lore raw

PATCH v7 37/47 qemu-img: Use child access functions

In reply to Max Reitz

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

---
 qemu-img.c                 | 36 ++++++++++++++++++++++--------------
 tests/qemu-iotests/204.out |  1 +
 2 files changed, 23 insertions(+), 14 deletions(-)
qemu-img.c +22 -14
index 381271a74e..947be6ffac 100644
@@ -1089,7 +1089,7 @@ static int img_commit(int argc, char **argv)
         /* This is different from QMP, which by default uses the deepest file in
          * the backing chain (i.e., the very base); however, the traditional
          * behavior of qemu-img commit is using the immediate backing file. */
-        base_bs = backing_bs(bs);
+        base_bs = bdrv_backing_chain_next(bs);
         if (!base_bs) {
             error_setg(&local_err, "Image does not have a backing file");
             goto done;
@@ -1737,18 +1737,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     if (s->sector_next_status <= sector_num) {
         uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
         int64_t count;
+        BlockDriverState *src_bs = blk_bs(s->src[src_cur]);
+        BlockDriverState *base;
+
+        if (s->target_has_backing) {
+            base = bdrv_cow_bs(bdrv_skip_filters(src_bs));
+        } else {
+            base = NULL;
+        }
 
         do {
             count = n * BDRV_SECTOR_SIZE;
 
-            if (s->target_has_backing) {
-                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
-                                        count, &count, NULL, NULL);
-            } else {
-                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                              offset, count, &count, NULL,
-                                              NULL);
-            }
+            ret = bdrv_block_status_above(src_bs, base, offset, count, &count,
+                                          NULL, NULL);
 
             if (ret < 0) {
                 if (s->salvage) {
@@ -2673,7 +2675,8 @@ static int img_convert(int argc, char **argv)
          * s.target_backing_sectors has to be negative, which it will
          * be automatically).  The backing file length is used only
          * for optimizations, so such a case is not fatal. */
-        s.target_backing_sectors = bdrv_nb_sectors(out_bs->backing->bs);
+        s.target_backing_sectors =
+            bdrv_nb_sectors(bdrv_backing_chain_next(out_bs));
     } else {
         s.target_backing_sectors = -1;
     }
@@ -3044,6 +3047,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 
     depth = 0;
     for (;;) {
+        bs = bdrv_skip_filters(bs);
         ret = bdrv_block_status(bs, offset, bytes, &bytes, &map, &file);
         if (ret < 0) {
             return ret;
@@ -3052,7 +3056,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
         if (ret & (BDRV_BLOCK_ZERO|BDRV_BLOCK_DATA)) {
             break;
         }
-        bs = backing_bs(bs);
+        bs = bdrv_cow_bs(bs);
         if (bs == NULL) {
             ret = 0;
             break;
@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)
     uint8_t *buf_old = NULL;
     uint8_t *buf_new = NULL;
     BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
     char *filename;
     const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
     int c, flags, src_flags, ret;
@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv)
     }
     bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_filters(bs);
+
     if (out_basefmt != NULL) {
         if (bdrv_find_format(out_basefmt) == NULL) {
             error_report("Invalid format name: '%s'", out_basefmt);
@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)
     /* For safe rebasing we need to compare old and new backing file */
     if (!unsafe) {
         QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_cow_bs(unfiltered_bs);
 
         if (base_bs) {
             blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)
                  * If cluster wasn't changed since prefix_chain, we don't need
                  * to take action
                  */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
-                                              false, offset, n, &n);
+                ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs),
+                                              prefix_chain_bs, false,
+                                              offset, n, &n);
                 if (ret < 0) {
                     error_report("error while reading image metadata: %s",
                                  strerror(-ret));
tests/qemu-iotests/204.out +1 -0
index f3a10fbe90..684774d763 100644
@@ -59,5 +59,6 @@ Offset          Length          File
 0x900000        0x2400000       TEST_DIR/t.IMGFMT
 0x3c00000       0x1100000       TEST_DIR/t.IMGFMT
 0x6a00000       0x400000        TEST_DIR/t.IMGFMT
+0x6e00000       0x1200000       TEST_DIR/t.IMGFMT.base
 No errors were found on the image.
 *** done
Signature · 1 line
2.26.2
Reviewed-by on v7 37/47
#133 lore raw

Re: [PATCH v7 37/47] qemu-img: Use child access functions

Quoted 123 lines · Max Reitz

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

---
  qemu-img.c                 | 36 ++++++++++++++++++++++--------------
  tests/qemu-iotests/204.out |  1 +
  2 files changed, 23 insertions(+), 14 deletions(-)
qemu-img.c +22 -14
index 381271a74e..947be6ffac 100644
@@ -1089,7 +1089,7 @@ static int img_commit(int argc, char **argv)
          /* This is different from QMP, which by default uses the deepest file in
           * the backing chain (i.e., the very base); however, the traditional
           * behavior of qemu-img commit is using the immediate backing file. */
-        base_bs = backing_bs(bs);
+        base_bs = bdrv_backing_chain_next(bs);
          if (!base_bs) {
              error_setg(&local_err, "Image does not have a backing file");
              goto done;
@@ -1737,18 +1737,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
      if (s->sector_next_status <= sector_num) {
          uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
          int64_t count;
+        BlockDriverState *src_bs = blk_bs(s->src[src_cur]);
+        BlockDriverState *base;
+
+        if (s->target_has_backing) {
+            base = bdrv_cow_bs(bdrv_skip_filters(src_bs));
+        } else {
+            base = NULL;
+        }
 
          do {
              count = n * BDRV_SECTOR_SIZE;
 
-            if (s->target_has_backing) {
-                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
-                                        count, &count, NULL, NULL);
-            } else {
-                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                              offset, count, &count, NULL,
-                                              NULL);
-            }
+            ret = bdrv_block_status_above(src_bs, base, offset, count, &count,
+                                          NULL, NULL);
 
              if (ret < 0) {
                  if (s->salvage) {
@@ -2673,7 +2675,8 @@ static int img_convert(int argc, char **argv)
           * s.target_backing_sectors has to be negative, which it will
           * be automatically).  The backing file length is used only
           * for optimizations, so such a case is not fatal. */
-        s.target_backing_sectors = bdrv_nb_sectors(out_bs->backing->bs);
+        s.target_backing_sectors =
+            bdrv_nb_sectors(bdrv_backing_chain_next(out_bs));
      } else {
          s.target_backing_sectors = -1;
      }
@@ -3044,6 +3047,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 
      depth = 0;
      for (;;) {
+        bs = bdrv_skip_filters(bs);
          ret = bdrv_block_status(bs, offset, bytes, &bytes, &map, &file);
          if (ret < 0) {
              return ret;
@@ -3052,7 +3056,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
          if (ret & (BDRV_BLOCK_ZERO|BDRV_BLOCK_DATA)) {
              break;
          }
-        bs = backing_bs(bs);
+        bs = bdrv_cow_bs(bs);
          if (bs == NULL) {
              ret = 0;
              break;
@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)
      uint8_t *buf_old = NULL;
      uint8_t *buf_new = NULL;
      BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
      char *filename;
      const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
      int c, flags, src_flags, ret;
@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv)
      }
      bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_filters(bs);
+
      if (out_basefmt != NULL) {
          if (bdrv_find_format(out_basefmt) == NULL) {
              error_report("Invalid format name: '%s'", out_basefmt);
@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)
      /* For safe rebasing we need to compare old and new backing file */
      if (!unsafe) {
          QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_cow_bs(unfiltered_bs);
 
          if (base_bs) {
              blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)
                   * If cluster wasn't changed since prefix_chain, we don't need
                   * to take action
                   */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
-                                              false, offset, n, &n);
+                ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs),
+                                              prefix_chain_bs, false,
+                                              offset, n, &n);
                  if (ret < 0) {
                      error_report("error while reading image metadata: %s",
                                   strerror(-ret));
tests/qemu-iotests/204.out +0 -0
index f3a10fbe90..684774d763 100644

tests/qemu-iotests/204.out Offset Length File · line 64

Quoted 6 lines
tests/qemu-iotests/204.out +1 -0
@@ -59,5 +59,6 @@ Offset          Length          File
  0x900000        0x2400000       TEST_DIR/t.IMGFMT
  0x3c00000       0x1100000       TEST_DIR/t.IMGFMT
  0x6a00000       0x400000        TEST_DIR/t.IMGFMT
+0x6e00000       0x1200000       TEST_DIR/t.IMGFMT.base
  No errors were found on the image.
  *** done
#134 lore raw

Re: [PATCH v7 37/47] qemu-img: Use child access functions

In reply to Max Reitz

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 35 lines

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

qemu-img.c +4 -1
@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)
     uint8_t *buf_old = NULL;
     uint8_t *buf_new = NULL;
     BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
     char *filename;
     const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
     int c, flags, src_flags, ret;
@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv)
     }
     bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_filters(bs);
+
     if (out_basefmt != NULL) {
         if (bdrv_find_format(out_basefmt) == NULL) {
             error_report("Invalid format name: '%s'", out_basefmt);
@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)
     /* For safe rebasing we need to compare old and new backing file */
     if (!unsafe) {
         QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_cow_bs(unfiltered_bs);
 
         if (base_bs) {
             blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)
                  * If cluster wasn't changed since prefix_chain, we don't need

qemu-img.c static int img_rebase(int argc, char **argv) · line 3753

Quoted 10 lines
qemu-img.c +3 -2
                  * to take action
                  */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
-                                              false, offset, n, &n);
+                ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs),
+                                              prefix_chain_bs, false,
+                                              offset, n, &n);
                 if (ret < 0) {
                     error_report("error while reading image metadata: %s",
                                  strerror(-ret));

img_rebase() has these additional calls:

    /* If the cluster is allocated, we don't need to take action */
    ret = bdrv_is_allocated(bs, offset, n, &n);

And:

    if (out_baseimg && *out_baseimg) {
        ret = bdrv_change_backing_file(bs, out_baseimg, out_basefmt);
    } else {
        ret = bdrv_change_backing_file(bs, NULL, NULL);
    }

Shouldn't they use unfiltered_bs?

(Not that it's likely that anyone would use 'qemu-img rebase' with a filter, but while you're touching it...)

Kevin

author
#135 lore raw

Re: [PATCH v7 37/47] qemu-img: Use child access functions

Quoted 54 lines · Kevin Wolf

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 35 lines

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

qemu-img.c +4 -1
@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)
     uint8_t *buf_old = NULL;
     uint8_t *buf_new = NULL;
     BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
     char *filename;
     const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
     int c, flags, src_flags, ret;
@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv)
     }
     bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_filters(bs);
+
     if (out_basefmt != NULL) {
         if (bdrv_find_format(out_basefmt) == NULL) {
             error_report("Invalid format name: '%s'", out_basefmt);
@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)
     /* For safe rebasing we need to compare old and new backing file */
     if (!unsafe) {
         QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_cow_bs(unfiltered_bs);
 
         if (base_bs) {
             blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)
                  * If cluster wasn't changed since prefix_chain, we don't need

qemu-img.c static int img_rebase(int argc, char **argv) · line 3753

Quoted 10 lines
qemu-img.c +3 -2
                  * to take action
                  */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
-                                              false, offset, n, &n);
+                ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs),
+                                              prefix_chain_bs, false,
+                                              offset, n, &n);
                 if (ret < 0) {
                     error_report("error while reading image metadata: %s",
                                  strerror(-ret));

img_rebase() has these additional calls:

    /* If the cluster is allocated, we don't need to take action */
    ret = bdrv_is_allocated(bs, offset, n, &n);

And:

Quoted 7 lines
    if (out_baseimg && *out_baseimg) {
        ret = bdrv_change_backing_file(bs, out_baseimg, out_basefmt);
    } else {
        ret = bdrv_change_backing_file(bs, NULL, NULL);
    }

Shouldn't they use unfiltered_bs?

Oh, yes, the second one definitely.

As for the first one, I don’t think there’s a difference. But why not, we really want to query unfiltered_bs, so it’s better to do so explicitly than through the implicit fall-through behavior of block_status.

Max

author
#136 lore raw

PATCH v7 38/47 block: Drop backing_bs()

In reply to Max Reitz

We want to make it explicit where bs->backing is used, and we have done so. The old role of backing_bs() is now effectively taken by bdrv_cow_bs().

---
 include/block/block_int.h | 5 -----
 1 file changed, 5 deletions(-)
include/block/block_int.h +0 -5
index c963ee9f28..6e09e15ed4 100644
@@ -991,11 +991,6 @@ typedef enum BlockMirrorBackingMode {
     MIRROR_LEAVE_BACKING_CHAIN,
 } BlockMirrorBackingMode;
 
-static inline BlockDriverState *backing_bs(BlockDriverState *bs)
-{
-    return bs->backing ? bs->backing->bs : NULL;
-}
-
 
 /* Essential block drivers which must always be statically linked into qemu, and
  * which therefore can be accessed without using bdrv_find_format() */
Signature · 1 line
2.26.2
Reviewed-by on v7 38/47
#137 lore raw

Re: [PATCH v7 38/47] block: Drop backing_bs()

Quoted 16 lines · Max Reitz

We want to make it explicit where bs->backing is used, and we have done so. The old role of backing_bs() is now effectively taken by bdrv_cow_bs().

---
  include/block/block_int.h | 5 -----
  1 file changed, 5 deletions(-)
include/block/block_int.h +0 -0
index c963ee9f28..6e09e15ed4 100644
@@ -991,11 +991,6 @@ typedef enum BlockMirrorBackingMode {
      MIRROR_LEAVE_BACKING_CHAIN,

include/block/block_int.h typedef enum BlockMirrorBackingMode { · line 996

Quoted 10 lines
include/block/block_int.h +0 -5
  } BlockMirrorBackingMode;
 
-static inline BlockDriverState *backing_bs(BlockDriverState *bs)
-{
-    return bs->backing ? bs->backing->bs : NULL;
-}
-
 
  /* Essential block drivers which must always be statically linked into qemu, and
   * which therefore can be accessed without using bdrv_find_format() */
author
#138 lore raw

PATCH v7 39/47 blockdev: Fix active commit choice

In reply to Max Reitz

We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.

---
 blockdev.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +21 -3
index 402f1d1df1..237fffbe53 100644
@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
     AioContext *aio_context;
     Error *local_err = NULL;
     int job_flags = JOB_DEFAULT;
+    uint64_t top_perm, top_shared;
 
     if (!has_speed) {
         speed = 0;
@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
         goto out;
     }
 
-    if (top_bs == bs) {
+    /*
+     * Active commit is required if and only if someone has taken a
+     * WRITE permission on the top node.  Historically, we have always
+     * used active commit for top nodes, so continue that practice.
+     * (Active commit is never really wrong.)
+     */
+    bdrv_get_cumulative_perm(top_bs, &top_perm, &top_shared);
+    if (top_perm & BLK_PERM_WRITE ||
+        bdrv_skip_filters(top_bs) == bdrv_skip_filters(bs))
+    {
         if (has_backing_file) {
             error_setg(errp, "'backing-file' specified,"
                              " but 'top' is the active layer");
             goto out;
         }
-        commit_active_start(has_job_id ? job_id : NULL, bs, base_bs,
-                            job_flags, speed, on_error,
+        if (!has_job_id) {
+            /*
+             * Emulate here what block_job_create() does, because it
+             * is possible that @bs != @top_bs (the block job should
+             * be named after @bs, even if @top_bs is the actual
+             * source)
+             */
+            job_id = bdrv_get_device_name(bs);
+        }
+        commit_active_start(job_id, top_bs, base_bs, job_flags, speed, on_error,
                             filter_node_name, NULL, NULL, false, &local_err);
     } else {
         BlockDriverState *overlay_bs = bdrv_find_overlay(bs, top_bs);
Signature · 1 line
2.26.2
#139 lore raw

Re: [PATCH v7 39/47] blockdev: Fix active commit choice

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 22 lines

We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.

---
 blockdev.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +1 -0
index 402f1d1df1..237fffbe53 100644
@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
     AioContext *aio_context;
     Error *local_err = NULL;
     int job_flags = JOB_DEFAULT;
+    uint64_t top_perm, top_shared;
 
     if (!has_speed) {
         speed = 0;

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710

Quoted 7 lines
blockdev.c +3 -1
@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
         goto out;
     }
 
-    if (top_bs == bs) {
+    /*
+     * Active commit is required if and only if someone has taken a
+     * WRITE permission on the top node.

...or if someone wants to take a WRITE permission while the job is running.

Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713

Quoted 4 lines
blockdev.c +3 -0
                                           Historically, we have always
+     * used active commit for top nodes, so continue that practice.
+     * (Active commit is never really wrong.)
+     */

Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".

This is even more problematic because the commit job doesn't unshare BLK_PERM_WRITE yet, so it would lead to silent corruption rather than an error.

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720

Quoted 7 lines
blockdev.c +4 -0
+    bdrv_get_cumulative_perm(top_bs, &top_perm, &top_shared);
+    if (top_perm & BLK_PERM_WRITE ||
+        bdrv_skip_filters(top_bs) == bdrv_skip_filters(bs))
+    {
         if (has_backing_file) {
             error_setg(errp, "'backing-file' specified,"
                              " but 'top' is the active layer");

Hm, this error message isn't accurate any more.

In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2729

Quoted 11 lines
blockdev.c +7 -2
             goto out;
         }
-        commit_active_start(has_job_id ? job_id : NULL, bs, base_bs,
-                            job_flags, speed, on_error,
+        if (!has_job_id) {
+            /*
+             * Emulate here what block_job_create() does, because it
+             * is possible that @bs != @top_bs (the block job should
+             * be named after @bs, even if @top_bs is the actual
+             * source)
+             */

Should it? Oh, yes, looks like it. block-commit is weird. :-)

Quoted 6 lines
blockdev.c +3 -0
+            job_id = bdrv_get_device_name(bs);
+        }
+        commit_active_start(job_id, top_bs, base_bs, job_flags, speed, on_error,
                             filter_node_name, NULL, NULL, false, &local_err);
     } else {
         BlockDriverState *overlay_bs = bdrv_find_overlay(bs, top_bs);

Kevin

author
#140 lore raw

Re: [PATCH v7 39/47] blockdev: Fix active commit choice

Quoted 43 lines · Kevin Wolf

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 22 lines

We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.

---
 blockdev.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +1 -0
index 402f1d1df1..237fffbe53 100644
@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
     AioContext *aio_context;
     Error *local_err = NULL;
     int job_flags = JOB_DEFAULT;
+    uint64_t top_perm, top_shared;
 
     if (!has_speed) {
         speed = 0;

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710

Quoted 7 lines
blockdev.c +3 -1
@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
         goto out;
     }
 
-    if (top_bs == bs) {
+    /*
+     * Active commit is required if and only if someone has taken a
+     * WRITE permission on the top node.

...or if someone wants to take a WRITE permission while the job is running.

Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713

Quoted 4 lines
blockdev.c +3 -0
                                           Historically, we have always
+     * used active commit for top nodes, so continue that practice.
+     * (Active commit is never really wrong.)
+     */
Quoted 4 lines

Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".

I think that’s what I meant by “historically”. Is “legacily” a word?

But sure, I can make it more explicit.

Quoted 12 lines

This is even more problematic because the commit job doesn't unshare BLK_PERM_WRITE yet, so it would lead to silent corruption rather than an error.

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720

Quoted 7 lines
blockdev.c +4 -0
+    bdrv_get_cumulative_perm(top_bs, &top_perm, &top_shared);
+    if (top_perm & BLK_PERM_WRITE ||
+        bdrv_skip_filters(top_bs) == bdrv_skip_filters(bs))
+    {
         if (has_backing_file) {
             error_setg(errp, "'backing-file' specified,"
                              " but 'top' is the active layer");
Quoted 4 lines

Hm, this error message isn't accurate any more.

In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.

Hm. I wanted to agree, and then I wanted to come up with a QAPI documentation that fits the new behavior (because I think it makes more sense to change the QAPI documentation along with the behavior change, rather than to force us to allow backing-file for anything that isn’t on the top layer).

But in the process of coming up with a better description, I noticed that this doesn’t say “is a root node”, it says “is the active layer”. I would say a node in the active layer is a node that has some parent that has taken a WRITE permission on it. So actually I think that the documentation is right, and this code only now fits.

Though I do think this wants for some clarification. Perhaps “If 'top' is the active layer (i.e., is a node that may be written to), specifying a backing [...]”?

There’s more wrong with the specification, namely the whole part under @backing-file past the “(Since 2.1)”, starting with “If top == base”. I think all of that should go to the top level. (And “If top == active” should be changed to “If top is active (i.e., may be written to)”.)

Max

#141 lore raw

Re: [PATCH v7 39/47] blockdev: Fix active commit choice

Am 24.08.2020 um 15:18 hat Max Reitz geschrieben:

Quoted 76 lines · Kevin Wolf
Quoted 43 lines

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 22 lines

We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.

---
 blockdev.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +1 -0
index 402f1d1df1..237fffbe53 100644
@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
     AioContext *aio_context;
     Error *local_err = NULL;
     int job_flags = JOB_DEFAULT;
+    uint64_t top_perm, top_shared;
 
     if (!has_speed) {
         speed = 0;

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710

Quoted 7 lines
blockdev.c +3 -1
@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
         goto out;
     }
 
-    if (top_bs == bs) {
+    /*
+     * Active commit is required if and only if someone has taken a
+     * WRITE permission on the top node.

...or if someone wants to take a WRITE permission while the job is running.

Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713

Quoted 4 lines
blockdev.c +3 -0
                                           Historically, we have always
+     * used active commit for top nodes, so continue that practice.
+     * (Active commit is never really wrong.)
+     */
Quoted 4 lines

Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".

I think that’s what I meant by “historically”. Is “legacily” a word?

But sure, I can make it more explicit.

Quoted 12 lines

This is even more problematic because the commit job doesn't unshare BLK_PERM_WRITE yet, so it would lead to silent corruption rather than an error.

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720

Quoted 7 lines
blockdev.c +4 -0
+    bdrv_get_cumulative_perm(top_bs, &top_perm, &top_shared);
+    if (top_perm & BLK_PERM_WRITE ||
+        bdrv_skip_filters(top_bs) == bdrv_skip_filters(bs))
+    {
         if (has_backing_file) {
             error_setg(errp, "'backing-file' specified,"
                              " but 'top' is the active layer");
Quoted 4 lines

Hm, this error message isn't accurate any more.

In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.

Hm. I wanted to agree, and then I wanted to come up with a QAPI documentation that fits the new behavior (because I think it makes more sense to change the QAPI documentation along with the behavior change, rather than to force us to allow backing-file for anything that isn’t on the top layer).

Quoted 5 lines

But in the process of coming up with a better description, I noticed that this doesn’t say “is a root node”, it says “is the active layer”. I would say a node in the active layer is a node that has some parent that has taken a WRITE permission on it. So actually I think that the documentation is right, and this code only now fits.

Then you may have not only "the" active layer, but multiple active layers. I find this a bit counterintuitive.

There is a simple reason why backing-file is an error for a root node:
It doesn't have overlays, so a value to write to the header of overlay images just doesn't make sense.

The same reasoning doesn't apply for writable images that do have overlays. Forbidding backing-file is a more arbitrary restriction there. I'm not saying that we can't make arbitrary restrictions where allowing an option is not worth the effort, but I feel they should be spelt out more explicitly instead of twisting words like "active layer" until they fit the code.

Quoted 3 lines

Though I do think this wants for some clarification. Perhaps “If 'top' is the active layer (i.e., is a node that may be written to), specifying a backing [...]”?

"If 'top' doesn't have an overlay image or is in use by a writer..."?

Quoted 4 lines

There’s more wrong with the specification, namely the whole part under @backing-file past the “(Since 2.1)”, starting with “If top == base”. I think all of that should go to the top level. (And “If top == active” should be changed to “If top is active (i.e., may be written to)”.)

At least the latter only becomes wrong with this patch, so I think it needs to be changed by this patch.

Kevin

author
#142 lore raw

Re: [PATCH v7 39/47] blockdev: Fix active commit choice

Quoted 77 lines · Kevin Wolf

Am 24.08.2020 um 15:18 hat Max Reitz geschrieben:

Quoted 76 lines · Kevin Wolf
Quoted 43 lines

Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:

Quoted 22 lines

We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.

---
 blockdev.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +1 -0
index 402f1d1df1..237fffbe53 100644
@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
     AioContext *aio_context;
     Error *local_err = NULL;
     int job_flags = JOB_DEFAULT;
+    uint64_t top_perm, top_shared;
 
     if (!has_speed) {
         speed = 0;

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710

Quoted 7 lines
blockdev.c +3 -1
@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
         goto out;
     }
 
-    if (top_bs == bs) {
+    /*
+     * Active commit is required if and only if someone has taken a
+     * WRITE permission on the top node.

...or if someone wants to take a WRITE permission while the job is running.

Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713

Quoted 4 lines
blockdev.c +3 -0
                                           Historically, we have always
+     * used active commit for top nodes, so continue that practice.
+     * (Active commit is never really wrong.)
+     */
Quoted 4 lines

Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".

I think that’s what I meant by “historically”. Is “legacily” a word?

But sure, I can make it more explicit.

Quoted 12 lines

This is even more problematic because the commit job doesn't unshare BLK_PERM_WRITE yet, so it would lead to silent corruption rather than an error.

blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720

Quoted 7 lines
blockdev.c +4 -0
+    bdrv_get_cumulative_perm(top_bs, &top_perm, &top_shared);
+    if (top_perm & BLK_PERM_WRITE ||
+        bdrv_skip_filters(top_bs) == bdrv_skip_filters(bs))
+    {
         if (has_backing_file) {
             error_setg(errp, "'backing-file' specified,"
                              " but 'top' is the active layer");
Quoted 4 lines

Hm, this error message isn't accurate any more.

In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.

Hm. I wanted to agree, and then I wanted to come up with a QAPI documentation that fits the new behavior (because I think it makes more sense to change the QAPI documentation along with the behavior change, rather than to force us to allow backing-file for anything that isn’t on the top layer).

Quoted 8 lines
Quoted 5 lines

But in the process of coming up with a better description, I noticed that this doesn’t say “is a root node”, it says “is the active layer”. I would say a node in the active layer is a node that has some parent that has taken a WRITE permission on it. So actually I think that the documentation is right, and this code only now fits.

Then you may have not only "the" active layer, but multiple active layers. I find this a bit counterintuitive.

Depends on what you count as a layer. I don’t think that’s a clearly defined term, is it? I only know of “active layer”, “format layer”, “protocol layer”, and you can at least have multiple format layers above each other. So I don’t find it counterintuitive.

But perhaps it’d be best to just get away from the term “active layer”, as you propose below.

Quoted 3 lines

There is a simple reason why backing-file is an error for a root node:
It doesn't have overlays, so a value to write to the header of overlay images just doesn't make sense.

Ah, yeah...

Quoted 6 lines

The same reasoning doesn't apply for writable images that do have overlays. Forbidding backing-file is a more arbitrary restriction there. I'm not saying that we can't make arbitrary restrictions where allowing an option is not worth the effort, but I feel they should be spelt out more explicitly instead of twisting words like "active layer" until they fit the code.

I’m all for spelling it out more explicitly. I just noticed that I couldn’t clearly distinguish “active layer” from “other” cases of nodes with writers on them, which is why I noted that “active” to me means the post-patch behavior already.

You’re right that there is no semantic reason for making it an error. So I just want it to be an error to be lazy. I hope you let me do that.
(I don’t think there’s much of a problem with it, considering that
commits on nodes that have the WRITE permission taken are basically just completely broken right now.)

Quoted 5 lines
Quoted 3 lines

Though I do think this wants for some clarification. Perhaps “If 'top' is the active layer (i.e., is a node that may be written to), specifying a backing [...]”?

"If 'top' doesn't have an overlay image or is in use by a writer..."?

I.e., avoiding the term “active layer” altogether? Sounds good. Only, I don’t know about “writer”... But it’s already used in BlockdevOptionsFile.dynamic-auto-read-only’s description, so I suppose we can use it here, too. (I just don’t know if as a non-block-layer-developer I’d know what it means.)

(Also, yes, you’re right, the current behavior of giving all root nodes an active commit of course remains, even when there are no writers.)

Quoted 7 lines
Quoted 4 lines

There’s more wrong with the specification, namely the whole part under @backing-file past the “(Since 2.1)”, starting with “If top == base”. I think all of that should go to the top level. (And “If top == active” should be changed to “If top is active (i.e., may be written to)”.)

At least the latter only becomes wrong with this patch, so I think it needs to be changed by this patch.

Sure. So I understand you agree with moving the whole chunk, right?

Max