Skip to contentSkip to conversation

block: Deal with filters

PATCH Superseded Max Reitz 4 participants 136 messages Aug 9, 2019 – Dec 11, 2019 Block layer Page 1 of 2

v6 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Max Reitz

author
#0 lore raw

v6 cover block: Deal with filters

Hi,

When we introduced filters, we did it a bit casually. Sure, we talked a lot about them before, but that was mostly discussion about where implicit filters should be added to the graph (note that we currently only have two implicit filters, those being mirror and commit). But in the end, we really just designated some drivers filters (Quorum, blkdebug, etc.) and added some specifically (throttle, COR), without really looking through the block layer to see where issues might occur.

It turns out vast areas of the block layer just don’t know about filters and cannot really handle them. Many cases will work in practice, in others, well, too bad, you cannot use some feature because some part deep inside the block layer looks at your filters and thinks they are format nodes.

This is one reason why this series is needed. Over time (since v1), a second reason has made its way in:

bs->file is not necessarily the place where a node’s data is stored. qcow2 now has external data files, and currently there is no way for the general block layer to know that the data is not stored in bs->file. Right now, I do not think that has any real consequences (all functions that need access to the actual data storage file should only do so as a fallback if the driver does not provide some functionality, but qcow2 should provide it all), but it still shows that we need some way to let the general block layer know about such data files. (Also, I will need this for v1 of my “Inquire images’ rotational info” series.)

I won’t go on and on about this series now, I think the patches pretty much speak for themselves now. If the cover letter gets too long, nobody reads it anyway (see previous versions).

*** I’ve based this series on John’s bitmap branch, which I’ve rebased
on my block-next branch. ***

I’ve pushed the patches here:

https://git.xanclic.moe/XanClic/qemu child-access-functions-v6
https://github.com/XanClic/qemu child-access-functions-v6

I’ve also pushed the base branch to each of those repos, it’s “child-access-functions-base”.

v6:
- Patch 9: Rename *freeze_backing_chain (etc.) to *freeze_chain (etc.)
- Patch 10: Drop bdrv_is_encrypted() instead of fixing it the wrong way
- Patch 15: Add a comment on why this works
- Patch 16: Just flush all children instead of one
- Patch 20: We have to snapshot all non-backing children, so both metadata and storage children; for simplification, just disallow the fallback path if there is more than one such child
- Patch 22: bdrv_get_allocated_file_size() should report all children’s sizes, not just the primary or storage child’s
- Patch 24: Make query-blockstats too report any filtered child under “backing”
- Patch 25:
- bdrv_is_allocated_above()’s new @include_base parameter makes things a bit simpler
- Forbid mirroring to a filter on top of the base
- Patch 27: bdrv_is_allocated_above()’s new @include_base parameter makes things a bit simpler
- Patch 28: Requires a few changes to keep stream independent of the base node
- Patch 30:
- Sprinkle in a few bdrv_skip_implicit_filters()s
- Conflicts with the convert --salvage patches

git backport-diff against v5:

Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively

001/42:[----] [-C] 'block: Mark commit and mirror as filter drivers' 002/42:[----] [--] 'copy-on-read: Support compressed writes' 003/42:[----] [--] 'throttle: Support compressed writes' 004/42:[----] [--] 'block: Add child access functions' 005/42:[----] [--] 'block: Add chain helper functions' 006/42:[----] [--] 'qcow2: Implement .bdrv_storage_child()' 007/42:[----] [--] 'block: *filtered_cow_child() for *has_zero_init()' 008/42:[----] [--] 'block: bdrv_set_backing_hd() is about bs->backing' 009/42:[0078] [FC] 'block: Include filters when freezing backing chain' 010/42:[down] 'block: Drop bdrv_is_encrypted()' 011/42:[----] [-C] 'block: Add bdrv_supports_compressed_writes()' 012/42:[----] [--] 'block: Use bdrv_filtered_rw* where obvious' 013/42:[----] [-C] 'block: Use CAFs in block status functions' 014/42:[----] [--] 'block: Use CAFs when working with backing chains' 015/42:[0017] [FC] 'block: Re-evaluate backing file handling in reopen' 016/42:[down] 'block: Flush all children in generic code' 017/42:[----] [--] 'block: Use CAFs in bdrv_refresh_limits()' 018/42:[----] [--] 'block: Use CAFs in bdrv_refresh_filename()' 019/42:[----] [--] 'block: Use CAF in bdrv_co_rw_vmstate()' 020/42:[down] 'block/snapshot: Fix fallback'
021/42:[----] [--] 'block: Use CAFs for debug breakpoints' 022/42:[down] 'block: Fix bdrv_get_allocated_file_size's fallback' 023/42:[----] [--] 'blockdev: Use CAF in external_snapshot_prepare()' 024/42:[0012] [FC] 'block: Use child access functions for QAPI queries' 025/42:[0019] [FC] 'mirror: Deal with filters'
026/42:[----] [--] 'backup: Deal with filters'
027/42:[0025] [FC] 'commit: Deal with filters'
028/42:[0048] [FC] 'stream: Deal with filters'
029/42:[----] [--] 'nbd: Use CAF when looking for dirty bitmap' 030/42:[0035] [FC] 'qemu-img: Use child access functions' 031/42:[----] [--] 'block: Drop backing_bs()'
032/42:[----] [-C] 'block: Make bdrv_get_cumulative_perm() public' 033/42:[----] [--] 'blockdev: Fix active commit choice' 034/42:[----] [-C] 'block: Inline bdrv_co_block_status_from_*()' 035/42:[----] [--] 'block: Fix check_to_replace_node()' 036/42:[----] [--] 'iotests: Add tests for mirror @replaces loops' 037/42:[----] [-C] 'block: Leave BDS.backing_file constant' 038/42:[----] [--] 'iotests: Let complete_and_wait() work with commit' 039/42:[----] [--] 'iotests: Add filter commit test cases' 040/42:[----] [--] 'iotests: Add filter mirror test cases' 041/42:[----] [--] 'iotests: Add test for commit in sub directory' 042/42:[----] [--] 'iotests: Test committing to overridden backing'

Max Reitz (42):
block: Mark commit and mirror as filter drivers
copy-on-read: Support compressed writes
throttle: Support compressed writes
block: Add child access functions
block: Add chain helper functions
qcow2: Implement .bdrv_storage_child()
block: *filtered_cow_child() for *has_zero_init()
block: bdrv_set_backing_hd() is about bs->backing
block: Include filters when freezing backing chain
block: Drop bdrv_is_encrypted()
block: Add bdrv_supports_compressed_writes()
block: Use bdrv_filtered_rw* where obvious
block: Use CAFs in block status functions
block: Use CAFs when working with backing chains
block: Re-evaluate backing file handling in reopen
block: Flush all children in generic code
block: Use CAFs in bdrv_refresh_limits()
block: Use CAFs in bdrv_refresh_filename()
block: Use CAF in bdrv_co_rw_vmstate() block/snapshot: Fix fallback
block: Use CAFs for debug breakpoints
block: Fix bdrv_get_allocated_file_size's fallback
blockdev: Use CAF in external_snapshot_prepare()
block: Use child access functions for QAPI queries
mirror: Deal with filters
backup: Deal with filters
commit: Deal with filters
stream: Deal with filters
nbd: Use CAF when looking for dirty bitmap
qemu-img: Use child access functions
block: Drop backing_bs()
block: Make bdrv_get_cumulative_perm() public
blockdev: Fix active commit choice
block: Inline bdrv_co_block_status_from_*()
block: Fix check_to_replace_node()
iotests: Add tests for mirror @replaces loops
block: Leave BDS.backing_file constant
iotests: Let complete_and_wait() work with commit
iotests: Add filter commit test cases
iotests: Add filter mirror test cases
iotests: Add test for commit in sub directory
iotests: Test committing to overridden backing

 qapi/block-core.json          |   4 +
 include/block/block.h         |  13 +-
 include/block/block_int.h     | 109 ++++---
 block.c                       | 552 ++++++++++++++++++++++++++++------
 block/backup.c                |   9 +-
 block/blkdebug.c              |   7 +-
 block/blklogwrites.c          |   1 -
 block/block-backend.c         |  16 +-
 block/commit.c                | 107 +++++--
 block/copy-on-read.c          |  13 +-
 block/io.c                    | 117 ++++---
 block/mirror.c                | 124 ++++++--
 block/qapi.c                  |  48 +--
 block/qcow2.c                 |   9 +
 block/snapshot.c              | 100 ++++--
 block/stream.c                |  52 ++--
 block/throttle.c              |  11 +-
 blockdev.c                    | 139 +++++++--
 nbd/server.c                  |   6 +-
 qemu-img.c                    |  33 +-
 tests/qemu-iotests/020        |  36 +++
 tests/qemu-iotests/020.out    |  10 +
 tests/qemu-iotests/040        | 238 +++++++++++++++
 tests/qemu-iotests/040.out    |   4 +-
 tests/qemu-iotests/041        | 270 ++++++++++++++++-
 tests/qemu-iotests/041.out    |   4 +-
 tests/qemu-iotests/184.out    |   7 +-
 tests/qemu-iotests/191.out    |   1 -
 tests/qemu-iotests/204.out    |   1 +
 tests/qemu-iotests/228        |   6 +-
 tests/qemu-iotests/228.out    |   6 +-
 tests/qemu-iotests/245        |   4 +-
 tests/qemu-iotests/iotests.py |  10 +-
 33 files changed, 1682 insertions(+), 385 deletions(-)
Signature · 1 line
2.21.0
author
#1 lore raw

PATCH v6 1/42 block: Mark commit and mirror as filter drivers

The commit and mirror block nodes are filters, so they should be marked as such. (Strictly speaking, BDS.is_filter's documentation states that a filter's child must be bs->file. The following patch will relax this restriction, however.)

---
 block/commit.c | 2 ++
 block/mirror.c | 2 ++
 2 files changed, 4 insertions(+)
block/commit.c +2 -0
index 408ae15389..19915603ae 100644
@@ -254,6 +254,8 @@ static BlockDriver bdrv_commit_top = {
     .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
     .bdrv_refresh_filename      = bdrv_commit_top_refresh_filename,
     .bdrv_child_perm            = bdrv_commit_top_child_perm,
+
+    .is_filter                  = true,
 };
 
 void commit_start(const char *job_id, BlockDriverState *bs,
block/mirror.c +2 -0
index 2b870683f1..a8f2d7a305 100644
@@ -1507,6 +1507,8 @@ static BlockDriver bdrv_mirror_top = {
     .bdrv_refresh_filename      = bdrv_mirror_top_refresh_filename,
     .bdrv_child_perm            = bdrv_mirror_top_child_perm,
     .bdrv_refresh_limits        = bdrv_mirror_top_refresh_limits,
+
+    .is_filter                  = true,
 };
 
 static BlockJob *mirror_start_job(
Signature · 1 line
2.21.0
author
#2 lore raw

PATCH v6 2/42 copy-on-read: Support compressed writes

In reply to Max Reitz
---
 block/copy-on-read.c | 11 +++++++++++
 1 file changed, 11 insertions(+)
block/copy-on-read.c +11 -0
index 6631f30205..16bdf630b6 100644
@@ -113,6 +113,16 @@ static int coroutine_fn cor_co_pdiscard(BlockDriverState *bs,
 }
 
 
+static int coroutine_fn cor_co_pwritev_compressed(BlockDriverState *bs,
+                                                  uint64_t offset,
+                                                  uint64_t bytes,
+                                                  QEMUIOVector *qiov)
+{
+    return bdrv_co_pwritev(bs->file, offset, bytes, qiov,
+                           BDRV_REQ_WRITE_COMPRESSED);
+}
+
+
 static void cor_eject(BlockDriverState *bs, bool eject_flag)
 {
     bdrv_eject(bs->file->bs, eject_flag);
@@ -145,6 +155,7 @@ static BlockDriver bdrv_copy_on_read = {
     .bdrv_co_pwritev                    = cor_co_pwritev,
     .bdrv_co_pwrite_zeroes              = cor_co_pwrite_zeroes,
     .bdrv_co_pdiscard                   = cor_co_pdiscard,
+    .bdrv_co_pwritev_compressed         = cor_co_pwritev_compressed,
 
     .bdrv_eject                         = cor_eject,
     .bdrv_lock_medium                   = cor_lock_medium,
Signature · 1 line
2.21.0
author
#3 lore raw

PATCH v6 3/42 throttle: Support compressed writes

In reply to Max Reitz
---
 block/throttle.c | 10 ++++++++++
 1 file changed, 10 insertions(+)
block/throttle.c +10 -0
index 0349f42257..958a2bcfa6 100644
@@ -153,6 +153,15 @@ static int coroutine_fn throttle_co_pdiscard(BlockDriverState *bs,
     return bdrv_co_pdiscard(bs->file, offset, bytes);
 }
 
+static int coroutine_fn throttle_co_pwritev_compressed(BlockDriverState *bs,
+                                                       uint64_t offset,
+                                                       uint64_t bytes,
+                                                       QEMUIOVector *qiov)
+{
+    return throttle_co_pwritev(bs, offset, bytes, qiov,
+                               BDRV_REQ_WRITE_COMPRESSED);
+}
+
 static int throttle_co_flush(BlockDriverState *bs)
 {
     return bdrv_co_flush(bs->file->bs);
@@ -251,6 +260,7 @@ static BlockDriver bdrv_throttle = {
 
     .bdrv_co_pwrite_zeroes              =   throttle_co_pwrite_zeroes,
     .bdrv_co_pdiscard                   =   throttle_co_pdiscard,
+    .bdrv_co_pwritev_compressed         =   throttle_co_pwritev_compressed,
 
     .bdrv_recurse_is_first_non_filter   =   throttle_recurse_is_first_non_filter,
 
Signature · 1 line
2.21.0
author
#4 lore raw

PATCH v6 4/42 block: Add child access functions

In reply to Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

---
 include/block/block_int.h | 57 ++++++++++++++++++++--
 block.c                   | 99 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 153 insertions(+), 3 deletions(-)
include/block/block_int.h +54 -3
index 60d9261f8e..6c60abc4c3 100644
@@ -90,9 +90,11 @@ struct BlockDriver {
     int instance_size;
 
     /* set to true if the BlockDriver is a block filter. Block filters pass
-     * certain callbacks that refer to data (see block.c) to their bs->file if
-     * the driver doesn't implement them. Drivers that do not wish to forward
-     * must implement them and return -ENOTSUP.
+     * certain callbacks that refer to data (see block.c) to their bs->file
+     * or bs->backing (whichever one exists) if the driver doesn't implement
+     * them. Drivers that do not wish to forward must implement them and return
+     * -ENOTSUP.
+     * Note that filters are not allowed to modify data.
      */
     bool is_filter;
     /* for snapshots block filter like Quorum can implement the
@@ -562,6 +564,13 @@ struct BlockDriver {
      * If this pointer is NULL, the array is considered empty.
      * "filename" and "driver" are always considered strong. */
     const char *const *strong_runtime_opts;
+
+    /**
+     * Return the data storage child, if there is exactly one.  If
+     * this function is not implemented, the block layer will assume
+     * bs->file to be this child.
+     */
+    BdrvChild *(*bdrv_storage_child)(BlockDriverState *bs);
 };
 
 typedef struct BlockLimits {
@@ -1276,4 +1285,46 @@ int coroutine_fn bdrv_co_copy_range_to(BdrvChild *src, uint64_t src_offset,
 
 int refresh_total_sectors(BlockDriverState *bs, int64_t hint);
 
+BdrvChild *bdrv_filtered_cow_child(BlockDriverState *bs);
+BdrvChild *bdrv_filtered_rw_child(BlockDriverState *bs);
+BdrvChild *bdrv_filtered_child(BlockDriverState *bs);
+BdrvChild *bdrv_metadata_child(BlockDriverState *bs);
+BdrvChild *bdrv_storage_child(BlockDriverState *bs);
+BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+
+static inline BlockDriverState *child_bs(BdrvChild *child)
+{
+    return child ? child->bs : NULL;
+}
+
+static inline BlockDriverState *bdrv_filtered_cow_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_filtered_cow_child(bs));
+}
+
+static inline BlockDriverState *bdrv_filtered_rw_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_filtered_rw_child(bs));
+}
+
+static inline BlockDriverState *bdrv_filtered_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_filtered_child(bs));
+}
+
+static inline BlockDriverState *bdrv_metadata_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_metadata_child(bs));
+}
+
+static inline BlockDriverState *bdrv_storage_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_storage_child(bs));
+}
+
+static inline BlockDriverState *bdrv_primary_bs(BlockDriverState *bs)
+{
+    return child_bs(bdrv_primary_child(bs));
+}
+
 #endif /* BLOCK_INT_H */
block.c +99 -0

102 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v6 4/42
#5 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Quoted 12 lines · Max Reitz

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Quoted 7 lines
---
 include/block/block_int.h | 57 ++++++++++++++++++++--
 block.c                   | 99 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 153 insertions(+), 3 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
#6 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 7 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

Quoted 7 lines

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Apparently any child of a filter node is "filtered" (which makes sense), but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

It looks like bdrv_filtered_child() is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.

Quoted 4 lines

+/*
+ * Return the child that @bs acts as an overlay for, and from which data may be
+ * copied in COW or COR operations. Usually this is the backing file.
+ */

Or NULL, if no such child exists.

It's relatively obvious here, but for some of the functions further down it would be really good to describe in which cases NULL is expected (or that NULL is even a possible return value).

Kevin

author
#7 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Quoted 13 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 12 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Quoted 8 lines
Quoted 2 lines

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Apparently any child of a filter node is "filtered" (which makes sense),

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.

qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.

raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.

(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.

Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)

Quoted 3 lines

It looks like bdrv_filtered_child() is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.

It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.

Quoted 10 lines
Quoted 4 lines

+/*
+ * Return the child that @bs acts as an overlay for, and from which data may be
+ * copied in COW or COR operations. Usually this is the backing file.
+ */

Or NULL, if no such child exists.

It's relatively obvious here, but for some of the functions further down it would be really good to describe in which cases NULL is expected (or that NULL is even a possible return value).

I’ll look into it.

Max

#8 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 20 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 7 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

Quoted 7 lines

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Quoted 5 lines
Quoted 1 line

Apparently any child of a filter node is "filtered" (which makes sense),

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?

Quoted 9 lines
Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Quoted 5 lines

The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.

We generally call this concept a "backing chain".

Quoted 6 lines

qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.

raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.

Quoted 8 lines

(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.

Agreed with offset, but with only size, it matches your definition of a filter.

Quoted 11 lines

Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)

Quoted 3 lines

It looks like bdrv_filtered_child() is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.

It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.

qcow2 doesn't fulfill the conditions for begin a filter driver. Two of its possible children fulfill the conditions for being a filtered child. You can pick either approach, talking about a "filter chain" just doesn't make sense there. Either the chain is broken by a non-filter driver like qcow2, or it must become a filter tree.

What we're really interested in is iterating the backing chain even across filter nodes, so your implementation achieves the right result. It just feels completely arbitrary, counterintuitive and confusing to call this a (or actually "the") "filter chain" and to pretend that the name tells anyone what it really is.

Kevin

author
#9 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Quoted 24 lines · Kevin Wolf

Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:

Quoted 23 lines · Kevin Wolf
Quoted 21 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 7 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

Quoted 7 lines

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Apparently any child of a filter node is "filtered" (which makes sense),

Quoted 7 lines
Quoted 3 lines

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?

Well, there’s Quorum...

Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

Quoted 5 lines

But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.

Which is why it currently returns NULL for Quorum.

Quoted 2 lines

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?

Quoted 7 lines
Quoted 5 lines

The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.

We generally call this concept a "backing chain".

I suppose that’s an exclusive “we”? Because I use ”backing chain” to refer to COW chains exclusively.

Such a chain may or may not include filters, but they are not really load-bearing nodes of the chain. As such, I generally want to skip them when looking at a backing chain (hence e.g. bdrv_backing_chain_next()).

From what I can tell nobody has ever formalized any terms regarding COW backing chains or R/W filter chains. This series is an attempt.

Quoted 15 lines
Quoted 6 lines

qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.

raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.

Quoted 8 lines

(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.

Quoted 2 lines

Agreed with offset, but with only size, it matches your definition of a filter.

So?

Should we treat it as a filter when @offset is 0 but otherwise not? That totally wouldn’t be confusing to users.

Quoted 9 lines
Quoted 9 lines

Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)

Quoted 3 lines

It looks like bdrv_filtered_child() is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.

Quoted 8 lines
Quoted 2 lines

It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.

qcow2 doesn't fulfill the conditions for begin a filter driver. Two of its possible children fulfill the conditions for being a filtered child. You can pick either approach, talking about a "filter chain" just doesn't make sense there. Either the chain is broken by a non-filter driver like qcow2, or it must become a filter tree.

I have no idea what your point is. There is no point in making it a filter tree at this point, just as we never had a backing tree.

And the good example is Quorum. qcow2 is a bad example because there is no benefit in marking it an R/W filter for its external data file, exactly like is the case for raw.

Quoted 5 lines

What we're really interested in is iterating the backing chain even across filter nodes, so your implementation achieves the right result. It just feels completely arbitrary, counterintuitive and confusing to call this a (or actually "the") "filter chain" and to pretend that the name tells anyone what it really is.

So exactly the same as “bs->backing” or “backing chain” for me.

You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.

The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.

Max

#10 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:

Quoted 29 lines · Kevin Wolf
Quoted 28 lines

Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 13 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 12 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Quoted 8 lines
Quoted 2 lines

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Apparently any child of a filter node is "filtered" (which makes sense),

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Quoted 5 lines
Quoted 3 lines

Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?

Well, there’s Quorum...

Ah, nice, quorum sets is_filter = true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.

So either quorum lies and isn't actually a filter driver, or our definition in the documentation of is_filter is wrong.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

Quoted 7 lines
Quoted 5 lines

But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.

Which is why it currently returns NULL for Quorum.

Which is about the only possible choice that breaks the contract...

* Return any filtered child, independently of how it reacts to write
* accesses and whether data is copied onto this BDS through COR.

Maybe the documentation of bdrv_filtered_child() needs to be rephrased?

Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too. So if quorum returns NULL to mean "no unambiguous result", why does it return bs->backing instead of NULL for a qcow2 node?

(Yes, I know, because it's useful. But I'm trying to get some basic consistency into these interfaces.)

Quoted 5 lines
Quoted 2 lines

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?

Wait, where is there even a place where this could be declared?

The once thing I see that a driver even can declare is drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.

Then you made some criteria above that tell us whether a specific child of a node is a filtered child or not. As it happens, qcow2 (which is not a filter driver) can have two children that match the criteria for being filtered children.

I already think this is a bit inconsistent, because why should a driver that declares itself a non-filter be considered to filter children? Okay, you say a broader definition of a filtered child is useful because you can then include all BdrvChild links in a backing/filter chain. Fair enough, it's not intuitive, but use a broader definition then.

But the point where you say that even though two of the children are filtered children under your broader definition, for the purpose of the API only one of them should be considered because the other one isn't that useful, that's really one inconsistency too much for me. You can't use a broad definition and then arbitrarily restrict the definition again so that it matches the special case you're currently interested in.

Either use a narrow definition, or use a broad one. But use only one and use it consistently.

Quoted 11 lines
Quoted 7 lines
Quoted 5 lines

The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.

We generally call this concept a "backing chain".

I suppose that’s an exclusive “we”? Because I use ”backing chain” to refer to COW chains exclusively.

Quoted 6 lines

Such a chain may or may not include filters, but they are not really load-bearing nodes of the chain. As such, I generally want to skip them when looking at a backing chain (hence e.g. bdrv_backing_chain_next()).

From what I can tell nobody has ever formalized any terms regarding COW backing chains or R/W filter chains. This series is an attempt.

Well, as you can see, this attempt feels confusing to me.

I agree with your naming of bdrv_backing_chain_next(), it's clear enough what path it will follow down the graph. I just disagree that "filter chain" is a good term for something that prefers backing file links when it has a choice.

Quoted 15 lines
Quoted 15 lines
Quoted 14 lines

qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.

raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.

(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.

Quoted 7 lines
Quoted 2 lines

Agreed with offset, but with only size, it matches your definition of a filter.

So?

Should we treat it as a filter when @offset is 0 but otherwise not? That totally wouldn’t be confusing to users.

No, I'm just applying your definitions to see if the contradictions between them and your explanations are of any importance. *shrug*

Quoted 18 lines
Quoted 9 lines
Quoted 9 lines

Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)

Quoted 3 lines

It looks like bdrv_filtered_child() is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.

Quoted 8 lines
Quoted 2 lines

It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.

qcow2 doesn't fulfill the conditions for begin a filter driver. Two of its possible children fulfill the conditions for being a filtered child. You can pick either approach, talking about a "filter chain" just doesn't make sense there. Either the chain is broken by a non-filter driver like qcow2, or it must become a filter tree.

Quoted 6 lines

I have no idea what your point is. There is no point in making it a filter tree at this point, just as we never had a backing tree.

And the good example is Quorum. qcow2 is a bad example because there is no benefit in marking it an R/W filter for its external data file, exactly like is the case for raw.

My point is not about changing the logic in the code, but about using names that actually describe accurately what the logic does.

And as I said above, neither is "not useful" a convincing argument for ignoring filtered children (as I think we're trying to build something rather generic, not something that works only for what we consider useful today) nor do I see how qcow2 could be marked or not marked an R/W filter (as mentioned above).

Quoted 6 lines
Quoted 5 lines

What we're really interested in is iterating the backing chain even across filter nodes, so your implementation achieves the right result. It just feels completely arbitrary, counterintuitive and confusing to call this a (or actually "the") "filter chain" and to pretend that the name tells anyone what it really is.

Quoted 8 lines

So exactly the same as “bs->backing” or “backing chain” for me.

You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.

The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.

Well, if "backing chain" is too narrow, "filter chain" is both too unspecific and inconsistent with the various definitions of "filter" and "filtered" we're using, and we can't think of anything more concise, we might have to use names that just mention both.

bdrv_cow_child() // don't call COW a filter, because .is_filter = false bdrv_filter_child() // your R/W filter, only for .is_filter = true nodes bdrv_filter_or_cow_child()

Or something like that. This would bring some more consistency into the way we use the words filter/filtered at least.

Kevin

author
#11 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Quoted 34 lines · Kevin Wolf

Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:

Quoted 33 lines · Kevin Wolf
Quoted 31 lines

Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 20 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 7 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

Quoted 7 lines

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Quoted 5 lines
Quoted 1 line

Apparently any child of a filter node is "filtered" (which makes sense),

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?

Quoted 8 lines
Quoted 1 line

Well, there’s Quorum...

Ah, nice, quorum sets is_filter = true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.

So either quorum lies and isn't actually a filter driver, or our definition in the documentation of is_filter is wrong.

You could say it lies because in FIFO mode it clearly isn’t a filter for all of its children.

There is a reason for lying, though, which is bdrv_recurse_is_first_non_filter(), which is necessary to use the whole to_replace mirror stuff.

(You mirror from a quorum with a failed child and then replace the failed child. mirror needs to ensure that there are only R/W filters between the child and the mirror source so that replacing it will not suddenly change any visible data. Which is actually a lie for quorum, because the child is clearly broken and thus precisely doesn’t show the same data...)

Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 9 lines
Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.

Quoted 5 lines
Quoted 1 line

Which is why it currently returns NULL for Quorum.

Which is about the only possible choice that breaks the contract...

* Return any filtered child, independently of how it reacts to write

I don’t know if you’re serious about this proposition, because I don’t know whether that could be useful in any way. :-?

Quoted 1 line

* accesses and whether data is copied onto this BDS through COR.

I meant the contract as “Return the single filtered child there is, or NULL”

Quoted 4 lines

Maybe the documentation of bdrv_filtered_child() needs to be rephrased?

Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.

Well, it doesn’t. It isn’t an R/W filter.

Maybe what we actually need to rephrase is the definition of .is_filter.
(Namely something along the lines of “Fulfills these guarantees (same
data, etc. pp.), *and* should be skipped for allocation information queries etc.”.

Quoted 6 lines

So if quorum returns NULL to mean "no
unambiguous result", why does it return bs->backing instead of NULL for a qcow2 node?

(Yes, I know, because it's useful. But I'm trying to get some basic consistency into these interfaces.)

Not precisely because it’s useful, but because qcow2 does not have .is_filter set. :-)
(And it doesn’t have it set because that wouldn’t be useful.)

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?

Wait, where is there even a place where this could be declared?

The once thing I see that a driver even can declare is drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.

That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.

Quoted 4 lines

Then you made some criteria above that tell us whether a specific child of a node is a filtered child or not. As it happens, qcow2 (which is not a filter driver) can have two children that match the criteria for being filtered children.

But just arguing that I’m incapable of giving a good definition won’t bring us along.

Quoted 2 lines

I already think this is a bit inconsistent, because why should a driver that declares itself a non-filter be considered to filter children?

.is_filter is for R/W filters. COW filters have .supports_backing for that.

Quoted 12 lines

Okay, you say a broader definition of a filtered child is useful because you can then include all BdrvChild links in a backing/filter chain. Fair enough, it's not intuitive, but use a broader definition then.

But the point where you say that even though two of the children are filtered children under your broader definition, for the purpose of the API only one of them should be considered because the other one isn't that useful, that's really one inconsistency too much for me. You can't use a broad definition and then arbitrarily restrict the definition again so that it matches the special case you're currently interested in.

Quoted 2 lines

Either use a narrow definition, or use a broad one. But use only one and use it consistently.

I think the problem appears because you restrict the process to a single step where there’s actually two.

Drivers can be either
(1) R/W filters (e.g. throttle)
(2) COW filters (e.g. qcow2)
(3) None of the above (e.g. vhdx, curl)

This choice is made on the driver level, not on the node level (for good reason, see below*).

And then we derive the node’s filtered children from what the driver is.
If it’s an R/W filter, bdrv_filtered_child() will return the R/W
filtered child. If it’s a COW filter, bdrv_filtered_child() will return the potentially existing COW backing child (or NULL, if there is no backing child).

*
What is clear to me is that it isn’t useful to treat nodes of a specific driver sometimes as R/W filter nodes and sometimes not. R/W filter nodes are handled differently from other nodes, and it would be confusing if a certain driver sometimes behaves this and sometimes that way. (For example, if you put a raw node on top of a qcow2 node, sometimes it would stop the backing chain, sometimes it wouldn’t. That makes no sense to me.)

OTOH, for COW filters, we do exactly that. Sometimes they have a backing file, sometimes they don’t. That’s completely fine because their overall behavior doesn’t change.

That makes me agree that there is indeed too much of a difference between R/W filters and COW filters to lump them together under the “filter” label.

[...]

Quoted 2 lines

My point is not about changing the logic in the code, but about using names that actually describe accurately what the logic does.

Again, naming things is hard.

[...]

Quoted 12 lines
Quoted 6 lines

You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.

The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.

Well, if "backing chain" is too narrow, "filter chain" is both too unspecific and inconsistent with the various definitions of "filter" and "filtered" we're using, and we can't think of anything more concise, we might have to use names that just mention both.

Quoted 6 lines

bdrv_cow_child() // don't call COW a filter, because .is_filter = false bdrv_filter_child() // your R/W filter, only for .is_filter = true nodes bdrv_filter_or_cow_child()

Or something like that. This would bring some more consistency into the way we use the words filter/filtered at least.

I’ll see how that looks overall, but why not. Sounds good to me.

Max

#12 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:

Quoted 44 lines · Kevin Wolf
Quoted 34 lines

Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:

Quoted 33 lines · Kevin Wolf
Quoted 31 lines

Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 20 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 7 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

Quoted 7 lines

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Quoted 5 lines
Quoted 1 line

Apparently any child of a filter node is "filtered" (which makes sense),

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?

Quoted 8 lines
Quoted 1 line

Well, there’s Quorum...

Ah, nice, quorum sets is_filter = true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.

So either quorum lies and isn't actually a filter driver, or our definition in the documentation of is_filter is wrong.

Quoted 6 lines

You could say it lies because in FIFO mode it clearly isn’t a filter for all of its children.

There is a reason for lying, though, which is bdrv_recurse_is_first_non_filter(), which is necessary to use the whole to_replace mirror stuff.

Hm, actually, now that you mention bdrv_recurse_is_first_non_filter(), quorum was the first driver to declare itself a filter, so strictly speaking, if there is an inconsistency, it's the other uses that are abusing the field...

Quoted 10 lines

(You mirror from a quorum with a failed child and then replace the failed child. mirror needs to ensure that there are only R/W filters between the child and the mirror source so that replacing it will not suddenly change any visible data. Which is actually a lie for quorum, because the child is clearly broken and thus precisely doesn’t show the same data...)

Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?

Why not.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 9 lines
Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Which is why it currently returns NULL for Quorum.

Which is about the only possible choice that breaks the contract...

* Return any filtered child, independently of how it reacts to write

I don’t know if you’re serious about this proposition, because I don’t know whether that could be useful in any way. :-?

Huh? This is just quoting the contract from your code?

Quoted 3 lines
Quoted 1 line

* accesses and whether data is copied onto this BDS through COR.

I meant the contract as “Return the single filtered child there is, or NULL”

Then that should probably be spelt out in the contract. Probably even explicitly "NULL if there is either no filtered child or multiple filtered children".

Quoted 6 lines
Quoted 4 lines

Maybe the documentation of bdrv_filtered_child() needs to be rephrased?

Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.

Well, it doesn’t. It isn’t an R/W filter.

What do I have to look at to see whether something is an R/W filter or not? qcow2 matches your criteria for an R/W filter. You say that it's not useful, so it's not an R/W filter anyway. But where in the code could I get this information?

This just doesn't make sense to me. If a driver matches the criteria for an R/W filter, then it should be one. If qcow2 should not be considered a R/W filter, then the criteria must be changed so that it isn't.

Quoted 4 lines

Maybe what we actually need to rephrase is the definition of .is_filter.
(Namely something along the lines of “Fulfills these guarantees (same
data, etc. pp.), *and* should be skipped for allocation information queries etc.”.

Hm - does this imply that .is_filter == this is a R/W filter? Because this was never spelt out, neither in code comments nor in commit messages.

If we called R/W filters just "filters" (which makes it obvious how it relates to .is_filter) and COW nodes something that doesn't include the word "filter", things might become a lot clearer.

Quoted 17 lines
Quoted 6 lines

So if quorum returns NULL to mean "no
unambiguous result", why does it return bs->backing instead of NULL for a qcow2 node?

(Yes, I know, because it's useful. But I'm trying to get some basic consistency into these interfaces.)

Not precisely because it’s useful, but because qcow2 does not have .is_filter set. :-)
(And it doesn’t have it set because that wouldn’t be useful.)

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?

Quoted 8 lines
Quoted 5 lines

Wait, where is there even a place where this could be declared?

The once thing I see that a driver even can declare is drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.

That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.

I was thinking per-child, actually, because you declare one BdrvChild filtered and another not filtered.

But by now I think most of the confusion is really just a result of COW being considered a filter in some respects (mainly just the names of the child access functions), but not in others (like .is_filter).

Quoted 5 lines
Quoted 4 lines

Then you made some criteria above that tell us whether a specific child of a node is a filtered child or not. As it happens, qcow2 (which is not a filter driver) can have two children that match the criteria for being filtered children.

Quoted 7 lines

But just arguing that I’m incapable of giving a good definition won’t bring us along.

Quoted 2 lines

I already think this is a bit inconsistent, because why should a driver that declares itself a non-filter be considered to filter children?

.is_filter is for R/W filters. COW filters have .supports_backing for that.

Okay, so you confirm what I concluded above.

Quoted 18 lines
Quoted 12 lines

Okay, you say a broader definition of a filtered child is useful because you can then include all BdrvChild links in a backing/filter chain. Fair enough, it's not intuitive, but use a broader definition then.

But the point where you say that even though two of the children are filtered children under your broader definition, for the purpose of the API only one of them should be considered because the other one isn't that useful, that's really one inconsistency too much for me. You can't use a broad definition and then arbitrarily restrict the definition again so that it matches the special case you're currently interested in.

Quoted 2 lines

Either use a narrow definition, or use a broad one. But use only one and use it consistently.

I think the problem appears because you restrict the process to a single step where there’s actually two.

Quoted 7 lines

Drivers can be either
(1) R/W filters (e.g. throttle)
(2) COW filters (e.g. qcow2)
(3) None of the above (e.g. vhdx, curl)

This choice is made on the driver level, not on the node level (for good reason, see below*).

What prevents a driver from being
(4) COW filter and R/W filter (e.g. qcow2 if it were useful)?

I mean, conceptually, not in the implementation.

Quoted 5 lines

And then we derive the node’s filtered children from what the driver is.
If it’s an R/W filter, bdrv_filtered_child() will return the R/W
filtered child. If it’s a COW filter, bdrv_filtered_child() will return the potentially existing COW backing child (or NULL, if there is no backing child).

I guess it boils down to me just not being able to get behind the concept that COW is some sort of filter (especially when other things like an external data file aren't).

Quoted 39 lines

*
What is clear to me is that it isn’t useful to treat nodes of a specific driver sometimes as R/W filter nodes and sometimes not. R/W filter nodes are handled differently from other nodes, and it would be confusing if a certain driver sometimes behaves this and sometimes that way. (For example, if you put a raw node on top of a qcow2 node, sometimes it would stop the backing chain, sometimes it wouldn’t. That makes no sense to me.)

OTOH, for COW filters, we do exactly that. Sometimes they have a backing file, sometimes they don’t. That’s completely fine because their overall behavior doesn’t change.

That makes me agree that there is indeed too much of a difference between R/W filters and COW filters to lump them together under the “filter” label.

[...]

Quoted 2 lines

My point is not about changing the logic in the code, but about using names that actually describe accurately what the logic does.

Again, naming things is hard.

[...]

Quoted 12 lines
Quoted 6 lines

You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.

The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.

Well, if "backing chain" is too narrow, "filter chain" is both too unspecific and inconsistent with the various definitions of "filter" and "filtered" we're using, and we can't think of anything more concise, we might have to use names that just mention both.

Quoted 8 lines
Quoted 6 lines

bdrv_cow_child() // don't call COW a filter, because .is_filter = false bdrv_filter_child() // your R/W filter, only for .is_filter = true nodes bdrv_filter_or_cow_child()

Or something like that. This would bring some more consistency into the way we use the words filter/filtered at least.

I’ll see how that looks overall, but why not. Sounds good to me.

Good. Or, well, good enough at least. ;-)

bdrv_filter_or_cow_child() is not a pretty name, but as long as we can't think of anything that accurately covers both in a single word, it will do the job...

Kevin

author
#13 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Quoted 64 lines · Kevin Wolf

Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:

Quoted 44 lines · Kevin Wolf
Quoted 34 lines

Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:

Quoted 33 lines · Kevin Wolf
Quoted 31 lines

Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 20 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 7 lines

There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.

Quoted 7 lines

This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.

Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?

Quoted 5 lines
Quoted 1 line

Apparently any child of a filter node is "filtered" (which makes sense),

It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.

Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?

Quoted 8 lines
Quoted 1 line

Well, there’s Quorum...

Ah, nice, quorum sets is_filter = true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.

So either quorum lies and isn't actually a filter driver, or our definition in the documentation of is_filter is wrong.

Quoted 6 lines

You could say it lies because in FIFO mode it clearly isn’t a filter for all of its children.

There is a reason for lying, though, which is bdrv_recurse_is_first_non_filter(), which is necessary to use the whole to_replace mirror stuff.

Hm, actually, now that you mention bdrv_recurse_is_first_non_filter(), quorum was the first driver to declare itself a filter, so strictly speaking, if there is an inconsistency, it's the other uses that are abusing the field...

Quoted 7 lines

(You mirror from a quorum with a failed child and then replace the failed child. mirror needs to ensure that there are only R/W filters between the child and the mirror source so that replacing it will not suddenly change any visible data. Which is actually a lie for quorum, because the child is clearly broken and thus precisely doesn’t show the same data...)

Quoted 5 lines
Quoted 3 lines

Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?

Why not.

It feels difficult to do in this series because this is a whole new can of worms.

In patch 35, I actually replace the mirror use case by is_filtered_child(). So it looks to me as if that should not be done, because I should instead fix bdrv_recurse_is_first_non_filter() (and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.

OTOH, there are other users of bdrv_is_first_non_filter(). Those are qmp_block_resize() and external_snapshot_prepare(), who throw an error if that returns false.

I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system). qmp_block_resize() should always work as long as it can get BLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines

but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?

“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).

Quoted 5 lines

But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.

Which is why it currently returns NULL for Quorum.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Which is about the only possible choice that breaks the contract...

* Return any filtered child, independently of how it reacts to write

I don’t know if you’re serious about this proposition, because I don’t know whether that could be useful in any way. :-?

Huh? This is just quoting the contract from your code?

I see. I was thinking about “any of COW/RW, of which only one exists”.
There is an assertion for that (that only one filtered child exists at
a time) in the code. (And I consider assertions part of the contract.)

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

* accesses and whether data is copied onto this BDS through COR.

I meant the contract as “Return the single filtered child there is, or NULL”

Then that should probably be spelt out in the contract.Probably even explicitly "NULL if there is either no filtered child or multiple filtered children".

Well, it’s spelled out through the assertion, but not in the documentation, yes.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

Maybe the documentation of bdrv_filtered_child() needs to be rephrased?

Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.

Well, it doesn’t. It isn’t an R/W filter.

What do I have to look at to see whether something is an R/W filter or not? qcow2 matches your criteria for an R/W filter.

No. Some qcow2 nodes match the criteria. But not all, which makes the qcow2 driver not a filter driver.

Quoted 3 lines

You say that it's
not useful, so it's not an R/W filter anyway. But where in the code could I get this information?

“Where in the code”? Do you want to add a comment to every BlockDriver structure on why it does or doesn’t set .is_filter?

Quoted 3 lines

This just doesn't make sense to me. If a driver matches the criteria for an R/W filter, then it should be one. If qcow2 should not be considered a R/W filter, then the criteria must be changed so that it isn't.

See below.

Quoted 8 lines
Quoted 4 lines

Maybe what we actually need to rephrase is the definition of .is_filter.
(Namely something along the lines of “Fulfills these guarantees (same
data, etc. pp.), *and* should be skipped for allocation information queries etc.”.

Hm - does this imply that .is_filter == this is a R/W filter? Because this was never spelt out, neither in code comments nor in commit messages.

While I’m not a fan of comment-less code, I do think that it’s possible to read code. Which clearly stated this.

Quoted 3 lines

If we called R/W filters just "filters" (which makes it obvious how it relates to .is_filter) and COW nodes something that doesn't include the word "filter", things might become a lot clearer.

Because you apparently wrote this before reading that I agreed to your renaming proposal, I now feel free to argue that I could just as well rename .is_filter to .is_rw_filter.

Obviously I won’t because I prefer your proposal.

[...]

Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?

Wait, where is there even a place where this could be declared?

The once thing I see that a driver even can declare is drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.

Quoted 5 lines
Quoted 2 lines

That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.

I was thinking per-child, actually, because you declare one BdrvChild filtered and another not filtered.

Why don’t you say so from the start then?

(Sorry, but honestly about 30 % of this discussion to me feels like you’re playing games with me. Please don’t take this the wrong way, I mean it very neutrally. It’s just that I feel like I’m explaining things to you that you very much know, but you just want me to say them.
And that feels unproductive and sometimes indeed frustrating.)

One thing is that this wouldn’t make the quorum case any easier because it actually doesn’t know for which children it acts as a filter and for which it doesn’t.

Quoted 3 lines

But by now I think most of the confusion is really just a result of COW being considered a filter in some respects (mainly just the names of the child access functions), but not in others (like .is_filter).

I don’t quite see how it’s “by now” when in your first mail you already basically wrote that functionally, everything works (leaving out quorum), but that you’re confused (or claim to be confused, I have no idea what’s real and what’s pretended anymore) by the names.

We have come to two results, as far as I can see:

First, naming COW backing nodes “COW filtered children” clashes with our existing use of ”filter”. There is no point in forcing the ”filter” label on everything. We can just keep calling (R/W) filters filters and COW backing children COW children. The names are succinct enough.

In some cases, we don’t care whether something is a COW or filtered child, in such a case a caller can be bothered to use the slightly longer bdrv_cow_or_filtered_child().

Second, most of the time we want a filter node to have a clear and unique path to go down. This is the important property of filters: That you can skip them and go to the node that actually has the data.

Quorum breaks this by having multiple children, and nobody knows which of them has the data we will see on the next read operation.

All “filters” who could have multiple children would have this problem.
Hence a filter must always have a single unique data child. I think.

[...]

Quoted 11 lines
Quoted 11 lines
Quoted 2 lines

Either use a narrow definition, or use a broad one. But use only one and use it consistently.

I think the problem appears because you restrict the process to a single step where there’s actually two.

Drivers can be either
(1) R/W filters (e.g. throttle)
(2) COW filters (e.g. qcow2)
(3) None of the above (e.g. vhdx, curl)

Quoted 7 lines
Quoted 2 lines

This choice is made on the driver level, not on the node level (for good reason, see below*).

What prevents a driver from being
(4) COW filter and R/W filter (e.g. qcow2 if it were useful)?

I mean, conceptually, not in the implementation.

An R/W filter always shows the same data as the filtered child. So the COW child‘s data can never be visible, and as such you couldn’t have a COW child at the same time.

Max

#14 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Am 10.09.2019 um 13:36 hat Max Reitz geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 6 lines

Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:

Quoted 3 lines

Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?

Why not.

It feels difficult to do in this series because this is a whole new can of worms.

In patch 35, I actually replace the mirror use case by is_filtered_child(). So it looks to me as if that should not be done, because I should instead fix bdrv_recurse_is_first_non_filter() (and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.

OTOH, there are other users of bdrv_is_first_non_filter(). Those are qmp_block_resize() and external_snapshot_prepare(), who throw an error if that returns false.

Quoted 5 lines

I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system). qmp_block_resize() should always work as long as it can get BLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.

Hm, doesn't it make sense in a way for qmp_block_resize() at least? It means that you can't resize just a filter, but you need to resize the image that actually provides the data for the filter.

Of course, there is no reason for it to be the _first_ non-filter as long as BLK_PERM_RESIZE is shared, but just some non-filter node.

Two more random observations:

* quorum uses bdrv_filter_default_perms(), which allows BLK_PERM_RESIZE. I think this is wrong and quorum should make sure that all children are always the same size because otherwise it can't tell what its own size is. (Or vote on size...? :-/) Probably not a problem in practice as long as we check bdrv_is_first_non_filter().

* child_file and child_backing don't implement .resize. So if you resize a non-top-level image, parents (in particular filters) don't get their size adjusted. This is probably a bug, too, but one that isn't prevented by bdrv_is_first_non_filter() and should be visible today.

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

Maybe the documentation of bdrv_filtered_child() needs to be rephrased?

Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.

Well, it doesn’t. It isn’t an R/W filter.

What do I have to look at to see whether something is an R/W filter or not? qcow2 matches your criteria for an R/W filter.

Quoted 8 lines

No. Some qcow2 nodes match the criteria. But not all, which makes the qcow2 driver not a filter driver.

Quoted 2 lines

You say that it's not useful, so it's not an R/W filter anyway. But where in the code could I get this information?

“Where in the code”? Do you want to add a comment to every BlockDriver structure on why it does or doesn’t set .is_filter?

Never mind, I just didn't understand that .is_filter is the thing that defines a R/W filter. In fact, I didn't really understand what .is_filter was supposed to mean at all because I was so confused. For some reason I was sure it had to mean any kind of filter, but that assumption just didn't match up with its use at all.

Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).

Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?

Wait, where is there even a place where this could be declared?

The once thing I see that a driver even can declare is drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.

I was thinking per-child, actually, because you declare one BdrvChild filtered and another not filtered.

Why don’t you say so from the start then?

Yes, I wrote "nodes", thought "child nodes" and should have said "children" because edges are not nodes. My bad, sorry.

Quoted 5 lines

(Sorry, but honestly about 30 % of this discussion to me feels like you’re playing games with me. Please don’t take this the wrong way, I mean it very neutrally. It’s just that I feel like I’m explaining things to you that you very much know, but you just want me to say them.
And that feels unproductive and sometimes indeed frustrating.)

No, certainly not. If my mails seemed confusing or pointless, it just shows how thoroughly confused I was.

Quoted 4 lines

One thing is that this wouldn’t make the quorum case any easier because it actually doesn’t know for which children it acts as a filter and for which it doesn’t.

Quoted 8 lines
Quoted 3 lines

But by now I think most of the confusion is really just a result of COW being considered a filter in some respects (mainly just the names of the child access functions), but not in others (like .is_filter).

I don’t quite see how it’s “by now” when in your first mail you already basically wrote that functionally, everything works (leaving out quorum), but that you’re confused (or claim to be confused, I have no idea what’s real and what’s pretended anymore) by the names.

Well, I saw that the special cases in the patches that I had reviewed so far seemed to be converted correctly, but I just didn't understand the whole concept behind it. It's possible to both understand that a transformation is correct and to fail to grasp the concept behind it.

And your first answer only confused me more because you gave definitions for R/W and COW filters that honestly ended up a bit misleading, possibly as a result of your endeavour to make R/W filters and COW sound like the same thing. (Which made me lose sight of basic facts like that R/W filters must forward _every_ request without exception to their filtered child even though COW doesn't.)

Quoted 10 lines

We have come to two results, as far as I can see:

First, naming COW backing nodes “COW filtered children” clashes with our existing use of ”filter”. There is no point in forcing the ”filter” label on everything. We can just keep calling (R/W) filters filters and COW backing children COW children. The names are succinct enough.

In some cases, we don’t care whether something is a COW or filtered child, in such a case a caller can be bothered to use the slightly longer bdrv_cow_or_filtered_child().

Aye.

Quoted 9 lines

Second, most of the time we want a filter node to have a clear and unique path to go down. This is the important property of filters: That you can skip them and go to the node that actually has the data.

Quorum breaks this by having multiple children, and nobody knows which of them has the data we will see on the next read operation.

All “filters” who could have multiple children would have this problem.
Hence a filter must always have a single unique data child. I think.

I agree, and this is the condition that I mentioned somewhere above, but failed to actually find guaranteed somewhere. We should probably make this explicit.

Of course, quorum and similar things intend all their children to provide the same data, but the whole point of the driver is that this is not always guaranteed, so they aren't actually filters.

Kevin

author
#15 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Quoted 28 lines · Kevin Wolf

Am 10.09.2019 um 13:36 hat Max Reitz geschrieben:

Quoted 21 lines · Kevin Wolf
Quoted 6 lines

Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:

Quoted 3 lines

Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?

Why not.

It feels difficult to do in this series because this is a whole new can of worms.

In patch 35, I actually replace the mirror use case by is_filtered_child(). So it looks to me as if that should not be done, because I should instead fix bdrv_recurse_is_first_non_filter() (and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.

OTOH, there are other users of bdrv_is_first_non_filter(). Those are qmp_block_resize() and external_snapshot_prepare(), who throw an error if that returns false.

Quoted 5 lines

I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system). qmp_block_resize() should always work as long as it can get BLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.

Quoted 3 lines

Hm, doesn't it make sense in a way for qmp_block_resize() at least? It means that you can't resize just a filter, but you need to resize the image that actually provides the data for the filter.

Filters generally implement .bdrv_truncate() by passing it through, so it should be fine.

Quoted 10 lines

Of course, there is no reason for it to be the _first_ non-filter as long as BLK_PERM_RESIZE is shared, but just some non-filter node.

Two more random observations:

* quorum uses bdrv_filter_default_perms(), which allows BLK_PERM_RESIZE. I think this is wrong and quorum should make sure that all children are always the same size because otherwise it can't tell what its own size is. (Or vote on size...? :-/) Probably not a problem in practice as long as we check bdrv_is_first_non_filter().

(“Quorum is broken” seems to be a recurring observation.)

I agree, it shouldn’t share that permission.

Quoted 4 lines

* child_file and child_backing don't implement .resize. So if you resize a non-top-level image, parents (in particular filters) don't get their size adjusted. This is probably a bug, too, but one that isn't prevented by bdrv_is_first_non_filter() and should be visible today.

Hm. :-/

The good news is that I can try to fix this independently of this series.

[...]

Quoted 23 lines
Quoted 10 lines

We have come to two results, as far as I can see:

First, naming COW backing nodes “COW filtered children” clashes with our existing use of ”filter”. There is no point in forcing the ”filter” label on everything. We can just keep calling (R/W) filters filters and COW backing children COW children. The names are succinct enough.

In some cases, we don’t care whether something is a COW or filtered child, in such a case a caller can be bothered to use the slightly longer bdrv_cow_or_filtered_child().

Aye.

Quoted 9 lines

Second, most of the time we want a filter node to have a clear and unique path to go down. This is the important property of filters: That you can skip them and go to the node that actually has the data.

Quorum breaks this by having multiple children, and nobody knows which of them has the data we will see on the next read operation.

All “filters” who could have multiple children would have this problem.
Hence a filter must always have a single unique data child. I think.

Quoted 7 lines

I agree, and this is the condition that I mentioned somewhere above, but failed to actually find guaranteed somewhere. We should probably make this explicit.

Of course, quorum and similar things intend all their children to provide the same data, but the whole point of the driver is that this is not always guaranteed, so they aren't actually filters.

OK, great, I’ll get cracking then.

Max

#16 lore raw

Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions

Am 10.09.2019 um 14:59 hat Max Reitz geschrieben:

Quoted 29 lines · Kevin Wolf
Quoted 28 lines

Am 10.09.2019 um 13:36 hat Max Reitz geschrieben:

Quoted 26 lines · Kevin Wolf
Quoted 6 lines

Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:

Quoted 3 lines

Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?

Why not.

It feels difficult to do in this series because this is a whole new can of worms.

In patch 35, I actually replace the mirror use case by is_filtered_child(). So it looks to me as if that should not be done, because I should instead fix bdrv_recurse_is_first_non_filter() (and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.

OTOH, there are other users of bdrv_is_first_non_filter(). Those are qmp_block_resize() and external_snapshot_prepare(), who throw an error if that returns false.

I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system). qmp_block_resize() should always work as long as it can get BLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.

Quoted 6 lines
Quoted 3 lines

Hm, doesn't it make sense in a way for qmp_block_resize() at least? It means that you can't resize just a filter, but you need to resize the image that actually provides the data for the filter.

Filters generally implement .bdrv_truncate() by passing it through, so it should be fine.

Good point.

Then checking bdrv_is_first_non_filter() probably just forbids the only command that would actually work correctly (resizing the top-level filter).

Kevin

author
#17 lore raw

PATCH v6 5/42 block: Add chain helper functions

In reply to Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
 include/block/block_int.h |  3 +++
 block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
index 6c60abc4c3..5bec3361fd 100644
@@ -1291,6 +1291,9 @@ BdrvChild *bdrv_filtered_child(BlockDriverState *bs);
 BdrvChild *bdrv_metadata_child(BlockDriverState *bs);
 BdrvChild *bdrv_storage_child(BlockDriverState *bs);
 BdrvChild *bdrv_primary_child(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_implicit_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_skip_rw_filters(BlockDriverState *bs);
+BlockDriverState *bdrv_backing_chain_next(BlockDriverState *bs);
 
 static inline BlockDriverState *child_bs(BdrvChild *child)
 {
block.c +55 -0

58 lines not on this page · raw message

Signature · 1 line
2.21.0
Reviewed-by on v6 5/42
#18 lore raw

Re: [Qemu-devel] [PATCH v6 05/42] block: Add chain helper functions

Quoted 9 lines · Max Reitz

Add some helper functions for skipping filters in a chain of block nodes.

---
 include/block/block_int.h |  3 +++
 block.c                   | 55 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#19 lore raw

PATCH v6 6/42 qcow2: Implement .bdrv_storage_child()

In reply to Max Reitz
---
 block/qcow2.c | 9 +++++++++
 1 file changed, 9 insertions(+)
block/qcow2.c +9 -0
index 039bdc2f7e..f8570d6210 100644
@@ -5086,6 +5086,13 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool fatal, int64_t offset,
     s->signaled_corruption = true;
 }
 
+static BdrvChild *qcow2_storage_child(BlockDriverState *bs)
+{
+    BDRVQcow2State *s = bs->opaque;
+
+    return s->data_file;
+}
+
 static QemuOptsList qcow2_create_opts = {
     .name = "qcow2-create-opts",
     .head = QTAILQ_HEAD_INITIALIZER(qcow2_create_opts.head),
@@ -5232,6 +5239,8 @@ BlockDriver bdrv_qcow2 = {
     .bdrv_reopen_bitmaps_rw = qcow2_reopen_bitmaps_rw,
     .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap,
     .bdrv_remove_persistent_dirty_bitmap = qcow2_remove_persistent_dirty_bitmap,
+
+    .bdrv_storage_child = qcow2_storage_child,
 };
 
 static void bdrv_qcow2_init(void)
Signature · 1 line
2.21.0
Reviewed-by on v6 6/42
#20 lore raw

Re: [Qemu-devel] [PATCH v6 06/42] qcow2: Implement .bdrv_storage_child()

Quoted 6 lines · Max Reitz
---
 block/qcow2.c | 9 +++++++++
 1 file changed, 9 insertions(+)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org
author
#21 lore raw

PATCH v6 7/42 block: *filtered_cow_child() for *has_zero_init()

In reply to Max Reitz

bdrv_has_zero_init() and the related bdrv_unallocated_blocks_are_zero() should use bdrv_filtered_cow_child() if they want to check whether the given BDS has a COW backing file.

---
 block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
index bfa5e27850..486c75d847 100644
@@ -5063,7 +5063,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)
 
     /* If BS is a copy on write image, it is initialized to
        the contents of the base image, which may not be zeroes.  */
-    if (bs->backing) {
+    if (bdrv_filtered_cow_child(bs)) {
         return 0;
     }
     if (bs->drv->bdrv_has_zero_init) {
@@ -5081,7 +5081,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)
 {
     BlockDriverInfo bdi;
 
-    if (bs->backing) {
+    if (bdrv_filtered_cow_child(bs)) {
         return false;
     }
 
Signature · 1 line
2.21.0
author
#22 lore raw

PATCH v6 8/42 block: bdrv_set_backing_hd() is about bs->backing

In reply to Max Reitz

bdrv_set_backing_hd() is a function that explicitly cares about the bs->backing child. Highlight that in its description and use child_bs(bs->backing) instead of backing_bs(bs) to make it more obvious.

---
 block.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
index 486c75d847..adf82efb0e 100644
@@ -2539,7 +2539,7 @@ static bool bdrv_inherits_from_recursive(BlockDriverState *child,
 }
 
 /*
- * Sets the backing file link of a BDS. A new reference is created; callers
+ * Sets the bs->backing link of a BDS. A new reference is created; callers
  * which don't need their own reference any more must call bdrv_unref().
  */
 void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
@@ -2548,7 +2548,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
     bool update_inherits_from = bdrv_chain_contains(bs, backing_hd) &&
         bdrv_inherits_from_recursive(backing_hd, bs);
 
-    if (bdrv_is_backing_chain_frozen(bs, backing_bs(bs), errp)) {
+    if (bdrv_is_backing_chain_frozen(bs, child_bs(bs->backing), errp)) {
         return;
     }
 
Signature · 1 line
2.21.0
author
#23 lore raw

PATCH v6 9/42 block: Include filters when freezing backing chain

In reply to Max Reitz

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.

In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

---
 include/block/block.h | 10 +++---
 block.c               | 81 +++++++++++++++++++++++++------------------
 block/commit.c        |  8 ++---
 block/mirror.c        |  4 +--
 block/stream.c        |  8 ++---
 5 files changed, 62 insertions(+), 49 deletions(-)
include/block/block.h +5 -5
index 50a07c1c33..f6f09b95cd 100644
@@ -364,11 +364,11 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs);
 BlockDriverState *bdrv_find_base(BlockDriverState *bs);
-bool bdrv_is_backing_chain_frozen(BlockDriverState *bs, BlockDriverState *base,
-                                  Error **errp);
-int bdrv_freeze_backing_chain(BlockDriverState *bs, BlockDriverState *base,
-                              Error **errp);
-void bdrv_unfreeze_backing_chain(BlockDriverState *bs, BlockDriverState *base);
+bool bdrv_is_chain_frozen(BlockDriverState *bs, BlockDriverState *base,
+                          Error **errp);
+int bdrv_freeze_chain(BlockDriverState *bs, BlockDriverState *base,
+                      Error **errp);
+void bdrv_unfreeze_chain(BlockDriverState *bs, BlockDriverState *base);
 
 
 typedef struct BdrvCheckResult {
block.c +47 -34

186 lines not on this page · raw message

block/commit.c +4 -4
index 19915603ae..5a7672c7c7 100644
@@ -68,7 +68,7 @@ static int commit_prepare(Job *job)
 {
     CommitBlockJob *s = container_of(job, CommitBlockJob, common.job);
 
-    bdrv_unfreeze_backing_chain(s->commit_top_bs, s->base_bs);
+    bdrv_unfreeze_chain(s->commit_top_bs, s->base_bs);
     s->chain_frozen = false;
 
     /* Remove base node parent that still uses BLK_PERM_WRITE/RESIZE before
@@ -88,7 +88,7 @@ static void commit_abort(Job *job)
     BlockDriverState *top_bs = blk_bs(s->top);
 
     if (s->chain_frozen) {
-        bdrv_unfreeze_backing_chain(s->commit_top_bs, s->base_bs);
+        bdrv_unfreeze_chain(s->commit_top_bs, s->base_bs);
     }
 
     /* Make sure commit_top_bs and top stay around until bdrv_replace_node() */
@@ -331,7 +331,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
         }
     }
 
-    if (bdrv_freeze_backing_chain(commit_top_bs, base, errp) < 0) {
+    if (bdrv_freeze_chain(commit_top_bs, base, errp) < 0) {
         goto fail;
     }
     s->chain_frozen = true;
@@ -372,7 +372,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
 
 fail:
     if (s->chain_frozen) {
-        bdrv_unfreeze_backing_chain(commit_top_bs, base);
+        bdrv_unfreeze_chain(commit_top_bs, base);
     }
     if (s->base) {
         blk_unref(s->base);
block/mirror.c +2 -2
index a8f2d7a305..54bafdf176 100644
@@ -633,7 +633,7 @@ static int mirror_exit_common(Job *job)
     s->prepared = true;
 
     if (bdrv_chain_contains(src, target_bs)) {
-        bdrv_unfreeze_backing_chain(mirror_top_bs, target_bs);
+        bdrv_unfreeze_chain(mirror_top_bs, target_bs);
     }
 
     bdrv_release_dirty_bitmap(src, s->dirty_bitmap);
@@ -1707,7 +1707,7 @@ static BlockJob *mirror_start_job(
             }
         }
 
-        if (bdrv_freeze_backing_chain(mirror_top_bs, target, errp) < 0) {
+        if (bdrv_freeze_chain(mirror_top_bs, target, errp) < 0) {
             goto fail;
         }
     }
block/stream.c +4 -4
index 6ac1e7bec4..4c8b89884a 100644
@@ -54,7 +54,7 @@ static void stream_abort(Job *job)
 
     if (s->chain_frozen) {
         BlockJob *bjob = &s->common;
-        bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_chain(blk_bs(bjob->blk), s->bottom);
     }
 }
 
@@ -67,7 +67,7 @@ static int stream_prepare(Job *job)
     Error *local_err = NULL;
     int ret = 0;
 
-    bdrv_unfreeze_backing_chain(bs, s->bottom);
+    bdrv_unfreeze_chain(bs, s->bottom);
     s->chain_frozen = false;
 
     if (bs->backing) {
@@ -233,7 +233,7 @@ void stream_start(const char *job_id, BlockDriverState *bs,
     int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
     BlockDriverState *bottom = bdrv_find_overlay(bs, base);
 
-    if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) {
+    if (bdrv_freeze_chain(bs, bottom, errp) < 0) {
         return;
     }
 
@@ -284,5 +284,5 @@ fail:
     if (bs_read_only) {
         bdrv_reopen_set_read_only(bs, true, NULL);
     }
-    bdrv_unfreeze_backing_chain(bs, bottom);
+    bdrv_unfreeze_chain(bs, bottom);
 }
Signature · 1 line
2.21.0
Reviewed-by on v6 9/42
#24 lore raw

Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain

Quoted 70 lines · Max Reitz

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.

In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

---
  include/block/block.h | 10 +++---
  block.c               | 81 +++++++++++++++++++++++++------------------
  block/commit.c        |  8 ++---
  block/mirror.c        |  4 +--
  block/stream.c        |  8 ++---
  5 files changed, 62 insertions(+), 49 deletions(-)
include/block/block.h +5 -5
index 50a07c1c33..f6f09b95cd 100644
@@ -364,11 +364,11 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
  BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                      BlockDriverState *bs);
  BlockDriverState *bdrv_find_base(BlockDriverState *bs);
-bool bdrv_is_backing_chain_frozen(BlockDriverState *bs, BlockDriverState *base,
-                                  Error **errp);
-int bdrv_freeze_backing_chain(BlockDriverState *bs, BlockDriverState *base,
-                              Error **errp);
-void bdrv_unfreeze_backing_chain(BlockDriverState *bs, BlockDriverState *base);
+bool bdrv_is_chain_frozen(BlockDriverState *bs, BlockDriverState *base,
+                          Error **errp);
+int bdrv_freeze_chain(BlockDriverState *bs, BlockDriverState *base,
+                      Error **errp);
+void bdrv_unfreeze_chain(BlockDriverState *bs, BlockDriverState *base);
 
 
  typedef struct BdrvCheckResult {
block.c +4 -0
index adf82efb0e..650c00d182 100644
@@ -2303,12 +2303,15 @@ static void bdrv_replace_child_noperm(BdrvChild *child,
   * If @new_bs is not NULL, bdrv_check_perm() must be called beforehand, as this
   * function uses bdrv_set_perm() to update the permissions according to the new
   * reference that @new_bs gets.
+ *
+ * Callers must ensure that child->frozen is false.
   */
  static void bdrv_replace_child(BdrvChild *child, BlockDriverState *new_bs)
  {
      BlockDriverState *old_bs = child->bs;
      uint64_t perm, shared_perm;
 
+    /* Asserts that child->frozen == false */
      bdrv_replace_child_noperm(child, new_bs);
 
      /*
@@ -2468,6 +2471,7 @@ static void bdrv_detach_child(BdrvChild *child)
      g_free(child);
  }
 
+/* Callers must ensure that child->frozen is false. */
  void bdrv_root_unref_child(BdrvChild *child)
  {
      BlockDriverState *child_bs;

block.c void bdrv_root_unref_child(BdrvChild *child) · line 2483

Quoted 7 lines
block.c +0 -4
@@ -2477,10 +2481,6 @@ void bdrv_root_unref_child(BdrvChild *child)
      bdrv_unref(child_bs);
  }
 
-/**
- * Clear all inherits_from pointers from children and grandchildren of
- * @root that point to @root, where necessary.
- */

Hmm, unrelated chunk? Without it:

Quoted 4 lines
block.c +0 -0
  static void bdrv_unset_inherits_from(BlockDriverState *root, BdrvChild *child)
  {
      BdrvChild *c;
@@ -2505,6 +2505,7 @@ static void bdrv_unset_inherits_from(BlockDriverState *root, BdrvChild *child)

[..]

Signature · 2 lines
Best regards,
Vladimir
author
#25 lore raw

Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain

Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines · Max Reitz

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.

In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

---
  include/block/block.h | 10 +++---
  block.c               | 81 +++++++++++++++++++++++++------------------
  block/commit.c        |  8 ++---
  block/mirror.c        |  4 +--
  block/stream.c        |  8 ++---
  5 files changed, 62 insertions(+), 49 deletions(-)

[...]

Quoted 11 lines

block.c void bdrv_root_unref_child(BdrvChild *child) · line 2483

Quoted 8 lines
block.c +0 -4
@@ -2477,10 +2481,6 @@ void bdrv_root_unref_child(BdrvChild *child)
      bdrv_unref(child_bs);
  }
 
-/**
- * Clear all inherits_from pointers from children and grandchildren of
- * @root that point to @root, where necessary.
- */

Hmm, unrelated chunk? Without it:

I don’t know how that slipped in, sorry...

Once again, thanks for reviewing!

Max

#26 lore raw

Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.

In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.

I don't think this is a good idea. The functions are still following the backing chain. A generic "chain" could mean following the bs->file links or any other children, so the new name is confusing because it doesn't really tell you any more what the function does. I'd prefer the name to stay specific.

Quoted 5 lines

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

Kevin

author
#27 lore raw

Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain

Quoted 5 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 4 lines

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.

Quoted 8 lines
Quoted 2 lines

In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.

I don't think this is a good idea. The functions are still following the backing chain. A generic "chain" could mean following the bs->file links or any other children, so the new name is confusing because it doesn't really tell you any more what the function does. I'd prefer the name to stay specific.

They’re following backing chains, among others.

It would make sense to rename s/backing_chain/filter_chain/, that is, in case you don‘t find lumping COW and R/W filters together under “filter” too offensive.

(Naming things is hard. I’m open for suggestions, but I found the “filter” concept succinct, even if it does not fully align with our existing parlance.)

Max

Quoted 8 lines
Quoted 5 lines

While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.

Kevin

#28 lore raw

Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain

Am 09.09.2019 um 10:02 hat Max Reitz geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 5 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 4 lines

In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.

Quoted 8 lines
Quoted 2 lines

In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.

I don't think this is a good idea. The functions are still following the backing chain. A generic "chain" could mean following the bs->file links or any other children, so the new name is confusing because it doesn't really tell you any more what the function does. I'd prefer the name to stay specific.

They’re following backing chains, among others.

Quoted 7 lines

It would make sense to rename s/backing_chain/filter_chain/, that is, in case you don‘t find lumping COW and R/W filters together under “filter” too offensive.

(Naming things is hard. I’m open for suggestions, but I found the “filter” concept succinct, even if it does not fully align with our existing parlance.)

As you could see in my reply to patch 4, I didn't. :-)

I think it makes a lot more sense to just broaden the meaning of "backing chain" to be what you call a "filter chain" (following the backing file links, but accept filter nodes in between), because of how unspecific "filter chain" is. The primary thing we're interested in is still the backing files.

Kevin

author
#29 lore raw

PATCH v6 10/42 block: Drop bdrv_is_encrypted()

In reply to Max Reitz

The original purpose of bdrv_is_encrypted() was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.

Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.

So now we have to decide in which direction we want to fix bdrv_is_encrypted(): Recursively query all children, or drop it and just use bs->encrypted to get the current node's status?

Nowadays, its only purpose is to report through bdrv_query_image_info() whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).

---
 include/block/block.h | 1 -
 block.c               | 8 --------
 block/qapi.c          | 2 +-
 3 files changed, 1 insertion(+), 10 deletions(-)
include/block/block.h +0 -1
index f6f09b95cd..9cfe77abaf 100644
@@ -487,7 +487,6 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);
 void bdrv_next_cleanup(BdrvNextIterator *it);
 
 BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
-bool bdrv_is_encrypted(BlockDriverState *bs);
 void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
                          void *opaque, bool read_only);
 const char *bdrv_get_node_name(const BlockDriverState *bs);
block.c +0 -8
index 650c00d182..415c555bf5 100644
@@ -4696,14 +4696,6 @@ bool bdrv_is_sg(BlockDriverState *bs)
     return bs->sg;
 }
 
-bool bdrv_is_encrypted(BlockDriverState *bs)
-{
-    if (bs->backing && bs->backing->bs->encrypted) {
-        return true;
-    }
-    return bs->encrypted;
-}
-
 const char *bdrv_get_format_name(BlockDriverState *bs)
 {
     return bs->drv ? bs->drv->format_name : NULL;
block/qapi.c +1 -1
index 15f1030264..9a185cba48 100644
@@ -281,7 +281,7 @@ void bdrv_query_image_info(BlockDriverState *bs,
     info->virtual_size    = size;
     info->actual_size     = bdrv_get_allocated_file_size(bs);
     info->has_actual_size = info->actual_size >= 0;
-    if (bdrv_is_encrypted(bs)) {
+    if (bs->encrypted) {
         info->encrypted = true;
         info->has_encrypted = true;
     }
Signature · 1 line
2.21.0
Reviewed-by on v6 10/42
#30 lore raw

Re: [Qemu-devel] [PATCH v6 10/42] block: Drop bdrv_is_encrypted()

Quoted 19 lines · Max Reitz

The original purpose of bdrv_is_encrypted() was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.

Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.

So now we have to decide in which direction we want to fix bdrv_is_encrypted(): Recursively query all children, or drop it and just use bs->encrypted to get the current node's status?

Nowadays, its only purpose is to report through bdrv_query_image_info() whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).

Quoted 2 lines
Signature · 2 lines
Best regards,
Vladimir
author
#31 lore raw

PATCH v6 11/42 block: Add bdrv_supports_compressed_writes()

In reply to Max Reitz

Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).

---
 include/block/block.h |  1 +
 block.c               | 22 ++++++++++++++++++++++
 2 files changed, 23 insertions(+)
include/block/block.h +1 -0
index 9cfe77abaf..6ba853fb90 100644
@@ -487,6 +487,7 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);
 void bdrv_next_cleanup(BdrvNextIterator *it);
 
 BlockDriverState *bdrv_next_monitor_owned(BlockDriverState *bs);
+bool bdrv_supports_compressed_writes(BlockDriverState *bs);
 void bdrv_iterate_format(void (*it)(void *opaque, const char *name),
                          void *opaque, bool read_only);
 const char *bdrv_get_node_name(const BlockDriverState *bs);
block.c +22 -0
index 415c555bf5..029c809a8e 100644
@@ -4696,6 +4696,28 @@ bool bdrv_is_sg(BlockDriverState *bs)
     return bs->sg;
 }
 
+/**
+ * Return whether the given node supports compressed writes.
+ */
+bool bdrv_supports_compressed_writes(BlockDriverState *bs)
+{
+    BlockDriverState *filtered = bdrv_filtered_rw_bs(bs);
+
+    if (!bs->drv || !bs->drv->bdrv_co_pwritev_compressed) {
+        return false;
+    }
+
+    if (filtered) {
+        /*
+         * Filters can only forward compressed writes, so we have to
+         * check the child.
+         */
+        return bdrv_supports_compressed_writes(filtered);
+    }
+
+    return true;
+}
+
 const char *bdrv_get_format_name(BlockDriverState *bs)
 {
     return bs->drv ? bs->drv->format_name : NULL;
Signature · 1 line
2.21.0
#32 lore raw

Re: [Qemu-devel] [PATCH v6 11/42] block: Add bdrv_supports_compressed_writes()

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 11 lines

Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).

Should patches 2 and 3 that add the .bdrv_co_pwritev_compressed() callback to filter drivers come only after this one? And should we also support it in the mirror filter?

Kevin

author
#33 lore raw

Re: [Qemu-devel] [PATCH v6 11/42] block: Add bdrv_supports_compressed_writes()

Quoted 10 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 9 lines

Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).

Quoted 5 lines
Quoted 2 lines

Should patches 2 and 3 that add the .bdrv_co_pwritev_compressed() callback to filter drivers come only after this one?

Why not.

Quoted 2 lines

And should we also
support it in the mirror filter?

Hm. AFAIU, compressed writes have very limited use. You can basically only use them when writing to a new image (where you’d never write anywhere you’ve already written something to), i.e. for qemu-img convert or the backup target. It makes sense to blockdev-backup to throttle, so that’s why it should be implemented there. I don’t really see how it would make sense for mirror.

OTOH, it doesn’t make sense for COR either. And it isn’t that hard. Now I don’t have a strong preference for either dropping the COR patch or adding it to mirror as well...

Max

author
#34 lore raw

PATCH v6 12/42 block: Use bdrv_filtered_rw* where obvious

In reply to Max Reitz

Places that use patterns like

    if (bs->drv->is_filter && bs->file) {
        ... something about bs->file->bs ...
    }

should be

    BlockDriverState *filtered = bdrv_filtered_rw_bs(bs);
    if (filtered) {
        ... something about @filtered ...
    }

instead.

---
 block.c    | 23 +++++++++++++++--------
 block/io.c |  5 +++--
 2 files changed, 18 insertions(+), 10 deletions(-)
block.c +15 -8
index 029c809a8e..86b84bea21 100644
@@ -556,11 +556,12 @@ int bdrv_create_file(const char *filename, QemuOpts *opts, Error **errp)
 int bdrv_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *filtered = bdrv_filtered_rw_bs(bs);
 
     if (drv && drv->bdrv_probe_blocksizes) {
         return drv->bdrv_probe_blocksizes(bs, bsz);
-    } else if (drv && drv->is_filter && bs->file) {
-        return bdrv_probe_blocksizes(bs->file->bs, bsz);
+    } else if (filtered) {
+        return bdrv_probe_blocksizes(filtered, bsz);
     }
 
     return -ENOTSUP;
@@ -575,11 +576,12 @@ int bdrv_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)
 int bdrv_probe_geometry(BlockDriverState *bs, HDGeometry *geo)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *filtered = bdrv_filtered_rw_bs(bs);
 
     if (drv && drv->bdrv_probe_geometry) {
         return drv->bdrv_probe_geometry(bs, geo);
-    } else if (drv && drv->is_filter && bs->file) {
-        return bdrv_probe_geometry(bs->file->bs, geo);
+    } else if (filtered) {
+        return bdrv_probe_geometry(filtered, geo);
     }
 
     return -ENOTSUP;
@@ -5084,6 +5086,8 @@ int bdrv_has_zero_init_1(BlockDriverState *bs)
 
 int bdrv_has_zero_init(BlockDriverState *bs)
 {
+    BlockDriverState *filtered;
+
     if (!bs->drv) {
         return 0;
     }
@@ -5096,8 +5100,10 @@ int bdrv_has_zero_init(BlockDriverState *bs)
     if (bs->drv->bdrv_has_zero_init) {
         return bs->drv->bdrv_has_zero_init(bs);
     }
-    if (bs->file && bs->drv->is_filter) {
-        return bdrv_has_zero_init(bs->file->bs);
+
+    filtered = bdrv_filtered_rw_bs(bs);
+    if (filtered) {
+        return bdrv_has_zero_init(filtered);
     }
 
     /* safe default */
@@ -5142,8 +5148,9 @@ int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
         return -ENOMEDIUM;
     }
     if (!drv->bdrv_get_info) {
-        if (bs->file && drv->is_filter) {
-            return bdrv_get_info(bs->file->bs, bdi);
+        BlockDriverState *filtered = bdrv_filtered_rw_bs(bs);
+        if (filtered) {
+            return bdrv_get_info(filtered, bdi);
         }
         return -ENOTSUP;
     }
block/io.c +3 -2
index 06305c6ea6..4d6cf4b3c2 100644
@@ -3186,8 +3186,9 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset,
     }
 
     if (!drv->bdrv_co_truncate) {
-        if (bs->file && drv->is_filter) {
-            ret = bdrv_co_truncate(bs->file, offset, prealloc, errp);
+        BdrvChild *filtered = bdrv_filtered_rw_child(bs);
+        if (filtered) {
+            ret = bdrv_co_truncate(filtered, offset, prealloc, errp);
             goto out;
         }
         error_setg(errp, "Image format driver does not support resize");
Signature · 1 line
2.21.0
author
#35 lore raw

PATCH v6 13/42 block: Use CAFs in block status functions

In reply to Max Reitz

Use the child access functions in the block status inquiry functions as appropriate.

---
 block/io.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)
block/io.c +10 -9
index 4d6cf4b3c2..c5a8e3e6a3 100644
@@ -2180,11 +2180,12 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,
     if (ret & (BDRV_BLOCK_DATA | BDRV_BLOCK_ZERO)) {
         ret |= BDRV_BLOCK_ALLOCATED;
     } else if (want_zero) {
+        BlockDriverState *cow_bs = bdrv_filtered_cow_bs(bs);
+
         if (bdrv_unallocated_blocks_are_zero(bs)) {
             ret |= BDRV_BLOCK_ZERO;
-        } else if (bs->backing) {
-            BlockDriverState *bs2 = bs->backing->bs;
-            int64_t size2 = bdrv_getlength(bs2);
+        } else if (cow_bs) {
+            int64_t size2 = bdrv_getlength(cow_bs);
 
             if (size2 >= 0 && offset >= size2) {
                 ret |= BDRV_BLOCK_ZERO;
@@ -2250,7 +2251,7 @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,
     bool first = true;
 
     assert(bs != base);
-    for (p = bs; p != base; p = backing_bs(p)) {
+    for (p = bs; p != base; p = bdrv_filtered_bs(p)) {
         ret = bdrv_co_block_status(p, want_zero, offset, bytes, pnum, map,
                                    file);
         if (ret < 0) {
@@ -2336,7 +2337,7 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,
 int bdrv_block_status(BlockDriverState *bs, int64_t offset, int64_t bytes,
                       int64_t *pnum, int64_t *map, BlockDriverState **file)
 {
-    return bdrv_block_status_above(bs, backing_bs(bs),
+    return bdrv_block_status_above(bs, bdrv_filtered_bs(bs),
                                    offset, bytes, pnum, map, file);
 }
 
@@ -2346,9 +2347,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,
     int ret;
     int64_t dummy;
 
-    ret = bdrv_common_block_status_above(bs, backing_bs(bs), false, offset,
-                                         bytes, pnum ? pnum : &dummy, NULL,
-                                         NULL);
+    ret = bdrv_common_block_status_above(bs, bdrv_filtered_bs(bs), false,
+                                         offset, bytes, pnum ? pnum : &dummy,
+                                         NULL, NULL);
     if (ret < 0) {
         return ret;
     }
@@ -2411,7 +2412,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,
             break;
         }
 
-        intermediate = backing_bs(intermediate);
+        intermediate = bdrv_filtered_bs(intermediate);
     }
 
     *pnum = n;
Signature · 1 line
2.21.0
author
#36 lore raw

PATCH v6 14/42 block: Use CAFs when working with backing chains

In reply to Max Reitz

Use child access functions when iterating through backing chains so filters do not break the chain.

---
 block.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)
block.c +28 -12
index 86b84bea21..42abbaf0ba 100644
@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 }
 
 /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.
  *
  * active is the current topmost image.
  *
@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs)
 {
-    while (active && bs != backing_bs(active)) {
-        active = backing_bs(active);
+    bs = bdrv_skip_rw_filters(bs);
+    active = bdrv_skip_rw_filters(active);
+
+    while (active) {
+        BlockDriverState *next = bdrv_backing_chain_next(active);
+        if (bs == next) {
+            return active;
+        }
+        active = next;
     }
 
-    return active;
+    return NULL;
 }
 
 /* Given a BDS, searches for the base layer. */
@@ -4544,9 +4552,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
      * other intermediate nodes have been dropped.
      * If 'top' is an implicit node (e.g. "commit_top") we should skip
      * it because no one inherits from it. We use explicit_top for that. */
-    while (explicit_top && explicit_top->implicit) {
-        explicit_top = backing_bs(explicit_top);
-    }
+    explicit_top = bdrv_skip_implicit_filters(explicit_top);
     update_inherits_from = bdrv_inherits_from_recursive(base, explicit_top);
 
     /* success - we can delete the intermediate states, and link top->base */
@@ -5014,7 +5020,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,
 bool bdrv_chain_contains(BlockDriverState *top, BlockDriverState *base)
 {
     while (top && top != base) {
-        top = backing_bs(top);
+        top = bdrv_filtered_bs(top);
     }
 
     return top != NULL;
@@ -5253,7 +5259,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     is_protocol = path_has_protocol(backing_file);
 
-    for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) {
+    /*
+     * Being largely a legacy function, skip any filters here
+     * (because filters do not have normal filenames, so they cannot
+     * match anyway; and allowing json:{} filenames is a bit out of
+     * scope).
+     */
+    for (curr_bs = bdrv_skip_rw_filters(bs);
+         bdrv_filtered_cow_child(curr_bs) != NULL;
+         curr_bs = bdrv_backing_chain_next(curr_bs))
+    {
+        BlockDriverState *bs_below = bdrv_backing_chain_next(curr_bs);
 
         /* If either of the filename paths is actually a protocol, then
          * compare unmodified paths; otherwise make paths relative */
@@ -5261,7 +5277,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
             char *backing_file_full_ret;
 
             if (strcmp(backing_file, curr_bs->backing_file) == 0) {
-                retval = curr_bs->backing->bs;
+                retval = bs_below;
                 break;
             }
             /* Also check against the full backing filename for the image */
@@ -5271,7 +5287,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
                 bool equal = strcmp(backing_file, backing_file_full_ret) == 0;
                 g_free(backing_file_full_ret);
                 if (equal) {
-                    retval = curr_bs->backing->bs;
+                    retval = bs_below;
                     break;
                 }
             }
@@ -5297,7 +5313,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
             g_free(filename_tmp);
 
             if (strcmp(backing_file_full, filename_full) == 0) {
-                retval = curr_bs->backing->bs;
+                retval = bs_below;
                 break;
             }
         }
Signature · 1 line
2.21.0
#38 lore raw

Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 23 lines

Use child access functions when iterating through backing chains so filters do not break the chain.

---
 block.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)
block.c +2 -1
index 86b84bea21..42abbaf0ba 100644
@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 }
 
 /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.
  *
  * active is the current topmost image.
  *

block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393

Quoted 7 lines
block.c +2 -2
@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs)
 {
-    while (active && bs != backing_bs(active)) {
-        active = backing_bs(active);
+    bs = bdrv_skip_rw_filters(bs);
+    active = bdrv_skip_rw_filters(active);

This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.

The block jobs seem to use it only for bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same. qmp_block_commit() will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.

All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.

Quoted 37 lines
block.c +9 -5
+    while (active) {
+        BlockDriverState *next = bdrv_backing_chain_next(active);
+        if (bs == next) {
+            return active;
+        }
+        active = next;
     }
 
-    return active;
+    return NULL;
 }
 
 /* Given a BDS, searches for the base layer. */
@@ -4544,9 +4552,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
      * other intermediate nodes have been dropped.
      * If 'top' is an implicit node (e.g. "commit_top") we should skip
      * it because no one inherits from it. We use explicit_top for that. */
-    while (explicit_top && explicit_top->implicit) {
-        explicit_top = backing_bs(explicit_top);
-    }
+    explicit_top = bdrv_skip_implicit_filters(explicit_top);
     update_inherits_from = bdrv_inherits_from_recursive(base, explicit_top);
 
     /* success - we can delete the intermediate states, and link top->base */
@@ -5014,7 +5020,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,
 bool bdrv_chain_contains(BlockDriverState *top, BlockDriverState *base)
 {
     while (top && top != base) {
-        top = backing_bs(top);
+        top = bdrv_filtered_bs(top);
     }
 
     return top != NULL;
@@ -5253,7 +5259,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     is_protocol = path_has_protocol(backing_file);
 

block.c BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, · line 5270

Quoted 10 lines
block.c +9 -1
-    for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) {
+    /*
+     * Being largely a legacy function, skip any filters here
+     * (because filters do not have normal filenames, so they cannot
+     * match anyway; and allowing json:{} filenames is a bit out of
+     * scope).
+     */
+    for (curr_bs = bdrv_skip_rw_filters(bs);
+         bdrv_filtered_cow_child(curr_bs) != NULL;
+         curr_bs = bdrv_backing_chain_next(curr_bs))

This could just use bs_below instead of recalculating the node if you moved the declaration of bs_below to the function scope.

Quoted 24 lines
block.c +4 -2
+    {
+        BlockDriverState *bs_below = bdrv_backing_chain_next(curr_bs);
 
         /* If either of the filename paths is actually a protocol, then
          * compare unmodified paths; otherwise make paths relative */
@@ -5261,7 +5277,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
             char *backing_file_full_ret;
 
             if (strcmp(backing_file, curr_bs->backing_file) == 0) {
-                retval = curr_bs->backing->bs;
+                retval = bs_below;
                 break;
             }
             /* Also check against the full backing filename for the image */
@@ -5271,7 +5287,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
                 bool equal = strcmp(backing_file, backing_file_full_ret) == 0;
                 g_free(backing_file_full_ret);
                 if (equal) {
-                    retval = curr_bs->backing->bs;
+                    retval = bs_below;
                     break;
                 }
             }

block.c BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, · line 5319

Quoted 8 lines
block.c +1 -1
@@ -5297,7 +5313,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
             g_free(filename_tmp);
 
             if (strcmp(backing_file_full, filename_full) == 0) {
-                retval = curr_bs->backing->bs;
+                retval = bs_below;
                 break;
             }
         }

Kevin

author
#39 lore raw

Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains

Quoted 28 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 27 lines

Use child access functions when iterating through backing chains so filters do not break the chain.

---
 block.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)
block.c +2 -2
index 86b84bea21..42abbaf0ba 100644
@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 }
 
 /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.
  *
  * active is the current topmost image.
  *
@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs)
 {
-    while (active && bs != backing_bs(active)) {
Quoted 8 lines

block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393

Quoted 3 lines
block.c +2 -1
-        active = backing_bs(active);
+    bs = bdrv_skip_rw_filters(bs);
+    active = bdrv_skip_rw_filters(active);

This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.

Which is to say the overlay.

(I think we only ever use “overlay” in the COW sense.)

Quoted 5 lines

The block jobs seem to use it only for bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same. qmp_block_commit() will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.

You mean that the op blocker could have been on a block job filter node before? I think that‘s pretty much the point of this fix; that that doesn’t make sense. (We didn’t have mirror_top_bs and the like at 058223a6e3b.)

Quoted 2 lines

All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.

I don’t know. The function was written when there were no filters. This change would have been a no-op then. The fact that it isn’t to me just means that introducing filters broke it.

So I don’t know what I would write. Maybe “bdrv_find_overlay() now actually finds the overlay, that is, it will not return a filter node. This is the behavior that all callers expect (because they work on COW backing chains).”

Quoted 37 lines
Quoted 37 lines
block.c +9 -5
+    while (active) {
+        BlockDriverState *next = bdrv_backing_chain_next(active);
+        if (bs == next) {
+            return active;
+        }
+        active = next;
     }
 
-    return active;
+    return NULL;
 }
 
 /* Given a BDS, searches for the base layer. */
@@ -4544,9 +4552,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,
      * other intermediate nodes have been dropped.
      * If 'top' is an implicit node (e.g. "commit_top") we should skip
      * it because no one inherits from it. We use explicit_top for that. */
-    while (explicit_top && explicit_top->implicit) {
-        explicit_top = backing_bs(explicit_top);
-    }
+    explicit_top = bdrv_skip_implicit_filters(explicit_top);
     update_inherits_from = bdrv_inherits_from_recursive(base, explicit_top);
 
     /* success - we can delete the intermediate states, and link top->base */
@@ -5014,7 +5020,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,
 bool bdrv_chain_contains(BlockDriverState *top, BlockDriverState *base)
 {
     while (top && top != base) {
-        top = backing_bs(top);
+        top = bdrv_filtered_bs(top);
     }
 
     return top != NULL;
@@ -5253,7 +5259,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,
 
     is_protocol = path_has_protocol(backing_file);
 
Quoted 13 lines

block.c BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, · line 5270

Quoted 10 lines
block.c +9 -1
-    for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) {
+    /*
+     * Being largely a legacy function, skip any filters here
+     * (because filters do not have normal filenames, so they cannot
+     * match anyway; and allowing json:{} filenames is a bit out of
+     * scope).
+     */
+    for (curr_bs = bdrv_skip_rw_filters(bs);
+         bdrv_filtered_cow_child(curr_bs) != NULL;
+         curr_bs = bdrv_backing_chain_next(curr_bs))

This could just use bs_below instead of recalculating the node if you moved the declaration of bs_below to the function scope.

Indeed, thanks.

Max

Quoted 5 lines
Quoted 5 lines
block.c +2 -0
+    {
+        BlockDriverState *bs_below = bdrv_backing_chain_next(curr_bs);
 
         /* If either of the filename paths is actually a protocol, then
          * compare unmodified paths; otherwise make paths relative */
#40 lore raw

Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains

Am 09.09.2019 um 10:25 hat Max Reitz geschrieben:

Quoted 38 lines · Kevin Wolf
Quoted 28 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 27 lines

Use child access functions when iterating through backing chains so filters do not break the chain.

---
 block.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)
block.c +2 -2
index 86b84bea21..42abbaf0ba 100644
@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 }
 
 /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.
  *
  * active is the current topmost image.
  *
@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs)
 {
-    while (active && bs != backing_bs(active)) {
Quoted 8 lines

block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393

Quoted 3 lines
block.c +2 -1
-        active = backing_bs(active);
+    bs = bdrv_skip_rw_filters(bs);
+    active = bdrv_skip_rw_filters(active);

This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.

Quoted 3 lines

Which is to say the overlay.

(I think we only ever use “overlay” in the COW sense.)

I think we do, but so far also only ever for immediate COW childs, not for skipping through intermediate node.

Quoted 10 lines
Quoted 5 lines

The block jobs seem to use it only for bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same. qmp_block_commit() will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.

You mean that the op blocker could have been on a block job filter node before? I think that‘s pretty much the point of this fix; that that doesn’t make sense. (We didn’t have mirror_top_bs and the like at 058223a6e3b.)

On the off chance that the op blocker actually works, it can't be a job filter. I was thinking more of throttling, blkdebug etc.

Quoted 4 lines
Quoted 2 lines

All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.

I don’t know. The function was written when there were no filters.

I doubt it. blkdebug is a really old filter.

Quoted 7 lines

This change would have been a no-op then. The fact that it isn’t to me just means that introducing filters broke it.

So I don’t know what I would write. Maybe “bdrv_find_overlay() now actually finds the overlay, that is, it will not return a filter node. This is the behavior that all callers expect (because they work on COW backing chains).”

Maybe just something like "In addition, filter nodes are not returned as overlays any more. Instead, the first non-filter node on top of bs is considered the overlay now."?

Kevin

author
#41 lore raw

Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains

Quoted 39 lines · Kevin Wolf

Am 09.09.2019 um 10:25 hat Max Reitz geschrieben:

Quoted 38 lines · Kevin Wolf
Quoted 36 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 23 lines

Use child access functions when iterating through backing chains so filters do not break the chain.

---
 block.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)
block.c +2 -1
index 86b84bea21..42abbaf0ba 100644
@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 }
 
 /*
- * Finds the image layer in the chain that has 'bs' as its backing file.
+ * Finds the image layer in the chain that has 'bs' (or a filter on
+ * top of it) as its backing file.
  *
  * active is the current topmost image.
  *

block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393

Quoted 7 lines
block.c +2 -2
@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,
 BlockDriverState *bdrv_find_overlay(BlockDriverState *active,
                                     BlockDriverState *bs)
 {
-    while (active && bs != backing_bs(active)) {
-        active = backing_bs(active);
+    bs = bdrv_skip_rw_filters(bs);
+    active = bdrv_skip_rw_filters(active);

This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.

Quoted 6 lines
Quoted 3 lines

Which is to say the overlay.

(I think we only ever use “overlay” in the COW sense.)

I think we do, but so far also only ever for immediate COW childs, not for skipping through intermediate node.

Yes, because it’s broken so far.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

The block jobs seem to use it only for bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same. qmp_block_commit() will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.

Quoted 7 lines
Quoted 4 lines

You mean that the op blocker could have been on a block job filter node before? I think that‘s pretty much the point of this fix; that that doesn’t make sense. (We didn’t have mirror_top_bs and the like at 058223a6e3b.)

On the off chance that the op blocker actually works, it can't be a job filter. I was thinking more of throttling, blkdebug etc.

Well, that’s just broken altogether right now.

Quoted 10 lines
Quoted 4 lines
Quoted 2 lines

All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.

I don’t know. The function was written when there were no filters.

I doubt it. blkdebug is a really old filter.

Quoted 3 lines

This change would have been a no-op then. The fact that it isn’t to me just means that introducing filters broke it.

Quoted 8 lines
Quoted 4 lines

So I don’t know what I would write. Maybe “bdrv_find_overlay() now actually finds the overlay, that is, it will not return a filter node. This is the behavior that all callers expect (because they work on COW backing chains).”

Maybe just something like "In addition, filter nodes are not returned as overlays any more. Instead, the first non-filter node on top of bs is considered the overlay now."?

Isn’t that basically the same? :-)

Max

author
#42 lore raw

PATCH v6 15/42 block: Re-evaluate backing file handling in reopen

In reply to Max Reitz

Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).

---
 block.c | 45 ++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 38 insertions(+), 7 deletions(-)
block.c +38 -7
index 42abbaf0ba..064cc99857 100644
@@ -3660,25 +3660,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state,
         }
     }
 
+    /*
+     * Ensure that @bs can really handle backing files, because we are
+     * about to give it one (or swap the existing one)
+     */
+    if (bs->drv->is_filter) {
+        /* Filters always have a file or a backing child */
+        if (!bs->backing) {
+            error_setg(errp, "'%s' is a %s filter node that does not support a "
+                       "backing child", bs->node_name, bs->drv->format_name);
+            return -EINVAL;
+        }
+    } else if (!bs->drv->supports_backing) {
+        error_setg(errp, "Driver '%s' of node '%s' does not support backing "
+                   "files", bs->drv->format_name, bs->node_name);
+        return -EINVAL;
+    }
+
     /*
      * Find the "actual" backing file by skipping all links that point
      * to an implicit node, if any (e.g. a commit filter node).
+     * We cannot use any of the bdrv_skip_*() functions here because
+     * those return the first explicit node, while we are looking for
+     * its overlay here.
      */
     overlay_bs = bs;
-    while (backing_bs(overlay_bs) && backing_bs(overlay_bs)->implicit) {
-        overlay_bs = backing_bs(overlay_bs);
+    while (bdrv_filtered_bs(overlay_bs) &&
+           bdrv_filtered_bs(overlay_bs)->implicit)
+    {
+        overlay_bs = bdrv_filtered_bs(overlay_bs);
     }
 
     /* If we want to replace the backing file we need some extra checks */
-    if (new_backing_bs != backing_bs(overlay_bs)) {
+    if (new_backing_bs != bdrv_filtered_bs(overlay_bs)) {
         /* Check for implicit nodes between bs and its backing file */
         if (bs != overlay_bs) {
             error_setg(errp, "Cannot change backing link if '%s' has "
                        "an implicit backing file", bs->node_name);
             return -EPERM;
         }
-        /* Check if the backing link that we want to replace is frozen */
-        if (bdrv_is_chain_frozen(overlay_bs, backing_bs(overlay_bs), errp)) {
+        /*
+         * Check if the backing link that we want to replace is frozen.
+         * Note that
+         * bdrv_filtered_child(overlay_bs) == overlay_bs->backing,
+         * because we know that overlay_bs == bs, and that @bs
+         * either is an R/W filter that uses ->backing or a COW format
+         * with bs->drv->supports_backing == true.
+         */
+        if (bdrv_is_chain_frozen(overlay_bs, child_bs(overlay_bs->backing),
+                                 errp))
+        {
             return -EPERM;
         }
         reopen_state->replace_backing_bs = true;
@@ -3829,7 +3860,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,
      * its metadata. Otherwise the 'backing' option can be omitted.
      */
     if (drv->supports_backing && reopen_state->backing_missing &&
-        (backing_bs(reopen_state->bs) || reopen_state->bs->backing_file[0])) {
+        (reopen_state->bs->backing || reopen_state->bs->backing_file[0])) {
         error_setg(errp, "backing is missing for '%s'",
                    reopen_state->bs->node_name);
         ret = -EINVAL;
@@ -3974,7 +4005,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)
      * from bdrv_set_backing_hd()) has the new values.
      */
     if (reopen_state->replace_backing_bs) {
-        BlockDriverState *old_backing_bs = backing_bs(bs);
+        BlockDriverState *old_backing_bs = child_bs(bs->backing);
         assert(!old_backing_bs || !old_backing_bs->implicit);
         /* Abort the permission update on the backing bs we're detaching */
         if (old_backing_bs) {
Signature · 1 line
2.21.0
Reviewed-by on v6 15/42
#43 lore raw

Re: [Qemu-devel] [PATCH v6 15/42] block: Re-evaluate backing file handling in reopen

Quoted 8 lines · Max Reitz

Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).

Signature · 2 lines
Best regards,
Vladimir
author
#44 lore raw

PATCH v6 16/42 block: Flush all children in generic code

In reply to Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
 block/io.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +17 -6
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
 int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
     int current_gen;
     int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     }
 
     /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
     if (bs->drv->bdrv_co_flush_to_os) {
         ret = bs->drv->bdrv_co_flush_to_os(bs);
         if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
     /* But don't actually force it to the disk with cache=unsafe */
     if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
     }
 
     /* Check if we really need to flush anything */
     if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
     }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
     if (!bs->drv) {
         /* bs->drv->bdrv_co_flush() might have ejected the BDS
          * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
     /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
      * in the case of cache=unsafe, so there are no useless flushes.
      */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0;
+    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }
+
 out:
     /* Notify any pending flushes that we have completed */
     if (ret == 0) {
Signature · 1 line
2.21.0
#45 lore raw

Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code

Quoted 59 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +6 -5
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
  int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
  {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
      int current_gen;
      int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      }
 
      /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
      if (bs->drv->bdrv_co_flush_to_os) {
          ret = bs->drv->bdrv_co_flush_to_os(bs);
          if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
      /* But don't actually force it to the disk with cache=unsafe */
      if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
      }
 
      /* Check if we really need to flush anything */
      if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
      }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
      if (!bs->drv) {
          /* bs->drv->bdrv_co_flush() might have ejected the BDS
           * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
-flush_parent:

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2676

Quoted 10 lines
block/io.c +9 -1
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0; > +    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }

Hmm, you said that we want to flush only children with write-access from parent.. Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?

Quoted 5 lines
block/io.c +1 -0
+
  out:
      /* Notify any pending flushes that we have completed */
      if (ret == 0) {
Signature · 2 lines
Best regards,
Vladimir
author
#46 lore raw

Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code

Quoted 64 lines · Vladimir Sementsov-Ogievskiy
Quoted 63 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +9 -6
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
  int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
  {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
      int current_gen;
      int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      }
 
      /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
      if (bs->drv->bdrv_co_flush_to_os) {
          ret = bs->drv->bdrv_co_flush_to_os(bs);
          if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
      /* But don't actually force it to the disk with cache=unsafe */
      if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
      }
 
      /* Check if we really need to flush anything */
      if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
      }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
      if (!bs->drv) {
          /* bs->drv->bdrv_co_flush() might have ejected the BDS
           * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0; > +    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677

Quoted 6 lines
block/io.c +6 -0
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }

Hmm, you said that we want to flush only children with write-access from parent..

Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)

Quoted 2 lines

Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?

I think it’s always safe. But checking it seems like a nice touch, yes.

Max

Quoted 7 lines
Quoted 5 lines
block/io.c +1 -0
+
  out:
      /* Notify any pending flushes that we have completed */
      if (ret == 0) {
#47 lore raw

Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code

Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:

Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +9 -6
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
  int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
  {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
      int current_gen;
      int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      }
 
      /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
      if (bs->drv->bdrv_co_flush_to_os) {
          ret = bs->drv->bdrv_co_flush_to_os(bs);
          if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
      /* But don't actually force it to the disk with cache=unsafe */
      if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
      }
 
      /* Check if we really need to flush anything */
      if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
      }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
      if (!bs->drv) {
          /* bs->drv->bdrv_co_flush() might have ejected the BDS
           * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0; > +    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677

Quoted 6 lines
block/io.c +6 -0
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }

Hmm, you said that we want to flush only children with write-access from parent..

Quoted 7 lines

Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)

Quoted 2 lines

Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?

I think it’s always safe. But checking it seems like a nice touch, yes.

I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.

But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe bs->write_gen saves us so that at least this doesn't result in an fdatasync() call for each, but still... Without a use case, I'd rather not do this.

Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?

Kevin

author
#48 lore raw

Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code

Quoted 95 lines · Kevin Wolf

Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:

Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +9 -6
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
  int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
  {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
      int current_gen;
      int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      }
 
      /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
      if (bs->drv->bdrv_co_flush_to_os) {
          ret = bs->drv->bdrv_co_flush_to_os(bs);
          if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
      /* But don't actually force it to the disk with cache=unsafe */
      if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
      }
 
      /* Check if we really need to flush anything */
      if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
      }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
      if (!bs->drv) {
          /* bs->drv->bdrv_co_flush() might have ejected the BDS
           * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0; > +    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677

Quoted 6 lines
block/io.c +6 -0
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }

Hmm, you said that we want to flush only children with write-access from parent..

Quoted 7 lines

Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)

Quoted 2 lines

Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?

I think it’s always safe. But checking it seems like a nice touch, yes.

I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.

But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe bs->write_gen saves us so that at least this doesn't result in an fdatasync() call for each, but still... Without a use case, I'd rather not do this.

Quoted 5 lines

Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?

And I should thus probably drop vmdk’s .bdrv_co_flush_to_disk() implementation.

I will indeed try to write a test, but to be completely honest, I feel like this series is long enough.

Max

#49 lore raw

Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code

Am 09.09.2019 um 10:31 hat Max Reitz geschrieben:

Quoted 102 lines · Kevin Wolf
Quoted 95 lines

Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:

Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +9 -6
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
  int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
  {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
      int current_gen;
      int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      }
 
      /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
      if (bs->drv->bdrv_co_flush_to_os) {
          ret = bs->drv->bdrv_co_flush_to_os(bs);
          if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
      /* But don't actually force it to the disk with cache=unsafe */
      if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
      }
 
      /* Check if we really need to flush anything */
      if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
      }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
      if (!bs->drv) {
          /* bs->drv->bdrv_co_flush() might have ejected the BDS
           * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0; > +    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677

Quoted 6 lines
block/io.c +6 -0
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }

Hmm, you said that we want to flush only children with write-access from parent..

Quoted 7 lines

Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)

Quoted 2 lines

Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?

I think it’s always safe. But checking it seems like a nice touch, yes.

I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.

But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe bs->write_gen saves us so that at least this doesn't result in an fdatasync() call for each, but still... Without a use case, I'd rather not do this.

Quoted 5 lines

Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?

Quoted 5 lines

And I should thus probably drop vmdk’s .bdrv_co_flush_to_disk() implementation.

I will indeed try to write a test, but to be completely honest, I feel like this series is long enough.

I guess I could already merge patch 1 to give you space for another test patch. ;-)

Kevin

author
#50 lore raw

Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code

Quoted 103 lines · Kevin Wolf

Am 09.09.2019 um 10:31 hat Max Reitz geschrieben:

Quoted 102 lines · Kevin Wolf
Quoted 100 lines

Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:

Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz

If the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.

In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.

---
  block/io.c | 23 +++++++++++++++++------
  1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +9 -6
index c5a8e3e6a3..bcc770d336 100644
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque)
 
  int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
  {
+    BdrvChild *primary_child = bdrv_primary_child(bs);
+    BdrvChild *child;
      int current_gen;
      int ret = 0;
 
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      }
 
      /* Write back cached data to the OS even with cache=unsafe */
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_OS);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_OS);
      if (bs->drv->bdrv_co_flush_to_os) {
          ret = bs->drv->bdrv_co_flush_to_os(bs);
          if (ret < 0) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
 
      /* But don't actually force it to the disk with cache=unsafe */
      if (bs->open_flags & BDRV_O_NO_FLUSH) {
-        goto flush_parent;
+        goto flush_children;
      }
 
      /* Check if we really need to flush anything */
      if (bs->flushed_gen == current_gen) {
-        goto flush_parent;
+        goto flush_children;
      }
 
-    BLKDBG_EVENT(bs->file, BLKDBG_FLUSH_TO_DISK);
+    BLKDBG_EVENT(primary_child, BLKDBG_FLUSH_TO_DISK);
      if (!bs->drv) {
          /* bs->drv->bdrv_co_flush() might have ejected the BDS
           * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)
      /* Now flush the underlying protocol.  It will also have BDRV_O_NO_FLUSH
       * in the case of cache=unsafe, so there are no useless flushes.
       */
-flush_parent:
-    ret = bs->file ? bdrv_co_flush(bs->file->bs) : 0;
+flush_children:
+    ret = 0; > +    QLIST_FOREACH(child, &bs->children, next) {
+        int this_child_ret;
Quoted 8 lines

block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677

Quoted 6 lines
block/io.c +6 -0
+
+        this_child_ret = bdrv_co_flush(child->bs);
+        if (!ret) {
+            ret = this_child_ret;
+        }
+    }

Hmm, you said that we want to flush only children with write-access from parent..

Quoted 7 lines

Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)

Quoted 2 lines

Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?

I think it’s always safe. But checking it seems like a nice touch, yes.

I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.

But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe bs->write_gen saves us so that at least this doesn't result in an fdatasync() call for each, but still... Without a use case, I'd rather not do this.

Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?

Quoted 8 lines
Quoted 5 lines

And I should thus probably drop vmdk’s .bdrv_co_flush_to_disk() implementation.

I will indeed try to write a test, but to be completely honest, I feel like this series is long enough.

I guess I could already merge patch 1 to give you space for another test patch. ;-)

Don’t forget the mirror-top patch. And AFAIR, there was some comment from Vladimir that also required an additional patch. So it would need to be three!

(Or I just start squashing from the back?)

Max

author
#51 lore raw

PATCH v6 17/42 block: Use CAFs in bdrv_refresh_limits()

In reply to Max Reitz
---
 block/io.c | 14 ++++++++------
 1 file changed, 8 insertions(+), 6 deletions(-)
block/io.c +8 -6
index bcc770d336..dca4689b2f 100644
@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)
 void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *storage_bs = bdrv_storage_bs(bs);
+    BlockDriverState *cow_bs = bdrv_filtered_cow_bs(bs);
     Error *local_err = NULL;
 
     memset(&bs->bl, 0, sizeof(bs->bl));
@@ -148,13 +150,13 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
                                 drv->bdrv_aio_preadv) ? 1 : 512;
 
     /* Take some limits from the children as a default */
-    if (bs->file) {
-        bdrv_refresh_limits(bs->file->bs, &local_err);
+    if (storage_bs) {
+        bdrv_refresh_limits(storage_bs, &local_err);
         if (local_err) {
             error_propagate(errp, local_err);
             return;
         }
-        bdrv_merge_limits(&bs->bl, &bs->file->bs->bl);
+        bdrv_merge_limits(&bs->bl, &storage_bs->bl);
     } else {
         bs->bl.min_mem_alignment = 512;
         bs->bl.opt_mem_alignment = getpagesize();
@@ -163,13 +165,13 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp)
         bs->bl.max_iov = IOV_MAX;
     }
 
-    if (bs->backing) {
-        bdrv_refresh_limits(bs->backing->bs, &local_err);
+    if (cow_bs) {
+        bdrv_refresh_limits(cow_bs, &local_err);
         if (local_err) {
             error_propagate(errp, local_err);
             return;
         }
-        bdrv_merge_limits(&bs->bl, &bs->backing->bs->bl);
+        bdrv_merge_limits(&bs->bl, &cow_bs->bl);
     }
 
     /* Then let the driver override it */
Signature · 1 line
2.21.0
author
#52 lore raw

PATCH v6 18/42 block: Use CAFs in bdrv_refresh_filename()

In reply to Max Reitz

bdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.

---
 block.c | 29 +++++++++++++++++++++--------
 1 file changed, 21 insertions(+), 8 deletions(-)
block.c +21 -8
index 064cc99857..a467b175c6 100644
@@ -6432,6 +6432,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
     BdrvChild *child;
+    BlockDriverState *primary_child_bs;
     QDict *opts;
     bool backing_overridden;
     bool generate_json_filename; /* Whether our default implementation should
@@ -6500,20 +6501,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)
     qobject_unref(bs->full_open_options);
     bs->full_open_options = opts;
 
+    primary_child_bs = bdrv_primary_bs(bs);
+
     if (drv->bdrv_refresh_filename) {
         /* Obsolete information is of no use here, so drop the old file name
          * information before refreshing it */
         bs->exact_filename[0] = '\0';
 
         drv->bdrv_refresh_filename(bs);
-    } else if (bs->file) {
-        /* Try to reconstruct valid information from the underlying file */
+    } else if (primary_child_bs) {
+        /*
+         * Try to reconstruct valid information from the underlying
+         * file -- this only works for format nodes (filter nodes
+         * cannot be probed and as such must be selected by the user
+         * either through an options dict, or through a special
+         * filename which the filter driver must construct in its
+         * .bdrv_refresh_filename() implementation).
+         */
 
         bs->exact_filename[0] = '\0';
 
         /*
          * We can use the underlying file's filename if:
          * - it has a filename,
+         * - the current BDS is not a filter,
          * - the file is a protocol BDS, and
          * - opening that file (as this BDS's format) will automatically create
          *   the BDS tree we have right now, that is:
@@ -6522,11 +6533,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)
          *   - no non-file child of this BDS has been overridden by the user
          *   Both of these conditions are represented by generate_json_filename.
          */
-        if (bs->file->bs->exact_filename[0] &&
-            bs->file->bs->drv->bdrv_file_open &&
-            !generate_json_filename)
+        if (primary_child_bs->exact_filename[0] &&
+            primary_child_bs->drv->bdrv_file_open &&
+            !drv->is_filter && !generate_json_filename)
         {
-            strcpy(bs->exact_filename, bs->file->bs->exact_filename);
+            strcpy(bs->exact_filename, primary_child_bs->exact_filename);
         }
     }
 
@@ -6543,6 +6554,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)
 char *bdrv_dirname(BlockDriverState *bs, Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs;
 
     if (!drv) {
         error_setg(errp, "Node '%s' is ejected", bs->node_name);
@@ -6553,8 +6565,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp)
         return drv->bdrv_dirname(bs, errp);
     }
 
-    if (bs->file) {
-        return bdrv_dirname(bs->file->bs, errp);
+    child_bs = bdrv_primary_bs(bs);
+    if (child_bs) {
+        return bdrv_dirname(child_bs, errp);
     }
 
     bdrv_refresh_filename(bs);
Signature · 1 line
2.21.0
author
#54 lore raw

PATCH v6 19/42 block: Use CAF in bdrv_co_rw_vmstate()

In reply to Max Reitz

If a node whose driver does not provide VM state functions has a metadata child, the VM state should probably go there; if it is a filter, the VM state should probably go there. It follows that we should generally go down to the primary child.

---
 block/io.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
block/io.c +3 -2
index dca4689b2f..e222d91893 100644
@@ -2434,6 +2434,7 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
                    bool is_read)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *child_bs = bdrv_primary_bs(bs);
     int ret = -ENOTSUP;
 
     bdrv_inc_in_flight(bs);
@@ -2446,8 +2447,8 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos,
         } else {
             ret = drv->bdrv_save_vmstate(bs, qiov, pos);
         }
-    } else if (bs->file) {
-        ret = bdrv_co_rw_vmstate(bs->file->bs, qiov, pos, is_read);
+    } else if (child_bs) {
+        ret = bdrv_co_rw_vmstate(child_bs, qiov, pos, is_read);
     }
 
     bdrv_dec_in_flight(bs);
Signature · 1 line
2.21.0
author
#55 lore raw

PATCH v6 20/42 block/snapshot: Fix fallback

In reply to Max Reitz

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
 block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +79 -21
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
     return ret;
 }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
+
+    return child_bs;
+}
+
 int bdrv_can_snapshot(BlockDriverState *bs)
 {
     BlockDriver *drv = bs->drv;
@@ -154,8 +180,9 @@ int bdrv_can_snapshot(BlockDriverState *bs)
     }
 
     if (!drv->bdrv_snapshot_create) {
-        if (bs->file != NULL) {
-            return bdrv_can_snapshot(bs->file->bs);
+        BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
+        if (fallback_bs) {
+            return bdrv_can_snapshot(fallback_bs);
         }
         return 0;
     }
@@ -167,14 +194,15 @@ int bdrv_snapshot_create(BlockDriverState *bs,
                          QEMUSnapshotInfo *sn_info)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
     if (!drv) {
         return -ENOMEDIUM;
     }
     if (drv->bdrv_snapshot_create) {
         return drv->bdrv_snapshot_create(bs, sn_info);
     }
-    if (bs->file) {
-        return bdrv_snapshot_create(bs->file->bs, sn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_create(fallback_bs, sn_info);
     }
     return -ENOTSUP;
 }
@@ -184,6 +212,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
                        Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs;
     int ret, open_ret;
 
     if (!drv) {
@@ -204,39 +233,66 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
         return ret;
     }
 
-    if (bs->file) {
-        BlockDriverState *file;
-        QDict *options = qdict_clone_shallow(bs->options);
+    fallback_bs = bdrv_snapshot_fallback(bs);
+    if (fallback_bs) {
+        QDict *options;
         QDict *file_options;
         Error *local_err = NULL;
+        bool is_backing_child;
+        BdrvChild **child_pointer;
+
+        /*
+         * We need a pointer to the fallback child pointer, so let us
+         * see whether the child is referenced by a field in the BDS
+         * object.
+         */
+        if (fallback_bs == bs->file->bs) {
+            is_backing_child = false;
+            child_pointer = &bs->file;
+        } else if (fallback_bs == bs->backing->bs) {
+            is_backing_child = true;
+            child_pointer = &bs->backing;
+        } else {
+            /*
+             * The fallback child is not referenced by a field in the
+             * BDS object.  We cannot go on then.
+             */
+            error_setg(errp, "Block driver does not support snapshots");
+            return -ENOTSUP;
+        }
+
+        options = qdict_clone_shallow(bs->options);
 
-        file = bs->file->bs;
         /* Prevent it from getting deleted when detached from bs */
-        bdrv_ref(file);
+        bdrv_ref(fallback_bs);
 
-        qdict_extract_subqdict(options, &file_options, "file.");
+        qdict_extract_subqdict(options, &file_options,
+                               is_backing_child ? "backing." : "file.");
         qobject_unref(file_options);
-        qdict_put_str(options, "file", bdrv_get_node_name(file));
+        qdict_put_str(options, is_backing_child ? "backing" : "file",
+                      bdrv_get_node_name(fallback_bs));
 
         if (drv->bdrv_close) {
             drv->bdrv_close(bs);
         }
-        bdrv_unref_child(bs, bs->file);
-        bs->file = NULL;
 
-        ret = bdrv_snapshot_goto(file, snapshot_id, errp);
+        assert(fallback_bs == (*child_pointer)->bs);
+        bdrv_unref_child(bs, *child_pointer);
+        *child_pointer = NULL;
+
+        ret = bdrv_snapshot_goto(fallback_bs, snapshot_id, errp);
         open_ret = drv->bdrv_open(bs, options, bs->open_flags, &local_err);
         qobject_unref(options);
         if (open_ret < 0) {
-            bdrv_unref(file);
+            bdrv_unref(fallback_bs);
             bs->drv = NULL;
             /* A bdrv_snapshot_goto() error takes precedence */
             error_propagate(errp, local_err);
             return ret < 0 ? ret : open_ret;
         }
 
-        assert(bs->file->bs == file);
-        bdrv_unref(file);
+        assert(fallback_bs == (*child_pointer)->bs);
+        bdrv_unref(fallback_bs);
         return ret;
     }
 
@@ -272,6 +328,7 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
                          Error **errp)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
     int ret;
 
     if (!drv) {
@@ -288,8 +345,8 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
 
     if (drv->bdrv_snapshot_delete) {
         ret = drv->bdrv_snapshot_delete(bs, snapshot_id, name, errp);
-    } else if (bs->file) {
-        ret = bdrv_snapshot_delete(bs->file->bs, snapshot_id, name, errp);
+    } else if (fallback_bs) {
+        ret = bdrv_snapshot_delete(fallback_bs, snapshot_id, name, errp);
     } else {
         error_setg(errp, "Block format '%s' used by device '%s' "
                    "does not support internal snapshot deletion",
@@ -305,14 +362,15 @@ int bdrv_snapshot_list(BlockDriverState *bs,
                        QEMUSnapshotInfo **psn_info)
 {
     BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
     if (!drv) {
         return -ENOMEDIUM;
     }
     if (drv->bdrv_snapshot_list) {
         return drv->bdrv_snapshot_list(bs, psn_info);
     }
-    if (bs->file) {
-        return bdrv_snapshot_list(bs->file->bs, psn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_list(fallback_bs, psn_info);
     }
     return -ENOTSUP;
 }
Signature · 1 line
2.21.0
#56 lore raw

Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback

Quoted 115 lines · Max Reitz

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
  block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
  1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +49 -7
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
      return ret;
  }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
+
+    return child_bs;
+}
+
  int bdrv_can_snapshot(BlockDriverState *bs)
  {
      BlockDriver *drv = bs->drv;
@@ -154,8 +180,9 @@ int bdrv_can_snapshot(BlockDriverState *bs)
      }
 
      if (!drv->bdrv_snapshot_create) {
-        if (bs->file != NULL) {
-            return bdrv_can_snapshot(bs->file->bs);
+        BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
+        if (fallback_bs) {
+            return bdrv_can_snapshot(fallback_bs);
          }
          return 0;
      }
@@ -167,14 +194,15 @@ int bdrv_snapshot_create(BlockDriverState *bs,
                           QEMUSnapshotInfo *sn_info)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
      if (!drv) {
          return -ENOMEDIUM;
      }
      if (drv->bdrv_snapshot_create) {
          return drv->bdrv_snapshot_create(bs, sn_info);
      }
-    if (bs->file) {
-        return bdrv_snapshot_create(bs->file->bs, sn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_create(fallback_bs, sn_info);
      }
      return -ENOTSUP;
  }
@@ -184,6 +212,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
                         Error **errp)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs;
      int ret, open_ret;
 
      if (!drv) {
@@ -204,39 +233,66 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
          return ret;
      }
 
-    if (bs->file) {
-        BlockDriverState *file;
-        QDict *options = qdict_clone_shallow(bs->options);
+    fallback_bs = bdrv_snapshot_fallback(bs);
+    if (fallback_bs) {
+        QDict *options;
          QDict *file_options;
          Error *local_err = NULL;
+        bool is_backing_child;
+        BdrvChild **child_pointer;
+
+        /*
+         * We need a pointer to the fallback child pointer, so let us
+         * see whether the child is referenced by a field in the BDS
+         * object.
+         */
+        if (fallback_bs == bs->file->bs) {
+            is_backing_child = false;
+            child_pointer = &bs->file;
+        } else if (fallback_bs == bs->backing->bs) {
+            is_backing_child = true;

block/snapshot.c int bdrv_snapshot_goto(BlockDriverState *bs, · line 263

Quoted 10 lines
block/snapshot.c +10 -0
+            child_pointer = &bs->backing;
+        } else {
+            /*
+             * The fallback child is not referenced by a field in the
+             * BDS object.  We cannot go on then.
+             */
+            error_setg(errp, "Block driver does not support snapshots");
+            return -ENOTSUP;
+        }
+

Hmm.. Should not this check be included into bdrv_snapshot_fallback(), to work only with file and backing?

And could we allow fallback only for filters? Is there real usecase except filters? Or may be, drop fallback at all?

Quoted 82 lines
block/snapshot.c +19 -14
-        file = bs->file->bs;
          /* Prevent it from getting deleted when detached from bs */
-        bdrv_ref(file);
+        bdrv_ref(fallback_bs);
 
-        qdict_extract_subqdict(options, &file_options, "file.");
+        qdict_extract_subqdict(options, &file_options,
+                               is_backing_child ? "backing." : "file.");
          qobject_unref(file_options);
-        qdict_put_str(options, "file", bdrv_get_node_name(file));
+        qdict_put_str(options, is_backing_child ? "backing" : "file",
+                      bdrv_get_node_name(fallback_bs));
 
          if (drv->bdrv_close) {
              drv->bdrv_close(bs);
          }
-        bdrv_unref_child(bs, bs->file);
-        bs->file = NULL;
 
-        ret = bdrv_snapshot_goto(file, snapshot_id, errp);
+        assert(fallback_bs == (*child_pointer)->bs);
+        bdrv_unref_child(bs, *child_pointer);
+        *child_pointer = NULL;
+
+        ret = bdrv_snapshot_goto(fallback_bs, snapshot_id, errp);
          open_ret = drv->bdrv_open(bs, options, bs->open_flags, &local_err);
          qobject_unref(options);
          if (open_ret < 0) {
-            bdrv_unref(file);
+            bdrv_unref(fallback_bs);
              bs->drv = NULL;
              /* A bdrv_snapshot_goto() error takes precedence */
              error_propagate(errp, local_err);
              return ret < 0 ? ret : open_ret;
          }
 
-        assert(bs->file->bs == file);
-        bdrv_unref(file);
+        assert(fallback_bs == (*child_pointer)->bs);
+        bdrv_unref(fallback_bs);
          return ret;
      }
 
@@ -272,6 +328,7 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
                           Error **errp)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
      int ret;
 
      if (!drv) {
@@ -288,8 +345,8 @@ int bdrv_snapshot_delete(BlockDriverState *bs,
 
      if (drv->bdrv_snapshot_delete) {
          ret = drv->bdrv_snapshot_delete(bs, snapshot_id, name, errp);
-    } else if (bs->file) {
-        ret = bdrv_snapshot_delete(bs->file->bs, snapshot_id, name, errp);
+    } else if (fallback_bs) {
+        ret = bdrv_snapshot_delete(fallback_bs, snapshot_id, name, errp);
      } else {
          error_setg(errp, "Block format '%s' used by device '%s' "
                     "does not support internal snapshot deletion",
@@ -305,14 +362,15 @@ int bdrv_snapshot_list(BlockDriverState *bs,
                         QEMUSnapshotInfo **psn_info)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
      if (!drv) {
          return -ENOMEDIUM;
      }
      if (drv->bdrv_snapshot_list) {
          return drv->bdrv_snapshot_list(bs, psn_info);
      }
-    if (bs->file) {
-        return bdrv_snapshot_list(bs->file->bs, psn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_list(fallback_bs, psn_info);
      }
      return -ENOTSUP;
  }
Signature · 2 lines
Best regards,
Vladimir
author
#57 lore raw

Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback

Quoted 121 lines · Vladimir Sementsov-Ogievskiy
Quoted 120 lines · Max Reitz

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
  block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
  1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +54 -7
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
      return ret;
  }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
+
+    return child_bs;
+}
+
  int bdrv_can_snapshot(BlockDriverState *bs)
  {
      BlockDriver *drv = bs->drv;
@@ -154,8 +180,9 @@ int bdrv_can_snapshot(BlockDriverState *bs)
      }
 
      if (!drv->bdrv_snapshot_create) {
-        if (bs->file != NULL) {
-            return bdrv_can_snapshot(bs->file->bs);
+        BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
+        if (fallback_bs) {
+            return bdrv_can_snapshot(fallback_bs);
          }
          return 0;
      }
@@ -167,14 +194,15 @@ int bdrv_snapshot_create(BlockDriverState *bs,
                           QEMUSnapshotInfo *sn_info)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs = bdrv_snapshot_fallback(bs);
      if (!drv) {
          return -ENOMEDIUM;
      }
      if (drv->bdrv_snapshot_create) {
          return drv->bdrv_snapshot_create(bs, sn_info);
      }
-    if (bs->file) {
-        return bdrv_snapshot_create(bs->file->bs, sn_info);
+    if (fallback_bs) {
+        return bdrv_snapshot_create(fallback_bs, sn_info);
      }
      return -ENOTSUP;
  }
@@ -184,6 +212,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
                         Error **errp)
  {
      BlockDriver *drv = bs->drv;
+    BlockDriverState *fallback_bs;
      int ret, open_ret;
 
      if (!drv) {
@@ -204,39 +233,66 @@ int bdrv_snapshot_goto(BlockDriverState *bs,
          return ret;
      }
 
-    if (bs->file) {
-        BlockDriverState *file;
-        QDict *options = qdict_clone_shallow(bs->options);
+    fallback_bs = bdrv_snapshot_fallback(bs);
+    if (fallback_bs) {
+        QDict *options;
          QDict *file_options;
          Error *local_err = NULL;
+        bool is_backing_child;
+        BdrvChild **child_pointer;
+
+        /*
+         * We need a pointer to the fallback child pointer, so let us
+         * see whether the child is referenced by a field in the BDS
+         * object.
+         */
+        if (fallback_bs == bs->file->bs) {
+            is_backing_child = false;
+            child_pointer = &bs->file;
+        } else if (fallback_bs == bs->backing->bs) {
+            is_backing_child = true;
+            child_pointer = &bs->backing;
+        } else {
+            /*
+             * The fallback child is not referenced by a field in the
+             * BDS object.  We cannot go on then.
Quoted 8 lines

block/snapshot.c int bdrv_snapshot_goto(BlockDriverState *bs, · line 263

Quoted 5 lines
block/snapshot.c +5 -0
+             */
+            error_setg(errp, "Block driver does not support snapshots");
+            return -ENOTSUP;
+        }
+

Hmm.. Should not this check be included into bdrv_snapshot_fallback(), to work only with file and backing?

I was under the impression that this was just special code for what turned out to be bdrv_snapshot_load_tmp() now, because it seems so weird. (So I thought just making the restriction here wouldn’t really be a limit.)

I was wrong. This is used when applying snapshots, so it is important.
If we make a restriction here, we should have it in all fallback code, yes.

Quoted 2 lines

And could we allow fallback only for filters? Is there real usecase except filters? Or may be, drop fallback at all?

raw isn’t a filter driver. And rbd as a protocol supports snapshotting.
Hence the fallback code, I presume.

Max

#58 lore raw

Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 38 lines

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
 block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +15 -0
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
     return ret;
 }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173

Quoted 10 lines
block/snapshot.c +10 -0
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
+
+    return child_bs;
+}

Why do we return child->bs here when bdrv_snapshot_goto() then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?

Kevin

author
#59 lore raw

Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback

Quoted 46 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 45 lines

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
 block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +22 -0
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
     return ret;
 }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
Quoted 8 lines

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173

Quoted 3 lines
block/snapshot.c +3 -0
+
+    return child_bs;
+}

Why do we return child->bs here when bdrv_snapshot_goto() then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?

What would you return instead? &child doesn’t work.

We could limit ourselves to bs->file and bs->backing. It just seemed like a bit of an artificial limit to me, because we only really have it for bdrv_snapshot_goto().

Max

#60 lore raw

Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback

Am 10.09.2019 um 14:04 hat Max Reitz geschrieben:

Quoted 56 lines · Kevin Wolf
Quoted 46 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 45 lines

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
 block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +22 -0
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
     return ret;
 }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
Quoted 8 lines

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173

Quoted 3 lines
block/snapshot.c +3 -0
+
+    return child_bs;
+}

Why do we return child->bs here when bdrv_snapshot_goto() then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?

Quoted 1 line

What would you return instead? &child doesn’t work.

Oops, brain fart. :-)

Quoted 3 lines

We could limit ourselves to bs->file and bs->backing. It just seemed like a bit of an artificial limit to me, because we only really have it for bdrv_snapshot_goto().

Hm, but then, what use is supporting other children for creating a snapshot when you can't load it any more afterwards?

Kevin

author
#61 lore raw

Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback

Quoted 59 lines · Kevin Wolf

Am 10.09.2019 um 14:04 hat Max Reitz geschrieben:

Quoted 56 lines · Kevin Wolf
Quoted 46 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 45 lines

If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.

bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).

---
 block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++----------
 1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +22 -0
index f2f48f926a..35403c167f 100644
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,
     return ret;
 }
 
+/**
+ * Return the child BDS to which we can fall back if the given BDS
+ * does not support snapshots.
+ * Return NULL if there is no BDS to (safely) fall back to.
+ */
+static BlockDriverState *bdrv_snapshot_fallback(BlockDriverState *bs)
+{
+    BlockDriverState *child_bs = NULL;
+    BdrvChild *child;
+
+    QLIST_FOREACH(child, &bs->children, next) {
+        if (child == bdrv_filtered_cow_child(bs)) {
+            /* Ignore: COW children need not be included in snapshots */
+            continue;
+        }
+
+        if (child_bs) {
+            /* Cannot fall back to a single child if there are multiple */
+            return NULL;
+        }
+        child_bs = child->bs;
+    }
Quoted 8 lines

block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173

Quoted 3 lines
block/snapshot.c +3 -0
+
+    return child_bs;
+}

Why do we return child->bs here when bdrv_snapshot_goto() then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?

Quoted 1 line

What would you return instead? &child doesn’t work.

Quoted 8 lines

Oops, brain fart. :-)

Quoted 3 lines

We could limit ourselves to bs->file and bs->backing. It just seemed like a bit of an artificial limit to me, because we only really have it for bdrv_snapshot_goto().

Hm, but then, what use is supporting other children for creating a snapshot when you can't load it any more afterwards?

Well, the snapshot is still there, it’s just on a different node. So in theory, you could take a snapshot in a live VM (where the snapshotting node is not at the top), and then later revert to it with qemu-img (by accessing the file with the snapshot directly).

Though in practice this is just a fallback anyway, and I don’t think we currently have anything where it would make sense to fall through some node to any child but .file or .backing. So why not. It would be shorter, too. (Well, plus the short comment why looking at .file and .backing is sufficient.)

Max

author
#62 lore raw

PATCH v6 21/42 block: Use CAFs for debug breakpoints

In reply to Max Reitz

When looking for a blkdebug node (which implements debug breakpoints), use bdrv_primary_bs() to iterate through the graph, because that is where a blkdebug node would be.

---
 block.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block.c +4 -4
index a467b175c6..1070aa1ba9 100644
@@ -5218,7 +5218,7 @@ int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event,
                           const char *tag)
 {
     while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {
@@ -5231,7 +5231,7 @@ int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event,
 int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)
 {
     while (bs && bs->drv && !bs->drv->bdrv_debug_remove_breakpoint) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_remove_breakpoint) {
@@ -5244,7 +5244,7 @@ int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)
 int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
 {
     while (bs && (!bs->drv || !bs->drv->bdrv_debug_resume)) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_resume) {
@@ -5257,7 +5257,7 @@ int bdrv_debug_resume(BlockDriverState *bs, const char *tag)
 bool bdrv_debug_is_suspended(BlockDriverState *bs, const char *tag)
 {
     while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) {
-        bs = bs->file ? bs->file->bs : NULL;
+        bs = bdrv_primary_bs(bs);
     }
 
     if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
Signature · 1 line
2.21.0
author
#63 lore raw

PATCH v6 22/42 block: Fix bdrv_get_allocated_file_size's fallback

In reply to Max Reitz

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

---
 block.c | 22 ++++++++++++++++++++--
 1 file changed, 20 insertions(+), 2 deletions(-)
block.c +20 -2
index 1070aa1ba9..6e1ddab056 100644
@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
     if (drv->bdrv_get_allocated_file_size) {
         return drv->bdrv_get_allocated_file_size(bs);
     }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (!QLIST_EMPTY(&bs->children)) {
+        BdrvChild *child;
+        int64_t child_size, total_size = 0;
+
+        QLIST_FOREACH(child, &bs->children, next) {
+            if (child == bdrv_filtered_cow_child(bs)) {
+                /* Ignore COW backing files */
+                continue;
+            }
+
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;
+            }
+            total_size += child_size;
+        }
+
+        return total_size;
     }
+
     return -ENOTSUP;
 }
 
Signature · 1 line
2.21.0
#64 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 35 lines · Max Reitz

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

---
  block.c | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)
block.c +14 -2
index 1070aa1ba9..6e1ddab056 100644
@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
      if (drv->bdrv_get_allocated_file_size) {
          return drv->bdrv_get_allocated_file_size(bs);
      }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (!QLIST_EMPTY(&bs->children)) {
+        BdrvChild *child;
+        int64_t child_size, total_size = 0;
+
+        QLIST_FOREACH(child, &bs->children, next) {
+            if (child == bdrv_filtered_cow_child(bs)) {
+                /* Ignore COW backing files */
+                continue;
+            }
+
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676

Quoted 10 lines
block.c +6 -0
+            }
+            total_size += child_size;
+        }
+
+        return total_size;
      }
+
      return -ENOTSUP;
  }
 

Hmm..

1. No children -> -ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUM

It's all arguable (the strictest way is -ENOTSUP in either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.

Signature · 2 lines
Best regards,
Vladimir
author
#65 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 35 lines · Max Reitz

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

---
  block.c | 22 ++++++++++++++++++++--
  1 file changed, 20 insertions(+), 2 deletions(-)
block.c +14 -2
index 1070aa1ba9..6e1ddab056 100644
@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
      if (drv->bdrv_get_allocated_file_size) {
          return drv->bdrv_get_allocated_file_size(bs);
      }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (!QLIST_EMPTY(&bs->children)) {
+        BdrvChild *child;
+        int64_t child_size, total_size = 0;
+
+        QLIST_FOREACH(child, &bs->children, next) {
+            if (child == bdrv_filtered_cow_child(bs)) {
+                /* Ignore COW backing files */
+                continue;
+            }
+
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676

Quoted 10 lines
block.c +6 -0
+            }
+            total_size += child_size;
+        }
+
+        return total_size;
      }
+
      return -ENOTSUP;
  }
 

Hmm..

Quoted 7 lines

1. No children -> -ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUM

It's all arguable (the strictest way is -ENOTSUP in either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.

I don’t think 2 is possible at all. If you have a COW child, you need some other child to COW to.

And in the weird (and probably impossible) case where a node really only has a COW child, I’d say it’s correct that it has a disk size of 0 – because it hasn’t COWed anything yet. (Just like a new qcow2 image with a backing file only has its metadata as its disk size.)

Max

#66 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 58 lines · Max Reitz
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 35 lines · Max Reitz

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

---
   block.c | 22 ++++++++++++++++++++--
   1 file changed, 20 insertions(+), 2 deletions(-)
block.c +14 -2
index 1070aa1ba9..6e1ddab056 100644
@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
       if (drv->bdrv_get_allocated_file_size) {
           return drv->bdrv_get_allocated_file_size(bs);
       }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (!QLIST_EMPTY(&bs->children)) {
+        BdrvChild *child;
+        int64_t child_size, total_size = 0;
+
+        QLIST_FOREACH(child, &bs->children, next) {
+            if (child == bdrv_filtered_cow_child(bs)) {
+                /* Ignore COW backing files */
+                continue;
+            }
+
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676

Quoted 10 lines
block.c +6 -0
+            }
+            total_size += child_size;
+        }
+
+        return total_size;
       }
+
       return -ENOTSUP;
   }
 

Hmm..

Quoted 7 lines

1. No children -> -ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUM

It's all arguable (the strictest way is -ENOTSUP in either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.

Quoted 8 lines

I don’t think 2 is possible at all. If you have a COW child, you need some other child to COW to.

And in the weird (and probably impossible) case where a node really only has a COW child, I’d say it’s correct that it has a disk size of 0 – because it hasn’t COWed anything yet. (Just like a new qcow2 image with a backing file only has its metadata as its disk size.)

Agreed. Then, why not return 0 on [1] ?

Also, another idea: shouldn't we return 0 for filters, i.e. skip filtered_rw_child too? [as filtered-child is more like backing child than file one, it's "less owned" by its parent]

with or without any of these suggestions:

Signature · 2 lines
Best regards,
Vladimir
author
#67 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 62 lines · Vladimir Sementsov-Ogievskiy
Quoted 61 lines · Max Reitz
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 35 lines · Max Reitz

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

---
   block.c | 22 ++++++++++++++++++++--
   1 file changed, 20 insertions(+), 2 deletions(-)
block.c +14 -2
index 1070aa1ba9..6e1ddab056 100644
@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)
       if (drv->bdrv_get_allocated_file_size) {
           return drv->bdrv_get_allocated_file_size(bs);
       }
-    if (bs->file) {
-        return bdrv_get_allocated_file_size(bs->file->bs);
+
+    if (!QLIST_EMPTY(&bs->children)) {
+        BdrvChild *child;
+        int64_t child_size, total_size = 0;
+
+        QLIST_FOREACH(child, &bs->children, next) {
+            if (child == bdrv_filtered_cow_child(bs)) {
+                /* Ignore COW backing files */
+                continue;
+            }
+
+            child_size = bdrv_get_allocated_file_size(child->bs);
+            if (child_size < 0) {
+                return child_size;

block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676

Quoted 10 lines
block.c +6 -0
+            }
+            total_size += child_size;
+        }
+
+        return total_size;
       }
+
       return -ENOTSUP;
   }
 

Hmm..

Quoted 7 lines

1. No children -> -ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUM

It's all arguable (the strictest way is -ENOTSUP in either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.

I don’t think 2 is possible at all. If you have a COW child, you need some other child to COW to.

Quoted 7 lines
Quoted 5 lines

And in the weird (and probably impossible) case where a node really only has a COW child, I’d say it’s correct that it has a disk size of 0 – because it hasn’t COWed anything yet. (Just like a new qcow2 image with a backing file only has its metadata as its disk size.)

Agreed. Then, why not return 0 on [1] ?

(1) Because that’s the current behavior. :-)

(2) Nodes that have no children are protocol nodes. Protocol nodes (apart from null) still have to store their data somewhere. Therefore, they must implement .bdrv_get_allocated_file_size() to report that. If they don’t, that doesn’t mean they don’t store any data, but only that we don’t know how much data they store.

Quoted 2 lines

Also, another idea: shouldn't we return 0 for filters, i.e. skip filtered_rw_child too? [as filtered-child is more like backing child than file one, it's "less owned" by its parent]

Why would we do that? If I have a block device with a throttle node attached to it and request how much space it uses, of course I will want to know how much space the whole tree below it uses.

(Otherwise, bdrv_get_allocated_file_size() should only report anything for protocol nodes, and 0 for everything else.)

Max

Quoted 2 lines

with or without any of these suggestions:

#68 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.

Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.

Or eventually the block job filter nodes.

Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.

And please adjust the comment for bdrv_get_allocated_file_size(), it only talks about a single file as if trees didn't exist. Actually, it doesn't even seem so easy to define. Maybe primary node + storage nodes? Then vmdk needs to expose its extents as storage nodes (plural!), but in the long run that might be needed anyway.

Kevin

author
#69 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 13 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.

Quoted 4 lines

Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.

Or eventually the block job filter nodes.

I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.

If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?

Quoted 5 lines

Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.

I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.

Max

Quoted 5 lines

And please adjust the comment for bdrv_get_allocated_file_size(), it only talks about a single file as if trees didn't exist. Actually, it doesn't even seem so easy to define. Maybe primary node + storage nodes? Then vmdk needs to expose its extents as storage nodes (plural!), but in the long run that might be needed anyway.

#70 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:

Quoted 14 lines · Kevin Wolf
Quoted 13 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.

Quoted 7 lines
Quoted 4 lines

Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.

Or eventually the block job filter nodes.

I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.

Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.

If you include the block job filter nodes, I have to disagree, though. If mirror_top_bs (or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.

Quoted 5 lines

If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?

Do it's the quorum way: Just vote!

No, you're right, of course. -ENOTSUP is probably the only other thing you could do then.

Quoted 8 lines
Quoted 5 lines

Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.

I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.

Well, by default the primary child, which should cover like 90% of the drivers?

Kevin

author
#71 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 29 lines · Kevin Wolf

Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:

Quoted 14 lines · Kevin Wolf
Quoted 13 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.

Quoted 7 lines
Quoted 4 lines

Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.

Or eventually the block job filter nodes.

I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.

Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.

Quoted 4 lines

If you include the block job filter nodes, I have to disagree, though. If mirror_top_bs (or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.

Hm, I don’t care much either way. I think it makes complete sense to add the target size there, but OTOH it’s only temporary while the job runs, so it may be a bit confusing if it suddenly goes up and then down again.

But I think this is the special case, so this is what should be handled in a driver callback.

Quoted 7 lines
Quoted 5 lines

If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?

Do it's the quorum way: Just vote!

Add an option for it? Average, maximum, median, majority, sum? :-)

Quoted 9 lines

No, you're right, of course. -ENOTSUP is probably the only other thing you could do then.

Quoted 6 lines
Quoted 5 lines

Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.

Quoted 5 lines
Quoted 2 lines

I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.

Well, by default the primary child, which should cover like 90% of the drivers?

Hm, yes.

But I still think that the drivers that do not want to count every single non-COW child are the exception.

Max

#72 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:

Quoted 35 lines · Kevin Wolf
Quoted 29 lines

Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:

Quoted 14 lines · Kevin Wolf
Quoted 13 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.

Quoted 7 lines
Quoted 4 lines

Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.

Or eventually the block job filter nodes.

I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.

Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.

Quoted 4 lines

If you include the block job filter nodes, I have to disagree, though. If mirror_top_bs (or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.

Quoted 4 lines

Hm, I don’t care much either way. I think it makes complete sense to add the target size there, but OTOH it’s only temporary while the job runs, so it may be a bit confusing if it suddenly goes up and then down again.

I think the number that most users are interested in is knowing how much space the image for their /dev/vda takes up on the host.

I can see how they might be interested in not only that one image file, but all other image files connected to it, i.e. their /dev/vda with all of its snapshots. This would mean counting backing files. I think adding up the numbers for this should be done in the management layer.

I can possibly also imagine users wanting to count everything that's even loosely connected to their /dev/vda, like copies of it. I doubt, however, they want to count only copies that are currently being made, but not snapshots and copies that have been completed earlier. So this is clearly a management layer thing, too.

Quoted 2 lines

But I think this is the special case, so this is what should be handled in a driver callback.

It's a special case, yes. But see below.

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines

If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?

Do it's the quorum way: Just vote!

Add an option for it? Average, maximum, median, majority, sum? :-)

We could also introduce a mode with an Electoral College so that sometimes an image that missed the majority has a chance to win anyway.

Quoted 12 lines
Quoted 12 lines

No, you're right, of course. -ENOTSUP is probably the only other thing you could do then.

Quoted 8 lines
Quoted 5 lines

Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.

I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.

Quoted 7 lines
Quoted 2 lines

Well, by default the primary child, which should cover like 90% of the drivers?

Hm, yes.

But I still think that the drivers that do not want to count every single non-COW child are the exception.

They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.

I really think we're likely to forget adding the callback where we need to disable the feature.

I can see two options that should address both of our views:

1. Just don't have a fallback at all, make the callback mandatory and provide implementations in block.c that can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).

2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).

Adding the size of storage children covers qcow2 and vmdk.

As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.

For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.

Kevin

author
#73 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 41 lines · Kevin Wolf

Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:

Quoted 35 lines · Kevin Wolf
Quoted 29 lines

Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:

Quoted 14 lines · Kevin Wolf
Quoted 13 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

If the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.

This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.

Quoted 7 lines
Quoted 4 lines

Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.

Or eventually the block job filter nodes.

I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.

Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.

Quoted 4 lines

If you include the block job filter nodes, I have to disagree, though. If mirror_top_bs (or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.

Quoted 4 lines

Hm, I don’t care much either way. I think it makes complete sense to add the target size there, but OTOH it’s only temporary while the job runs, so it may be a bit confusing if it suddenly goes up and then down again.

Quoted 7 lines

I think the number that most users are interested in is knowing how much space the image for their /dev/vda takes up on the host.

I can see how they might be interested in not only that one image file, but all other image files connected to it, i.e. their /dev/vda with all of its snapshots. This would mean counting backing files. I think adding up the numbers for this should be done in the management layer.

My main argument against counting backing files is that we’ve never done it.

(Whereas for quorum, I’d argue we just forgot to adjust bdrv_get_allocated_file_size() for it.)

Quoted 5 lines

I can possibly also imagine users wanting to count everything that's even loosely connected to their /dev/vda, like copies of it. I doubt, however, they want to count only copies that are currently being made, but not snapshots and copies that have been completed earlier. So this is clearly a management layer thing, too.

OK.

Quoted 11 lines
Quoted 2 lines

But I think this is the special case, so this is what should be handled in a driver callback.

It's a special case, yes. But see below.

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Do it's the quorum way: Just vote!

Add an option for it? Average, maximum, median, majority, sum? :-)

We could also introduce a mode with an Electoral College so that sometimes an image that missed the majority has a chance to win anyway.

That’s actually a good idea for a quorum mode in general. Who says the majority is right? Better let someone with more authority cross-check the result.

Quoted 20 lines
Quoted 12 lines
Quoted 12 lines

No, you're right, of course. -ENOTSUP is probably the only other thing you could do then.

Quoted 8 lines
Quoted 5 lines

Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.

I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.

Quoted 7 lines
Quoted 2 lines

Well, by default the primary child, which should cover like 90% of the drivers?

Hm, yes.

But I still think that the drivers that do not want to count every single non-COW child are the exception.

Quoted 7 lines

They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.

I really think we're likely to forget adding the callback where we need to disable the feature.

Well, I mean, we did forget adding it for qcow2.

Quoted 8 lines

I can see two options that should address both of our views:

1. Just don't have a fallback at all, make the callback mandatory and provide implementations in block.c that can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).

Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.

Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.

Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?

Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)

Quoted 5 lines

2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).

I didn’t answer that because I didn’t disagree.

Quoted 1 line

Adding the size of storage children covers qcow2 and vmdk.

That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.

Quoted 8 lines

As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.

For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.

I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.

The better example is replication, but that has a clear filtered child (the primary node).

So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.

Max

#74 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 15 lines

Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:

Quoted 8 lines · Kevin Wolf
Quoted 2 lines

Well, by default the primary child, which should cover like 90% of the drivers?

Hm, yes.

But I still think that the drivers that do not want to count every single non-COW child are the exception.

They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.

Quoted 4 lines
Quoted 2 lines

I really think we're likely to forget adding the callback where we need to disable the feature.

Well, I mean, we did forget adding it for qcow2.

I'm afraid I have to agree. So the conclusion is that we won't get it right anyway?

Quoted 9 lines
Quoted 8 lines

I can see two options that should address both of our views:

1. Just don't have a fallback at all, make the callback mandatory and provide implementations in block.c that can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).

Quoted 7 lines

Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.

Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.

Yes, I think this is the same as I meant by "provide implementations in block.c".

Quoted 9 lines

Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?

Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)

The job filter nodes have only one child currently, which should be counted. We'll add other children that shouldn't be counted only later.

But we already have an idea of what possible extensions look like, so we can probably choose the right function from the start.

Quoted 6 lines
Quoted 5 lines

2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).

Quoted 8 lines

I didn’t answer that because I didn’t disagree.

Quoted 1 line

Adding the size of storage children covers qcow2 and vmdk.

That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.

Well, you added bdrv_storage_child(). I'd argue this interface is wrong because it assumes that only one storage child exists. You just didn't implement it for vmdk so that the problem didn't become apparent. It would have to return a list rather than a single child. So fixing the interface and then using it is what I was thinking.

Now that you mention a per-BdrvChild attribute, however, I start to wonder if the distinction between COW children, filter children, storage children, metadata children, etc. isn't really what BdrvChildRole was supposed to represent?

Maybe we want to split off child_storage from child_file, though it's not strictly necessary for this specific case because we want to treat both metadata and storage nodes the same. But it could be useful for other users of bdrv_storage_child(), if there are any.

Quoted 12 lines
Quoted 8 lines

As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.

For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.

I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.

Quoted 8 lines

The better example is replication, but that has a clear filtered child (the primary node).

So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.

Yes, that should work.

We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.

Kevin

author
#75 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Quoted 55 lines · Kevin Wolf

Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 15 lines

Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:

Quoted 8 lines · Kevin Wolf
Quoted 2 lines

Well, by default the primary child, which should cover like 90% of the drivers?

Hm, yes.

But I still think that the drivers that do not want to count every single non-COW child are the exception.

They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.

Quoted 4 lines
Quoted 2 lines

I really think we're likely to forget adding the callback where we need to disable the feature.

Well, I mean, we did forget adding it for qcow2.

I'm afraid I have to agree. So the conclusion is that we won't get it right anyway?

Quoted 9 lines
Quoted 8 lines

I can see two options that should address both of our views:

1. Just don't have a fallback at all, make the callback mandatory and provide implementations in block.c that can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).

Quoted 7 lines

Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.

Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.

Yes, I think this is the same as I meant by "provide implementations in block.c".

Quoted 9 lines

Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?

Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)

Quoted 5 lines

The job filter nodes have only one child currently, which should be counted. We'll add other children that shouldn't be counted only later.

But we already have an idea of what possible extensions look like, so we can probably choose the right function from the start.

Yep.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).

I didn’t answer that because I didn’t disagree.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Adding the size of storage children covers qcow2 and vmdk.

That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.

Well, you added bdrv_storage_child().I'd argue this interface is wrong

Yes, it probably is.

Quoted 9 lines

because it assumes that only one storage child exists. You just didn't implement it for vmdk so that the problem didn't become apparent. It would have to return a list rather than a single child. So fixing the interface and then using it is what I was thinking.

Now that you mention a per-BdrvChild attribute, however, I start to wonder if the distinction between COW children, filter children, storage children, metadata children, etc. isn't really what BdrvChildRole was supposed to represent?

That’s a good point.

Quoted 4 lines

Maybe we want to split off child_storage from child_file, though it's not strictly necessary for this specific case because we want to treat both metadata and storage nodes the same. But it could be useful for other users of bdrv_storage_child(), if there are any.

Possible. Maybe it turns out that at least for this series I don’t need bdrv_storage_child() at all.

Quoted 21 lines
Quoted 12 lines
Quoted 8 lines

As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.

For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.

I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.

Quoted 8 lines

The better example is replication, but that has a clear filtered child (the primary node).

So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.

Quoted 5 lines

Yes, that should work.

We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.

Hm. It feels like something that should be done before this series, actually.

So I think we should add at least a child role per child access function so that they match? And then maybe in bdrv_attach_child() assert that a BDS never has more than one primary or filtered child (a filtered child acts as a primary child, too), or more than one COW child. (And that these are always in bs->file or bs->backing so the child access functions do work.)

Max

#76 lore raw

Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

Am 11.09.2019 um 13:00 hat Max Reitz geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 11 lines

Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:

Quoted 4 lines

So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.

Yes, that should work.

We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.

Hm. It feels like something that should be done before this series, actually.

Quoted 6 lines

So I think we should add at least a child role per child access function so that they match? And then maybe in bdrv_attach_child() assert that a BDS never has more than one primary or filtered child (a filtered child acts as a primary child, too), or more than one COW child. (And that these are always in bs->file or bs->backing so the child access functions do work.)

Makes sense to me.

Kevin

author
#77 lore raw

Re: [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback

In reply to Max Reitz
Quoted 133 lines · Max Reitz
Quoted 55 lines · Kevin Wolf

Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:

Quoted 16 lines · Kevin Wolf
Quoted 15 lines

Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:

Quoted 8 lines · Kevin Wolf
Quoted 2 lines

Well, by default the primary child, which should cover like 90% of the drivers?

Hm, yes.

But I still think that the drivers that do not want to count every single non-COW child are the exception.

They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.

Quoted 4 lines
Quoted 2 lines

I really think we're likely to forget adding the callback where we need to disable the feature.

Well, I mean, we did forget adding it for qcow2.

I'm afraid I have to agree. So the conclusion is that we won't get it right anyway?

Quoted 9 lines
Quoted 8 lines

I can see two options that should address both of our views:

1. Just don't have a fallback at all, make the callback mandatory and provide implementations in block.c that can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).

Quoted 7 lines

Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.

Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.

Yes, I think this is the same as I meant by "provide implementations in block.c".

Quoted 9 lines

Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?

Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)

Quoted 5 lines

The job filter nodes have only one child currently, which should be counted. We'll add other children that shouldn't be counted only later.

But we already have an idea of what possible extensions look like, so we can probably choose the right function from the start.

Yep.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).

I didn’t answer that because I didn’t disagree.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Adding the size of storage children covers qcow2 and vmdk.

That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.

Well, you added bdrv_storage_child().I'd argue this interface is wrong

Yes, it probably is.

Quoted 9 lines

because it assumes that only one storage child exists. You just didn't implement it for vmdk so that the problem didn't become apparent. It would have to return a list rather than a single child. So fixing the interface and then using it is what I was thinking.

Now that you mention a per-BdrvChild attribute, however, I start to wonder if the distinction between COW children, filter children, storage children, metadata children, etc. isn't really what BdrvChildRole was supposed to represent?

That’s a good point.

Quoted 4 lines

Maybe we want to split off child_storage from child_file, though it's not strictly necessary for this specific case because we want to treat both metadata and storage nodes the same. But it could be useful for other users of bdrv_storage_child(), if there are any.

Possible. Maybe it turns out that at least for this series I don’t need bdrv_storage_child() at all.

Quoted 21 lines
Quoted 12 lines
Quoted 8 lines

As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.

For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.

I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.

Quoted 8 lines

The better example is replication, but that has a clear filtered child (the primary node).

So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.

Quoted 5 lines

Yes, that should work.

We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.

Hm. It feels like something that should be done before this series, actually.

Quoted 6 lines

So I think we should add at least a child role per child access function so that they match? And then maybe in bdrv_attach_child() assert that a BDS never has more than one primary or filtered child (a filtered child acts as a primary child, too), or more than one COW child. (And that these are always in bs->file or bs->backing so the child access functions do work.)

I’ve been trying to make this work, but I don’t think it does. It just feels all wrong and I need up with things like “child_metadata_and_data”. The last straw was that blkverify should have the raw file be the filtered child (because, well, it’s bs->file), but then the format file would need to be a non-filtered child, and those would default to BDRV_O_PROTOCOL (which we decidedly don’t want).

Anyway, I’m currently attempting to solve this differently:
BdrvChildRole isn’t suitable for the job, I think. The name is completely what we want, but it actually doesn’t look like something that describes the child role to me.

Instead, I’m introducing a new BdrvChildRole enum mask that describes how the child is going to be used: stay-at-node, cow, metadata, data, etc.

I’m going to rename the current BdrvChildRole structure to BdrvChildParent (in want of a better name), because really most of what it does is describe the parent, but precisely not the child. I’m moving .stay_as_node to the new BdrvChildRole enum.

I hope this lets me unify child_file, child_backing, and child_format into a child_of_bds object. The callbacks should then decide the particularities based on the BdrvChildRole enum.

Hope that makes sense. (? :S)

At least I feel much happier implementing it this way, which I suppose is a good sign.

Max

author
#78 lore raw

PATCH v6 23/42 blockdev: Use CAF in external_snapshot_prepare()

In reply to Max Reitz

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

---
 blockdev.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +6 -1
index 29c6c6044a..c540802127 100644
@@ -1664,7 +1664,12 @@ static void external_snapshot_prepare(BlkActionState *common,
         goto out;
     }
 
-    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {
+        error_setg(errp, "Filters cannot be used as overlays");
+        goto out;
+    }
+
+    if (bdrv_filtered_cow_child(state->new_bs)) {
         error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
Signature · 1 line
2.21.0
#79 lore raw

Re: [Qemu-devel] [PATCH v6 23/42] blockdev: Use CAF in external_snapshot_prepare()

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

Didn't we occasionally advertise blockdev-snapshot as the way to insert filters on top at runtime? Though it seems it has always only worked for filters that use bs->backing, among which I think there aren't any user-creatable ones. So we're probably good.

Kevin

Quoted 24 lines
 blockdev.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +6 -1
index 29c6c6044a..c540802127 100644
@@ -1664,7 +1664,12 @@ static void external_snapshot_prepare(BlkActionState *common,
         goto out;
     }
 
-    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {
+        error_setg(errp, "Filters cannot be used as overlays");
+        goto out;
+    }
+
+    if (bdrv_filtered_cow_child(state->new_bs)) {
         error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
Signature · 1 line
2.21.0
author
#80 lore raw

Re: [Qemu-devel] [PATCH v6 23/42] blockdev: Use CAF in external_snapshot_prepare()

Quoted 10 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 6 lines

This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).

Didn't we occasionally advertise blockdev-snapshot as the way to insert filters on top at runtime?

I can only remember advertising for it as the only graph manipulation tool we had, and maybe saying “We’d want something like blockdev-snapshot for filters, too”.

Max

Quoted 30 lines

Though it seems it has always only worked for
filters that use bs->backing, among which I think there aren't any user-creatable ones. So we're probably good.

Kevin

Quoted 24 lines
 blockdev.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +6 -1
index 29c6c6044a..c540802127 100644
@@ -1664,7 +1664,12 @@ static void external_snapshot_prepare(BlkActionState *common,
         goto out;
     }
 
-    if (state->new_bs->backing != NULL) {
+    if (state->new_bs->drv->is_filter) {
+        error_setg(errp, "Filters cannot be used as overlays");
+        goto out;
+    }
+
+    if (bdrv_filtered_cow_child(state->new_bs)) {
         error_setg(errp, "The overlay already has a backing image");
         goto out;
     }
Signature · 1 line
2.21.0
author
#81 lore raw

PATCH v6 24/42 block: Use child access functions for QAPI queries

In reply to Max Reitz

query-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just bs->backing or COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.

---
 block/qapi.c               | 39 +++++++++++++++++++++++---------------
 tests/qemu-iotests/184.out |  7 ++++++-
 2 files changed, 30 insertions(+), 16 deletions(-)
block/qapi.c +24 -15
index 9a185cba48..4f59ac1c0f 100644
@@ -156,9 +156,13 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
             return NULL;
         }
 
-        if (bs0->drv && bs0->backing) {
+        if (bs0->drv && bdrv_filtered_child(bs0)) {
+            /*
+             * Put any filtered child here (for backwards compatibility to when
+             * we put bs0->backing here, which might be any filtered child).
+             */
             info->backing_file_depth++;
-            bs0 = bs0->backing->bs;
+            bs0 = bdrv_filtered_bs(bs0);
             (*p_image_info)->has_backing_image = true;
             p_image_info = &((*p_image_info)->backing_image);
         } else {
@@ -167,9 +171,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
 
         /* Skip automatically inserted nodes that the user isn't aware of for
          * query-block (blk != NULL), but not for query-named-block-nodes */
-        while (blk && bs0->drv && bs0->implicit) {
-            bs0 = backing_bs(bs0);
-            assert(bs0);
+        if (blk) {
+            bs0 = bdrv_skip_implicit_filters(bs0);
         }
     }
 
@@ -354,9 +357,9 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,
     BlockDriverState *bs = blk_bs(blk);
     char *qdev;
 
-    /* Skip automatically inserted nodes that the user isn't aware of */
-    while (bs && bs->drv && bs->implicit) {
-        bs = backing_bs(bs);
+    if (bs) {
+        /* Skip automatically inserted nodes that the user isn't aware of */
+        bs = bdrv_skip_implicit_filters(bs);
     }
 
     info->device = g_strdup(blk_name(blk));
@@ -513,6 +516,7 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk)
 static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
                                         bool blk_level)
 {
+    BlockDriverState *storage_bs, *filtered_bs;
     BlockStats *s = NULL;
 
     s = g_malloc0(sizeof(*s));
@@ -525,9 +529,8 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
     /* Skip automatically inserted nodes that the user isn't aware of in
      * a BlockBackend-level command. Stay at the exact node for a node-level
      * command. */
-    while (blk_level && bs->drv && bs->implicit) {
-        bs = backing_bs(bs);
-        assert(bs);
+    if (blk_level) {
+        bs = bdrv_skip_implicit_filters(bs);
     }
 
     if (bdrv_get_node_name(bs)[0]) {
@@ -537,14 +540,20 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,
 
     s->stats->wr_highest_offset = stat64_get(&bs->wr_highest_offset);
 
-    if (bs->file) {
+    storage_bs = bdrv_storage_bs(bs);
+    if (storage_bs) {
         s->has_parent = true;
-        s->parent = bdrv_query_bds_stats(bs->file->bs, blk_level);
+        s->parent = bdrv_query_bds_stats(storage_bs, blk_level);
     }
 
-    if (blk_level && bs->backing) {
+    filtered_bs = bdrv_filtered_bs(bs);
+    if (blk_level && filtered_bs) {
+        /*
+         * Put any filtered child here (for backwards compatibility to when
+         * we put bs0->backing here, which might be any filtered child).
+         */
         s->has_backing = true;
-        s->backing = bdrv_query_bds_stats(bs->backing->bs, blk_level);
+        s->backing = bdrv_query_bds_stats(filtered_bs, blk_level);
     }
 
     return s;
tests/qemu-iotests/184.out +6 -1
index 3deb3cfb94..1d61f7e224 100644
@@ -27,6 +27,11 @@ Testing:
             "iops_rd": 0,
             "detect_zeroes": "off",
             "image": {
+                "backing-image": {
+                    "virtual-size": 1073741824,
+                    "filename": "null-co://",
+                    "format": "null-co"
+                },
                 "virtual-size": 1073741824,
                 "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
                 "format": "throttle"
@@ -34,7 +39,7 @@ Testing:
             "iops_wr": 0,
             "ro": false,
             "node-name": "throttle0",
-            "backing_file_depth": 0,
+            "backing_file_depth": 1,
             "drv": "throttle",
             "iops": 0,
             "bps_wr": 0,
Signature · 1 line
2.21.0
Reviewed-by on v6 24/42
#82 lore raw

Re: [Qemu-devel] [PATCH v6 24/42] block: Use child access functions for QAPI queries

Quoted 16 lines · Max Reitz

query-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just bs->backing or COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.

---
  block/qapi.c               | 39 +++++++++++++++++++++++---------------
  tests/qemu-iotests/184.out |  7 ++++++-
  2 files changed, 30 insertions(+), 16 deletions(-)
block/qapi.c +0 -0
index 9a185cba48..4f59ac1c0f 100644

[..]

block/qapi.c static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info, · line 363

Quoted 11 lines
block/qapi.c +3 -3
@@ -354,9 +357,9 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,
      BlockDriverState *bs = blk_bs(blk);
      char *qdev;
 
-    /* Skip automatically inserted nodes that the user isn't aware of */
-    while (bs && bs->drv && bs->implicit) {
-        bs = backing_bs(bs);
+    if (bs) {
+        /* Skip automatically inserted nodes that the user isn't aware of */
+        bs = bdrv_skip_implicit_filters(bs);
      }

bdrv_skip_implicit_filters supports NULL, so it may be written without "if"

Anyway:

[..]

Signature · 2 lines
Best regards,
Vladimir
author
#83 lore raw

PATCH v6 25/42 mirror: Deal with filters

In reply to Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
 block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
 blockdev.c     |  47 +++++++++++++++++---
 2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +91 -26
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
     BlockBackend *target;
     BlockDriverState *mirror_top_bs;
     BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
     /* The name of the graph node to replace */
     char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                              &error_abort);
     if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
         BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
             if (local_err) {
                 error_report_err(local_err);
                 ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
      * valid.
      */
     block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
     /* We just changed the BDS the job BB refers to (with either or both of the
      * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
             return 0;
         }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
         if (ret < 0) {
             return ret;
         }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
     } else {
         s->target_cluster_size = BDRV_SECTOR_SIZE;
     }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
         s->granularity < s->target_cluster_size) {
         s->buf_size = MAX(s->buf_size, s->target_cluster_size);
         s->cow_bitmap = bitmap_new(length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
     if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
         int ret;
 
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
         if (ret < 0) {
             return;
         }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(
     MirrorBlockJob *s;
     MirrorBDSOpaque *bs_opaque;
     BlockDriverState *mirror_top_bs;
-    bool target_graph_mod;
     bool target_is_backing;
+    uint64_t target_perms, target_shared_perms;
     Error *local_err = NULL;
     int ret;
 
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job(
         buf_size = DEFAULT_MIRROR_BUF_SIZE;
     }
 
-    if (bs == target) {
+    if (bdrv_skip_rw_filters(bs) == bdrv_skip_rw_filters(target)) {
         error_setg(errp, "Can't mirror node into itself");
         return NULL;
     }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(
      * In the case of active commit, things look a bit different, though,
      * because the target is an already populated backing file in active use.
      * We can allow anything except resize there.*/
+
+    target_perms = BLK_PERM_WRITE;
+    target_shared_perms = BLK_PERM_WRITE_UNCHANGED;
+
     target_is_backing = bdrv_chain_contains(bs, target);
-    target_graph_mod = (backing_mode != MIRROR_LEAVE_BACKING_CHAIN);
+    if (target_is_backing) {
+        int64_t bs_size, target_size;
+        bs_size = bdrv_getlength(bs);
+        if (bs_size < 0) {
+            error_setg_errno(errp, -bs_size,
+                             "Could not inquire top image size");
+            goto fail;
+        }
+
+        target_size = bdrv_getlength(target);
+        if (target_size < 0) {
+            error_setg_errno(errp, -target_size,
+                             "Could not inquire base image size");
+            goto fail;
+        }
+
+        if (target_size < bs_size) {
+            target_perms |= BLK_PERM_RESIZE;
+        }
+
+        target_shared_perms |= BLK_PERM_CONSISTENT_READ
+                            |  BLK_PERM_WRITE
+                            |  BLK_PERM_GRAPH_MOD;
+    } else if (bdrv_chain_contains(bs, bdrv_skip_rw_filters(target))) {
+        /*
+         * We may want to allow this in the future, but it would
+         * require taking some extra care.
+         */
+        error_setg(errp, "Cannot mirror to a filter on top of a node in the "
+                   "source's backing chain");
+        goto fail;
+    }
+
+    if (backing_mode != MIRROR_LEAVE_BACKING_CHAIN) {
+        target_perms |= BLK_PERM_GRAPH_MOD;
+    }
+
     s->target = blk_new(s->common.job.aio_context,
-                        BLK_PERM_WRITE | BLK_PERM_RESIZE |
-                        (target_graph_mod ? BLK_PERM_GRAPH_MOD : 0),
-                        BLK_PERM_WRITE_UNCHANGED |
-                        (target_is_backing ? BLK_PERM_CONSISTENT_READ |
-                                             BLK_PERM_WRITE |
-                                             BLK_PERM_GRAPH_MOD : 0));
+                        target_perms, target_shared_perms);
     ret = blk_insert_bs(s->target, target, errp);
     if (ret < 0) {
         goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job(
     s->backing_mode = backing_mode;
     s->copy_mode = copy_mode;
     s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
     s->granularity = granularity;
     s->buf_size = ROUND_UP(buf_size, granularity);
     s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
     /* In commit_active_start() all intermediate nodes disappear, so
      * any jobs in them must be blocked */
     if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
             ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
             if (ret < 0) {
                 goto fail;
             }
@@ -1737,7 +1802,7 @@ fail:
     bs_opaque->stop = true;
     bdrv_child_refresh_perms(mirror_top_bs, mirror_top_bs->backing,
                              &error_abort);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
     bdrv_unref(mirror_top_bs);
 
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
         return;
     }
     is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
-    base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;
+    base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
     mirror_start_job(job_id, bs, creation_flags, target, replaces,
                      speed, granularity, buf_size, backing_mode,
                      on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +40 -7

114 lines not on this page · raw message

Signature · 1 line
2.21.0
#84 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 14 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
  block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
  blockdev.c     |  47 +++++++++++++++++---
  2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +0 -0
index 54bafdf176..6ddbfb9708 100644

[..]

Quoted 29 lines
block/mirror.c +19 -6
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
      /* In commit_active_start() all intermediate nodes disappear, so
       * any jobs in them must be blocked */
      if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;

block/mirror.c static BlockJob *mirror_start_job( · line 1765

Quoted 10 lines
block/mirror.c +10 -0
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;

Hmm, I don't understand, why read from upper nodes is not shared?

Quoted 9 lines
block/mirror.c +3 -2
+            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
              if (ret < 0) {
                  goto fail;
              }

[..]

Signature · 2 lines
Best regards,
Vladimir
author
#85 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
  block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
  blockdev.c     |  47 +++++++++++++++++---
  2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +0 -0
index 54bafdf176..6ddbfb9708 100644

[..]

Quoted 34 lines
block/mirror.c +24 -6
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
      /* In commit_active_start() all intermediate nodes disappear, so
       * any jobs in them must be blocked */
      if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1765

Quoted 5 lines
block/mirror.c +5 -0
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;

Hmm, I don't understand, why read from upper nodes is not shared?

Because they don’t represent a consistent disk state during the commit.

Please don’t ask me details about CONSISTENT_READ, because I always pretend I understand it, but I never really do, actually.

(My problem is that I do understand why the intermediate nodes shouldn’t share CONSISTENT_READ: It’s because they only read garbage, effectively.
But I don’t understand how any block job target (like our base here)
can have CONSISTENT_READ. Block job targets are mostly written front to back (except with sync=none), so they too don’t “[represent] the contents of a disk at a specific point.”
But that is how it was, so that is how it should be kept.)

If it makes you any happier, BLK_PERM_CONSISTENT_READ’s description explicitly notes that it will not be shared on intermediate nodes of a commit job.

Max

Quoted 12 lines
Quoted 9 lines
block/mirror.c +3 -2
+            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
              if (ret < 0) {
                  goto fail;
              }

[..]

#86 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 65 lines · Max Reitz
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
   block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
   blockdev.c     |  47 +++++++++++++++++---
   2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +0 -0
index 54bafdf176..6ddbfb9708 100644

[..]

Quoted 34 lines
block/mirror.c +24 -6
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
       /* In commit_active_start() all intermediate nodes disappear, so
        * any jobs in them must be blocked */
       if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1765

Quoted 5 lines
block/mirror.c +5 -0
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;

Hmm, I don't understand, why read from upper nodes is not shared?

Because they don’t represent a consistent disk state during the commit.

Quoted 7 lines

Please don’t ask me details about CONSISTENT_READ, because I always pretend I understand it, but I never really do, actually.

(My problem is that I do understand why the intermediate nodes shouldn’t share CONSISTENT_READ: It’s because they only read garbage, effectively.
But I don’t understand how any block job target (like our base here)
can have CONSISTENT_READ.

I know such example: it's image fleecing scheme, when for backup job source is a backing for target. If serialization of requests works well target represents consistent state of disk ate backup-start point in time.

But yes, it's not about mirror or commit.

Quoted 25 lines

Block job targets are mostly written front to
back (except with sync=none), so they too don’t “[represent] the contents of a disk at a specific point.”
But that is how it was, so that is how it should be kept.)

If it makes you any happier, BLK_PERM_CONSISTENT_READ’s description explicitly notes that it will not be shared on intermediate nodes of a commit job.

Max

Quoted 12 lines
Quoted 9 lines
block/mirror.c +3 -2
+            }
+
               ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
               if (ret < 0) {
                   goto fail;
               }

[..]

Signature · 2 lines
Best regards,
Vladimir
#87 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

In reply to Max Reitz
Quoted 64 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
  block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
  blockdev.c     |  47 +++++++++++++++++---
  2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +9 -5
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
      BlockBackend *target;
      BlockDriverState *mirror_top_bs;
      BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
      /* The name of the graph node to replace */
      char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                               &error_abort);
      if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
          BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
              if (local_err) {
                  error_report_err(local_err);
                  ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
       * valid.
       */
      block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
      /* We just changed the BDS the job BB refers to (with either or both of the
       * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
              return 0;
          }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
          if (ret < 0) {
              return ret;
          }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
      } else {
          s->target_cluster_size = BDRV_SECTOR_SIZE;
      }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
          s->granularity < s->target_cluster_size) {
          s->buf_size = MAX(s->buf_size, s->target_cluster_size);
          s->cow_bitmap = bitmap_new(length);

block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095

Quoted 6 lines
block/mirror.c +1 -2
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
      if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
          int ret;
 
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));

Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.

Quoted 26 lines
block/mirror.c +4 -2
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
          if (ret < 0) {
              return;
          }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(
      MirrorBlockJob *s;
      MirrorBDSOpaque *bs_opaque;
      BlockDriverState *mirror_top_bs;
-    bool target_graph_mod;
      bool target_is_backing;
+    uint64_t target_perms, target_shared_perms;
      Error *local_err = NULL;
      int ret;
 
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job(
          buf_size = DEFAULT_MIRROR_BUF_SIZE;
      }
 
-    if (bs == target) {
+    if (bdrv_skip_rw_filters(bs) == bdrv_skip_rw_filters(target)) {
          error_setg(errp, "Can't mirror node into itself");
          return NULL;
      }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(
       * In the case of active commit, things look a bit different, though,

block/mirror.c static BlockJob *mirror_start_job( · line 1629

Quoted 10 lines
block/mirror.c +6 -1
       * because the target is an already populated backing file in active use.
       * We can allow anything except resize there.*/
+
+    target_perms = BLK_PERM_WRITE;
+    target_shared_perms = BLK_PERM_WRITE_UNCHANGED;
+
      target_is_backing = bdrv_chain_contains(bs, target);
-    target_graph_mod = (backing_mode != MIRROR_LEAVE_BACKING_CHAIN);
+    if (target_is_backing) {
+        int64_t bs_size, target_size;

<empty after definitions>

Quoted 81 lines
block/mirror.c +54 -12
+        bs_size = bdrv_getlength(bs);
+        if (bs_size < 0) {
+            error_setg_errno(errp, -bs_size,
+                             "Could not inquire top image size");
+            goto fail;
+        }
+
+        target_size = bdrv_getlength(target);
+        if (target_size < 0) {
+            error_setg_errno(errp, -target_size,
+                             "Could not inquire base image size");
+            goto fail;
+        }
+
+        if (target_size < bs_size) {
+            target_perms |= BLK_PERM_RESIZE;
+        }
+
+        target_shared_perms |= BLK_PERM_CONSISTENT_READ
+                            |  BLK_PERM_WRITE
+                            |  BLK_PERM_GRAPH_MOD;
+    } else if (bdrv_chain_contains(bs, bdrv_skip_rw_filters(target))) {
+        /*
+         * We may want to allow this in the future, but it would
+         * require taking some extra care.
+         */
+        error_setg(errp, "Cannot mirror to a filter on top of a node in the "
+                   "source's backing chain");
+        goto fail;
+    }
+
+    if (backing_mode != MIRROR_LEAVE_BACKING_CHAIN) {
+        target_perms |= BLK_PERM_GRAPH_MOD;
+    }
+
      s->target = blk_new(s->common.job.aio_context,
-                        BLK_PERM_WRITE | BLK_PERM_RESIZE |
-                        (target_graph_mod ? BLK_PERM_GRAPH_MOD : 0),
-                        BLK_PERM_WRITE_UNCHANGED |
-                        (target_is_backing ? BLK_PERM_CONSISTENT_READ |
-                                             BLK_PERM_WRITE |
-                                             BLK_PERM_GRAPH_MOD : 0));
+                        target_perms, target_shared_perms);
      ret = blk_insert_bs(s->target, target, errp);
      if (ret < 0) {
          goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job(
      s->backing_mode = backing_mode;
      s->copy_mode = copy_mode;
      s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
      s->granularity = granularity;
      s->buf_size = ROUND_UP(buf_size, granularity);
      s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
      /* In commit_active_start() all intermediate nodes disappear, so
       * any jobs in them must be blocked */
      if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).

block/mirror.c static BlockJob *mirror_start_job( · line 1763

Quoted 10 lines
block/mirror.c +10 -0
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.

I'd prefere to add something like: "because we share it on target (see target BlockBackend creation and corresponding comment above)".

blockdev.c

Quoted 31 lines
block/mirror.c +7 -4
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
              if (ret < 0) {
                  goto fail;
              }
@@ -1737,7 +1802,7 @@ fail:
      bs_opaque->stop = true;
      bdrv_child_refresh_perms(mirror_top_bs, mirror_top_bs->backing,
                               &error_abort);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
      bdrv_unref(mirror_top_bs);
 
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
          return;
      }
      is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
-    base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;
+    base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
      mirror_start_job(job_id, bs, creation_flags, target, replaces,
                       speed, granularity, buf_size, backing_mode,
                       on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +0 -0
index c540802127..c451f553f7 100644

block/mirroc.c is OK for me. Continue with blockdev.c...

Quoted 46 lines
blockdev.c +13 -2
@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
          return;
      }
 
-    if (!bs->backing && sync == MIRROR_SYNC_MODE_TOP) {
+    if (!bdrv_backing_chain_next(bs) && sync == MIRROR_SYNC_MODE_TOP) {
          sync = MIRROR_SYNC_MODE_FULL;
      }
 
@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
 
  void qmp_drive_mirror(DriveMirror *arg, Error **errp)
  {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
      BlockDriverState *source, *target_bs;
      AioContext *aio_context;
      BlockMirrorBackingMode backing_mode;
@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
      int flags;
      int64_t size;
      const char *format = arg->format;
+    const char *replaces_node_name = NULL;
      int ret;
 
      bs = qmp_get_root_bs(arg->device, errp);
@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
          return;
      }
 
+    /*
+     * If the user has not instructed us otherwise, we should let the
+     * block job run from @bs (thus taking into account all filters on
+     * it) but replace @unfiltered_bs when it finishes (thus not
+     * removing those filters).
+     * (And if there are any explicit filters, we should assume the
+     *  user knows how to use the @replaces option.)
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);
+
      aio_context = bdrv_get_aio_context(bs);
      aio_context_acquire(aio_context);
 

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3949

Quoted 6 lines
blockdev.c +1 -1
@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
      }
 
      flags = bs->open_flags | BDRV_O_RDWR;
-    source = backing_bs(bs);
+    source = bdrv_filtered_cow_bs(unfiltered_bs);
      if (!source && arg->sync == MIRROR_SYNC_MODE_TOP) {

Hmm, you handle this case a bit differently here and in blockdev_mirror_common.. Can we handle it only in blockdev_mirror_common, to be consistent with qmp_blockdev_mirror?

Quoted 59 lines
blockdev.c +16 -4
+        if (bdrv_filtered_bs(unfiltered_bs)) {
+            /* @unfiltered_bs is an explicit filter */
+            error_setg(errp, "Cannot perform sync=top mirror through an "
+                       "explicitly added filter node on the source");
+            goto out;
+        }
          arg->sync = MIRROR_SYNC_MODE_FULL;
      }
      if (arg->sync == MIRROR_SYNC_MODE_NONE) {
@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                               " named node of the graph");
              goto out;
          }
+        replaces_node_name = arg->replaces;
+    } else if (unfiltered_bs != bs) {
+        replaces_node_name = unfiltered_bs->node_name;
      }
 
      if (arg->mode == NEW_IMAGE_MODE_ABSOLUTE_PATHS) {
@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
          bdrv_img_create(arg->target, format,
                          NULL, NULL, NULL, size, flags, false, &local_err);
      } else {
+        /* Implicit filters should not appear in the filename */
+        BlockDriverState *explicit_backing = bdrv_skip_implicit_filters(source);
+
          switch (arg->mode) {
          case NEW_IMAGE_MODE_EXISTING:
              break;
@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
              /* create new image with backing file */
              bdrv_refresh_filename(source);
              bdrv_img_create(arg->target, format,
-                            source->filename,
-                            source->drv->format_name,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name,
                              NULL, size, flags, false, &local_err);
              break;
          default:
@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
      }
 
      blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
-                           arg->has_replaces, arg->replaces, arg->sync,
+                           !!replaces_node_name, replaces_node_name, arg->sync,
                             backing_mode, arg->has_speed, arg->speed,
                             arg->has_granularity, arg->granularity,
                             arg->has_buf_size, arg->buf_size,
@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                           bool has_auto_dismiss, bool auto_dismiss,
                           Error **errp)
  {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
      BlockDriverState *target_bs;
      AioContext *aio_context;
      BlockMirrorBackingMode backing_mode = MIRROR_LEAVE_BACKING_CHAIN;

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4092

Quoted 5 lines
blockdev.c +2 -0
@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
          return;
      }
 
+    /*
+     * Same as in qmp_drive_mirror():

Then, may be better do it in blockdev_mirror_common ?

Quoted 13 lines
blockdev.c +8 -0
 We want to run the job from @bs,
+     * but we want to replace @unfiltered_bs on completion.
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);
+    if (!has_replaces && unfiltered_bs != bs) {
+        replaces = unfiltered_bs->node_name;
+        has_replaces = true;
+    }
+
      target_bs = bdrv_lookup_bs(target, target, errp);
      if (!target_bs) {
          return;
Signature · 2 lines
Best regards,
Vladimir
author
#88 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
  block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
  blockdev.c     |  47 +++++++++++++++++---
  2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +9 -5
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
      BlockBackend *target;
      BlockDriverState *mirror_top_bs;
      BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
      /* The name of the graph node to replace */
      char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                               &error_abort);
      if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
          BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
              if (local_err) {
                  error_report_err(local_err);
                  ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
       * valid.
       */
      block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
      /* We just changed the BDS the job BB refers to (with either or both of the
       * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
              return 0;
          }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
          if (ret < 0) {
              return ret;
          }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
      } else {
          s->target_cluster_size = BDRV_SECTOR_SIZE;
      }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
          s->granularity < s->target_cluster_size) {
          s->buf_size = MAX(s->buf_size, s->target_cluster_size);
          s->cow_bitmap = bitmap_new(length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
      if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
          int ret;
 
Quoted 7 lines

block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095

Quoted 3 lines
block/mirror.c +1 -2
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));

Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.

The idea is that MIRROR_OPEN_BACKING_CHAIN is set only when using drive-mirror with mode=existing. In this case, we also set BDRV_O_NO_BACKING for the target.

You’re right that a user could add a backing chain to the target during the operation. They really have to make an effort to shoot themselves in the foot for this because the target must have an auto-generated node name.

I suppose the best would be not to open the backing chain if the target node already has a backing child?

Quoted 30 lines
Quoted 30 lines
block/mirror.c +6 -2
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
          if (ret < 0) {
              return;
          }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(
      MirrorBlockJob *s;
      MirrorBDSOpaque *bs_opaque;
      BlockDriverState *mirror_top_bs;
-    bool target_graph_mod;
      bool target_is_backing;
+    uint64_t target_perms, target_shared_perms;
      Error *local_err = NULL;
      int ret;
 
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job(
          buf_size = DEFAULT_MIRROR_BUF_SIZE;
      }
 
-    if (bs == target) {
+    if (bdrv_skip_rw_filters(bs) == bdrv_skip_rw_filters(target)) {
          error_setg(errp, "Can't mirror node into itself");
          return NULL;
      }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(
       * In the case of active commit, things look a bit different, though,
       * because the target is an already populated backing file in active use.
       * We can allow anything except resize there.*/
+
+    target_perms = BLK_PERM_WRITE;
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1629

Quoted 6 lines
block/mirror.c +4 -1
+    target_shared_perms = BLK_PERM_WRITE_UNCHANGED;
+
      target_is_backing = bdrv_chain_contains(bs, target);
-    target_graph_mod = (backing_mode != MIRROR_LEAVE_BACKING_CHAIN);
+    if (target_is_backing) {
+        int64_t bs_size, target_size;

<empty after definitions>

Is that part of any of our guidelines? :-)

Sure, will add.

Quoted 86 lines
Quoted 86 lines
block/mirror.c +59 -12
+        bs_size = bdrv_getlength(bs);
+        if (bs_size < 0) {
+            error_setg_errno(errp, -bs_size,
+                             "Could not inquire top image size");
+            goto fail;
+        }
+
+        target_size = bdrv_getlength(target);
+        if (target_size < 0) {
+            error_setg_errno(errp, -target_size,
+                             "Could not inquire base image size");
+            goto fail;
+        }
+
+        if (target_size < bs_size) {
+            target_perms |= BLK_PERM_RESIZE;
+        }
+
+        target_shared_perms |= BLK_PERM_CONSISTENT_READ
+                            |  BLK_PERM_WRITE
+                            |  BLK_PERM_GRAPH_MOD;
+    } else if (bdrv_chain_contains(bs, bdrv_skip_rw_filters(target))) {
+        /*
+         * We may want to allow this in the future, but it would
+         * require taking some extra care.
+         */
+        error_setg(errp, "Cannot mirror to a filter on top of a node in the "
+                   "source's backing chain");
+        goto fail;
+    }
+
+    if (backing_mode != MIRROR_LEAVE_BACKING_CHAIN) {
+        target_perms |= BLK_PERM_GRAPH_MOD;
+    }
+
      s->target = blk_new(s->common.job.aio_context,
-                        BLK_PERM_WRITE | BLK_PERM_RESIZE |
-                        (target_graph_mod ? BLK_PERM_GRAPH_MOD : 0),
-                        BLK_PERM_WRITE_UNCHANGED |
-                        (target_is_backing ? BLK_PERM_CONSISTENT_READ |
-                                             BLK_PERM_WRITE |
-                                             BLK_PERM_GRAPH_MOD : 0));
+                        target_perms, target_shared_perms);
      ret = blk_insert_bs(s->target, target, errp);
      if (ret < 0) {
          goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job(
      s->backing_mode = backing_mode;
      s->copy_mode = copy_mode;
      s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
      s->granularity = granularity;
      s->buf_size = ROUND_UP(buf_size, granularity);
      s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
      /* In commit_active_start() all intermediate nodes disappear, so
       * any jobs in them must be blocked */
      if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1763

Quoted 5 lines
block/mirror.c +5 -0
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.

I'd prefere to add something like: "because we share it on target (see target BlockBackend creation and corresponding comment above)".

I’d rather not refer to other comments in case they change… Maybe just “This allows us to share BLK_PERM_CONSISTENT_READ, as we do on the target.”? I think if someone is interested, they will scan the file for what permissions are shared on the target anyway.

Quoted 83 lines

blockdev.c

Quoted 31 lines
block/mirror.c +7 -4
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
              ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
              if (ret < 0) {
                  goto fail;
              }
@@ -1737,7 +1802,7 @@ fail:
      bs_opaque->stop = true;
      bdrv_child_refresh_perms(mirror_top_bs, mirror_top_bs->backing,
                               &error_abort);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
      bdrv_unref(mirror_top_bs);
 
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
          return;
      }
      is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
-    base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;
+    base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
      mirror_start_job(job_id, bs, creation_flags, target, replaces,
                       speed, granularity, buf_size, backing_mode,
                       on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +0 -0
index c540802127..c451f553f7 100644

block/mirroc.c is OK for me. Continue with blockdev.c...

Quoted 48 lines
blockdev.c +13 -2
@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
          return;
      }
 
-    if (!bs->backing && sync == MIRROR_SYNC_MODE_TOP) {
+    if (!bdrv_backing_chain_next(bs) && sync == MIRROR_SYNC_MODE_TOP) {
          sync = MIRROR_SYNC_MODE_FULL;
      }
 
@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
 
  void qmp_drive_mirror(DriveMirror *arg, Error **errp)
  {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
      BlockDriverState *source, *target_bs;
      AioContext *aio_context;
      BlockMirrorBackingMode backing_mode;
@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
      int flags;
      int64_t size;
      const char *format = arg->format;
+    const char *replaces_node_name = NULL;
      int ret;
 
      bs = qmp_get_root_bs(arg->device, errp);
@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
          return;
      }
 
+    /*
+     * If the user has not instructed us otherwise, we should let the
+     * block job run from @bs (thus taking into account all filters on
+     * it) but replace @unfiltered_bs when it finishes (thus not
+     * removing those filters).
+     * (And if there are any explicit filters, we should assume the
+     *  user knows how to use the @replaces option.)
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);
+
      aio_context = bdrv_get_aio_context(bs);
      aio_context_acquire(aio_context);
 
@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
      }
 
Quoted 8 lines

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3949

Quoted 4 lines
blockdev.c +1 -1
      flags = bs->open_flags | BDRV_O_RDWR;
-    source = backing_bs(bs);
+    source = bdrv_filtered_cow_bs(unfiltered_bs);
      if (!source && arg->sync == MIRROR_SYNC_MODE_TOP) {

Hmm, you handle this case a bit differently here and in blockdev_mirror_common.. Can we handle it only in blockdev_mirror_common, to be consistent with qmp_blockdev_mirror?

What exactly do you mean? The difference between skipping all filters and just skipping implicit filters? Hm.

First, the check in blockdev_mirror_common() should actually be unnecessary. In qmp_{blockdev,drive}_mirror(), we do nearly the same check anyway (and then force sync=full if there is no backing file). So if all three functions did the same check, we wouldn’t need it in blockdev_mirror_common().

Second, let’s look at the difference in an example: One where blockdev_mirror_common() would not decide to enforce mode=full, but qmp_{blockdev,drive}_mirror() would.
This happens when @bs is an explicit filter over some overlay with a backing file, e.g.:

throttle --file--> qcow2 --backing--> raw

It’s correct to run the mirror job from the throttle node; but @source should be bdrv_backing_chain_next() so it will point to the raw node. Currently, it is NULL (because the throttle node does not have a COW child).

But then again, I’ve made qmp_{blockdev,drive}_mirror() throw an error in such a case:

Quoted 6 lines

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3955

Quoted 6 lines
blockdev.c +6 -0
+        if (bdrv_filtered_bs(unfiltered_bs)) {
+            /* @unfiltered_bs is an explicit filter */
+            error_setg(errp, "Cannot perform sync=top mirror through an "
+                       "explicitly added filter node on the source");
+            goto out;
+        }

So it isn’t really a problem. Still, does the error make sense? Should we just allow that case by letting source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs))?

(BTW, I just noticed that @base seems to be pretty much unused in block/mirror.c. It only really uses @base_overlay now. So I suppose it makes sense to remove it in v7.)

Quoted 56 lines
Quoted 56 lines
blockdev.c +10 -4
          arg->sync = MIRROR_SYNC_MODE_FULL;
      }
      if (arg->sync == MIRROR_SYNC_MODE_NONE) {
@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                               " named node of the graph");
              goto out;
          }
+        replaces_node_name = arg->replaces;
+    } else if (unfiltered_bs != bs) {
+        replaces_node_name = unfiltered_bs->node_name;
      }
 
      if (arg->mode == NEW_IMAGE_MODE_ABSOLUTE_PATHS) {
@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
          bdrv_img_create(arg->target, format,
                          NULL, NULL, NULL, size, flags, false, &local_err);
      } else {
+        /* Implicit filters should not appear in the filename */
+        BlockDriverState *explicit_backing = bdrv_skip_implicit_filters(source);
+
          switch (arg->mode) {
          case NEW_IMAGE_MODE_EXISTING:
              break;
@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
              /* create new image with backing file */
              bdrv_refresh_filename(source);
              bdrv_img_create(arg->target, format,
-                            source->filename,
-                            source->drv->format_name,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name,
                              NULL, size, flags, false, &local_err);
              break;
          default:
@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
      }
 
      blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
-                           arg->has_replaces, arg->replaces, arg->sync,
+                           !!replaces_node_name, replaces_node_name, arg->sync,
                             backing_mode, arg->has_speed, arg->speed,
                             arg->has_granularity, arg->granularity,
                             arg->has_buf_size, arg->buf_size,
@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                           bool has_auto_dismiss, bool auto_dismiss,
                           Error **errp)
  {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
      BlockDriverState *target_bs;
      AioContext *aio_context;
      BlockMirrorBackingMode backing_mode = MIRROR_LEAVE_BACKING_CHAIN;
@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
          return;
      }
 
Quoted 4 lines

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4092

Quoted 2 lines
blockdev.c +2 -0
+    /*
+     * Same as in qmp_drive_mirror():

Then, may be better do it in blockdev_mirror_common ?

Hm, maybe. Should we decide to let @source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs)) in qmp_drive_mirror(), I don’t think we need @unfiltered_bs there to determine @source.

Max

Quoted 15 lines
Quoted 13 lines
blockdev.c +8 -0
 We want to run the job from @bs,
+     * but we want to replace @unfiltered_bs on completion.
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);
+    if (!has_replaces && unfiltered_bs != bs) {
+        replaces = unfiltered_bs->node_name;
+        has_replaces = true;
+    }
+
      target_bs = bdrv_lookup_bs(target, target, errp);
      if (!target_bs) {
          return;
#89 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 81 lines · Max Reitz
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
   block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
   blockdev.c     |  47 +++++++++++++++++---
   2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +9 -5
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
       BlockBackend *target;
       BlockDriverState *mirror_top_bs;
       BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
       /* The name of the graph node to replace */
       char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                                &error_abort);
       if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
           BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
               if (local_err) {
                   error_report_err(local_err);
                   ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
        * valid.
        */
       block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
       /* We just changed the BDS the job BB refers to (with either or both of the
        * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
               return 0;
           }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
           if (ret < 0) {
               return ret;
           }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
       } else {
           s->target_cluster_size = BDRV_SECTOR_SIZE;
       }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
           s->granularity < s->target_cluster_size) {
           s->buf_size = MAX(s->buf_size, s->target_cluster_size);
           s->cow_bitmap = bitmap_new(length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
       if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
           int ret;
 
Quoted 7 lines

block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095

Quoted 3 lines
block/mirror.c +1 -2
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));

Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.

The idea is that MIRROR_OPEN_BACKING_CHAIN is set only when using drive-mirror with mode=existing. In this case, we also set BDRV_O_NO_BACKING for the target.

Quoted 7 lines

You’re right that a user could add a backing chain to the target during the operation. They really have to make an effort to shoot themselves in the foot for this because the target must have an auto-generated node name.

I suppose the best would be not to open the backing chain if the target node already has a backing child?

Hmm, but we still should generate an error, as we can't do what was requested.

Quoted 40 lines
Quoted 31 lines
Quoted 31 lines
block/mirror.c +6 -2
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
           if (ret < 0) {
               return;
           }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(
       MirrorBlockJob *s;
       MirrorBDSOpaque *bs_opaque;
       BlockDriverState *mirror_top_bs;
-    bool target_graph_mod;
       bool target_is_backing;
+    uint64_t target_perms, target_shared_perms;
       Error *local_err = NULL;
       int ret;
 
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job(
           buf_size = DEFAULT_MIRROR_BUF_SIZE;
       }
 
-    if (bs == target) {
+    if (bdrv_skip_rw_filters(bs) == bdrv_skip_rw_filters(target)) {
           error_setg(errp, "Can't mirror node into itself");
           return NULL;
       }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(
        * In the case of active commit, things look a bit different, though,
        * because the target is an already populated backing file in active use.
        * We can allow anything except resize there.*/
+
+    target_perms = BLK_PERM_WRITE;
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1629

Quoted 6 lines
block/mirror.c +4 -1
+    target_shared_perms = BLK_PERM_WRITE_UNCHANGED;
+
       target_is_backing = bdrv_chain_contains(bs, target);
-    target_graph_mod = (backing_mode != MIRROR_LEAVE_BACKING_CHAIN);
+    if (target_is_backing) {
+        int64_t bs_size, target_size;

<empty after definitions>

Quoted 3 lines

Is that part of any of our guidelines? :-)

Sure, will add.

Not sure. Someone asked me about it on list in past and I'm used to.

Quoted 96 lines
Quoted 87 lines
Quoted 87 lines
block/mirror.c +59 -12
+        bs_size = bdrv_getlength(bs);
+        if (bs_size < 0) {
+            error_setg_errno(errp, -bs_size,
+                             "Could not inquire top image size");
+            goto fail;
+        }
+
+        target_size = bdrv_getlength(target);
+        if (target_size < 0) {
+            error_setg_errno(errp, -target_size,
+                             "Could not inquire base image size");
+            goto fail;
+        }
+
+        if (target_size < bs_size) {
+            target_perms |= BLK_PERM_RESIZE;
+        }
+
+        target_shared_perms |= BLK_PERM_CONSISTENT_READ
+                            |  BLK_PERM_WRITE
+                            |  BLK_PERM_GRAPH_MOD;
+    } else if (bdrv_chain_contains(bs, bdrv_skip_rw_filters(target))) {
+        /*
+         * We may want to allow this in the future, but it would
+         * require taking some extra care.
+         */
+        error_setg(errp, "Cannot mirror to a filter on top of a node in the "
+                   "source's backing chain");
+        goto fail;
+    }
+
+    if (backing_mode != MIRROR_LEAVE_BACKING_CHAIN) {
+        target_perms |= BLK_PERM_GRAPH_MOD;
+    }
+
       s->target = blk_new(s->common.job.aio_context,
-                        BLK_PERM_WRITE | BLK_PERM_RESIZE |
-                        (target_graph_mod ? BLK_PERM_GRAPH_MOD : 0),
-                        BLK_PERM_WRITE_UNCHANGED |
-                        (target_is_backing ? BLK_PERM_CONSISTENT_READ |
-                                             BLK_PERM_WRITE |
-                                             BLK_PERM_GRAPH_MOD : 0));
+                        target_perms, target_shared_perms);
       ret = blk_insert_bs(s->target, target, errp);
       if (ret < 0) {
           goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job(
       s->backing_mode = backing_mode;
       s->copy_mode = copy_mode;
       s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
       s->granularity = granularity;
       s->buf_size = ROUND_UP(buf_size, granularity);
       s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
       /* In commit_active_start() all intermediate nodes disappear, so
        * any jobs in them must be blocked */
       if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1763

Quoted 5 lines
block/mirror.c +5 -0
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.

I'd prefere to add something like: "because we share it on target (see target BlockBackend creation and corresponding comment above)".

Quoted 4 lines

I’d rather not refer to other comments in case they change… Maybe just “This allows us to share BLK_PERM_CONSISTENT_READ, as we do on the target.”? I think if someone is interested, they will scan the file for what permissions are shared on the target anyway.

OK. Yes I just wanted to stress that we just duplicate behavior about target, as it helped me to understand.

Quoted 93 lines
Quoted 84 lines

blockdev.c

Quoted 32 lines
block/mirror.c +7 -4
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
               ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
               if (ret < 0) {
                   goto fail;
               }
@@ -1737,7 +1802,7 @@ fail:
       bs_opaque->stop = true;
       bdrv_child_refresh_perms(mirror_top_bs, mirror_top_bs->backing,
                                &error_abort);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
       bdrv_unref(mirror_top_bs);
 
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
           return;
       }
       is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
-    base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;
+    base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
       mirror_start_job(job_id, bs, creation_flags, target, replaces,
                        speed, granularity, buf_size, backing_mode,
                        on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +0 -0
index c540802127..c451f553f7 100644

block/mirroc.c is OK for me. Continue with blockdev.c...

Quoted 48 lines
blockdev.c +13 -2
@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
           return;
       }
 
-    if (!bs->backing && sync == MIRROR_SYNC_MODE_TOP) {
+    if (!bdrv_backing_chain_next(bs) && sync == MIRROR_SYNC_MODE_TOP) {
           sync = MIRROR_SYNC_MODE_FULL;
       }
 
@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
 
   void qmp_drive_mirror(DriveMirror *arg, Error **errp)
   {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
       BlockDriverState *source, *target_bs;
       AioContext *aio_context;
       BlockMirrorBackingMode backing_mode;
@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
       int flags;
       int64_t size;
       const char *format = arg->format;
+    const char *replaces_node_name = NULL;
       int ret;
 
       bs = qmp_get_root_bs(arg->device, errp);
@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
           return;
       }
 
+    /*
+     * If the user has not instructed us otherwise, we should let the
+     * block job run from @bs (thus taking into account all filters on
+     * it) but replace @unfiltered_bs when it finishes (thus not
+     * removing those filters).
+     * (And if there are any explicit filters, we should assume the
+     *  user knows how to use the @replaces option.)
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);
+
       aio_context = bdrv_get_aio_context(bs);
       aio_context_acquire(aio_context);
 
@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
       }
 
Quoted 8 lines

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3949

Quoted 4 lines
blockdev.c +1 -1
       flags = bs->open_flags | BDRV_O_RDWR;
-    source = backing_bs(bs);
+    source = bdrv_filtered_cow_bs(unfiltered_bs);
       if (!source && arg->sync == MIRROR_SYNC_MODE_TOP) {

Hmm, you handle this case a bit differently here and in blockdev_mirror_common.. Can we handle it only in blockdev_mirror_common, to be consistent with qmp_blockdev_mirror?

Quoted 6 lines

What exactly do you mean? The difference between skipping all filters and just skipping implicit filters? Hm.

First, the check in blockdev_mirror_common() should actually be unnecessary. In qmp_{blockdev,drive}_mirror(), we do nearly the same check anyway (and then force sync=full if there is no backing file).

Hmm, I see it only in qmp_drive_mirror, not in _blockdev_

Quoted 3 lines

So
if all three functions did the same check, we wouldn’t need it in blockdev_mirror_common().

And if it was so, better have one check in _common than two duplicated.

Quoted 23 lines

Second, let’s look at the difference in an example: One where blockdev_mirror_common() would not decide to enforce mode=full, but qmp_{blockdev,drive}_mirror() would.
This happens when @bs is an explicit filter over some overlay with a backing file, e.g.:

throttle --file--> qcow2 --backing--> raw

It’s correct to run the mirror job from the throttle node; but @source should be bdrv_backing_chain_next() so it will point to the raw node. Currently, it is NULL (because the throttle node does not have a COW child).

But then again, I’ve made qmp_{blockdev,drive}_mirror() throw an error in such a case:

Quoted 6 lines

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3955

Quoted 6 lines
blockdev.c +6 -0
+        if (bdrv_filtered_bs(unfiltered_bs)) {
+            /* @unfiltered_bs is an explicit filter */
+            error_setg(errp, "Cannot perform sync=top mirror through an "
+                       "explicitly added filter node on the source");
+            goto out;
+        }
Quoted 3 lines

So it isn’t really a problem. Still, does the error make sense? Should we just allow that case by letting source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs))?

Looks good. As I understand, you have the test (40/42) for this case and it works for _blockdev_ and for for _drive_ version of command. Of course, they'd better behave in same manner.

Quoted 89 lines

(BTW, I just noticed that @base seems to be pretty much unused in block/mirror.c. It only really uses @base_overlay now. So I suppose it makes sense to remove it in v7.)

Quoted 56 lines
Quoted 56 lines
blockdev.c +10 -4
           arg->sync = MIRROR_SYNC_MODE_FULL;
       }
       if (arg->sync == MIRROR_SYNC_MODE_NONE) {
@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                                " named node of the graph");
               goto out;
           }
+        replaces_node_name = arg->replaces;
+    } else if (unfiltered_bs != bs) {
+        replaces_node_name = unfiltered_bs->node_name;
       }
 
       if (arg->mode == NEW_IMAGE_MODE_ABSOLUTE_PATHS) {
@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
           bdrv_img_create(arg->target, format,
                           NULL, NULL, NULL, size, flags, false, &local_err);
       } else {
+        /* Implicit filters should not appear in the filename */
+        BlockDriverState *explicit_backing = bdrv_skip_implicit_filters(source);
+
           switch (arg->mode) {
           case NEW_IMAGE_MODE_EXISTING:
               break;
@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
               /* create new image with backing file */
               bdrv_refresh_filename(source);
               bdrv_img_create(arg->target, format,
-                            source->filename,
-                            source->drv->format_name,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name,
                               NULL, size, flags, false, &local_err);
               break;
           default:
@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
       }
 
       blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
-                           arg->has_replaces, arg->replaces, arg->sync,
+                           !!replaces_node_name, replaces_node_name, arg->sync,
                              backing_mode, arg->has_speed, arg->speed,
                              arg->has_granularity, arg->granularity,
                              arg->has_buf_size, arg->buf_size,
@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                            bool has_auto_dismiss, bool auto_dismiss,
                            Error **errp)
   {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
       BlockDriverState *target_bs;
       AioContext *aio_context;
       BlockMirrorBackingMode backing_mode = MIRROR_LEAVE_BACKING_CHAIN;
@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
           return;
       }
 
Quoted 4 lines

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4092

Quoted 2 lines
blockdev.c +2 -0
+    /*
+     * Same as in qmp_drive_mirror():

Then, may be better do it in blockdev_mirror_common ?

Hm, maybe. Should we decide to let @source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs)) in qmp_drive_mirror(), I don’t think we need @unfiltered_bs there to determine @source.

Max

Quoted 15 lines
Quoted 13 lines
blockdev.c +8 -0
 We want to run the job from @bs,
+     * but we want to replace @unfiltered_bs on completion.
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);
+    if (!has_replaces && unfiltered_bs != bs) {
+        replaces = unfiltered_bs->node_name;
+        has_replaces = true;
+    }
+
       target_bs = bdrv_lookup_bs(target, target, errp);
       if (!target_bs) {
           return;
Signature · 2 lines
Best regards,
Vladimir
author
#90 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
   block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
   blockdev.c     |  47 +++++++++++++++++---
   2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +9 -5
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
       BlockBackend *target;
       BlockDriverState *mirror_top_bs;
       BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
       /* The name of the graph node to replace */
       char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                                &error_abort);
       if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
           BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
               if (local_err) {
                   error_report_err(local_err);
                   ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
        * valid.
        */
       block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
       /* We just changed the BDS the job BB refers to (with either or both of the
        * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
               return 0;
           }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
           if (ret < 0) {
               return ret;
           }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
       } else {
           s->target_cluster_size = BDRV_SECTOR_SIZE;
       }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
           s->granularity < s->target_cluster_size) {
           s->buf_size = MAX(s->buf_size, s->target_cluster_size);
           s->cow_bitmap = bitmap_new(length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
       if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
           int ret;
 
Quoted 7 lines

block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095

Quoted 3 lines
block/mirror.c +1 -2
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));

Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.

The idea is that MIRROR_OPEN_BACKING_CHAIN is set only when using drive-mirror with mode=existing. In this case, we also set BDRV_O_NO_BACKING for the target.

You’re right that a user could add a backing chain to the target during the operation. They really have to make an effort to shoot themselves in the foot for this because the target must have an auto-generated node name.

Quoted 4 lines
Quoted 2 lines

I suppose the best would be not to open the backing chain if the target node already has a backing child?

Hmm, but we still should generate an error, as we can't do what was requested.

But the user didn’t request anything. They just specified an existing file as the target with mode=existing, then (for whatever reason) made a real effort to add a backing node to it (i.e. they had to look up the target’s auto-generated node name), and then the job finishes.

The user didn’t request us to open the backing chain of the target. They just requested to mirror to an existing file. If they manually override that existing file’s backing chain, I think it makes sense to keep it that way.

Max

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
block/mirror.c +2 -0
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
           if (ret < 0) {
               return;
           }
#91 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 130 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
 block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
 blockdev.c     |  47 +++++++++++++++++---
 2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +44 -10
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
     BlockBackend *target;
     BlockDriverState *mirror_top_bs;
     BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
     /* The name of the graph node to replace */
     char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                              &error_abort);
     if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
         BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
             if (local_err) {
                 error_report_err(local_err);
                 ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
      * valid.
      */
     block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
     /* We just changed the BDS the job BB refers to (with either or both of the
      * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
             return 0;
         }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
         if (ret < 0) {
             return ret;
         }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
     } else {
         s->target_cluster_size = BDRV_SECTOR_SIZE;
     }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
         s->granularity < s->target_cluster_size) {
         s->buf_size = MAX(s->buf_size, s->target_cluster_size);
         s->cow_bitmap = bitmap_new(length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
     if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
         int ret;
 
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
         if (ret < 0) {
             return;
         }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(
     MirrorBlockJob *s;
     MirrorBDSOpaque *bs_opaque;
     BlockDriverState *mirror_top_bs;
-    bool target_graph_mod;
     bool target_is_backing;
+    uint64_t target_perms, target_shared_perms;
     Error *local_err = NULL;
     int ret;
 
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job(
         buf_size = DEFAULT_MIRROR_BUF_SIZE;
     }
 
-    if (bs == target) {
+    if (bdrv_skip_rw_filters(bs) == bdrv_skip_rw_filters(target)) {
         error_setg(errp, "Can't mirror node into itself");
         return NULL;
     }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(
      * In the case of active commit, things look a bit different, though,
      * because the target is an already populated backing file in active use.
      * We can allow anything except resize there.*/
+
+    target_perms = BLK_PERM_WRITE;
+    target_shared_perms = BLK_PERM_WRITE_UNCHANGED;
+
     target_is_backing = bdrv_chain_contains(bs, target);
-    target_graph_mod = (backing_mode != MIRROR_LEAVE_BACKING_CHAIN);
+    if (target_is_backing) {
+        int64_t bs_size, target_size;
+        bs_size = bdrv_getlength(bs);
+        if (bs_size < 0) {
+            error_setg_errno(errp, -bs_size,
+                             "Could not inquire top image size");
+            goto fail;
+        }
+
+        target_size = bdrv_getlength(target);
+        if (target_size < 0) {
+            error_setg_errno(errp, -target_size,
+                             "Could not inquire base image size");
+            goto fail;
+        }
+
+        if (target_size < bs_size) {
+            target_perms |= BLK_PERM_RESIZE;
+        }
+
+        target_shared_perms |= BLK_PERM_CONSISTENT_READ
+                            |  BLK_PERM_WRITE
+                            |  BLK_PERM_GRAPH_MOD;
+    } else if (bdrv_chain_contains(bs, bdrv_skip_rw_filters(target))) {
+        /*
+         * We may want to allow this in the future, but it would

block/mirror.c static BlockJob *mirror_start_job( · line 1663

Quoted 10 lines
block/mirror.c +10 -0
+         * require taking some extra care.
+         */
+        error_setg(errp, "Cannot mirror to a filter on top of a node in the "
+                   "source's backing chain");
+        goto fail;
+    }
+
+    if (backing_mode != MIRROR_LEAVE_BACKING_CHAIN) {
+        target_perms |= BLK_PERM_GRAPH_MOD;
+    }

This is getting absurd. We keep moving GRAPH_MOD around, but still nobody knows what it's actually supposed to mean. Maybe it would be better to just remove it finally?

Of course, not a reason to stop this patch, after all it's moving the nonsensical piece of code correctly...

Quoted 118 lines
block/mirror.c +36 -16
     s->target = blk_new(s->common.job.aio_context,
-                        BLK_PERM_WRITE | BLK_PERM_RESIZE |
-                        (target_graph_mod ? BLK_PERM_GRAPH_MOD : 0),
-                        BLK_PERM_WRITE_UNCHANGED |
-                        (target_is_backing ? BLK_PERM_CONSISTENT_READ |
-                                             BLK_PERM_WRITE |
-                                             BLK_PERM_GRAPH_MOD : 0));
+                        target_perms, target_shared_perms);
     ret = blk_insert_bs(s->target, target, errp);
     if (ret < 0) {
         goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job(
     s->backing_mode = backing_mode;
     s->copy_mode = copy_mode;
     s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
     s->granularity = granularity;
     s->buf_size = ROUND_UP(buf_size, granularity);
     s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
     /* In commit_active_start() all intermediate nodes disappear, so
      * any jobs in them must be blocked */
     if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
             ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
             if (ret < 0) {
                 goto fail;
             }
@@ -1737,7 +1802,7 @@ fail:
     bs_opaque->stop = true;
     bdrv_child_refresh_perms(mirror_top_bs, mirror_top_bs->backing,
                              &error_abort);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
     bdrv_unref(mirror_top_bs);
 
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
         return;
     }
     is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
-    base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;
+    base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
     mirror_start_job(job_id, bs, creation_flags, target, replaces,
                      speed, granularity, buf_size, backing_mode,
                      on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +3 -2
index c540802127..c451f553f7 100644
@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
         return;
     }
 
-    if (!bs->backing && sync == MIRROR_SYNC_MODE_TOP) {
+    if (!bdrv_backing_chain_next(bs) && sync == MIRROR_SYNC_MODE_TOP) {
         sync = MIRROR_SYNC_MODE_FULL;
     }
 
@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
 
 void qmp_drive_mirror(DriveMirror *arg, Error **errp)
 {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
     BlockDriverState *source, *target_bs;
     AioContext *aio_context;
     BlockMirrorBackingMode backing_mode;
@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     int flags;
     int64_t size;
     const char *format = arg->format;
+    const char *replaces_node_name = NULL;
     int ret;
 
     bs = qmp_get_root_bs(arg->device, errp);
@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
         return;
     }

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3933

Quoted 10 lines
blockdev.c +9 -0
 
+    /*
+     * If the user has not instructed us otherwise, we should let the
+     * block job run from @bs (thus taking into account all filters on
+     * it) but replace @unfiltered_bs when it finishes (thus not
+     * removing those filters).
+     * (And if there are any explicit filters, we should assume the
+     *  user knows how to use the @replaces option.)
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);

Should this behaviour be documented in the QAPI schema for drive-mirror?

Quoted 69 lines
blockdev.c +17 -5
     aio_context = bdrv_get_aio_context(bs);
     aio_context_acquire(aio_context);
 
@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     }
 
     flags = bs->open_flags | BDRV_O_RDWR;
-    source = backing_bs(bs);
+    source = bdrv_filtered_cow_bs(unfiltered_bs);
     if (!source && arg->sync == MIRROR_SYNC_MODE_TOP) {
+        if (bdrv_filtered_bs(unfiltered_bs)) {
+            /* @unfiltered_bs is an explicit filter */
+            error_setg(errp, "Cannot perform sync=top mirror through an "
+                       "explicitly added filter node on the source");
+            goto out;
+        }
         arg->sync = MIRROR_SYNC_MODE_FULL;
     }
     if (arg->sync == MIRROR_SYNC_MODE_NONE) {
@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                              " named node of the graph");
             goto out;
         }
+        replaces_node_name = arg->replaces;
+    } else if (unfiltered_bs != bs) {
+        replaces_node_name = unfiltered_bs->node_name;
     }
 
     if (arg->mode == NEW_IMAGE_MODE_ABSOLUTE_PATHS) {
@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
         bdrv_img_create(arg->target, format,
                         NULL, NULL, NULL, size, flags, false, &local_err);
     } else {
+        /* Implicit filters should not appear in the filename */
+        BlockDriverState *explicit_backing = bdrv_skip_implicit_filters(source);
+
         switch (arg->mode) {
         case NEW_IMAGE_MODE_EXISTING:
             break;
@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
             /* create new image with backing file */
             bdrv_refresh_filename(source);
             bdrv_img_create(arg->target, format,
-                            source->filename,
-                            source->drv->format_name,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name,
                             NULL, size, flags, false, &local_err);
             break;
         default:
@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     }
 
     blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
-                           arg->has_replaces, arg->replaces, arg->sync,
+                           !!replaces_node_name, replaces_node_name, arg->sync,
                            backing_mode, arg->has_speed, arg->speed,
                            arg->has_granularity, arg->granularity,
                            arg->has_buf_size, arg->buf_size,
@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                          bool has_auto_dismiss, bool auto_dismiss,
                          Error **errp)
 {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
     BlockDriverState *target_bs;
     AioContext *aio_context;
     BlockMirrorBackingMode backing_mode = MIRROR_LEAVE_BACKING_CHAIN;

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4095

Quoted 8 lines
blockdev.c +5 -0
@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
         return;
     }
 
+    /*
+     * Same as in qmp_drive_mirror(): We want to run the job from @bs,
+     * but we want to replace @unfiltered_bs on completion.
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);

Do we? I thought the idea with blockdev-mirror was that the client tells us the exact node it is interested in, without any magic skipping nodes.

Skipping implicit nodes is a feature for compatibility with legacy clients, but a client using blockdev-mirror isn't a legacy client.

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4103

Quoted 8 lines
blockdev.c +5 -0
+    if (!has_replaces && unfiltered_bs != bs) {
+        replaces = unfiltered_bs->node_name;
+        has_replaces = true;
+    }
+
     target_bs = bdrv_lookup_bs(target, target, errp);
     if (!target_bs) {
         return;

Kevin

author
#92 lore raw

Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters

Quoted 137 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 136 lines

This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).

---
 block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++-----------
 blockdev.c     |  47 +++++++++++++++++---
 2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +50 -10
index 54bafdf176..6ddbfb9708 100644
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {
     BlockBackend *target;
     BlockDriverState *mirror_top_bs;
     BlockDriverState *base;
+    BlockDriverState *base_overlay;
 
     /* The name of the graph node to replace */
     char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job)
                              &error_abort);
     if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) {
         BlockDriverState *backing = s->is_none_mode ? src : s->base;
-        if (backing_bs(target_bs) != backing) {
-            bdrv_set_backing_hd(target_bs, backing, &local_err);
+        BlockDriverState *unfiltered_target = bdrv_skip_rw_filters(target_bs);
+
+        if (bdrv_filtered_cow_bs(unfiltered_target) != backing) {
+            bdrv_set_backing_hd(unfiltered_target, backing, &local_err);
             if (local_err) {
                 error_report_err(local_err);
                 ret = -EPERM;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)
      * valid.
      */
     block_job_remove_all_bdrv(bjob);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
     /* We just changed the BDS the job BB refers to (with either or both of the
      * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)
             return 0;
         }
 
-        ret = bdrv_is_allocated_above(bs, base, false, offset, bytes, &count);
+        ret = bdrv_is_allocated_above(bs, s->base_overlay, true, offset, bytes,
+                                      &count);
         if (ret < 0) {
             return ret;
         }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
     } else {
         s->target_cluster_size = BDRV_SECTOR_SIZE;
     }
-    if (backing_filename[0] && !target_bs->backing &&
+    if (backing_filename[0] && !bdrv_backing_chain_next(target_bs) &&
         s->granularity < s->target_cluster_size) {
         s->buf_size = MAX(s->buf_size, s->target_cluster_size);
         s->cow_bitmap = bitmap_new(length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)
     if (s->backing_mode == MIRROR_OPEN_BACKING_CHAIN) {
         int ret;
 
-        assert(!target->backing);
-        ret = bdrv_open_backing_file(target, NULL, "backing", errp);
+        assert(!bdrv_backing_chain_next(target));
+        ret = bdrv_open_backing_file(bdrv_skip_rw_filters(target), NULL,
+                                     "backing", errp);
         if (ret < 0) {
             return;
         }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(
     MirrorBlockJob *s;
     MirrorBDSOpaque *bs_opaque;
     BlockDriverState *mirror_top_bs;
-    bool target_graph_mod;
     bool target_is_backing;
+    uint64_t target_perms, target_shared_perms;
     Error *local_err = NULL;
     int ret;
 
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job(
         buf_size = DEFAULT_MIRROR_BUF_SIZE;
     }
 
-    if (bs == target) {
+    if (bdrv_skip_rw_filters(bs) == bdrv_skip_rw_filters(target)) {
         error_setg(errp, "Can't mirror node into itself");
         return NULL;
     }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(
      * In the case of active commit, things look a bit different, though,
      * because the target is an already populated backing file in active use.
      * We can allow anything except resize there.*/
+
+    target_perms = BLK_PERM_WRITE;
+    target_shared_perms = BLK_PERM_WRITE_UNCHANGED;
+
     target_is_backing = bdrv_chain_contains(bs, target);
-    target_graph_mod = (backing_mode != MIRROR_LEAVE_BACKING_CHAIN);
+    if (target_is_backing) {
+        int64_t bs_size, target_size;
+        bs_size = bdrv_getlength(bs);
+        if (bs_size < 0) {
+            error_setg_errno(errp, -bs_size,
+                             "Could not inquire top image size");
+            goto fail;
+        }
+
+        target_size = bdrv_getlength(target);
+        if (target_size < 0) {
+            error_setg_errno(errp, -target_size,
+                             "Could not inquire base image size");
+            goto fail;
+        }
+
+        if (target_size < bs_size) {
+            target_perms |= BLK_PERM_RESIZE;
+        }
+
+        target_shared_perms |= BLK_PERM_CONSISTENT_READ
+                            |  BLK_PERM_WRITE
+                            |  BLK_PERM_GRAPH_MOD;
+    } else if (bdrv_chain_contains(bs, bdrv_skip_rw_filters(target))) {
+        /*
+         * We may want to allow this in the future, but it would
+         * require taking some extra care.
+         */
+        error_setg(errp, "Cannot mirror to a filter on top of a node in the "
+                   "source's backing chain");
+        goto fail;
+    }
Quoted 8 lines

block/mirror.c static BlockJob *mirror_start_job( · line 1663

Quoted 4 lines
block/mirror.c +4 -0
+
+    if (backing_mode != MIRROR_LEAVE_BACKING_CHAIN) {
+        target_perms |= BLK_PERM_GRAPH_MOD;
+    }

This is getting absurd. We keep moving GRAPH_MOD around, but still nobody knows what it's actually supposed to mean. Maybe it would be better to just remove it finally?

I suppose even if we ever needed it, we no longer do now with .freeze.

Quoted 122 lines

Of course, not a reason to stop this patch, after all it's moving the nonsensical piece of code correctly...

Quoted 119 lines
block/mirror.c +36 -16
     s->target = blk_new(s->common.job.aio_context,
-                        BLK_PERM_WRITE | BLK_PERM_RESIZE |
-                        (target_graph_mod ? BLK_PERM_GRAPH_MOD : 0),
-                        BLK_PERM_WRITE_UNCHANGED |
-                        (target_is_backing ? BLK_PERM_CONSISTENT_READ |
-                                             BLK_PERM_WRITE |
-                                             BLK_PERM_GRAPH_MOD : 0));
+                        target_perms, target_shared_perms);
     ret = blk_insert_bs(s->target, target, errp);
     if (ret < 0) {
         goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job(
     s->backing_mode = backing_mode;
     s->copy_mode = copy_mode;
     s->base = base;
+    s->base_overlay = bdrv_find_overlay(bs, base);
     s->granularity = granularity;
     s->buf_size = ROUND_UP(buf_size, granularity);
     s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(
     /* In commit_active_start() all intermediate nodes disappear, so
      * any jobs in them must be blocked */
     if (target_is_backing) {
-        BlockDriverState *iter;
-        for (iter = backing_bs(bs); iter != target; iter = backing_bs(iter)) {
-            /* XXX BLK_PERM_WRITE needs to be allowed so we don't block
-             * ourselves at s->base (if writes are blocked for a node, they are
-             * also blocked for its backing file). The other options would be a
-             * second filter driver above s->base (== target). */
+        BlockDriverState *iter, *filtered_target;
+        uint64_t iter_shared_perms;
+
+        /*
+         * The topmost node with
+         * bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target)
+         */
+        filtered_target = bdrv_filtered_cow_bs(bdrv_find_overlay(bs, target));
+
+        assert(bdrv_skip_rw_filters(filtered_target) ==
+               bdrv_skip_rw_filters(target));
+
+        /*
+         * XXX BLK_PERM_WRITE needs to be allowed so we don't block
+         * ourselves at s->base (if writes are blocked for a node, they are
+         * also blocked for its backing file). The other options would be a
+         * second filter driver above s->base (== target).
+         */
+        iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+        for (iter = bdrv_filtered_bs(bs); iter != target;
+             iter = bdrv_filtered_bs(iter))
+        {
+            if (iter == filtered_target) {
+                /*
+                 * From here on, all nodes are filters on the base.
+                 * This allows us to share BLK_PERM_CONSISTENT_READ.
+                 */
+                iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+            }
+
             ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                     BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                     errp);
+                                     iter_shared_perms, errp);
             if (ret < 0) {
                 goto fail;
             }
@@ -1737,7 +1802,7 @@ fail:
     bs_opaque->stop = true;
     bdrv_child_refresh_perms(mirror_top_bs, mirror_top_bs->backing,
                              &error_abort);
-    bdrv_replace_node(mirror_top_bs, backing_bs(mirror_top_bs), &error_abort);
+    bdrv_replace_node(mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);
 
     bdrv_unref(mirror_top_bs);
 
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
         return;
     }
     is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
-    base = mode == MIRROR_SYNC_MODE_TOP ? backing_bs(bs) : NULL;
+    base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
     mirror_start_job(job_id, bs, creation_flags, target, replaces,
                      speed, granularity, buf_size, backing_mode,
                      on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +3 -2
index c540802127..c451f553f7 100644
@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
         return;
     }
 
-    if (!bs->backing && sync == MIRROR_SYNC_MODE_TOP) {
+    if (!bdrv_backing_chain_next(bs) && sync == MIRROR_SYNC_MODE_TOP) {
         sync = MIRROR_SYNC_MODE_FULL;
     }
 
@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
 
 void qmp_drive_mirror(DriveMirror *arg, Error **errp)
 {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
     BlockDriverState *source, *target_bs;
     AioContext *aio_context;
     BlockMirrorBackingMode backing_mode;
@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     int flags;
     int64_t size;
     const char *format = arg->format;
+    const char *replaces_node_name = NULL;
     int ret;
 
     bs = qmp_get_root_bs(arg->device, errp);
@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
         return;
     }
 
Quoted 11 lines

blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3933

Quoted 9 lines
blockdev.c +9 -0
+    /*
+     * If the user has not instructed us otherwise, we should let the
+     * block job run from @bs (thus taking into account all filters on
+     * it) but replace @unfiltered_bs when it finishes (thus not
+     * removing those filters).
+     * (And if there are any explicit filters, we should assume the
+     *  user knows how to use the @replaces option.)
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);

Should this behaviour be documented in the QAPI schema for drive-mirror?

Hm. I’d document it for @replaces. But what would I write? “By default, @device is replaced, though implicitly created nodes on it are kept”?

I feel bad about referencing implicit nodes, especially if our plan is to remove them anyway. OTOH, if I dropped special handling of implicit nodes here, I should probably drop it in the whole series. And I don’t feel like that’s right as long as we haven’t actually removed implicit nodes.

So I suppose I’ll go for the documentation addendum? :-/

Quoted 81 lines
Quoted 69 lines
blockdev.c +17 -5
     aio_context = bdrv_get_aio_context(bs);
     aio_context_acquire(aio_context);
 
@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     }
 
     flags = bs->open_flags | BDRV_O_RDWR;
-    source = backing_bs(bs);
+    source = bdrv_filtered_cow_bs(unfiltered_bs);
     if (!source && arg->sync == MIRROR_SYNC_MODE_TOP) {
+        if (bdrv_filtered_bs(unfiltered_bs)) {
+            /* @unfiltered_bs is an explicit filter */
+            error_setg(errp, "Cannot perform sync=top mirror through an "
+                       "explicitly added filter node on the source");
+            goto out;
+        }
         arg->sync = MIRROR_SYNC_MODE_FULL;
     }
     if (arg->sync == MIRROR_SYNC_MODE_NONE) {
@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
                              " named node of the graph");
             goto out;
         }
+        replaces_node_name = arg->replaces;
+    } else if (unfiltered_bs != bs) {
+        replaces_node_name = unfiltered_bs->node_name;
     }
 
     if (arg->mode == NEW_IMAGE_MODE_ABSOLUTE_PATHS) {
@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
         bdrv_img_create(arg->target, format,
                         NULL, NULL, NULL, size, flags, false, &local_err);
     } else {
+        /* Implicit filters should not appear in the filename */
+        BlockDriverState *explicit_backing = bdrv_skip_implicit_filters(source);
+
         switch (arg->mode) {
         case NEW_IMAGE_MODE_EXISTING:
             break;
@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
             /* create new image with backing file */
             bdrv_refresh_filename(source);
             bdrv_img_create(arg->target, format,
-                            source->filename,
-                            source->drv->format_name,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name,
                             NULL, size, flags, false, &local_err);
             break;
         default:
@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)
     }
 
     blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs,
-                           arg->has_replaces, arg->replaces, arg->sync,
+                           !!replaces_node_name, replaces_node_name, arg->sync,
                            backing_mode, arg->has_speed, arg->speed,
                            arg->has_granularity, arg->granularity,
                            arg->has_buf_size, arg->buf_size,
@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
                          bool has_auto_dismiss, bool auto_dismiss,
                          Error **errp)
 {
-    BlockDriverState *bs;
+    BlockDriverState *bs, *unfiltered_bs;
     BlockDriverState *target_bs;
     AioContext *aio_context;
     BlockMirrorBackingMode backing_mode = MIRROR_LEAVE_BACKING_CHAIN;

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4095

Quoted 8 lines
blockdev.c +5 -0
@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,
         return;
     }
 
+    /*
+     * Same as in qmp_drive_mirror(): We want to run the job from @bs,
+     * but we want to replace @unfiltered_bs on completion.
+     */
+    unfiltered_bs = bdrv_skip_implicit_filters(bs);

Do we? I thought the idea with blockdev-mirror was that the client tells us the exact node it is interested in, without any magic skipping nodes.

Quoted 2 lines

Skipping implicit nodes is a feature for compatibility with legacy clients, but a client using blockdev-mirror isn't a legacy client.

And I thought legacy clients didn’t let implicit nodes be created.

We could return an error if @bs is an implicit node. But I don’t think that would help anyone, and it’d be more complicated in my opinion.

Again, the best thing (IMO) is to remove the concept of implicit nodes altogether and then we can drop this piece of code anyway.

Max

Quoted 11 lines

blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4103

Quoted 8 lines
blockdev.c +5 -0
+    if (!has_replaces && unfiltered_bs != bs) {
+        replaces = unfiltered_bs->node_name;
+        has_replaces = true;
+    }
+
     target_bs = bdrv_lookup_bs(target, target, errp);
     if (!target_bs) {
         return;

Kevin

author
#93 lore raw

PATCH v6 26/42 backup: Deal with filters

In reply to Max Reitz
---
 block/backup.c |  9 +++++----
 blockdev.c     | 19 +++++++++++++++----
 2 files changed, 20 insertions(+), 8 deletions(-)
block/backup.c +5 -4
index ecadb61af3..7854d7575b 100644
@@ -611,6 +611,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
 {
     int ret;
     BlockDriverInfo bdi;
+    bool target_does_cow = bdrv_backing_chain_next(target);
 
     /*
      * If there is no backing file on the target, we cannot rely on COW if our
@@ -618,7 +619,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
      * targets with a backing file, try to avoid COW if possible.
      */
     ret = bdrv_get_info(target, &bdi);
-    if (ret == -ENOTSUP && !target->backing) {
+    if (ret == -ENOTSUP && !target_does_cow) {
         /* Cluster size is not defined */
         warn_report("The target block device doesn't provide "
                     "information about the block size and it doesn't have a "
@@ -627,14 +628,14 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,
                     "this default, the backup may be unusable",
                     BACKUP_CLUSTER_SIZE_DEFAULT);
         return BACKUP_CLUSTER_SIZE_DEFAULT;
-    } else if (ret < 0 && !target->backing) {
+    } else if (ret < 0 && !target_does_cow) {
         error_setg_errno(errp, -ret,
             "Couldn't determine the cluster size of the target image, "
             "which has no backing file");
         error_append_hint(errp,
             "Aborting, since this may create an unusable destination image\n");
         return ret;
-    } else if (ret < 0 && target->backing) {
+    } else if (ret < 0 && target_does_cow) {
         /* Not fatal; just trudge on ahead. */
         return BACKUP_CLUSTER_SIZE_DEFAULT;
     }
@@ -683,7 +684,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,
         return NULL;
     }
 
-    if (compress && target->drv->bdrv_co_pwritev_compressed == NULL) {
+    if (compress && !bdrv_supports_compressed_writes(target)) {
         error_setg(errp, "Compression is not supported for this drive %s",
                    bdrv_get_device_name(target));
         return NULL;
blockdev.c +15 -4
index c451f553f7..c6f79b4e0e 100644
@@ -3656,7 +3656,13 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
     /* See if we have a backing HD we can use to create our new image
      * on top of. */
     if (backup->sync == MIRROR_SYNC_MODE_TOP) {
-        source = backing_bs(bs);
+        /*
+         * Backup will not replace the source by the target, so none
+         * of the filters skipped here will be removed (in contrast to
+         * mirror).  Therefore, we can skip all of them when looking
+         * for the first COW relationship.
+         */
+        source = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs));
         if (!source) {
             backup->sync = MIRROR_SYNC_MODE_FULL;
         }
@@ -3676,9 +3682,14 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,
     if (backup->mode != NEW_IMAGE_MODE_EXISTING) {
         assert(backup->format);
         if (source) {
-            bdrv_refresh_filename(source);
-            bdrv_img_create(backup->target, backup->format, source->filename,
-                            source->drv->format_name, NULL,
+            /* Implicit filters should not appear in the filename */
+            BlockDriverState *explicit_backing =
+                bdrv_skip_implicit_filters(source);
+
+            bdrv_refresh_filename(explicit_backing);
+            bdrv_img_create(backup->target, backup->format,
+                            explicit_backing->filename,
+                            explicit_backing->drv->format_name, NULL,
                             size, flags, false, &local_err);
         } else {
             bdrv_img_create(backup->target, backup->format, NULL, NULL, NULL,
Signature · 1 line
2.21.0
author
#94 lore raw

PATCH v6 27/42 commit: Deal with filters

In reply to Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
 block/block-backend.c | 16 +++++---
 block/commit.c        | 96 +++++++++++++++++++++++++++++++------------
 blockdev.c            |  6 ++-
 3 files changed, 85 insertions(+), 33 deletions(-)
block/block-backend.c +11 -5
index c13c5c83b0..0bc592d023 100644
@@ -2180,11 +2180,17 @@ int blk_commit_all(void)
         AioContext *aio_context = blk_get_aio_context(blk);
 
         aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);
-            if (ret < 0) {
-                aio_context_release(aio_context);
-                return ret;
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            if (bdrv_filtered_cow_child(non_filter)) {
+                int ret = bdrv_commit(non_filter);
+                if (ret < 0) {
+                    aio_context_release(aio_context);
+                    return ret;
+                }
             }
         }
         aio_context_release(aio_context);
block/commit.c +70 -26

173 lines not on this page · raw message

blockdev.c +4 -2
index c6f79b4e0e..7bef41c0b0 100644
@@ -1094,7 +1094,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)
             return;
         }
 
-        bs = blk_bs(blk);
+        bs = bdrv_skip_implicit_filters(blk_bs(blk));
         aio_context = bdrv_get_aio_context(bs);
         aio_context_acquire(aio_context);
 
@@ -3454,7 +3454,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
 
     assert(bdrv_get_aio_context(base_bs) == aio_context);
 
-    for (iter = top_bs; iter != backing_bs(base_bs); iter = backing_bs(iter)) {
+    for (iter = top_bs; iter != bdrv_filtered_bs(base_bs);
+         iter = bdrv_filtered_bs(iter))
+    {
         if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_COMMIT_TARGET, errp)) {
             goto out;
         }
Signature · 1 line
2.21.0
Reviewed-by on v6 27/42
#95 lore raw

Re: [Qemu-devel] [PATCH v6 27/42] commit: Deal with filters

Quoted 25 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
  block/block-backend.c | 16 +++++---
  block/commit.c        | 96 +++++++++++++++++++++++++++++++------------
  blockdev.c            |  6 ++-
  3 files changed, 85 insertions(+), 33 deletions(-)
block/block-backend.c +2 -5
index c13c5c83b0..0bc592d023 100644
@@ -2180,11 +2180,17 @@ int blk_commit_all(void)
          AioContext *aio_context = blk_get_aio_context(blk);
 
          aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);
-            if (ret < 0) {
-                aio_context_release(aio_context);
-                return ret;
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;

block/block-backend.c int blk_commit_all(void) · line 2194

Quoted 10 lines
block/block-backend.c +9 -0
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            if (bdrv_filtered_cow_child(non_filter)) {
+                int ret = bdrv_commit(non_filter);
+                if (ret < 0) {
+                    aio_context_release(aio_context);
+                    return ret;
+                }
              }

and if non_filter is explicit filter we just skip it. I think we'd better return error in this case. For example, just drop if (bdrv_filtered_cow_child) and get ENOTSUP from bdrv_commit in this case.

And with at least this fixed I'm OK with this patch:

However some comments below:

block/commit.c typedef struct CommitBlockJob { · line 40

Quoted 11 lines
block/block-backend.c +0 -0
          }
          aio_context_release(aio_context);
block/commit.c +1 -0
index 5a7672c7c7..40d1c8eeac 100644
@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {
      BlockBackend *top;
      BlockBackend *base;
      BlockDriverState *base_bs;
+    BlockDriverState *above_base;

you called it base_overlay in mirror, seems better to keep same naming

Quoted 60 lines
block/commit.c +23 -3
      BlockdevOnError on_error;
      bool base_read_only;
      bool chain_frozen;
@@ -110,7 +111,7 @@ static void commit_abort(Job *job)
       * XXX Can (or should) we somehow keep 'consistent read' blocked even
       * after the failed/cancelled commit job is gone? If we already wrote
       * something to base, the intermediate images aren't valid any more. */
-    bdrv_replace_node(s->commit_top_bs, backing_bs(s->commit_top_bs),
+    bdrv_replace_node(s->commit_top_bs, s->commit_top_bs->backing->bs,
                        &error_abort);
 
      bdrv_unref(s->commit_top_bs);
@@ -174,7 +175,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)
              break;
          }
          /* Copy if allocated above the base */
-        ret = bdrv_is_allocated_above(blk_bs(s->top), blk_bs(s->base), false,
+        ret = bdrv_is_allocated_above(blk_bs(s->top), s->above_base, true,
                                        offset, COMMIT_BUFFER_SIZE, &n);
          copy = (ret == 1);
          trace_commit_one_iteration(s, offset, n, ret);
@@ -267,15 +268,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,
      CommitBlockJob *s;
      BlockDriverState *iter;
      BlockDriverState *commit_top_bs = NULL;
+    BlockDriverState *filtered_base;
      Error *local_err = NULL;
+    int64_t base_size, top_size;
+    uint64_t perms, iter_shared_perms;
      int ret;
 
      assert(top != bs);
-    if (top == base) {
+    if (bdrv_skip_rw_filters(top) == bdrv_skip_rw_filters(base)) {
          error_setg(errp, "Invalid files for merge: top and base are the same");
          return;
      }
 
+    base_size = bdrv_getlength(base);
+    if (base_size < 0) {
+        error_setg_errno(errp, -base_size, "Could not inquire base image size");
+        return;
+    }
+
+    top_size = bdrv_getlength(top);
+    if (top_size < 0) {
+        error_setg_errno(errp, -top_size, "Could not inquire top image size");
+        return;
+    }
+
+    perms = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE;
+    if (base_size < top_size) {
+        perms |= BLK_PERM_RESIZE;
+    }
+
      s = block_job_create(job_id, &commit_job_driver, NULL, bs, 0, BLK_PERM_ALL,
                           speed, creation_flags, NULL, NULL, errp);
      if (!s) {
@@ -315,17 +336,43 @@ void commit_start(const char *job_id, BlockDriverState *bs,
 

block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 325

Quoted 10 lines
block/commit.c +0 -8
      s->commit_top_bs = commit_top_bs;
 
-    /* Block all nodes between top and base, because they will
-     * disappear from the chain after this operation. */
-    assert(bdrv_chain_contains(top, base));
-    for (iter = top; iter != base; iter = backing_bs(iter)) {
-        /* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
-         * at s->base (if writes are blocked for a node, they are also blocked
-         * for its backing file). The other options would be a second filter
-         * driver above s->base. */

This code part is absolutely equal to corresponding in block/mirror.c.. It would be great to put it into a function and reuse. However its not about these series.

Quoted 54 lines
block/commit.c +37 -5
+    /*
+     * Block all nodes between top and base, because they will
+     * disappear from the chain after this operation.
+     * Note that this assumes that the user is fine with removing all
+     * nodes (including R/W filters) between top and base.  Assuring
+     * this is the responsibility of the interface (i.e. whoever calls
+     * commit_start()).
+     */
+    s->above_base = bdrv_find_overlay(top, base);
+    assert(s->above_base);
+
+    /*
+     * The topmost node with
+     * bdrv_skip_rw_filters(filtered_base) == bdrv_skip_rw_filters(base)
+     */
+    filtered_base = bdrv_filtered_cow_bs(s->above_base);
+    assert(bdrv_skip_rw_filters(filtered_base) == bdrv_skip_rw_filters(base));
+
+    /*
+     * XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
+     * at s->base (if writes are blocked for a node, they are also blocked
+     * for its backing file). The other options would be a second filter
+     * driver above s->base.
+     */
+    iter_shared_perms = BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE;
+
+    for (iter = top; iter != base; iter = bdrv_filtered_bs(iter)) {
+        if (iter == filtered_base) {
+            /*
+             * From here on, all nodes are filters on the base.  This
+             * allows us to share BLK_PERM_CONSISTENT_READ.
+             */
+            iter_shared_perms |= BLK_PERM_CONSISTENT_READ;
+        }
+
          ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-                                 BLK_PERM_WRITE_UNCHANGED | BLK_PERM_WRITE,
-                                 errp);
+                                 iter_shared_perms, errp);
          if (ret < 0) {
              goto fail;
          }
@@ -342,9 +389,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
      }
 
      s->base = blk_new(s->common.job.aio_context,
-                      BLK_PERM_CONSISTENT_READ
-                      | BLK_PERM_WRITE
-                      | BLK_PERM_RESIZE,
+                      perms,
                        BLK_PERM_CONSISTENT_READ
                        | BLK_PERM_GRAPH_MOD
                        | BLK_PERM_WRITE_UNCHANGED);

block/commit.c int bdrv_commit(BlockDriverState *bs) · line 460

Quoted 5 lines
block/commit.c +1 -1
@@ -412,19 +457,22 @@ int bdrv_commit(BlockDriverState *bs)
      if (!drv)
          return -ENOMEDIUM;
 
-    if (!bs->backing) {
+    backing_file_bs = bdrv_filtered_cow_bs(bs);

Hmm just note: if in future we'll have cow child which is not bs->backing, a lot of code will fail, as we always assume that cow child is bs->backing. May be, this should be commented in bdrv_filtered_cow_child implementation.

Quoted 31 lines
block/commit.c +6 -5
+
+    if (!backing_file_bs) {
          return -ENOTSUP;
      }
 
      if (bdrv_op_is_blocked(bs, BLOCK_OP_TYPE_COMMIT_SOURCE, NULL) ||
-        bdrv_op_is_blocked(bs->backing->bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL)) {
+        bdrv_op_is_blocked(backing_file_bs, BLOCK_OP_TYPE_COMMIT_TARGET, NULL))
+    {
          return -EBUSY;
      }
 
-    ro = bs->backing->bs->read_only;
+    ro = backing_file_bs->read_only;
 
      if (ro) {
-        if (bdrv_reopen_set_read_only(bs->backing->bs, false, NULL)) {
+        if (bdrv_reopen_set_read_only(backing_file_bs, false, NULL)) {
              return -EACCES;
          }
      }
@@ -440,8 +488,6 @@ int bdrv_commit(BlockDriverState *bs)
      }
 
      /* Insert commit_top block node above backing, so we can write to it */
-    backing_file_bs = backing_bs(bs);
-
      commit_top_bs = bdrv_new_open_driver(&bdrv_commit_top, NULL, BDRV_O_RDWR,
                                           &local_err);
      if (commit_top_bs == NULL) {

block/commit.c ro_cleanup: · line 575

Quoted 7 lines
block/commit.c +1 -3
@@ -526,15 +572,13 @@ ro_cleanup:
      qemu_vfree(buf);
 
      blk_unref(backing);
-    if (backing_file_bs) {
-        bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);
-    }
+    bdrv_set_backing_hd(bs, backing_file_bs, &error_abort);

Preexisting, but we should not drop filter if we didn't added it (if we failed above filter insertion). You increased amount of such cases. No damage still.

Quoted 35 lines
block/commit.c +1 -1
      bdrv_unref(commit_top_bs);
      blk_unref(src);
 
      if (ro) {
          /* ignoring error return here */
-        bdrv_reopen_set_read_only(bs->backing->bs, true, NULL);
+        bdrv_reopen_set_read_only(backing_file_bs, true, NULL);
      }
 
      return ret;
blockdev.c +4 -2
index c6f79b4e0e..7bef41c0b0 100644
@@ -1094,7 +1094,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)
              return;
          }
 
-        bs = blk_bs(blk);
+        bs = bdrv_skip_implicit_filters(blk_bs(blk));
          aio_context = bdrv_get_aio_context(bs);
          aio_context_acquire(aio_context);
 
@@ -3454,7 +3454,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
 
      assert(bdrv_get_aio_context(base_bs) == aio_context);
 
-    for (iter = top_bs; iter != backing_bs(base_bs); iter = backing_bs(iter)) {
+    for (iter = top_bs; iter != bdrv_filtered_bs(base_bs);
+         iter = bdrv_filtered_bs(iter))
+    {
          if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_COMMIT_TARGET, errp)) {
              goto out;
          }
Signature · 2 lines
Best regards,
Vladimir
author
#96 lore raw

Re: [Qemu-devel] [PATCH v6 27/42] commit: Deal with filters

Quoted 32 lines · Vladimir Sementsov-Ogievskiy
Quoted 31 lines · Max Reitz

This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).

---
  block/block-backend.c | 16 +++++---
  block/commit.c        | 96 +++++++++++++++++++++++++++++++------------
  blockdev.c            |  6 ++-
  3 files changed, 85 insertions(+), 33 deletions(-)
block/block-backend.c +8 -5
index c13c5c83b0..0bc592d023 100644
@@ -2180,11 +2180,17 @@ int blk_commit_all(void)
          AioContext *aio_context = blk_get_aio_context(blk);
 
          aio_context_acquire(aio_context);
-        if (blk_is_inserted(blk) && blk->root->bs->backing) {
-            int ret = bdrv_commit(blk->root->bs);
-            if (ret < 0) {
-                aio_context_release(aio_context);
-                return ret;
+        if (blk_is_inserted(blk)) {
+            BlockDriverState *non_filter;
+
+            /* Legacy function, so skip implicit filters */
+            non_filter = bdrv_skip_implicit_filters(blk->root->bs);
+            if (bdrv_filtered_cow_child(non_filter)) {
+                int ret = bdrv_commit(non_filter);
+                if (ret < 0) {
Quoted 8 lines

block/block-backend.c int blk_commit_all(void) · line 2194

Quoted 4 lines
block/block-backend.c +3 -0
+                    aio_context_release(aio_context);
+                    return ret;
+                }
              }

and if non_filter is explicit filter we just skip it. I think we'd better return error in this case. For example, just drop if (bdrv_filtered_cow_child) and get ENOTSUP from bdrv_commit in this case.

Sounds good, yes.

Quoted 6 lines

And with at least this fixed I'm OK with this patch:

However some comments below:

Quoted 13 lines

block/commit.c typedef struct CommitBlockJob { · line 40

Quoted 11 lines
block/block-backend.c +0 -0
          }
          aio_context_release(aio_context);
block/commit.c +1 -0
index 5a7672c7c7..40d1c8eeac 100644
@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {
      BlockBackend *top;
      BlockBackend *base;
      BlockDriverState *base_bs;
+    BlockDriverState *above_base;

you called it base_overlay in mirror, seems better to keep same naming

Indeed.

[...]

Quoted 4 lines
Quoted 4 lines
block/commit.c +0 -0
@@ -315,17 +336,43 @@ void commit_start(const char *job_id, BlockDriverState *bs,
 
      s->commit_top_bs = commit_top_bs;
 
Quoted 11 lines

block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 325

Quoted 8 lines
block/commit.c +0 -8
-    /* Block all nodes between top and base, because they will
-     * disappear from the chain after this operation. */
-    assert(bdrv_chain_contains(top, base));
-    for (iter = top; iter != base; iter = backing_bs(iter)) {
-        /* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves
-         * at s->base (if writes are blocked for a node, they are also blocked
-         * for its backing file). The other options would be a second filter
-         * driver above s->base. */

This code part is absolutely equal to corresponding in block/mirror.c.. It would be great to put it into a function and reuse. However its not about these series.

It would probably be great to just drop block/commit.c altogether and fully merge it into block/mirror.c at some point.

(I suppose we’d just have to check whether there’s any parent who’s taken the WRITE permission on the top node, and if so, emit READY (and if not, skip to COMPLETED).)

[...]

Quoted 10 lines

block/commit.c int bdrv_commit(BlockDriverState *bs) · line 460

Quoted 6 lines
block/commit.c +1 -1
@@ -412,19 +457,22 @@ int bdrv_commit(BlockDriverState *bs)
      if (!drv)
          return -ENOMEDIUM;
 
-    if (!bs->backing) {
+    backing_file_bs = bdrv_filtered_cow_bs(bs);

Hmm just note: if in future we'll have cow child which is not bs->backing, a lot of code will fail, as we always assume that cow child is bs->backing. May be, this should be commented in bdrv_filtered_cow_child implementation.

I couldn’t see why we’d ever do this. I hope we never do.

(Aside from just removing bs->file and bs->backing altogether.)

Max

author
#97 lore raw

PATCH v6 28/42 stream: Deal with filters

In reply to Max Reitz

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)
qapi/block-core.json +4 -0
index 38c4dbd7c3..3c54717870 100644
@@ -2516,6 +2516,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.
+#
 # @job-id: identifier for the newly-created block job. If
 #          omitted, the device name will be used. (Since 2.7)
 #
block/stream.c +33 -19

126 lines not on this page · raw message

blockdev.c +1 -1
index 7bef41c0b0..ee8b951154 100644
@@ -3297,7 +3297,7 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device,
     }
 
     /* Check for op blockers in the whole chain between bs and base */
-    for (iter = bs; iter && iter != base_bs; iter = backing_bs(iter)) {
+    for (iter = bs; iter && iter != base_bs; iter = bdrv_filtered_bs(iter)) {
         if (bdrv_op_is_blocked(iter, BLOCK_OP_TYPE_STREAM, errp)) {
             goto out;
         }
Signature · 1 line
2.21.0
Reviewed-by on v6 28/42
#98 lore raw

Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters

Quoted 6 lines · Max Reitz

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

Signature · 2 lines
Best regards,
Vladimir
#99 lore raw

Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters

In reply to Max Reitz

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 32 lines

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)
qapi/block-core.json +4 -0
index 38c4dbd7c3..3c54717870 100644
@@ -2516,6 +2516,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.
+#
 # @job-id: identifier for the newly-created block job. If
 #          omitted, the device name will be used. (Since 2.7)
 #
block/stream.c +0 -0
index 4c8b89884a..bd4a351dae 100644

block/stream.c enum { · line 35

Quoted 6 lines
block/stream.c +2 -1
@@ -31,7 +31,8 @@ enum {
 
 typedef struct StreamBlockJob {
     BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *bottom_cow_node;
+    BlockDriverState *above_base;

Confusing naming, especially because in commit you used above_base for what is bottom_cow_node here. Vladimir already suggested using base_overlay consistently, so we should do this here too (for bottom_cow_node). above_base can keep its name because the different above_base in commit is going to be renamed).

Quoted 79 lines
block/stream.c +13 -10
     BlockdevOnError on_error;
     char *backing_file_str;
     bool bs_read_only;
@@ -54,7 +55,7 @@ static void stream_abort(Job *job)
 
     if (s->chain_frozen) {
         BlockJob *bjob = &s->common;
-        bdrv_unfreeze_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_chain(blk_bs(bjob->blk), s->above_base);
     }
 }
 
@@ -63,14 +64,15 @@ static int stream_prepare(Job *job)
     StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
     BlockJob *bjob = &s->common;
     BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_rw_filters(bs);
+    BlockDriverState *base = bdrv_filtered_bs(s->above_base);
     Error *local_err = NULL;
     int ret = 0;
 
-    bdrv_unfreeze_chain(bs, s->bottom);
+    bdrv_unfreeze_chain(bs, s->above_base);
     s->chain_frozen = false;
 
-    if (bs->backing) {
+    if (bdrv_filtered_cow_child(unfiltered_bs)) {
         const char *base_id = NULL, *base_fmt = NULL;
         if (base) {
             base_id = s->backing_file_str;
@@ -78,8 +80,8 @@ static int stream_prepare(Job *job)
                 base_fmt = base->drv->format_name;
             }
         }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt);
         if (local_err) {
             error_report_err(local_err);
             return -EPERM;
@@ -110,7 +112,8 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
     BlockBackend *blk = s->common.blk;
     BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_rw_filters(bs);
+    bool enable_cor = !bdrv_filtered_bs(s->above_base);
     int64_t len;
     int64_t offset = 0;
     uint64_t delay_ns = 0;
@@ -119,7 +122,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     int64_t n = 0; /* bytes */
     void *buf;
 
-    if (bs == s->bottom) {
+    if (unfiltered_bs == s->bottom_cow_node) {
         /* Nothing to stream */
         return 0;
     }
@@ -154,13 +157,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 
         copy = false;
 
-        ret = bdrv_is_allocated(bs, offset, STREAM_BUFFER_SIZE, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_BUFFER_SIZE, &n);
         if (ret == 1) {
             /* Allocated in the top, no need to copy.  */
         } else if (ret >= 0) {
             /* Copy if allocated in the intermediate images.  Limit to the
              * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_filtered_cow_bs(unfiltered_bs),
+                                          s->bottom_cow_node, true,
                                           offset, n, &n);
             /* Finish early if end of backing file has been reached */
             if (ret == 0 && n == 0) {

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 239

Quoted 6 lines
block/stream.c +2 -1
@@ -231,9 +235,16 @@ void stream_start(const char *job_id, BlockDriverState *bs,
     BlockDriverState *iter;
     bool bs_read_only;
     int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base);
+    BlockDriverState *bottom_cow_node = bdrv_find_overlay(bs, base);
+    BlockDriverState *above_base;

Do we need to check for bottom_cow_node == NULL?

I think you could get a bs that is a filter of bottom_cow_node, and then bdrv_find_overlay() returns NULL and...

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 245

Quoted 6 lines
block/stream.c +5 -1
-    if (bdrv_freeze_chain(bs, bottom, errp) < 0) {
+    /* Find the node directly above @base */
+    for (above_base = bottom_cow_node;
+         bdrv_filtered_bs(above_base) != base;
+         above_base = bdrv_filtered_bs(above_base))
+    {}

...bottom_cow_node == NULL turns this into an endless loop.

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 249

Quoted 3 lines
block/stream.c +1 -0
+    if (bdrv_freeze_chain(bs, above_base, errp) < 0) {
         return;
     }

Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.

So should this use bottom_cow_node/base_overlay instead of above_base?

You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 278

Quoted 12 lines
block/stream.c +3 -4
@@ -261,16 +272,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,
      * disappear from the chain after this operation. The streaming job reads
      * every block only once, assuming that it doesn't change, so forbid writes
      * and resizes. Reassign the base node pointer because the backing BS of the
-     * bottom node might change after the call to bdrv_reopen_set_read_only()
-     * due to parallel block jobs running.
+     * above_base node might change after the call to
+     * bdrv_reopen_set_read_only() due to parallel block jobs running.
      */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filtered_bs(above_base);

We just calculated above_base such that it's the parent of base. Why would base not already have the value we're assigning it again here?

Quoted 14 lines
block/stream.c +5 -1
+    for (iter = bdrv_filtered_bs(bs); iter && iter != base;
+         iter = bdrv_filtered_bs(iter))
+    {
         block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
                            basic_flags, &error_abort);
     }
 
-    s->bottom = bottom;
+    s->bottom_cow_node = bottom_cow_node;
+    s->above_base = above_base;
     s->backing_file_str = g_strdup(backing_file_str);
     s->bs_read_only = bs_read_only;
     s->chain_frozen = true;

block/stream.c fail: · line 302

Quoted 6 lines
block/stream.c +1 -1
@@ -284,5 +298,5 @@ fail:
     if (bs_read_only) {
         bdrv_reopen_set_read_only(bs, true, NULL);
     }
-    bdrv_unfreeze_chain(bs, bottom);
+    bdrv_unfreeze_chain(bs, above_base);
 }

Kevin

author
#100 lore raw

Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters

Quoted 34 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 33 lines

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)
qapi/block-core.json +4 -0
index 38c4dbd7c3..3c54717870 100644
@@ -2516,6 +2516,10 @@
 # On successful completion the image file is updated to drop the backing file
 # and the BLOCK_JOB_COMPLETED event is emitted.
 #
+# In case @device is a filter node, block-stream modifies the first non-filter
+# overlay node below it to point to base's backing node (or NULL if @base was
+# not specified) instead of modifying @device itself.
+#
 # @job-id: identifier for the newly-created block job. If
 #          omitted, the device name will be used. (Since 2.7)
 #
block/stream.c +0 -0
index 4c8b89884a..bd4a351dae 100644
@@ -31,7 +31,8 @@ enum {
 
Quoted 11 lines

block/stream.c enum { · line 35

Quoted 5 lines
block/stream.c +2 -1
 typedef struct StreamBlockJob {
     BlockJob common;
-    BlockDriverState *bottom;
+    BlockDriverState *bottom_cow_node;
+    BlockDriverState *above_base;

Confusing naming, especially because in commit you used above_base for what is bottom_cow_node here. Vladimir already suggested using base_overlay consistently, so we should do this here too (for bottom_cow_node). above_base can keep its name because the different above_base in commit is going to be renamed).

Sure.

Quoted 88 lines
Quoted 79 lines
block/stream.c +13 -10
     BlockdevOnError on_error;
     char *backing_file_str;
     bool bs_read_only;
@@ -54,7 +55,7 @@ static void stream_abort(Job *job)
 
     if (s->chain_frozen) {
         BlockJob *bjob = &s->common;
-        bdrv_unfreeze_chain(blk_bs(bjob->blk), s->bottom);
+        bdrv_unfreeze_chain(blk_bs(bjob->blk), s->above_base);
     }
 }
 
@@ -63,14 +64,15 @@ static int stream_prepare(Job *job)
     StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
     BlockJob *bjob = &s->common;
     BlockDriverState *bs = blk_bs(bjob->blk);
-    BlockDriverState *base = backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_rw_filters(bs);
+    BlockDriverState *base = bdrv_filtered_bs(s->above_base);
     Error *local_err = NULL;
     int ret = 0;
 
-    bdrv_unfreeze_chain(bs, s->bottom);
+    bdrv_unfreeze_chain(bs, s->above_base);
     s->chain_frozen = false;
 
-    if (bs->backing) {
+    if (bdrv_filtered_cow_child(unfiltered_bs)) {
         const char *base_id = NULL, *base_fmt = NULL;
         if (base) {
             base_id = s->backing_file_str;
@@ -78,8 +80,8 @@ static int stream_prepare(Job *job)
                 base_fmt = base->drv->format_name;
             }
         }
-        bdrv_set_backing_hd(bs, base, &local_err);
-        ret = bdrv_change_backing_file(bs, base_id, base_fmt);
+        bdrv_set_backing_hd(unfiltered_bs, base, &local_err);
+        ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt);
         if (local_err) {
             error_report_err(local_err);
             return -EPERM;
@@ -110,7 +112,8 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);
     BlockBackend *blk = s->common.blk;
     BlockDriverState *bs = blk_bs(blk);
-    bool enable_cor = !backing_bs(s->bottom);
+    BlockDriverState *unfiltered_bs = bdrv_skip_rw_filters(bs);
+    bool enable_cor = !bdrv_filtered_bs(s->above_base);
     int64_t len;
     int64_t offset = 0;
     uint64_t delay_ns = 0;
@@ -119,7 +122,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
     int64_t n = 0; /* bytes */
     void *buf;
 
-    if (bs == s->bottom) {
+    if (unfiltered_bs == s->bottom_cow_node) {
         /* Nothing to stream */
         return 0;
     }
@@ -154,13 +157,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp)
 
         copy = false;
 
-        ret = bdrv_is_allocated(bs, offset, STREAM_BUFFER_SIZE, &n);
+        ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_BUFFER_SIZE, &n);
         if (ret == 1) {
             /* Allocated in the top, no need to copy.  */
         } else if (ret >= 0) {
             /* Copy if allocated in the intermediate images.  Limit to the
              * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE).  */
-            ret = bdrv_is_allocated_above(backing_bs(bs), s->bottom, true,
+            ret = bdrv_is_allocated_above(bdrv_filtered_cow_bs(unfiltered_bs),
+                                          s->bottom_cow_node, true,
                                           offset, n, &n);
             /* Finish early if end of backing file has been reached */
             if (ret == 0 && n == 0) {

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 239

Quoted 6 lines
block/stream.c +2 -1
@@ -231,9 +235,16 @@ void stream_start(const char *job_id, BlockDriverState *bs,
     BlockDriverState *iter;
     bool bs_read_only;
     int basic_flags = BLK_PERM_CONSISTENT_READ | BLK_PERM_WRITE_UNCHANGED;
-    BlockDriverState *bottom = bdrv_find_overlay(bs, base);
+    BlockDriverState *bottom_cow_node = bdrv_find_overlay(bs, base);
+    BlockDriverState *above_base;

Do we need to check for bottom_cow_node == NULL?

Quoted 2 lines

I think you could get a bs that is a filter of bottom_cow_node, and then bdrv_find_overlay() returns NULL and...

Ah, yes. It isn’t even about the infinite loop, it’s just a case of “Nothing to stream” (if @bs is just a filter chain away from @base).

Also, I just noticed that bdrv_find_overlay() in the version of this series won’t work if the BDS passed to it (@base here) is a filter, so that’s something else to be fixed.

Quoted 13 lines

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 245

Quoted 6 lines
block/stream.c +5 -1
-    if (bdrv_freeze_chain(bs, bottom, errp) < 0) {
+    /* Find the node directly above @base */
+    for (above_base = bottom_cow_node;
+         bdrv_filtered_bs(above_base) != base;
+         above_base = bdrv_filtered_bs(above_base))
+    {}

...bottom_cow_node == NULL turns this into an endless loop.

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 249

Quoted 3 lines
block/stream.c +1 -0
+    if (bdrv_freeze_chain(bs, above_base, errp) < 0) {
         return;
     }
Quoted 7 lines

Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.

So should this use bottom_cow_node/base_overlay instead of above_base?

I suppose I thought “Better be safe than sorry”.

Quoted 5 lines

You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.

Great, I’ll see to it.

Quoted 7 lines
Quoted 7 lines
block/stream.c +1 -2
@@ -261,16 +272,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,
      * disappear from the chain after this operation. The streaming job reads
      * every block only once, assuming that it doesn't change, so forbid writes
      * and resizes. Reassign the base node pointer because the backing BS of the
-     * bottom node might change after the call to bdrv_reopen_set_read_only()
-     * due to parallel block jobs running.
+     * above_base node might change after the call to
Quoted 8 lines

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 278

Quoted 5 lines
block/stream.c +2 -2
+     * bdrv_reopen_set_read_only() due to parallel block jobs running.
      */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filtered_bs(above_base);

We just calculated above_base such that it's the parent of base. Why would base not already have the value we're assigning it again here?

That’s no change to existing code, whose reasoning is explained in the comment above: bdrv_reopen_set_read_only() can yield, which might lead to children of the bottom node changing.

If you feel like either that’s superfluous, or that if something like that were to happen we’d have much bigger problems, be my guest to drop both.

But in this series I’d rather just not change it.

Max

#101 lore raw

Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters

Am 16.09.2019 um 11:52 hat Max Reitz geschrieben:

Quoted 22 lines · Kevin Wolf
Quoted 16 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 278

Quoted 12 lines
block/stream.c +3 -4
@@ -261,16 +272,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,
      * disappear from the chain after this operation. The streaming job reads
      * every block only once, assuming that it doesn't change, so forbid writes
      * and resizes. Reassign the base node pointer because the backing BS of the
-     * bottom node might change after the call to bdrv_reopen_set_read_only()
-     * due to parallel block jobs running.
+     * above_base node might change after the call to
+     * bdrv_reopen_set_read_only() due to parallel block jobs running.
      */
-    base = backing_bs(bottom);
-    for (iter = backing_bs(bs); iter && iter != base; iter = backing_bs(iter)) {
+    base = bdrv_filtered_bs(above_base);

We just calculated above_base such that it's the parent of base. Why would base not already have the value we're assigning it again here?

That’s no change to existing code, whose reasoning is explained in the comment above: bdrv_reopen_set_read_only() can yield, which might lead to children of the bottom node changing.

Quoted 5 lines

If you feel like either that’s superfluous, or that if something like that were to happen we’d have much bigger problems, be my guest to drop both.

But in this series I’d rather just not change it.

Ah, you mean comments are there to be read?

But actually, I think iterating down to base is too much anyway. The reasoning in the comment for block_job_add_bdrv() is that the nodes will be dropped at the end. But base with all of its filter will be kept after this patch.

So I think the for loop should stop after bs->base_overlay. And then concurrently changing links aren't even a problem any more because that's exactly the place up to which we've frozen the chain.

Kevin

author
#102 lore raw

Re: [PATCH v6 28/42] stream: Deal with filters

In reply to Max Reitz
Quoted 13 lines · Max Reitz
Quoted 12 lines · Kevin Wolf

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 11 lines

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)

[...]

Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+    if (bdrv_freeze_chain(bs, above_base, errp) < 0) {
         return;
     }

Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.

So should this use bottom_cow_node/base_overlay instead of above_base?

I suppose I thought “Better be safe than sorry”.

Quoted 7 lines
Quoted 5 lines

You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.

Great, I’ll see to it.

On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.

If @base is a filter, then bdrv_backing_chain_next(s->base_overlay) will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.

(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)

So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?

What this patch did in this version worked because the graph was frozen until @above_base.

Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between s->base_overlay and bdrv_backing_chain_next(s->base_overlay) that matches that at the end of streaming.

Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.

The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of base_overlay with that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.

But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...

Max

#103 lore raw

Re: [PATCH v6 28/42] stream: Deal with filters

Am 11.12.2019 um 13:52 hat Max Reitz geschrieben:

Quoted 43 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
Quoted 12 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 11 lines

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)

[...]

Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+    if (bdrv_freeze_chain(bs, above_base, errp) < 0) {
         return;
     }

Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.

So should this use bottom_cow_node/base_overlay instead of above_base?

I suppose I thought “Better be safe than sorry”.

Quoted 7 lines
Quoted 5 lines

You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.

Great, I’ll see to it.

On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.

Quoted 6 lines

If @base is a filter, then bdrv_backing_chain_next(s->base_overlay) will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.

(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)

Hm... Okay, let's try an example:

... <- backing <- filter1 <- filter2 <- filter3 <- top
| | |
@base above_base @device
base_overlay

The expected result after the job has completed is:

    ... <- backing <- filter1 <- top

This means that filter1 must not go away until the job has completed. In other words, we would need to freeze the link between @base and above_base.

If we use backing_bs(above_base) as we currently do, we wouldn't necessarily get filter1 as the new backing child of top (as the documentation promises), but whatever is the backing child of filter2 when the job completes. In other words, documentation and code don't match currently.

Let's look at a few more examples to see which of the options makes more sense:

1. Assume we drop filter1 while the stream job is running, i.e. backing is now the backing child of filter 2. I think it's obvious that when the stream job completes, you want backing to be the new backing child of top and not add filter1 back to the chain.

2. If we add filter1b between filter1 and filter2, do we want to drop it when the job completes? It's not entirely clear, but there are probably cases where keeping it makes sense. The user can always drop it manually again, but if we delete a node, some requests will go through unfiltered.

3. Imagine we replace filter1 wholesale, for example because a concurrently running mirror job performs a storage migration for everything up to filter1. Do we then want to reinstate the whole old subtree when the stream job completes? Certainly not.

So from this I would probably conclude that the bug is the documentation, not necessarily in the code.

Quoted 6 lines

So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?

What this patch did in this version worked because the graph was frozen until @above_base.

No, it didn't provide the promised semantics, because "unless the user changed the graph" isn't part of the documentation. But the promised semantics are probably not what we want, so it's okay.

Quoted 15 lines

Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between s->base_overlay and bdrv_backing_chain_next(s->base_overlay) that matches that at the end of streaming.

Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.

The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of base_overlay with that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.

Quoted 4 lines

But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...

Using node names feels completely wrong to me. If we can't keep a pointer (with a reference) because we want the user to be able to delete the node, then we certainly can't keep the node name either because that could refer to an entirely different node when the job completes.

I don't think it's useful to waste too much thought on how to implement the behaviour required by the documentation. The requirement seems to be just wrong.

So in the end, maybe your current code is fine, and the way to address the "doesn't make obvious sense" part is having a comment that explains why above_base is where we stop freezing.

Kevin

author
#104 lore raw

Re: [PATCH v6 28/42] stream: Deal with filters

Quoted 67 lines · Kevin Wolf

Am 11.12.2019 um 13:52 hat Max Reitz geschrieben:

Quoted 43 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
Quoted 12 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 11 lines

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)

[...]

Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+    if (bdrv_freeze_chain(bs, above_base, errp) < 0) {
         return;
     }

Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.

So should this use bottom_cow_node/base_overlay instead of above_base?

I suppose I thought “Better be safe than sorry”.

Quoted 7 lines
Quoted 5 lines

You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.

Great, I’ll see to it.

On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.

Quoted 6 lines

If @base is a filter, then bdrv_backing_chain_next(s->base_overlay) will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.

(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)

Hm... Okay, let's try an example:

... <- backing <- filter1 <- filter2 <- filter3 <- top
| | |
@base above_base @device
base_overlay

The expected result after the job has completed is:

    ... <- backing <- filter1 <- top

This means that filter1 must not go away until the job has completed. In other words, we would need to freeze the link between @base and above_base.

Quoted 5 lines

If we use backing_bs(above_base) as we currently do, we wouldn't necessarily get filter1 as the new backing child of top (as the documentation promises), but whatever is the backing child of filter2 when the job completes. In other words, documentation and code don't match currently.

Correct.

Quoted 8 lines

Let's look at a few more examples to see which of the options makes more sense:

1. Assume we drop filter1 while the stream job is running, i.e. backing is now the backing child of filter 2. I think it's obvious that when the stream job completes, you want backing to be the new backing child of top and not add filter1 back to the chain.

Quoted 5 lines

2. If we add filter1b between filter1 and filter2, do we want to drop it when the job completes? It's not entirely clear, but there are probably cases where keeping it makes sense. The user can always drop it manually again, but if we delete a node, some requests will go through unfiltered.

Depends. If the base-node was given to be literally "backing", then I’d say the user wants us to use "backing" as the new base still.

Quoted 4 lines

3. Imagine we replace filter1 wholesale, for example because a concurrently running mirror job performs a storage migration for everything up to filter1. Do we then want to reinstate the whole old subtree when the stream job completes? Certainly not.

I’m not sure. Again, if I as the user specified the "backing" node-name as the base, I’d expect that to be the new backing file in all cases.

Quoted 2 lines

So from this I would probably conclude that the bug is the documentation, not necessarily in the code.

It certainly is true that it does not address what happens when you meddle with base.

(Except it can be argued (and I suppose I did argue that) that as it is it does say the base file should be the backing file after stream, so if the base file is still there, I do interpret that to mean that the stream job always uses that as the backing node. Of course even if it says that, that doesn’t mean that it makes sense. And if it doesn’t make sense, then that’s the definition of a bug in the documentation, yes.)

Quoted 10 lines
Quoted 6 lines

So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?

What this patch did in this version worked because the graph was frozen until @above_base.

No, it didn't provide the promised semantics, because "unless the user changed the graph" isn't part of the documentation.But the promised semantics are probably not what we want, so it's okay.

I would have said that no semantics are promised for when the node is no longer a valid base, so we can basically do what we think makes sense.

Quoted 20 lines
Quoted 15 lines

Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between s->base_overlay and bdrv_backing_chain_next(s->base_overlay) that matches that at the end of streaming.

Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.

The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of base_overlay with that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.

Quoted 4 lines

But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...

Quoted 5 lines

Using node names feels completely wrong to me. If we can't keep a pointer (with a reference) because we want the user to be able to delete the node, then we certainly can't keep the node name either because that could refer to an entirely different node when the job completes.

I was thinking that if the user does graph manipulation, we can expect them to have given base-node. So they actually did refer to a node name and it makes sense to me to keep looking it up.

(As a side note: I originally intended to say “storing a pointer or the node-name are the only things that come to mind, but obviously both are stupid” – but then I changed my mind and realized that the node name maybe actually isn’t that stupid.)

Quoted 7 lines

I don't think it's useful to waste too much thought on how to implement the behaviour required by the documentation. The requirement seems to be just wrong.

So in the end, maybe your current code is fine, and the way to address the "doesn't make obvious sense" part is having a comment that explains why above_base is where we stop freezing.

Let’s maybe talk about it tomorrow. :-)

Max

#105 lore raw

Re: [PATCH v6 28/42] stream: Deal with filters

Am 11.12.2019 um 17:12 hat Max Reitz geschrieben:

Quoted 98 lines · Kevin Wolf
Quoted 67 lines

Am 11.12.2019 um 13:52 hat Max Reitz geschrieben:

Quoted 43 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
Quoted 12 lines

Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:

Quoted 11 lines

Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).

---
 qapi/block-core.json |  4 ++++
 block/stream.c       | 52 ++++++++++++++++++++++++++++----------------
 blockdev.c           |  2 +-
 3 files changed, 38 insertions(+), 20 deletions(-)

[...]

Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+    if (bdrv_freeze_chain(bs, above_base, errp) < 0) {
         return;
     }

Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.

So should this use bottom_cow_node/base_overlay instead of above_base?

I suppose I thought “Better be safe than sorry”.

Quoted 7 lines
Quoted 5 lines

You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.

Great, I’ll see to it.

On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.

Quoted 6 lines

If @base is a filter, then bdrv_backing_chain_next(s->base_overlay) will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.

(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)

Hm... Okay, let's try an example:

... <- backing <- filter1 <- filter2 <- filter3 <- top
| | |
@base above_base @device
base_overlay

The expected result after the job has completed is:

    ... <- backing <- filter1 <- top

This means that filter1 must not go away until the job has completed. In other words, we would need to freeze the link between @base and above_base.

Quoted 5 lines

If we use backing_bs(above_base) as we currently do, we wouldn't necessarily get filter1 as the new backing child of top (as the documentation promises), but whatever is the backing child of filter2 when the job completes. In other words, documentation and code don't match currently.

Correct.

Quoted 8 lines

Let's look at a few more examples to see which of the options makes more sense:

1. Assume we drop filter1 while the stream job is running, i.e. backing is now the backing child of filter 2. I think it's obvious that when the stream job completes, you want backing to be the new backing child of top and not add filter1 back to the chain.

Quoted 5 lines

2. If we add filter1b between filter1 and filter2, do we want to drop it when the job completes? It's not entirely clear, but there are probably cases where keeping it makes sense. The user can always drop it manually again, but if we delete a node, some requests will go through unfiltered.

Depends. If the base-node was given to be literally "backing", then I’d say the user wants us to use "backing" as the new base still.

Quoted 4 lines

3. Imagine we replace filter1 wholesale, for example because a concurrently running mirror job performs a storage migration for everything up to filter1. Do we then want to reinstate the whole old subtree when the stream job completes? Certainly not.

Quoted 8 lines

I’m not sure. Again, if I as the user specified the "backing" node-name as the base, I’d expect that to be the new backing file in all cases.

Quoted 2 lines

So from this I would probably conclude that the bug is the documentation, not necessarily in the code.

It certainly is true that it does not address what happens when you meddle with base.

Which is because it wasn't relevant then: You were not allowed to meddle with base when block-stream documentation was written. It's only a possible case since commit c624b015.

If we really want to guarantee that it's exactly base that will be used at the end, then we need to freeze all links down to base, essentially reverting commit c624b015 and disallowing some operations that would be possible if we didn't make that promise. I don't think that's the right thing to do, but at least it would be self-consistent.

Promising base, but not freezing the links, leads to weird results in some of the cases mentioned above.

Quoted 14 lines

(Except it can be argued (and I suppose I did argue that) that as it is it does say the base file should be the backing file after stream, so if the base file is still there, I do interpret that to mean that the stream job always uses that as the backing node. Of course even if it says that, that doesn’t mean that it makes sense. And if it doesn’t make sense, then that’s the definition of a bug in the documentation, yes.)

Quoted 7 lines
Quoted 6 lines

So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?

What this patch did in this version worked because the graph was frozen until @above_base.

Quoted 6 lines
Quoted 3 lines

No, it didn't provide the promised semantics, because "unless the user changed the graph" isn't part of the documentation.But the promised semantics are probably not what we want, so it's okay.

I would have said that no semantics are promised for when the node is no longer a valid base, so we can basically do what we think makes sense.

Like undefined behaviour in C?

Sounds great, let's reboot the guest into some 512-byte demo then.

Quoted 26 lines
Quoted 20 lines
Quoted 15 lines

Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between s->base_overlay and bdrv_backing_chain_next(s->base_overlay) that matches that at the end of streaming.

Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.

The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of base_overlay with that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.

Quoted 4 lines

But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...

Quoted 5 lines

Using node names feels completely wrong to me. If we can't keep a pointer (with a reference) because we want the user to be able to delete the node, then we certainly can't keep the node name either because that could refer to an entirely different node when the job completes.

Quoted 8 lines

I was thinking that if the user does graph manipulation, we can expect them to have given base-node. So they actually did refer to a node name and it makes sense to me to keep looking it up.

(As a side note: I originally intended to say “storing a pointer or the node-name are the only things that come to mind, but obviously both are stupid” – but then I changed my mind and realized that the node name maybe actually isn’t that stupid.)

I think it is. A node name refers to the node with that name at the time of the QMP command. What node was referenced in that command doesn't change later because of graph manipulations.

Quoted 9 lines
Quoted 7 lines

I don't think it's useful to waste too much thought on how to implement the behaviour required by the documentation. The requirement seems to be just wrong.

So in the end, maybe your current code is fine, and the way to address the "doesn't make obvious sense" part is having a comment that explains why above_base is where we stop freezing.

Let’s maybe talk about it tomorrow. :-)

Okay, added to my preliminary agenda.

Kevin

author
#106 lore raw

PATCH v6 29/42 nbd: Use CAF when looking for dirty bitmap

In reply to Max Reitz

When looking for a dirty bitmap to share, we should handle filters by just including them in the search (so they do not break backing chains).

---
 nbd/server.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
nbd/server.c +3 -3
index fbd51b48a7..a1c73ce957 100644
@@ -1511,13 +1511,13 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,
     if (bitmap) {
         BdrvDirtyBitmap *bm = NULL;
 
-        while (true) {
+        while (bs) {
             bm = bdrv_find_dirty_bitmap(bs, bitmap);
-            if (bm != NULL || bs->backing == NULL) {
+            if (bm != NULL) {
                 break;
             }
 
-            bs = bs->backing->bs;
+            bs = bdrv_filtered_bs(bs);
         }
 
         if (bm == NULL) {
Signature · 1 line
2.21.0
author
#107 lore raw

PATCH v6 30/42 qemu-img: Use child access functions

In reply to Max Reitz

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

---
 qemu-img.c                 | 33 ++++++++++++++++++++-------------
 tests/qemu-iotests/204.out |  1 +
 2 files changed, 21 insertions(+), 13 deletions(-)
qemu-img.c +20 -13
index 79983772de..3b30c5ae70 100644
@@ -1012,7 +1012,7 @@ static int img_commit(int argc, char **argv)
         /* This is different from QMP, which by default uses the deepest file in
          * the backing chain (i.e., the very base); however, the traditional
          * behavior of qemu-img commit is using the immediate backing file. */
-        base_bs = backing_bs(bs);
+        base_bs = bdrv_backing_chain_next(bs);
         if (!base_bs) {
             error_setg(&local_err, "Image does not have a backing file");
             goto done;
@@ -1632,18 +1632,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
     if (s->sector_next_status <= sector_num) {
         uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
         int64_t count;
+        BlockDriverState *src_bs = blk_bs(s->src[src_cur]);
+        BlockDriverState *base;
+
+        if (s->target_has_backing) {
+            base = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(src_bs));
+        } else {
+            base = NULL;
+        }
 
         do {
             count = n * BDRV_SECTOR_SIZE;
 
-            if (s->target_has_backing) {
-                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
-                                        count, &count, NULL, NULL);
-            } else {
-                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                              offset, count, &count, NULL,
-                                              NULL);
-            }
+            ret = bdrv_block_status_above(src_bs, base, offset, count, &count,
+                                          NULL, NULL);
 
             if (ret < 0) {
                 if (s->salvage) {
@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)
          * s.target_backing_sectors has to be negative, which it will
          * be automatically).  The backing file length is used only
          * for optimizations, so such a case is not fatal. */
-        s.target_backing_sectors = bdrv_nb_sectors(out_bs->backing->bs);
+        s.target_backing_sectors =
+            bdrv_nb_sectors(bdrv_filtered_cow_bs(out_bs));
     } else {
         s.target_backing_sectors = -1;
     }
@@ -2853,6 +2856,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 
     depth = 0;
     for (;;) {
+        bs = bdrv_skip_rw_filters(bs);
         ret = bdrv_block_status(bs, offset, bytes, &bytes, &map, &file);
         if (ret < 0) {
             return ret;
@@ -2861,7 +2865,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
         if (ret & (BDRV_BLOCK_ZERO|BDRV_BLOCK_DATA)) {
             break;
         }
-        bs = backing_bs(bs);
+        bs = bdrv_filtered_cow_bs(bs);
         if (bs == NULL) {
             ret = 0;
             break;
@@ -3216,6 +3220,7 @@ static int img_rebase(int argc, char **argv)
     uint8_t *buf_old = NULL;
     uint8_t *buf_new = NULL;
     BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
     char *filename;
     const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
     int c, flags, src_flags, ret;
@@ -3350,6 +3355,8 @@ static int img_rebase(int argc, char **argv)
     }
     bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_rw_filters(bs);
+
     if (out_basefmt != NULL) {
         if (bdrv_find_format(out_basefmt) == NULL) {
             error_report("Invalid format name: '%s'", out_basefmt);
@@ -3361,7 +3368,7 @@ static int img_rebase(int argc, char **argv)
     /* For safe rebasing we need to compare old and new backing file */
     if (!unsafe) {
         QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_filtered_cow_bs(unfiltered_bs);
 
         if (base_bs) {
             blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3517,7 +3524,7 @@ static int img_rebase(int argc, char **argv)
                  * If cluster wasn't changed since prefix_chain, we don't need
                  * to take action
                  */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
+                ret = bdrv_is_allocated_above(unfiltered_bs, prefix_chain_bs,
                                               false, offset, n, &n);
                 if (ret < 0) {
                     error_report("error while reading image metadata: %s",
tests/qemu-iotests/204.out +1 -0
index f3a10fbe90..684774d763 100644
@@ -59,5 +59,6 @@ Offset          Length          File
 0x900000        0x2400000       TEST_DIR/t.IMGFMT
 0x3c00000       0x1100000       TEST_DIR/t.IMGFMT
 0x6a00000       0x400000        TEST_DIR/t.IMGFMT
+0x6e00000       0x1200000       TEST_DIR/t.IMGFMT.base
 No errors were found on the image.
 *** done
Signature · 1 line
2.21.0
#108 lore raw

Re: [Qemu-devel] [PATCH v6 30/42] qemu-img: Use child access functions

Quoted 54 lines · Max Reitz

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

---
  qemu-img.c                 | 33 ++++++++++++++++++++-------------
  tests/qemu-iotests/204.out |  1 +
  2 files changed, 21 insertions(+), 13 deletions(-)
qemu-img.c +11 -9
index 79983772de..3b30c5ae70 100644
@@ -1012,7 +1012,7 @@ static int img_commit(int argc, char **argv)
          /* This is different from QMP, which by default uses the deepest file in
           * the backing chain (i.e., the very base); however, the traditional
           * behavior of qemu-img commit is using the immediate backing file. */
-        base_bs = backing_bs(bs);
+        base_bs = bdrv_backing_chain_next(bs);
          if (!base_bs) {
              error_setg(&local_err, "Image does not have a backing file");
              goto done;
@@ -1632,18 +1632,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
      if (s->sector_next_status <= sector_num) {
          uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
          int64_t count;
+        BlockDriverState *src_bs = blk_bs(s->src[src_cur]);
+        BlockDriverState *base;
+
+        if (s->target_has_backing) {
+            base = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(src_bs));
+        } else {
+            base = NULL;
+        }
 
          do {
              count = n * BDRV_SECTOR_SIZE;
 
-            if (s->target_has_backing) {
-                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
-                                        count, &count, NULL, NULL);
-            } else {
-                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                              offset, count, &count, NULL,
-                                              NULL);
-            }
+            ret = bdrv_block_status_above(src_bs, base, offset, count, &count,
+                                          NULL, NULL);
 
              if (ret < 0) {
                  if (s->salvage) {

qemu-img.c static int img_convert(int argc, char **argv) · line 2496

Quoted 6 lines
qemu-img.c +2 -1
@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)
           * s.target_backing_sectors has to be negative, which it will
           * be automatically).  The backing file length is used only
           * for optimizations, so such a case is not fatal. */
-        s.target_backing_sectors = bdrv_nb_sectors(out_bs->backing->bs);
+        s.target_backing_sectors =
+            bdrv_nb_sectors(bdrv_filtered_cow_bs(out_bs));

why not skip_rw_filters? It will fail if out_bs is filter..

Quoted 67 lines
qemu-img.c +7 -3
      } else {
          s.target_backing_sectors = -1;
      }
@@ -2853,6 +2856,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 
      depth = 0;
      for (;;) {
+        bs = bdrv_skip_rw_filters(bs);
          ret = bdrv_block_status(bs, offset, bytes, &bytes, &map, &file);
          if (ret < 0) {
              return ret;
@@ -2861,7 +2865,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
          if (ret & (BDRV_BLOCK_ZERO|BDRV_BLOCK_DATA)) {
              break;
          }
-        bs = backing_bs(bs);
+        bs = bdrv_filtered_cow_bs(bs);
          if (bs == NULL) {
              ret = 0;
              break;
@@ -3216,6 +3220,7 @@ static int img_rebase(int argc, char **argv)
      uint8_t *buf_old = NULL;
      uint8_t *buf_new = NULL;
      BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
      char *filename;
      const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
      int c, flags, src_flags, ret;
@@ -3350,6 +3355,8 @@ static int img_rebase(int argc, char **argv)
      }
      bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_rw_filters(bs);
+
      if (out_basefmt != NULL) {
          if (bdrv_find_format(out_basefmt) == NULL) {
              error_report("Invalid format name: '%s'", out_basefmt);
@@ -3361,7 +3368,7 @@ static int img_rebase(int argc, char **argv)
      /* For safe rebasing we need to compare old and new backing file */
      if (!unsafe) {
          QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_filtered_cow_bs(unfiltered_bs);
 
          if (base_bs) {
              blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3517,7 +3524,7 @@ static int img_rebase(int argc, char **argv)
                   * If cluster wasn't changed since prefix_chain, we don't need
                   * to take action
                   */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
+                ret = bdrv_is_allocated_above(unfiltered_bs, prefix_chain_bs,
                                                false, offset, n, &n);
                  if (ret < 0) {
                      error_report("error while reading image metadata: %s",
tests/qemu-iotests/204.out +1 -0
index f3a10fbe90..684774d763 100644
@@ -59,5 +59,6 @@ Offset          Length          File
  0x900000        0x2400000       TEST_DIR/t.IMGFMT
  0x3c00000       0x1100000       TEST_DIR/t.IMGFMT
  0x6a00000       0x400000        TEST_DIR/t.IMGFMT
+0x6e00000       0x1200000       TEST_DIR/t.IMGFMT.base
  No errors were found on the image.
  *** done
Signature · 2 lines
Best regards,
Vladimir
author
#109 lore raw

Re: [Qemu-devel] [PATCH v6 30/42] qemu-img: Use child access functions

Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Max Reitz

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

---
  qemu-img.c                 | 33 ++++++++++++++++++++-------------
  tests/qemu-iotests/204.out |  1 +
  2 files changed, 21 insertions(+), 13 deletions(-)
qemu-img.c +0 -0
index 79983772de..3b30c5ae70 100644

[...]

Quoted 9 lines

qemu-img.c static int img_convert(int argc, char **argv) · line 2496

Quoted 7 lines
qemu-img.c +2 -1
@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)
           * s.target_backing_sectors has to be negative, which it will
           * be automatically).  The backing file length is used only
           * for optimizations, so such a case is not fatal. */
-        s.target_backing_sectors = bdrv_nb_sectors(out_bs->backing->bs);
+        s.target_backing_sectors =
+            bdrv_nb_sectors(bdrv_filtered_cow_bs(out_bs));

why not skip_rw_filters? It will fail if out_bs is filter..

Because I forgot this place. :-) Although backing_chain_next() would be simpler here and do the same, effectively.

Max

Reviewed-by on v6 30/42
#110 lore raw

Re: [Qemu-devel] [PATCH v6 30/42] qemu-img: Use child access functions

In reply to Max Reitz
Quoted 54 lines · Max Reitz

This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.

---
  qemu-img.c                 | 33 ++++++++++++++++++++-------------
  tests/qemu-iotests/204.out |  1 +
  2 files changed, 21 insertions(+), 13 deletions(-)
qemu-img.c +11 -9
index 79983772de..3b30c5ae70 100644
@@ -1012,7 +1012,7 @@ static int img_commit(int argc, char **argv)
          /* This is different from QMP, which by default uses the deepest file in
           * the backing chain (i.e., the very base); however, the traditional
           * behavior of qemu-img commit is using the immediate backing file. */
-        base_bs = backing_bs(bs);
+        base_bs = bdrv_backing_chain_next(bs);
          if (!base_bs) {
              error_setg(&local_err, "Image does not have a backing file");
              goto done;
@@ -1632,18 +1632,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)
      if (s->sector_next_status <= sector_num) {
          uint64_t offset = (sector_num - src_cur_offset) * BDRV_SECTOR_SIZE;
          int64_t count;
+        BlockDriverState *src_bs = blk_bs(s->src[src_cur]);
+        BlockDriverState *base;
+
+        if (s->target_has_backing) {
+            base = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(src_bs));
+        } else {
+            base = NULL;
+        }
 
          do {
              count = n * BDRV_SECTOR_SIZE;
 
-            if (s->target_has_backing) {
-                ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset,
-                                        count, &count, NULL, NULL);
-            } else {
-                ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL,
-                                              offset, count, &count, NULL,
-                                              NULL);
-            }
+            ret = bdrv_block_status_above(src_bs, base, offset, count, &count,
+                                          NULL, NULL);
 
              if (ret < 0) {
                  if (s->salvage) {

qemu-img.c static int img_convert(int argc, char **argv) · line 2496

Quoted 6 lines
qemu-img.c +2 -1
@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)
           * s.target_backing_sectors has to be negative, which it will
           * be automatically).  The backing file length is used only
           * for optimizations, so such a case is not fatal. */
-        s.target_backing_sectors = bdrv_nb_sectors(out_bs->backing->bs);
+        s.target_backing_sectors =
+            bdrv_nb_sectors(bdrv_filtered_cow_bs(out_bs));

bdrv_nb_sectors(bdrv_backing_chain_next(out_bs))

Quoted 47 lines
qemu-img.c +6 -2
      } else {
          s.target_backing_sectors = -1;
      }
@@ -2853,6 +2856,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
 
      depth = 0;
      for (;;) {
+        bs = bdrv_skip_rw_filters(bs);
          ret = bdrv_block_status(bs, offset, bytes, &bytes, &map, &file);
          if (ret < 0) {
              return ret;
@@ -2861,7 +2865,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,
          if (ret & (BDRV_BLOCK_ZERO|BDRV_BLOCK_DATA)) {
              break;
          }
-        bs = backing_bs(bs);
+        bs = bdrv_filtered_cow_bs(bs);
          if (bs == NULL) {
              ret = 0;
              break;
@@ -3216,6 +3220,7 @@ static int img_rebase(int argc, char **argv)
      uint8_t *buf_old = NULL;
      uint8_t *buf_new = NULL;
      BlockDriverState *bs = NULL, *prefix_chain_bs = NULL;
+    BlockDriverState *unfiltered_bs;
      char *filename;
      const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;
      int c, flags, src_flags, ret;
@@ -3350,6 +3355,8 @@ static int img_rebase(int argc, char **argv)
      }
      bs = blk_bs(blk);
 
+    unfiltered_bs = bdrv_skip_rw_filters(bs);
+
      if (out_basefmt != NULL) {
          if (bdrv_find_format(out_basefmt) == NULL) {
              error_report("Invalid format name: '%s'", out_basefmt);
@@ -3361,7 +3368,7 @@ static int img_rebase(int argc, char **argv)
      /* For safe rebasing we need to compare old and new backing file */
      if (!unsafe) {
          QDict *options = NULL;
-        BlockDriverState *base_bs = backing_bs(bs);
+        BlockDriverState *base_bs = bdrv_filtered_cow_bs(unfiltered_bs);
 
          if (base_bs) {
              blk_old_backing = blk_new(qemu_get_aio_context(),

qemu-img.c static int img_rebase(int argc, char **argv) · line 3527

Quoted 5 lines
qemu-img.c +1 -1
@@ -3517,7 +3524,7 @@ static int img_rebase(int argc, char **argv)
                   * If cluster wasn't changed since prefix_chain, we don't need
                   * to take action
                   */
-                ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs,
+                ret = bdrv_is_allocated_above(unfiltered_bs, prefix_chain_bs,

s/unfiltered_bs/bdrv_filtered_cow_bs(unfiltered_bs)/

Quoted 9 lines
qemu-img.c +0 -0
                                                false, offset, n, &n);
                  if (ret < 0) {
                      error_report("error while reading image metadata: %s",
tests/qemu-iotests/204.out +0 -0
index f3a10fbe90..684774d763 100644

tests/qemu-iotests/204.out Offset Length File · line 64

Quoted 6 lines
tests/qemu-iotests/204.out +1 -0
@@ -59,5 +59,6 @@ Offset          Length          File
  0x900000        0x2400000       TEST_DIR/t.IMGFMT
  0x3c00000       0x1100000       TEST_DIR/t.IMGFMT
  0x6a00000       0x400000        TEST_DIR/t.IMGFMT
+0x6e00000       0x1200000       TEST_DIR/t.IMGFMT.base
  No errors were found on the image.
  *** done

With two fixes:

Signature · 2 lines
Best regards,
Vladimir
author
#111 lore raw

PATCH v6 31/42 block: Drop backing_bs()

In reply to Max Reitz

We want to make it explicit where bs->backing is used, and we have done so. The old role of backing_bs() is now effectively taken by bdrv_filtered_cow_bs().

---
 include/block/block_int.h | 5 -----
 1 file changed, 5 deletions(-)
include/block/block_int.h +0 -5
index 5bec3361fd..786801c32f 100644
@@ -932,11 +932,6 @@ typedef enum BlockMirrorBackingMode {
     MIRROR_LEAVE_BACKING_CHAIN,
 } BlockMirrorBackingMode;
 
-static inline BlockDriverState *backing_bs(BlockDriverState *bs)
-{
-    return bs->backing ? bs->backing->bs : NULL;
-}
-
 
 /* Essential block drivers which must always be statically linked into qemu, and
  * which therefore can be accessed without using bdrv_find_format() */
Signature · 1 line
2.21.0
author
#112 lore raw

PATCH v6 32/42 block: Make bdrv_get_cumulative_perm() public

In reply to Max Reitz

This is useful in other files like blockdev.c to determine for example whether a node can be written to or not.

---
 include/block/block_int.h | 3 +++
 block.c                   | 6 ++----
 2 files changed, 5 insertions(+), 4 deletions(-)
include/block/block_int.h +3 -0
index 786801c32f..c17df3808a 100644
@@ -1205,6 +1205,9 @@ int bdrv_child_try_set_perm(BdrvChild *c, uint64_t perm, uint64_t shared,
  */
 int bdrv_child_refresh_perms(BlockDriverState *bs, BdrvChild *c, Error **errp);
 
+void bdrv_get_cumulative_perm(BlockDriverState *bs,
+                              uint64_t *perm, uint64_t *shared_perm);
+
 /* Default implementation for BlockDriver.bdrv_child_perm() that can be used by
  * block filters: Forward CONSISTENT_READ, WRITE, WRITE_UNCHANGED and RESIZE to
  * all children */
block.c +2 -4
index 6e1ddab056..915b80153c 100644
@@ -1713,8 +1713,6 @@ static int bdrv_child_check_perm(BdrvChild *c, BlockReopenQueue *q,
                                  bool *tighten_restrictions, Error **errp);
 static void bdrv_child_abort_perm_update(BdrvChild *c);
 static void bdrv_child_set_perm(BdrvChild *c, uint64_t perm, uint64_t shared);
-static void bdrv_get_cumulative_perm(BlockDriverState *bs, uint64_t *perm,
-                                     uint64_t *shared_perm);
 
 typedef struct BlockReopenQueueEntry {
      bool prepared;
@@ -1938,8 +1936,8 @@ static void bdrv_set_perm(BlockDriverState *bs, uint64_t cumulative_perms,
     }
 }
 
-static void bdrv_get_cumulative_perm(BlockDriverState *bs, uint64_t *perm,
-                                     uint64_t *shared_perm)
+void bdrv_get_cumulative_perm(BlockDriverState *bs,
+                              uint64_t *perm, uint64_t *shared_perm)
 {
     BdrvChild *c;
     uint64_t cumulative_perms = 0;
Signature · 1 line
2.21.0
author
#113 lore raw

PATCH v6 33/42 blockdev: Fix active commit choice

In reply to Max Reitz

We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.

---
 blockdev.c | 24 +++++++++++++++++++++---
 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +21 -3
index ee8b951154..4e72f6f701 100644
@@ -3356,6 +3356,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
      */
     BlockdevOnError on_error = BLOCKDEV_ON_ERROR_REPORT;
     int job_flags = JOB_DEFAULT;
+    uint64_t top_perm, top_shared;
 
     if (!has_speed) {
         speed = 0;
@@ -3468,14 +3469,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,
         goto out;
     }
 
-    if (top_bs == bs) {
+    /*
+     * Active commit is required if and only if someone has taken a
+     * WRITE permission on the top node.  Historically, we have always
+     * used active commit for top nodes, so continue that practice.
+     * (Active commit is never really wrong.)
+     */
+    bdrv_get_cumulative_perm(top_bs, &top_perm, &top_shared);
+    if (top_perm & BLK_PERM_WRITE ||
+        bdrv_skip_rw_filters(top_bs) == bdrv_skip_rw_filters(bs))
+    {
         if (has_backing_file) {
             error_setg(errp, "'backing-file' specified,"
                              " but 'top' is the active layer");
             goto out;
         }
-        commit_active_start(has_job_id ? job_id : NULL, bs, base_bs,
-                            job_flags, speed, on_error,
+        if (!has_job_id) {
+            /*
+             * Emulate here what block_job_create() does, because it
+             * is possible that @bs != @top_bs (the block job should
+             * be named after @bs, even if @top_bs is the actual
+             * source)
+             */
+            job_id = bdrv_get_device_name(bs);
+        }
+        commit_active_start(job_id, top_bs, base_bs, job_flags, speed, on_error,
                             filter_node_name, NULL, NULL, false, &local_err);
     } else {
         BlockDriverState *overlay_bs = bdrv_find_overlay(bs, top_bs);
Signature · 1 line
2.21.0
author
#114 lore raw

PATCH v6 34/42 block: Inline bdrv_co_block_status_from_*()

In reply to Max Reitz

With bdrv_filtered_rw_bs(), we can easily handle this default filter behavior in bdrv_co_block_status().

blkdebug wants to have an additional assertion, so it keeps its own implementation, except bdrv_co_block_status_from_file() needs to be inlined there.

---
 include/block/block_int.h | 22 -----------------
 block/blkdebug.c          |  7 ++++--
 block/blklogwrites.c      |  1 -
 block/commit.c            |  1 -
 block/copy-on-read.c      |  2 --
 block/io.c                | 51 +++++++++++++--------------------------
 block/mirror.c            |  1 -
 block/throttle.c          |  1 -
 8 files changed, 22 insertions(+), 64 deletions(-)
include/block/block_int.h +0 -22
index c17df3808a..42ee2fcf7f 100644
@@ -1227,28 +1227,6 @@ void bdrv_format_default_perms(BlockDriverState *bs, BdrvChild *c,
                                uint64_t perm, uint64_t shared,
                                uint64_t *nperm, uint64_t *nshared);
 
-/*
- * Default implementation for drivers to pass bdrv_co_block_status() to
- * their file.
- */
-int coroutine_fn bdrv_co_block_status_from_file(BlockDriverState *bs,
-                                                bool want_zero,
-                                                int64_t offset,
-                                                int64_t bytes,
-                                                int64_t *pnum,
-                                                int64_t *map,
-                                                BlockDriverState **file);
-/*
- * Default implementation for drivers to pass bdrv_co_block_status() to
- * their backing file.
- */
-int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,
-                                                   bool want_zero,
-                                                   int64_t offset,
-                                                   int64_t bytes,
-                                                   int64_t *pnum,
-                                                   int64_t *map,
-                                                   BlockDriverState **file);
 const char *bdrv_get_parent_name(const BlockDriverState *bs);
 void blk_dev_change_media_cb(BlockBackend *blk, bool load, Error **errp);
 bool blk_dev_has_removable_media(BlockBackend *blk);
block/blkdebug.c +5 -2
index 5ae96c52b0..0011e068ce 100644
@@ -679,8 +679,11 @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,
         return err;
     }
 
-    return bdrv_co_block_status_from_file(bs, want_zero, offset, bytes,
-                                          pnum, map, file);
+    assert(bs->file && bs->file->bs);
+    *pnum = bytes;
+    *map = offset;
+    *file = bs->file->bs;
+    return BDRV_BLOCK_RAW | BDRV_BLOCK_OFFSET_VALID;
 }
 
 static void blkdebug_close(BlockDriverState *bs)
block/blklogwrites.c +0 -1
index 04d8b33607..8982fd15c4 100644
@@ -519,7 +519,6 @@ static BlockDriver bdrv_blk_log_writes = {
     .bdrv_co_pwrite_zeroes  = blk_log_writes_co_pwrite_zeroes,
     .bdrv_co_flush_to_disk  = blk_log_writes_co_flush_to_disk,
     .bdrv_co_pdiscard       = blk_log_writes_co_pdiscard,
-    .bdrv_co_block_status   = bdrv_co_block_status_from_file,
 
     .is_filter              = true,
     .strong_runtime_opts    = blk_log_writes_strong_runtime_opts,
block/commit.c +0 -1
index 40d1c8eeac..c94678bea3 100644
@@ -252,7 +252,6 @@ static void bdrv_commit_top_child_perm(BlockDriverState *bs, BdrvChild *c,
 static BlockDriver bdrv_commit_top = {
     .format_name                = "commit_top",
     .bdrv_co_preadv             = bdrv_commit_top_preadv,
-    .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
     .bdrv_refresh_filename      = bdrv_commit_top_refresh_filename,
     .bdrv_child_perm            = bdrv_commit_top_child_perm,
 
block/copy-on-read.c +0 -2
index 16bdf630b6..ad6577d078 100644
@@ -160,8 +160,6 @@ static BlockDriver bdrv_copy_on_read = {
     .bdrv_eject                         = cor_eject,
     .bdrv_lock_medium                   = cor_lock_medium,
 
-    .bdrv_co_block_status               = bdrv_co_block_status_from_file,
-
     .bdrv_recurse_is_first_non_filter   = cor_recurse_is_first_non_filter,
 
     .has_variable_length                = true,
block/io.c +17 -34

75 lines not on this page · raw message

block/mirror.c +0 -1
index 6ddbfb9708..88155faaec 100644
@@ -1508,7 +1508,6 @@ static BlockDriver bdrv_mirror_top = {
     .bdrv_co_pwrite_zeroes      = bdrv_mirror_top_pwrite_zeroes,
     .bdrv_co_pdiscard           = bdrv_mirror_top_pdiscard,
     .bdrv_co_flush              = bdrv_mirror_top_flush,
-    .bdrv_co_block_status       = bdrv_co_block_status_from_backing,
     .bdrv_refresh_filename      = bdrv_mirror_top_refresh_filename,
     .bdrv_child_perm            = bdrv_mirror_top_child_perm,
     .bdrv_refresh_limits        = bdrv_mirror_top_refresh_limits,
block/throttle.c +0 -1
index 958a2bcfa6..d0436f875b 100644
@@ -270,7 +270,6 @@ static BlockDriver bdrv_throttle = {
     .bdrv_reopen_prepare                =   throttle_reopen_prepare,
     .bdrv_reopen_commit                 =   throttle_reopen_commit,
     .bdrv_reopen_abort                  =   throttle_reopen_abort,
-    .bdrv_co_block_status               =   bdrv_co_block_status_from_file,
 
     .bdrv_co_drain_begin                =   throttle_co_drain_begin,
     .bdrv_co_drain_end                  =   throttle_co_drain_end,
Signature · 1 line
2.21.0
author
#115 lore raw

PATCH v6 35/42 block: Fix check_to_replace_node()

In reply to Max Reitz

Currently, check_to_replace_node() only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.

This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.

This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:

$ qemu-img create -f qcow2 /tmp/source.qcow2 64M
$ qemu-system-x86_64 -qmp stdio
{"execute": "qmp_capabilities"}
{"execute": "object-add",
 "arguments": {"qom-type": "throttle-group", "id": "tg0"}}
{"execute": "blockdev-add",
 "arguments": {
     "node-name": "source",
     "driver": "throttle",
     "throttle-group": "tg0",
     "file": {
         "node-name": "filtered",
         "driver": "qcow2",
         "file": {
             "driver": "file",
             "filename": "/tmp/source.qcow2"
         } } } }
{"execute": "drive-mirror",
 "arguments": {
     "job-id": "mirror",
     "device": "source",
     "target": "/tmp/target.qcow2",
     "format": "qcow2",
     "node-name": "target",
     "sync" :"none",
     "replaces": "filtered"
 } }
{"execute": "block-job-complete", "arguments": {"device": "mirror"}}

And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).

(blockdev-mirror can be broken similarly.)

So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.

---
 include/block/block.h |  1 +
 block.c               | 83 +++++++++++++++++++++++++++++++++++++++----
 blockdev.c            | 34 ++++++++++++++++--
 3 files changed, 110 insertions(+), 8 deletions(-)
include/block/block.h +1 -0
index 6ba853fb90..8da706cd89 100644
@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);
 
 /* check if a named node can be replaced when doing drive-mirror */
 BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                         const char *node_name, Error **errp);
 
 /* async block I/O */
block.c +77 -6

97 lines not on this page · raw message

blockdev.c +32 -2
index 4e72f6f701..758e0b5431 100644
@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
     }
 
     if (has_replaces) {
-        BlockDriverState *to_replace_bs;
+        BlockDriverState *to_replace_bs, *backing_bs;
         AioContext *replace_aio_context;
         int64_t bs_size, replace_size;
 
@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
             return;
         }
 
-        to_replace_bs = check_to_replace_node(bs, replaces, errp);
+        if (backing_mode == MIRROR_SOURCE_BACKING_CHAIN ||
+            backing_mode == MIRROR_OPEN_BACKING_CHAIN)
+        {
+            /*
+             * While we do not quite know what OPEN_BACKING_CHAIN
+             * (used for mode=existing) will yield, it is probably
+             * best to restrict it exactly like SOURCE_BACKING_CHAIN,
+             * because that is our best guess.
+             */
+            switch (sync) {
+            case MIRROR_SYNC_MODE_FULL:
+                backing_bs = NULL;
+                break;
+
+            case MIRROR_SYNC_MODE_TOP:
+                backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs));
+                break;
+
+            case MIRROR_SYNC_MODE_NONE:
+                backing_bs = bs;
+                break;
+
+            default:
+                abort();
+            }
+        } else {
+            assert(backing_mode == MIRROR_LEAVE_BACKING_CHAIN);
+            backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(target));
+        }
+
+        to_replace_bs = check_to_replace_node(bs, backing_bs, replaces, errp);
         if (!to_replace_bs) {
             return;
         }
Signature · 1 line
2.21.0
#116 lore raw

Re: [Qemu-devel] [PATCH v6 35/42] block: Fix check_to_replace_node()

Quoted 92 lines · Max Reitz

Currently, check_to_replace_node() only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.

This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.

This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:

$ qemu-img create -f qcow2 /tmp/source.qcow2 64M
$ qemu-system-x86_64 -qmp stdio
{"execute": "qmp_capabilities"}
{"execute": "object-add",
  "arguments": {"qom-type": "throttle-group", "id": "tg0"}}
{"execute": "blockdev-add",
  "arguments": {
      "node-name": "source",
      "driver": "throttle",
      "throttle-group": "tg0",
      "file": {
          "node-name": "filtered",
          "driver": "qcow2",
          "file": {
              "driver": "file",
              "filename": "/tmp/source.qcow2"
          } } } }
{"execute": "drive-mirror",
  "arguments": {
      "job-id": "mirror",
      "device": "source",
      "target": "/tmp/target.qcow2",
      "format": "qcow2",
      "node-name": "target",
      "sync" :"none",
      "replaces": "filtered"
  } }
{"execute": "block-job-complete", "arguments": {"device": "mirror"}}

And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).

(blockdev-mirror can be broken similarly.)

So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.

---
  include/block/block.h |  1 +
  block.c               | 83 +++++++++++++++++++++++++++++++++++++++----
  blockdev.c            | 34 ++++++++++++++++--
  3 files changed, 110 insertions(+), 8 deletions(-)
include/block/block.h +1 -0
index 6ba853fb90..8da706cd89 100644
@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);
 
  /* check if a named node can be replaced when doing drive-mirror */
  BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                          const char *node_name, Error **errp);
 
  /* async block I/O */
block.c +11 -0
index 915b80153c..4858d3e718 100644
@@ -6290,7 +6290,59 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate)
      return false;
  }
 
+static bool is_child_of(BlockDriverState *child, BlockDriverState *parent)
+{
+    BdrvChild *c;
+
+    if (!parent) {
+        return false;
+    }
+
+    QLIST_FOREACH(c, &parent->children, next) {
+        if (c->bs == child || is_child_of(child, c->bs)) {
+            return true;

block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6313

Quoted 10 lines
block.c +10 -0
+        }
+    }
+
+    return false;
+}
+
+/*
+ * Return true if there are only filters in [@top, @base).  Note that
+ * this may include quorum (which bdrv_chain_contains() cannot
+ * handle).

More presizely: return true if exists chain of filters from top to base or if top == base.

I keep in mind backup-top filter:

[backup-top]
| \target
|backing -------->[target]
V /
[source] <---------/backing

Quoted 12 lines
block.c +12 -0
+ */
+static bool is_filtered_child(BlockDriverState *top, BlockDriverState *base)
+{
+    BdrvChild *c;
+
+    if (!top) {
+        return false;
+    }
+
+    if (top == base) {
+        return true;
+    }

block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6335

Quoted 10 lines
block.c +10 -0
+
+    if (!top->drv->is_filter) {
+        return false;
+    }
+
+    QLIST_FOREACH(c, &top->children, next) {
+        if (is_filtered_child(c->bs, base)) {
+            return true;
+        }
+    }

interesting, how much is it better to somehow reuse DFS search written in should_update_child().. [just note, don't do it in these series please]

Quoted 18 lines
block.c +9 -1
+
+    return false;
+}
+
+/*
+ * @parent_bs is mirror's source BDS, @backing_bs is the BDS which
+ * will be attached to the target when mirror completes.
+ */
  BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                          const char *node_name, Error **errp)
  {
      BlockDriverState *to_replace_bs = bdrv_find_node(node_name);
@@ -6309,13 +6361,32 @@ BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
          goto out;
      }
 
-    /* We don't want arbitrary node of the BDS chain to be replaced only the top

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6368

Quoted 10 lines
block.c +4 -5
-     * most non filter in order to prevent data corruption.
-     * Another benefit is that this tests exclude backing files which are
-     * blocked by the backing blockers.
+    /*
+     * If to_replace_bs is (recursively) a child of backing_bs,
+     * replacing it may create a loop.  We cannot allow that.
       */
-    if (!bdrv_recurse_is_first_non_filter(parent_bs, to_replace_bs)) {
-        error_setg(errp, "Only top most non filter can be replaced");
+    if (to_replace_bs == backing_bs || is_child_of(to_replace_bs, backing_bs)) {

first condition is covered by second, so first may be omitted.

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,

Quoted 8 lines
block.c +8 -0
+        error_setg(errp, "Replacing this node would result in a loop");
+        to_replace_bs = NULL;
+        goto out;
+    }
+
+    /*
+     * Mirror is designed in such a way that when it completes, the
+     * source BDS is seamlessly replaced.  

Not source but to_replace_bs is replaced?

Quoted 4 lines

It is therefore not allowed

block.c +3 -0
+     * to replace a BDS where this condition would be violated, as that
+     * would defeat the purpose of mirror and could lead to data
+     * corruption.

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6389

Quoted 10 lines
block.c +10 -0
+     * Therefore, between parent_bs and to_replace_bs there may be
+     * only filters (and the one on top must be a filter, too), so
+     * their data always stays in sync and mirror can complete and
+     * replace to_replace_bs without any possible corruptions.
+     */
+    if (!is_filtered_child(parent_bs, to_replace_bs) &&
+        !is_filtered_child(to_replace_bs, parent_bs))
+    {
+        error_setg(errp, "The node to be replaced must be connected to the "
+                   "source through filter nodes only");

"and the one on top must be a filter, too" not mentioned in the error..

Quoted 27 lines
          to_replace_bs = NULL;
          goto out;
      }
blockdev.c +7 -2
index 4e72f6f701..758e0b5431 100644
@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
      }
 
      if (has_replaces) {
-        BlockDriverState *to_replace_bs;
+        BlockDriverState *to_replace_bs, *backing_bs;
          AioContext *replace_aio_context;
          int64_t bs_size, replace_size;
 
@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
              return;
          }
 
-        to_replace_bs = check_to_replace_node(bs, replaces, errp);
+        if (backing_mode == MIRROR_SOURCE_BACKING_CHAIN ||
+            backing_mode == MIRROR_OPEN_BACKING_CHAIN)
+        {
+            /*
+             * While we do not quite know what OPEN_BACKING_CHAIN
+             * (used for mode=existing) will yield, it is probably

blockdev.c static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, · line 3915

Quoted 10 lines
blockdev.c +10 -0
+             * best to restrict it exactly like SOURCE_BACKING_CHAIN,
+             * because that is our best guess.
+             */
+            switch (sync) {
+            case MIRROR_SYNC_MODE_FULL:
+                backing_bs = NULL;
+                break;
+
+            case MIRROR_SYNC_MODE_TOP:
+                backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs));

why not bdrv_backing_chain_next(bs) like in mirror_start?

Quoted 19 lines
blockdev.c +15 -0
+                break;
+
+            case MIRROR_SYNC_MODE_NONE:
+                backing_bs = bs;
+                break;
+
+            default:
+                abort();
+            }
+        } else {
+            assert(backing_mode == MIRROR_LEAVE_BACKING_CHAIN);
+            backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(target));
+        }
+
+        to_replace_bs = check_to_replace_node(bs, backing_bs, replaces, errp);
          if (!to_replace_bs) {
              return;
          }
Signature · 2 lines
Best regards,
Vladimir
author
#117 lore raw

Re: [Qemu-devel] [PATCH v6 35/42] block: Fix check_to_replace_node()

Quoted 107 lines · Vladimir Sementsov-Ogievskiy
Quoted 92 lines · Max Reitz

Currently, check_to_replace_node() only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.

This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.

This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:

$ qemu-img create -f qcow2 /tmp/source.qcow2 64M
$ qemu-system-x86_64 -qmp stdio
{"execute": "qmp_capabilities"}
{"execute": "object-add",
  "arguments": {"qom-type": "throttle-group", "id": "tg0"}}
{"execute": "blockdev-add",
  "arguments": {
      "node-name": "source",
      "driver": "throttle",
      "throttle-group": "tg0",
      "file": {
          "node-name": "filtered",
          "driver": "qcow2",
          "file": {
              "driver": "file",
              "filename": "/tmp/source.qcow2"
          } } } }
{"execute": "drive-mirror",
  "arguments": {
      "job-id": "mirror",
      "device": "source",
      "target": "/tmp/target.qcow2",
      "format": "qcow2",
      "node-name": "target",
      "sync" :"none",
      "replaces": "filtered"
  } }
{"execute": "block-job-complete", "arguments": {"device": "mirror"}}

And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).

(blockdev-mirror can be broken similarly.)

So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.

---
  include/block/block.h |  1 +
  block.c               | 83 +++++++++++++++++++++++++++++++++++++++----
  blockdev.c            | 34 ++++++++++++++++--
  3 files changed, 110 insertions(+), 8 deletions(-)
include/block/block.h +1 -0
index 6ba853fb90..8da706cd89 100644
@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);
 
  /* check if a named node can be replaced when doing drive-mirror */
  BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                          const char *node_name, Error **errp);
 
  /* async block I/O */
block.c +11 -0
index 915b80153c..4858d3e718 100644
@@ -6290,7 +6290,59 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate)
      return false;
  }
 
+static bool is_child_of(BlockDriverState *child, BlockDriverState *parent)
+{
+    BdrvChild *c;
+
+    if (!parent) {
+        return false;
+    }
+
+    QLIST_FOREACH(c, &parent->children, next) {
+        if (c->bs == child || is_child_of(child, c->bs)) {
+            return true;

block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6313

Quoted 10 lines
block.c +10 -0
+        }
+    }
+
+    return false;
+}
+
+/*
+ * Return true if there are only filters in [@top, @base).  Note that
+ * this may include quorum (which bdrv_chain_contains() cannot
+ * handle).

More presizely: return true if exists chain of filters from top to base or if top == base.

Quoted 4 lines

I keep in mind backup-top filter:

[backup-top]
| \target

backup-top can’t be a filter if it has two children with different contents, though.

(commit-top and mirror-top aren’t filters either.)

That’s why there must be a unique chain [@top, @base).

I should probably not that it will return true if top == base, though, yes.

Quoted 47 lines

|backing -------->[target]
V /
[source] <---------/backing

Quoted 12 lines
block.c +12 -0
+ */
+static bool is_filtered_child(BlockDriverState *top, BlockDriverState *base)
+{
+    BdrvChild *c;
+
+    if (!top) {
+        return false;
+    }
+
+    if (top == base) {
+        return true;
+    }

block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6335

Quoted 10 lines
block.c +10 -0
+
+    if (!top->drv->is_filter) {
+        return false;
+    }
+
+    QLIST_FOREACH(c, &top->children, next) {
+        if (is_filtered_child(c->bs, base)) {
+            return true;
+        }
+    }

interesting, how much is it better to somehow reuse DFS search written in should_update_child().. [just note, don't do it in these series please]

Quoted 17 lines
block.c +9 -0
+
+    return false;
+}
+
+/*
+ * @parent_bs is mirror's source BDS, @backing_bs is the BDS which
+ * will be attached to the target when mirror completes.
+ */
  BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                          const char *node_name, Error **errp)
  {
      BlockDriverState *to_replace_bs = bdrv_find_node(node_name);
@@ -6309,13 +6361,32 @@ BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
          goto out;
      }
 
Quoted 13 lines

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6368

Quoted 11 lines
block.c +4 -6
-    /* We don't want arbitrary node of the BDS chain to be replaced only the top
-     * most non filter in order to prevent data corruption.
-     * Another benefit is that this tests exclude backing files which are
-     * blocked by the backing blockers.
+    /*
+     * If to_replace_bs is (recursively) a child of backing_bs,
+     * replacing it may create a loop.  We cannot allow that.
       */
-    if (!bdrv_recurse_is_first_non_filter(parent_bs, to_replace_bs)) {
-        error_setg(errp, "Only top most non filter can be replaced");
+    if (to_replace_bs == backing_bs || is_child_of(to_replace_bs, backing_bs)) {

first condition is covered by second, so first may be omitted.

It is not. is_child_of() does not return true if child == parent.

Quoted 10 lines

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,

Quoted 8 lines
block.c +8 -0
+        error_setg(errp, "Replacing this node would result in a loop");
+        to_replace_bs = NULL;
+        goto out;
+    }
+
+    /*
+     * Mirror is designed in such a way that when it completes, the
+     * source BDS is seamlessly replaced.  

Not source but to_replace_bs is replaced?

It has originally been designed to replace the source. If it could replace any arbitrary BDS, all of this would be moot.

Quoted 15 lines
Quoted 4 lines

It is therefore not allowed

block.c +3 -0
+     * to replace a BDS where this condition would be violated, as that
+     * would defeat the purpose of mirror and could lead to data
+     * corruption.

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6389

Quoted 10 lines
block.c +10 -0
+     * Therefore, between parent_bs and to_replace_bs there may be
+     * only filters (and the one on top must be a filter, too), so
+     * their data always stays in sync and mirror can complete and
+     * replace to_replace_bs without any possible corruptions.
+     */
+    if (!is_filtered_child(parent_bs, to_replace_bs) &&
+        !is_filtered_child(to_replace_bs, parent_bs))
+    {
+        error_setg(errp, "The node to be replaced must be connected to the "
+                   "source through filter nodes only");
Quoted 1 line

"and the one on top must be a filter, too" not mentioned in the error..

Well, unless the source node is the node to be replaced. Hm... This gets very hard to express. I think I’d prefer to keep this as it is, even though it is not quite correct, unless you have a better suggestion of what to report. :-/

Quoted 31 lines
Quoted 31 lines
          to_replace_bs = NULL;
          goto out;
      }
blockdev.c +11 -2
index 4e72f6f701..758e0b5431 100644
@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
      }
 
      if (has_replaces) {
-        BlockDriverState *to_replace_bs;
+        BlockDriverState *to_replace_bs, *backing_bs;
          AioContext *replace_aio_context;
          int64_t bs_size, replace_size;
 
@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
              return;
          }
 
-        to_replace_bs = check_to_replace_node(bs, replaces, errp);
+        if (backing_mode == MIRROR_SOURCE_BACKING_CHAIN ||
+            backing_mode == MIRROR_OPEN_BACKING_CHAIN)
+        {
+            /*
+             * While we do not quite know what OPEN_BACKING_CHAIN
+             * (used for mode=existing) will yield, it is probably
+             * best to restrict it exactly like SOURCE_BACKING_CHAIN,
+             * because that is our best guess.
+             */
+            switch (sync) {
Quoted 8 lines

blockdev.c static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, · line 3915

Quoted 6 lines
blockdev.c +6 -0
+            case MIRROR_SYNC_MODE_FULL:
+                backing_bs = NULL;
+                break;
+
+            case MIRROR_SYNC_MODE_TOP:
+                backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs));

why not bdrv_backing_chain_next(bs) like in mirror_start?

Good question. I suppose it should be bdrv_filtered_cow_bs(bdrv_backing_chain_next(bs)) in mirror_start()? Because with sync=top, we just want to remove the topmost COW node (and filters on top), but keep filters behind it.

Max

Quoted 21 lines
Quoted 19 lines
blockdev.c +15 -0
+                break;
+
+            case MIRROR_SYNC_MODE_NONE:
+                backing_bs = bs;
+                break;
+
+            default:
+                abort();
+            }
+        } else {
+            assert(backing_mode == MIRROR_LEAVE_BACKING_CHAIN);
+            backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(target));
+        }
+
+        to_replace_bs = check_to_replace_node(bs, backing_bs, replaces, errp);
          if (!to_replace_bs) {
              return;
          }
#118 lore raw

Re: [Qemu-devel] [PATCH v6 35/42] block: Fix check_to_replace_node()

Quoted 108 lines · Max Reitz
Quoted 107 lines · Vladimir Sementsov-Ogievskiy
Quoted 92 lines · Max Reitz

Currently, check_to_replace_node() only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.

This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.

This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:

$ qemu-img create -f qcow2 /tmp/source.qcow2 64M
$ qemu-system-x86_64 -qmp stdio
{"execute": "qmp_capabilities"}
{"execute": "object-add",
   "arguments": {"qom-type": "throttle-group", "id": "tg0"}}
{"execute": "blockdev-add",
   "arguments": {
       "node-name": "source",
       "driver": "throttle",
       "throttle-group": "tg0",
       "file": {
           "node-name": "filtered",
           "driver": "qcow2",
           "file": {
               "driver": "file",
               "filename": "/tmp/source.qcow2"
           } } } }
{"execute": "drive-mirror",
   "arguments": {
       "job-id": "mirror",
       "device": "source",
       "target": "/tmp/target.qcow2",
       "format": "qcow2",
       "node-name": "target",
       "sync" :"none",
       "replaces": "filtered"
   } }
{"execute": "block-job-complete", "arguments": {"device": "mirror"}}

And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).

(blockdev-mirror can be broken similarly.)

So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.

---
   include/block/block.h |  1 +
   block.c               | 83 +++++++++++++++++++++++++++++++++++++++----
   blockdev.c            | 34 ++++++++++++++++--
   3 files changed, 110 insertions(+), 8 deletions(-)
include/block/block.h +1 -0
index 6ba853fb90..8da706cd89 100644
@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);
 
   /* check if a named node can be replaced when doing drive-mirror */
   BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                           const char *node_name, Error **errp);
 
   /* async block I/O */
block.c +11 -0
index 915b80153c..4858d3e718 100644
@@ -6290,7 +6290,59 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate)
       return false;
   }
 
+static bool is_child_of(BlockDriverState *child, BlockDriverState *parent)
+{
+    BdrvChild *c;
+
+    if (!parent) {
+        return false;
+    }
+
+    QLIST_FOREACH(c, &parent->children, next) {
+        if (c->bs == child || is_child_of(child, c->bs)) {
+            return true;

block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6313

Quoted 10 lines
block.c +10 -0
+        }
+    }
+
+    return false;
+}
+
+/*
+ * Return true if there are only filters in [@top, @base).  Note that
+ * this may include quorum (which bdrv_chain_contains() cannot
+ * handle).

More presizely: return true if exists chain of filters from top to base or if top == base.

Quoted 7 lines
Quoted 4 lines

I keep in mind backup-top filter:

[backup-top]
| \target

backup-top can’t be a filter if it has two children with different contents, though.

Why? target is special child, unrelated to what is read/written over backup-top. It's an own business of backup-top.

Quoted 2 lines

(commit-top and mirror-top aren’t filters either.)

Ahm, I missed something. They have is_filter = true and their children considered to be filtered-rw children in your series? And than, who they are? Format nodes? And how they appears in backing chains than?

Quoted 77 lines

That’s why there must be a unique chain [@top, @base).

I should probably not that it will return true if top == base, though, yes.

Quoted 47 lines

|backing -------->[target]
V /
[source] <---------/backing

Quoted 12 lines
block.c +12 -0
+ */
+static bool is_filtered_child(BlockDriverState *top, BlockDriverState *base)
+{
+    BdrvChild *c;
+
+    if (!top) {
+        return false;
+    }
+
+    if (top == base) {
+        return true;
+    }

block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6335

Quoted 10 lines
block.c +10 -0
+
+    if (!top->drv->is_filter) {
+        return false;
+    }
+
+    QLIST_FOREACH(c, &top->children, next) {
+        if (is_filtered_child(c->bs, base)) {
+            return true;
+        }
+    }

interesting, how much is it better to somehow reuse DFS search written in should_update_child().. [just note, don't do it in these series please]

Quoted 17 lines
block.c +9 -0
+
+    return false;
+}
+
+/*
+ * @parent_bs is mirror's source BDS, @backing_bs is the BDS which
+ * will be attached to the target when mirror completes.
+ */
   BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
+                                        BlockDriverState *backing_bs,
                                           const char *node_name, Error **errp)
   {
       BlockDriverState *to_replace_bs = bdrv_find_node(node_name);
@@ -6309,13 +6361,32 @@ BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
           goto out;
       }
 
Quoted 13 lines

block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6368

Quoted 11 lines
block.c +4 -6
-    /* We don't want arbitrary node of the BDS chain to be replaced only the top
-     * most non filter in order to prevent data corruption.
-     * Another benefit is that this tests exclude backing files which are
-     * blocked by the backing blockers.
+    /*
+     * If to_replace_bs is (recursively) a child of backing_bs,
+     * replacing it may create a loop.  We cannot allow that.
        */
-    if (!bdrv_recurse_is_first_non_filter(parent_bs, to_replace_bs)) {
-        error_setg(errp, "Only top most non filter can be replaced");
+    if (to_replace_bs == backing_bs || is_child_of(to_replace_bs, backing_bs)) {

first condition is covered by second, so first may be omitted.

It is not. is_child_of() does not return true if child == parent.

Quoted 9 lines
Quoted 8 lines
block.c +8 -0
+        error_setg(errp, "Replacing this node would result in a loop");
+        to_replace_bs = NULL;
+        goto out;
+    }
+
+    /*
+     * Mirror is designed in such a way that when it completes, the
+     * source BDS is seamlessly replaced.
Quoted 4 lines
Quoted 1 line

Not source but to_replace_bs is replaced?

It has originally been designed to replace the source. If it could replace any arbitrary BDS, all of this would be moot.

quorum child, you saying about in commit message?

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

It is therefore not allowed

block.c +13 -0
+     * to replace a BDS where this condition would be violated, as that
+     * would defeat the purpose of mirror and could lead to data
+     * corruption.
+     * Therefore, between parent_bs and to_replace_bs there may be
+     * only filters (and the one on top must be a filter, too), so
+     * their data always stays in sync and mirror can complete and
+     * replace to_replace_bs without any possible corruptions.
+     */
+    if (!is_filtered_child(parent_bs, to_replace_bs) &&
+        !is_filtered_child(to_replace_bs, parent_bs))
+    {
+        error_setg(errp, "The node to be replaced must be connected to the "
+                   "source through filter nodes only");
Quoted 6 lines
Quoted 1 line

"and the one on top must be a filter, too" not mentioned in the error..

Well, unless the source node is the node to be replaced. Hm... This gets very hard to express. I think I’d prefer to keep this as it is, even though it is not quite correct, unless you have a better suggestion of what to report. :-/

I can't imaging something better than just add "(and the one on top must be a filter, too)"

Quoted 41 lines
Quoted 32 lines
Quoted 32 lines
           to_replace_bs = NULL;
           goto out;
       }
blockdev.c +11 -2
index 4e72f6f701..758e0b5431 100644
@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
       }
 
       if (has_replaces) {
-        BlockDriverState *to_replace_bs;
+        BlockDriverState *to_replace_bs, *backing_bs;
           AioContext *replace_aio_context;
           int64_t bs_size, replace_size;
 
@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,
               return;
           }
 
-        to_replace_bs = check_to_replace_node(bs, replaces, errp);
+        if (backing_mode == MIRROR_SOURCE_BACKING_CHAIN ||
+            backing_mode == MIRROR_OPEN_BACKING_CHAIN)
+        {
+            /*
+             * While we do not quite know what OPEN_BACKING_CHAIN
+             * (used for mode=existing) will yield, it is probably
+             * best to restrict it exactly like SOURCE_BACKING_CHAIN,
+             * because that is our best guess.
+             */
+            switch (sync) {
Quoted 8 lines

blockdev.c static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, · line 3915

Quoted 6 lines
blockdev.c +6 -0
+            case MIRROR_SYNC_MODE_FULL:
+                backing_bs = NULL;
+                break;
+
+            case MIRROR_SYNC_MODE_TOP:
+                backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs));

why not bdrv_backing_chain_next(bs) like in mirror_start?

Quoted 2 lines

Good question. I suppose it should be bdrv_filtered_cow_bs(bdrv_backing_chain_next(bs)) in mirror_start()?

You mean bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs)), I hope)

Quoted 3 lines

Because with sync=top, we just want to remove the topmost COW node (and filters on top), but keep filters behind it.

Agreed.

Quoted 24 lines
Quoted 22 lines
Quoted 20 lines
blockdev.c +15 -0
+                break;
+
+            case MIRROR_SYNC_MODE_NONE:
+                backing_bs = bs;
+                break;
+
+            default:
+                abort();
+            }
+        } else {
+            assert(backing_mode == MIRROR_LEAVE_BACKING_CHAIN);
+            backing_bs = bdrv_filtered_cow_bs(bdrv_skip_rw_filters(target));
+        }
+
+        to_replace_bs = check_to_replace_node(bs, backing_bs, replaces, errp);
           if (!to_replace_bs) {
               return;
           }
Signature · 2 lines
Best regards,
Vladimir