block: Deal with filters
v6 cover block: Deal with filters
Hi,
When we introduced filters, we did it a bit casually. Sure, we talked a lot about them before, but that was mostly discussion about where implicit filters should be added to the graph (note that we currently only have two implicit filters, those being mirror and commit). But in the end, we really just designated some drivers filters (Quorum, blkdebug, etc.) and added some specifically (throttle, COR), without really looking through the block layer to see where issues might occur.
It turns out vast areas of the block layer just don’t know about filters and cannot really handle them. Many cases will work in practice, in others, well, too bad, you cannot use some feature because some part deep inside the block layer looks at your filters and thinks they are format nodes.
This is one reason why this series is needed. Over time (since v1), a second reason has made its way in:
bs->file is not necessarily the place where a node’s data is stored. qcow2 now has external data files, and currently there is no way for the general block layer to know that the data is not stored in bs->file. Right now, I do not think that has any real consequences (all functions that need access to the actual data storage file should only do so as a fallback if the driver does not provide some functionality, but qcow2 should provide it all), but it still shows that we need some way to let the general block layer know about such data files. (Also, I will need this for v1 of my “Inquire images’ rotational info” series.)
I won’t go on and on about this series now, I think the patches pretty much speak for themselves now. If the cover letter gets too long, nobody reads it anyway (see previous versions).
*** I’ve based this series on John’s bitmap branch, which I’ve rebased
on my block-next branch. ***
I’ve pushed the patches here:
https://git.xanclic.moe/XanClic/qemu child-access-functions-v6
https://github.com/XanClic/qemu child-access-functions-v6
I’ve also pushed the base branch to each of those repos, it’s “child-access-functions-base”.
v6:
- Patch 9: Rename *freeze_backing_chain (etc.) to *freeze_chain (etc.)
- Patch 10: Drop bdrv_is_encrypted() instead of fixing it the wrong way
- Patch 15: Add a comment on why this works
- Patch 16: Just flush all children instead of one
- Patch 20: We have to snapshot all non-backing children, so both metadata and storage children; for simplification, just disallow the fallback path if there is more than one such child
- Patch 22: bdrv_get_allocated_file_size() should report all children’s sizes, not just the primary or storage child’s
- Patch 24: Make query-blockstats too report any filtered child under “backing”
- Patch 25:
- bdrv_is_allocated_above()’s new @include_base parameter makes things a bit simpler
- Forbid mirroring to a filter on top of the base
- Patch 27: bdrv_is_allocated_above()’s new @include_base parameter makes things a bit simpler
- Patch 28: Requires a few changes to keep stream independent of the base node
- Patch 30:
- Sprinkle in a few bdrv_skip_implicit_filters()s
- Conflicts with the convert --salvage patches
git backport-diff against v5:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/42:[----] [-C] 'block: Mark commit and mirror as filter drivers' 002/42:[----] [--] 'copy-on-read: Support compressed writes' 003/42:[----] [--] 'throttle: Support compressed writes' 004/42:[----] [--] 'block: Add child access functions' 005/42:[----] [--] 'block: Add chain helper functions' 006/42:[----] [--] 'qcow2: Implement .bdrv_storage_child()' 007/42:[----] [--] 'block: *filtered_cow_child() for *has_zero_init()' 008/42:[----] [--] 'block: bdrv_set_backing_hd() is about bs->backing' 009/42:[0078] [FC] 'block: Include filters when freezing backing chain' 010/42:[down] 'block: Drop bdrv_is_encrypted()' 011/42:[----] [-C] 'block: Add bdrv_supports_compressed_writes()' 012/42:[----] [--] 'block: Use bdrv_filtered_rw* where obvious' 013/42:[----] [-C] 'block: Use CAFs in block status functions' 014/42:[----] [--] 'block: Use CAFs when working with backing chains' 015/42:[0017] [FC] 'block: Re-evaluate backing file handling in reopen' 016/42:[down] 'block: Flush all children in generic code' 017/42:[----] [--] 'block: Use CAFs in bdrv_refresh_limits()' 018/42:[----] [--] 'block: Use CAFs in bdrv_refresh_filename()' 019/42:[----] [--] 'block: Use CAF in bdrv_co_rw_vmstate()' 020/42:[down] 'block/snapshot: Fix fallback'
021/42:[----] [--] 'block: Use CAFs for debug breakpoints' 022/42:[down] 'block: Fix bdrv_get_allocated_file_size's fallback' 023/42:[----] [--] 'blockdev: Use CAF in external_snapshot_prepare()' 024/42:[0012] [FC] 'block: Use child access functions for QAPI queries' 025/42:[0019] [FC] 'mirror: Deal with filters'
026/42:[----] [--] 'backup: Deal with filters'
027/42:[0025] [FC] 'commit: Deal with filters'
028/42:[0048] [FC] 'stream: Deal with filters'
029/42:[----] [--] 'nbd: Use CAF when looking for dirty bitmap' 030/42:[0035] [FC] 'qemu-img: Use child access functions' 031/42:[----] [--] 'block: Drop backing_bs()'
032/42:[----] [-C] 'block: Make bdrv_get_cumulative_perm() public' 033/42:[----] [--] 'blockdev: Fix active commit choice' 034/42:[----] [-C] 'block: Inline bdrv_co_block_status_from_*()' 035/42:[----] [--] 'block: Fix check_to_replace_node()' 036/42:[----] [--] 'iotests: Add tests for mirror @replaces loops' 037/42:[----] [-C] 'block: Leave BDS.backing_file constant' 038/42:[----] [--] 'iotests: Let complete_and_wait() work with commit' 039/42:[----] [--] 'iotests: Add filter commit test cases' 040/42:[----] [--] 'iotests: Add filter mirror test cases' 041/42:[----] [--] 'iotests: Add test for commit in sub directory' 042/42:[----] [--] 'iotests: Test committing to overridden backing'
Max Reitz (42):
block: Mark commit and mirror as filter drivers
copy-on-read: Support compressed writes
throttle: Support compressed writes
block: Add child access functions
block: Add chain helper functions
qcow2: Implement .bdrv_storage_child()
block: *filtered_cow_child() for *has_zero_init()
block: bdrv_set_backing_hd() is about bs->backing
block: Include filters when freezing backing chain
block: Drop bdrv_is_encrypted()
block: Add bdrv_supports_compressed_writes()
block: Use bdrv_filtered_rw* where obvious
block: Use CAFs in block status functions
block: Use CAFs when working with backing chains
block: Re-evaluate backing file handling in reopen
block: Flush all children in generic code
block: Use CAFs in bdrv_refresh_limits()
block: Use CAFs in bdrv_refresh_filename()
block: Use CAF in bdrv_co_rw_vmstate() block/snapshot: Fix fallback
block: Use CAFs for debug breakpoints
block: Fix bdrv_get_allocated_file_size's fallback
blockdev: Use CAF in external_snapshot_prepare()
block: Use child access functions for QAPI queries
mirror: Deal with filters
backup: Deal with filters
commit: Deal with filters
stream: Deal with filters
nbd: Use CAF when looking for dirty bitmap
qemu-img: Use child access functions
block: Drop backing_bs()
block: Make bdrv_get_cumulative_perm() public
blockdev: Fix active commit choice
block: Inline bdrv_co_block_status_from_*()
block: Fix check_to_replace_node()
iotests: Add tests for mirror @replaces loops
block: Leave BDS.backing_file constant
iotests: Let complete_and_wait() work with commit
iotests: Add filter commit test cases
iotests: Add filter mirror test cases
iotests: Add test for commit in sub directory
iotests: Test committing to overridden backing
qapi/block-core.json | 4 + include/block/block.h | 13 +- include/block/block_int.h | 109 ++++--- block.c | 552 ++++++++++++++++++++++++++++------ block/backup.c | 9 +- block/blkdebug.c | 7 +- block/blklogwrites.c | 1 - block/block-backend.c | 16 +- block/commit.c | 107 +++++-- block/copy-on-read.c | 13 +- block/io.c | 117 ++++--- block/mirror.c | 124 ++++++-- block/qapi.c | 48 +-- block/qcow2.c | 9 + block/snapshot.c | 100 ++++-- block/stream.c | 52 ++-- block/throttle.c | 11 +- blockdev.c | 139 +++++++-- nbd/server.c | 6 +- qemu-img.c | 33 +- tests/qemu-iotests/020 | 36 +++ tests/qemu-iotests/020.out | 10 + tests/qemu-iotests/040 | 238 +++++++++++++++ tests/qemu-iotests/040.out | 4 +- tests/qemu-iotests/041 | 270 ++++++++++++++++- tests/qemu-iotests/041.out | 4 +- tests/qemu-iotests/184.out | 7 +- tests/qemu-iotests/191.out | 1 - tests/qemu-iotests/204.out | 1 + tests/qemu-iotests/228 | 6 +- tests/qemu-iotests/228.out | 6 +- tests/qemu-iotests/245 | 4 +- tests/qemu-iotests/iotests.py | 10 +- 33 files changed, 1682 insertions(+), 385 deletions(-)
Signature · 1 line
2.21.0
PATCH v6 1/42 block: Mark commit and mirror as filter drivers
The commit and mirror block nodes are filters, so they should be marked as such. (Strictly speaking, BDS.is_filter's documentation states that a filter's child must be bs->file. The following patch will relax this restriction, however.)
--- block/commit.c | 2 ++ block/mirror.c | 2 ++ 2 files changed, 4 insertions(+)
block/commit.c +2 -0
@@ -254,6 +254,8 @@ static BlockDriver bdrv_commit_top = { .bdrv_co_block_status = bdrv_co_block_status_from_backing, .bdrv_refresh_filename = bdrv_commit_top_refresh_filename, .bdrv_child_perm = bdrv_commit_top_child_perm, + + .is_filter = true, }; void commit_start(const char *job_id, BlockDriverState *bs,
block/mirror.c +2 -0
@@ -1507,6 +1507,8 @@ static BlockDriver bdrv_mirror_top = { .bdrv_refresh_filename = bdrv_mirror_top_refresh_filename, .bdrv_child_perm = bdrv_mirror_top_child_perm, .bdrv_refresh_limits = bdrv_mirror_top_refresh_limits, + + .is_filter = true, }; static BlockJob *mirror_start_job(
Signature · 1 line
2.21.0
PATCH v6 2/42 copy-on-read: Support compressed writes
In reply to Max Reitz--- block/copy-on-read.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
block/copy-on-read.c +11 -0
@@ -113,6 +113,16 @@ static int coroutine_fn cor_co_pdiscard(BlockDriverState *bs, } +static int coroutine_fn cor_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_co_pwritev (bs->file, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} + +static void cor_eject (BlockDriverState *bs,bool eject_flag) {bdrv_eject (bs->file->bs, eject_flag);
@@ -145,6 +155,7 @@ static BlockDriver bdrv_copy_on_read = { .bdrv_co_pwritev = cor_co_pwritev, .bdrv_co_pwrite_zeroes = cor_co_pwrite_zeroes, .bdrv_co_pdiscard = cor_co_pdiscard, + .bdrv_co_pwritev_compressed = cor_co_pwritev_compressed, .bdrv_eject = cor_eject, .bdrv_lock_medium = cor_lock_medium,
Signature · 1 line
2.21.0
PATCH v6 3/42 throttle: Support compressed writes
In reply to Max Reitz--- block/throttle.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
block/throttle.c +10 -0
@@ -153,6 +153,15 @@ static int coroutine_fn throttle_co_pdiscard(BlockDriverState *bs,return bdrv_co_pdiscard (bs->file, offset, bytes); } +static int coroutine_fn throttle_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return throttle_co_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} +static int throttle_co_flush (BlockDriverState *bs) {return bdrv_co_flush (bs->file->bs);
@@ -251,6 +260,7 @@ static BlockDriver bdrv_throttle = { .bdrv_co_pwrite_zeroes = throttle_co_pwrite_zeroes, .bdrv_co_pdiscard = throttle_co_pdiscard, + .bdrv_co_pwritev_compressed = throttle_co_pwritev_compressed, .bdrv_recurse_is_first_non_filter = throttle_recurse_is_first_non_filter,
Signature · 1 line
2.21.0
PATCH v6 4/42 block: Add child access functions
In reply to Max ReitzThere are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be an R/W-filtered child; bs->file can be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
--- include/block/block_int.h | 57 ++++++++++++++++++++-- block.c | 99 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+), 3 deletions(-)
include/block/block_int.h +54 -3
@@ -90,9 +90,11 @@ struct BlockDriver {int instance_size;/* set to true if the BlockDriver is a block filter. Block filters pass -* certain callbacks that refer to data (see block.c) to their bs->file if -* the driver doesn't implement them. Drivers that do not wish to forward -* must implement them and return -ENOTSUP. +* certain callbacks that refer to data (see block.c) to their bs->file +* or bs->backing (whichever one exists) if the driver doesn't implement +* them. Drivers that do not wish to forward must implement them and return +* -ENOTSUP. +* Note that filters are not allowed to modify data. */ bool is_filter;/* for snapshots block filter like Quorum can implement the
@@ -562,6 +564,13 @@ struct BlockDriver {* If this pointer is NULL, the array is considered empty. * "filename" and "driver" are always considered strong. */ const char *const *strong_runtime_opts; + +/** +* Return the data storage child, if there is exactly one. If +* this function is not implemented, the block layer will assume +* bs->file to be this child. +*/ + BdrvChild *(*bdrv_storage_child)(BlockDriverState *bs); }; typedef struct BlockLimits {
@@ -1276,4 +1285,46 @@ int coroutine_fn bdrv_co_copy_range_to(BdrvChild *src, uint64_t src_offset,int refresh_total_sectors (BlockDriverState *bs,int64_t hint); +BdrvChild *bdrv_filtered_cow_child (BlockDriverState *bs); +BdrvChild *bdrv_filtered_rw_child (BlockDriverState *bs); +BdrvChild *bdrv_filtered_child (BlockDriverState *bs); +BdrvChild *bdrv_metadata_child (BlockDriverState *bs); +BdrvChild *bdrv_storage_child (BlockDriverState *bs); +BdrvChild *bdrv_primary_child (BlockDriverState *bs); + +static inline BlockDriverState *child_bs (BdrvChild *child) +{ +return child ? child->bs :NULL ; +} + +static inline BlockDriverState *bdrv_filtered_cow_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_filtered_cow_child (bs)); +} + +static inline BlockDriverState *bdrv_filtered_rw_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_filtered_rw_child (bs)); +} + +static inline BlockDriverState *bdrv_filtered_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_filtered_child (bs)); +} + +static inline BlockDriverState *bdrv_metadata_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_metadata_child (bs)); +} + +static inline BlockDriverState *bdrv_storage_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_storage_child (bs)); +} + +static inline BlockDriverState *bdrv_primary_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_primary_child (bs)); +} +#endif /* BLOCK_INT_H */
block.c +99 -0
102 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Quoted 12 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Quoted 7 lines
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block_int.h | 57 ++++++++++++++++++++-- block.c | 99 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 153 insertions(+), 3 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 7 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.
Quoted 7 lines
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?
Apparently any child of a filter node is "filtered" (which makes sense), but also bs->backing of a qcow2 image, while bs->file of qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?
It looks like bdrv_filtered_child() is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.
Quoted 4 lines
+/*
+ * Return the child that @bs acts as an overlay for, and from which data may be
+ * copied in COW or COR operations. Usually this is the backing file.
+ */
Or NULL, if no such child exists.
It's relatively obvious here, but for some of the functions further down it would be really good to describe in which cases NULL is expected (or that NULL is even a possible return value).
Kevin
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Quoted 13 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 12 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?Apparently any child of a filter node is "filtered" (which makes sense),
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?
“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.
qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.
raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.
(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.
Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)
Quoted 3 lines
It looks like
bdrv_filtered_child()is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.
It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.
Quoted 10 lines
Quoted 4 lines
+/*
+ * Return the child that @bs acts as an overlay for, and from which data may be
+ * copied in COW or COR operations. Usually this is the backing file.
+ */Or NULL, if no such child exists.
It's relatively obvious here, but for some of the functions further down it would be really good to describe in which cases NULL is expected (or that NULL is even a possible return value).
I’ll look into it.
Max
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 20 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 7 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.Quoted 7 lines
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?
Quoted 5 lines
Quoted 1 line
Apparently any child of a filter node is "filtered" (which makes sense),
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?
Quoted 9 lines
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
But there is no reason why a node couldn't fulfill this condition for more than one child node. bdrv_filtered_child() isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Quoted 5 lines
The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.
We generally call this concept a "backing chain".
Quoted 6 lines
qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.
raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.
Quoted 8 lines
(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.
Agreed with offset, but with only size, it matches your definition of a filter.
Quoted 11 lines
Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)Quoted 3 lines
It looks like
bdrv_filtered_child()is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.
qcow2 doesn't fulfill the conditions for begin a filter driver. Two of its possible children fulfill the conditions for being a filtered child. You can pick either approach, talking about a "filter chain" just doesn't make sense there. Either the chain is broken by a non-filter driver like qcow2, or it must become a filter tree.
What we're really interested in is iterating the backing chain even across filter nodes, so your implementation achieves the right result. It just feels completely arbitrary, counterintuitive and confusing to call this a (or actually "the") "filter chain" and to pretend that the name tells anyone what it really is.
Kevin
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Quoted 24 lines · Kevin Wolf
Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:
Quoted 23 lines · Kevin Wolf
Quoted 21 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 7 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.Quoted 7 lines
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?Apparently any child of a filter node is "filtered" (which makes sense),
Quoted 7 lines
Quoted 3 lines
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?
Well, there’s Quorum...
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
Quoted 5 lines
But there is no reason why a node couldn't fulfill this condition for more than one child node.
bdrv_filtered_child()isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.
Which is why it currently returns NULL for Quorum.
Quoted 2 lines
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?
Quoted 7 lines
Quoted 5 lines
The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.
We generally call this concept a "backing chain".
I suppose that’s an exclusive “we”? Because I use ”backing chain” to refer to COW chains exclusively.
Such a chain may or may not include filters, but they are not really load-bearing nodes of the chain. As such, I generally want to skip them when looking at a backing chain (hence e.g. bdrv_backing_chain_next()).
From what I can tell nobody has ever formalized any terms regarding COW backing chains or R/W filter chains. This series is an attempt.
Quoted 15 lines
Quoted 6 lines
qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.
raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.
Quoted 8 lines
(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.
Quoted 2 lines
Agreed with offset, but with only size, it matches your definition of a filter.
So?
Should we treat it as a filter when @offset is 0 but otherwise not? That totally wouldn’t be confusing to users.
Quoted 9 lines
Quoted 9 lines
Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)Quoted 3 lines
It looks like
bdrv_filtered_child()is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.
Quoted 8 lines
Quoted 2 lines
It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.
qcow2 doesn't fulfill the conditions for begin a filter driver. Two of its possible children fulfill the conditions for being a filtered child. You can pick either approach, talking about a "filter chain" just doesn't make sense there. Either the chain is broken by a non-filter driver like qcow2, or it must become a filter tree.
I have no idea what your point is. There is no point in making it a filter tree at this point, just as we never had a backing tree.
And the good example is Quorum. qcow2 is a bad example because there is no benefit in marking it an R/W filter for its external data file, exactly like is the case for raw.
Quoted 5 lines
What we're really interested in is iterating the backing chain even across filter nodes, so your implementation achieves the right result. It just feels completely arbitrary, counterintuitive and confusing to call this a (or actually "the") "filter chain" and to pretend that the name tells anyone what it really is.
So exactly the same as “bs->backing” or “backing chain” for me.
You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.
The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.
Max
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:
Quoted 29 lines · Kevin Wolf
Quoted 28 lines
Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:
Quoted 26 lines · Kevin Wolf
Quoted 13 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 12 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Quoted 8 lines
Quoted 2 lines
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?Apparently any child of a filter node is "filtered" (which makes sense),
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Quoted 5 lines
Quoted 3 lines
Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?
Well, there’s Quorum...
Ah, nice, quorum sets is_filter = true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.
So either quorum lies and isn't actually a filter driver, or our definition in the documentation of is_filter is wrong.
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
Quoted 7 lines
Quoted 5 lines
But there is no reason why a node couldn't fulfill this condition for more than one child node.
bdrv_filtered_child()isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.Which is why it currently returns NULL for Quorum.
Which is about the only possible choice that breaks the contract...
* Return any filtered child, independently of how it reacts to write
* accesses and whether data is copied onto this BDS through COR.
Maybe the documentation of bdrv_filtered_child() needs to be rephrased?
Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too. So if quorum returns NULL to mean "no unambiguous result", why does it return bs->backing instead of NULL for a qcow2 node?
(Yes, I know, because it's useful. But I'm trying to get some basic consistency into these interfaces.)
Quoted 5 lines
Quoted 2 lines
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?
Wait, where is there even a place where this could be declared?
The once thing I see that a driver even can declare is drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.
Then you made some criteria above that tell us whether a specific child of a node is a filtered child or not. As it happens, qcow2 (which is not a filter driver) can have two children that match the criteria for being filtered children.
I already think this is a bit inconsistent, because why should a driver that declares itself a non-filter be considered to filter children? Okay, you say a broader definition of a filtered child is useful because you can then include all BdrvChild links in a backing/filter chain. Fair enough, it's not intuitive, but use a broader definition then.
But the point where you say that even though two of the children are filtered children under your broader definition, for the purpose of the API only one of them should be considered because the other one isn't that useful, that's really one inconsistency too much for me. You can't use a broad definition and then arbitrarily restrict the definition again so that it matches the special case you're currently interested in.
Either use a narrow definition, or use a broad one. But use only one and use it consistently.
Quoted 11 lines
Quoted 7 lines
Quoted 5 lines
The main reason behind the common “filtered” name is for the generic functions that work on both COW and true filter (R/W filters) chains. We need such functionality sometimes. I personally felt like the concept of true (R/W) filters and COW children was similar enough to share a common name base.
We generally call this concept a "backing chain".
I suppose that’s an exclusive “we”? Because I use ”backing chain” to refer to COW chains exclusively.
Quoted 6 lines
Such a chain may or may not include filters, but they are not really load-bearing nodes of the chain. As such, I generally want to skip them when looking at a backing chain (hence e.g.
bdrv_backing_chain_next()).From what I can tell nobody has ever formalized any terms regarding COW backing chains or R/W filter chains. This series is an attempt.
Well, as you can see, this attempt feels confusing to me.
I agree with your naming of bdrv_backing_chain_next(), it's clear enough what path it will follow down the graph. I just disagree that "filter chain" is a good term for something that prefers backing file links when it has a choice.
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
qcow2 has a COW child. As such, it acts as a COW filter in the sense of the function names.
raw has neither a COW child nor acts as an R/W filter. As such, it has no filtered child. My opinion on this hasn’t changed.
(To reiterate, in practice I see no way anyone would ever use raw as an R/W filter.
Either you use it without offset/size, in which case you simply use it in lieu of a format node, so you precisely don’t want it to act as a filter when it comes to allocation information and so on (even though it can be classified a filter here).
Or you use it as kind of a filter with offset/size, but then it no longer is a filter.
Quoted 7 lines
Quoted 2 lines
Agreed with offset, but with only size, it matches your definition of a filter.
So?
Should we treat it as a filter when @offset is 0 but otherwise not? That totally wouldn’t be confusing to users.
No, I'm just applying your definitions to see if the contradictions between them and your explanations are of any importance. *shrug*
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Filters are defined by “Every filter must fulfill these conditions: ...”
– not by “Everything that fulfills these conditions is a filter”. Marking a driver as a filter has consequences, and I don’t see why we would want those consequences for raw.)Quoted 3 lines
It looks like
bdrv_filtered_child()is the right function to iterate along a backing file chain, but I just still fail to connect that and the name of the function in a meaningful way.Quoted 8 lines
Quoted 2 lines
It‘s the right function to iterate along a filter chain. This includes COW backing children and R/W filtered children.
qcow2 doesn't fulfill the conditions for begin a filter driver. Two of its possible children fulfill the conditions for being a filtered child. You can pick either approach, talking about a "filter chain" just doesn't make sense there. Either the chain is broken by a non-filter driver like qcow2, or it must become a filter tree.
Quoted 6 lines
I have no idea what your point is. There is no point in making it a filter tree at this point, just as we never had a backing tree.
And the good example is Quorum. qcow2 is a bad example because there is no benefit in marking it an R/W filter for its external data file, exactly like is the case for raw.
My point is not about changing the logic in the code, but about using names that actually describe accurately what the logic does.
And as I said above, neither is "not useful" a convincing argument for ignoring filtered children (as I think we're trying to build something rather generic, not something that works only for what we consider useful today) nor do I see how qcow2 could be marked or not marked an R/W filter (as mentioned above).
Quoted 6 lines
Quoted 5 lines
What we're really interested in is iterating the backing chain even across filter nodes, so your implementation achieves the right result. It just feels completely arbitrary, counterintuitive and confusing to call this a (or actually "the") "filter chain" and to pretend that the name tells anyone what it really is.
Quoted 8 lines
So exactly the same as “bs->backing” or “backing chain” for me.
You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.
The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.
Well, if "backing chain" is too narrow, "filter chain" is both too unspecific and inconsistent with the various definitions of "filter" and "filtered" we're using, and we can't think of anything more concise, we might have to use names that just mention both.
bdrv_cow_child() // don't call COW a filter, because .is_filter = false bdrv_filter_child() // your R/W filter, only for .is_filter = true nodes bdrv_filter_or_cow_child()
Or something like that. This would bring some more consistency into the way we use the words filter/filtered at least.
Kevin
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Quoted 34 lines · Kevin Wolf
Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 31 lines
Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 20 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 7 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.Quoted 7 lines
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?Quoted 5 lines
Quoted 1 line
Apparently any child of a filter node is "filtered" (which makes sense),
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?
Quoted 8 lines
Quoted 1 line
Well, there’s Quorum...
Ah, nice, quorum sets
is_filter= true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.So either quorum lies and isn't actually a filter driver, or our definition in the documentation of
is_filteris wrong.
You could say it lies because in FIFO mode it clearly isn’t a filter for all of its children.
There is a reason for lying, though, which is bdrv_recurse_is_first_non_filter(), which is necessary to use the whole to_replace mirror stuff.
(You mirror from a quorum with a failed child and then replace the failed child. mirror needs to ensure that there are only R/W filters between the child and the mirror source so that replacing it will not suddenly change any visible data. Which is actually a lie for quorum, because the child is clearly broken and thus precisely doesn’t show the same data...)
Maybe we should stop declaring Quorum a filter and then rename the bdrv_recurse_is_first_non_filter() to, I don’t know, bdrv_recurse_can_be_replaced_by_mirror()?
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 9 lines
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
But there is no reason why a node couldn't fulfill this condition for more than one child node.
bdrv_filtered_child()isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.
Quoted 5 lines
Quoted 1 line
Which is why it currently returns NULL for Quorum.
Which is about the only possible choice that breaks the contract...
* Return any filtered child, independently of how it reacts to write
I don’t know if you’re serious about this proposition, because I don’t know whether that could be useful in any way. :-?
Quoted 1 line
* accesses and whether data is copied onto this BDS through COR.
I meant the contract as “Return the single filtered child there is, or NULL”
Quoted 4 lines
Maybe the documentation of
bdrv_filtered_child()needs to be rephrased?Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.
Well, it doesn’t. It isn’t an R/W filter.
Maybe what we actually need to rephrase is the definition of .is_filter.
(Namely something along the lines of “Fulfills these guarantees (same
data, etc. pp.), *and* should be skipped for allocation information queries etc.”.
Quoted 6 lines
So if quorum returns NULL to mean "no
unambiguous result", why does it returnbs->backinginstead of NULL for a qcow2 node?(Yes, I know, because it's useful. But I'm trying to get some basic consistency into these interfaces.)
Not precisely because it’s useful, but because qcow2 does not have .is_filter set. :-)
(And it doesn’t have it set because that wouldn’t be useful.)
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?
Wait, where is there even a place where this could be declared?
The once thing I see that a driver even can declare is
drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.
That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.
Quoted 4 lines
Then you made some criteria above that tell us whether a specific child of a node is a filtered child or not. As it happens, qcow2 (which is not a filter driver) can have two children that match the criteria for being filtered children.
But just arguing that I’m incapable of giving a good definition won’t bring us along.
Quoted 2 lines
I already think this is a bit inconsistent, because why should a driver that declares itself a non-filter be considered to filter children?
.is_filter is for R/W filters. COW filters have .supports_backing for that.
Quoted 12 lines
Okay, you say a broader definition of a filtered child is useful because you can then include all BdrvChild links in a backing/filter chain. Fair enough, it's not intuitive, but use a broader definition then.
But the point where you say that even though two of the children are filtered children under your broader definition, for the purpose of the API only one of them should be considered because the other one isn't that useful, that's really one inconsistency too much for me. You can't use a broad definition and then arbitrarily restrict the definition again so that it matches the special case you're currently interested in.
Quoted 2 lines
Either use a narrow definition, or use a broad one. But use only one and use it consistently.
I think the problem appears because you restrict the process to a single step where there’s actually two.
Drivers can be either
(1) R/W filters (e.g. throttle)
(2) COW filters (e.g. qcow2)
(3) None of the above (e.g. vhdx, curl)
This choice is made on the driver level, not on the node level (for good reason, see below*).
And then we derive the node’s filtered children from what the driver is.
If it’s an R/W filter, bdrv_filtered_child() will return the R/W
filtered child. If it’s a COW filter, bdrv_filtered_child() will return the potentially existing COW backing child (or NULL, if there is no backing child).
*
What is clear to me is that it isn’t useful to treat nodes of a specific driver sometimes as R/W filter nodes and sometimes not. R/W filter nodes are handled differently from other nodes, and it would be confusing if a certain driver sometimes behaves this and sometimes that way. (For example, if you put a raw node on top of a qcow2 node, sometimes it would stop the backing chain, sometimes it wouldn’t. That makes no sense to me.)
OTOH, for COW filters, we do exactly that. Sometimes they have a backing file, sometimes they don’t. That’s completely fine because their overall behavior doesn’t change.
That makes me agree that there is indeed too much of a difference between R/W filters and COW filters to lump them together under the “filter” label.
[...]
Quoted 2 lines
My point is not about changing the logic in the code, but about using names that actually describe accurately what the logic does.
Again, naming things is hard.
[...]
Quoted 12 lines
Quoted 6 lines
You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.
The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.
Well, if "backing chain" is too narrow, "filter chain" is both too unspecific and inconsistent with the various definitions of "filter" and "filtered" we're using, and we can't think of anything more concise, we might have to use names that just mention both.
Quoted 6 lines
bdrv_cow_child()// don't call COW a filter, because .is_filter = falsebdrv_filter_child()// your R/W filter, only for .is_filter = true nodesbdrv_filter_or_cow_child()Or something like that. This would bring some more consistency into the way we use the words filter/filtered at least.
I’ll see how that looks overall, but why not. Sounds good to me.
Max
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:
Quoted 44 lines · Kevin Wolf
Quoted 34 lines
Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 31 lines
Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 20 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 7 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.Quoted 7 lines
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?Quoted 5 lines
Quoted 1 line
Apparently any child of a filter node is "filtered" (which makes sense),
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?
Quoted 8 lines
Quoted 1 line
Well, there’s Quorum...
Ah, nice, quorum sets
is_filter= true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.So either quorum lies and isn't actually a filter driver, or our definition in the documentation of
is_filteris wrong.
Quoted 6 lines
You could say it lies because in FIFO mode it clearly isn’t a filter for all of its children.
There is a reason for lying, though, which is
bdrv_recurse_is_first_non_filter(), which is necessary to use the wholeto_replacemirror stuff.
Hm, actually, now that you mention bdrv_recurse_is_first_non_filter(), quorum was the first driver to declare itself a filter, so strictly speaking, if there is an inconsistency, it's the other uses that are abusing the field...
Quoted 10 lines
(You mirror from a quorum with a failed child and then replace the failed child. mirror needs to ensure that there are only R/W filters between the child and the mirror source so that replacing it will not suddenly change any visible data. Which is actually a lie for quorum, because the child is clearly broken and thus precisely doesn’t show the same data...)
Maybe we should stop declaring Quorum a filter and then rename the
bdrv_recurse_is_first_non_filter()to, I don’t know,bdrv_recurse_can_be_replaced_by_mirror()?
Why not.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Quoted 9 lines
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
But there is no reason why a node couldn't fulfill this condition for more than one child node.
bdrv_filtered_child()isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Which is why it currently returns NULL for Quorum.
Which is about the only possible choice that breaks the contract...
* Return any filtered child, independently of how it reacts to write
I don’t know if you’re serious about this proposition, because I don’t know whether that could be useful in any way. :-?
Huh? This is just quoting the contract from your code?
Quoted 3 lines
Quoted 1 line
* accesses and whether data is copied onto this BDS through COR.
I meant the contract as “Return the single filtered child there is, or NULL”
Then that should probably be spelt out in the contract. Probably even explicitly "NULL if there is either no filtered child or multiple filtered children".
Quoted 6 lines
Quoted 4 lines
Maybe the documentation of
bdrv_filtered_child()needs to be rephrased?Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.
Well, it doesn’t. It isn’t an R/W filter.
What do I have to look at to see whether something is an R/W filter or not? qcow2 matches your criteria for an R/W filter. You say that it's not useful, so it's not an R/W filter anyway. But where in the code could I get this information?
This just doesn't make sense to me. If a driver matches the criteria for an R/W filter, then it should be one. If qcow2 should not be considered a R/W filter, then the criteria must be changed so that it isn't.
Quoted 4 lines
Maybe what we actually need to rephrase is the definition of .is_filter.
(Namely something along the lines of “Fulfills these guarantees (same
data, etc. pp.), *and* should be skipped for allocation information queries etc.”.
Hm - does this imply that .is_filter == this is a R/W filter? Because this was never spelt out, neither in code comments nor in commit messages.
If we called R/W filters just "filters" (which makes it obvious how it relates to .is_filter) and COW nodes something that doesn't include the word "filter", things might become a lot clearer.
Quoted 17 lines
Quoted 6 lines
So if quorum returns NULL to mean "no
unambiguous result", why does it returnbs->backinginstead of NULL for a qcow2 node?(Yes, I know, because it's useful. But I'm trying to get some basic consistency into these interfaces.)
Not precisely because it’s useful, but because qcow2 does not have .is_filter set. :-)
(And it doesn’t have it set because that wouldn’t be useful.)Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?
Quoted 8 lines
Quoted 5 lines
Wait, where is there even a place where this could be declared?
The once thing I see that a driver even can declare is
drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.
I was thinking per-child, actually, because you declare one BdrvChild filtered and another not filtered.
But by now I think most of the confusion is really just a result of COW being considered a filter in some respects (mainly just the names of the child access functions), but not in others (like .is_filter).
Quoted 5 lines
Quoted 4 lines
Then you made some criteria above that tell us whether a specific child of a node is a filtered child or not. As it happens, qcow2 (which is not a filter driver) can have two children that match the criteria for being filtered children.
Quoted 7 lines
But just arguing that I’m incapable of giving a good definition won’t bring us along.
Quoted 2 lines
I already think this is a bit inconsistent, because why should a driver that declares itself a non-filter be considered to filter children?
.is_filter is for R/W filters. COW filters have .supports_backing for that.
Okay, so you confirm what I concluded above.
Quoted 18 lines
Quoted 12 lines
Okay, you say a broader definition of a filtered child is useful because you can then include all BdrvChild links in a backing/filter chain. Fair enough, it's not intuitive, but use a broader definition then.
But the point where you say that even though two of the children are filtered children under your broader definition, for the purpose of the API only one of them should be considered because the other one isn't that useful, that's really one inconsistency too much for me. You can't use a broad definition and then arbitrarily restrict the definition again so that it matches the special case you're currently interested in.
Quoted 2 lines
Either use a narrow definition, or use a broad one. But use only one and use it consistently.
I think the problem appears because you restrict the process to a single step where there’s actually two.
Quoted 7 lines
Drivers can be either
(1) R/W filters (e.g. throttle)
(2) COW filters (e.g. qcow2)
(3) None of the above (e.g. vhdx, curl)This choice is made on the driver level, not on the node level (for good reason, see below*).
What prevents a driver from being
(4) COW filter and R/W filter (e.g. qcow2 if it were useful)?
I mean, conceptually, not in the implementation.
Quoted 5 lines
And then we derive the node’s filtered children from what the driver is.
If it’s an R/W filter,bdrv_filtered_child()will return the R/W
filtered child. If it’s a COW filter,bdrv_filtered_child()will return the potentially existing COW backing child (or NULL, if there is no backing child).
I guess it boils down to me just not being able to get behind the concept that COW is some sort of filter (especially when other things like an external data file aren't).
Quoted 39 lines
*
What is clear to me is that it isn’t useful to treat nodes of a specific driver sometimes as R/W filter nodes and sometimes not. R/W filter nodes are handled differently from other nodes, and it would be confusing if a certain driver sometimes behaves this and sometimes that way. (For example, if you put a raw node on top of a qcow2 node, sometimes it would stop the backing chain, sometimes it wouldn’t. That makes no sense to me.)OTOH, for COW filters, we do exactly that. Sometimes they have a backing file, sometimes they don’t. That’s completely fine because their overall behavior doesn’t change.
That makes me agree that there is indeed too much of a difference between R/W filters and COW filters to lump them together under the “filter” label.
[...]
Quoted 2 lines
My point is not about changing the logic in the code, but about using names that actually describe accurately what the logic does.
Again, naming things is hard.
[...]
Quoted 12 lines
Quoted 6 lines
You disagreeing with me on these terms to me shows that there is a need to formalize. This is precisely what I want to do in this series.
The fact that we don’t use the term “filter chain” so far is the reason why I introduce it. Because it comes as a clean slate. “backing chain” already means something to me, and it means something different.
Well, if "backing chain" is too narrow, "filter chain" is both too unspecific and inconsistent with the various definitions of "filter" and "filtered" we're using, and we can't think of anything more concise, we might have to use names that just mention both.
Quoted 8 lines
Quoted 6 lines
bdrv_cow_child()// don't call COW a filter, because .is_filter = falsebdrv_filter_child()// your R/W filter, only for .is_filter = true nodesbdrv_filter_or_cow_child()Or something like that. This would bring some more consistency into the way we use the words filter/filtered at least.
I’ll see how that looks overall, but why not. Sounds good to me.
Good. Or, well, good enough at least. ;-)
bdrv_filter_or_cow_child() is not a pretty name, but as long as we can't think of anything that accurately covers both in a single word, it will do the job...
Kevin
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Quoted 64 lines · Kevin Wolf
Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:
Quoted 44 lines · Kevin Wolf
Quoted 34 lines
Am 09.09.2019 um 16:04 hat Max Reitz geschrieben:
Quoted 33 lines · Kevin Wolf
Quoted 31 lines
Am 09.09.2019 um 09:56 hat Max Reitz geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 20 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 7 lines
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be an R/W-filtered child;bs->filecan be an R/W-filtered child, it can be data and metadata storage, or it can be just metadata storage.Quoted 7 lines
This overloading really is not helpful. This patch adds function that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes and external data files in a meaningful way.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Each time I look at this patch, I'm confused by the function names. Maybe I should just ask what the idea there was, or more specifically:
What does the "filtered" in "filtered child" really mean?Quoted 5 lines
Quoted 1 line
Apparently any child of a filter node is "filtered" (which makes sense),
It isn’t, filters can have non-filter children. For example, backup-top could have the source as a filtered child and the target as a non-filter child.
Hm, okay, makes sense. I had a definition in mind that says that filter nodes only have a single child node. Is it that a filter may have only a single _filtered_ child node?
Quoted 8 lines
Quoted 1 line
Well, there’s Quorum...
Ah, nice, quorum sets
is_filter= true even though it neither fulfulls the conditions for it before this series, nor the changed conditions after this series.So either quorum lies and isn't actually a filter driver, or our definition in the documentation of
is_filteris wrong.Quoted 6 lines
You could say it lies because in FIFO mode it clearly isn’t a filter for all of its children.
There is a reason for lying, though, which is
bdrv_recurse_is_first_non_filter(), which is necessary to use the wholeto_replacemirror stuff.Hm, actually, now that you mention
bdrv_recurse_is_first_non_filter(), quorum was the first driver to declare itself a filter, so strictly speaking, if there is an inconsistency, it's the other uses that are abusing the field...Quoted 7 lines
(You mirror from a quorum with a failed child and then replace the failed child. mirror needs to ensure that there are only R/W filters between the child and the mirror source so that replacing it will not suddenly change any visible data. Which is actually a lie for quorum, because the child is clearly broken and thus precisely doesn’t show the same data...)
Quoted 5 lines
Quoted 3 lines
Maybe we should stop declaring Quorum a filter and then rename the
bdrv_recurse_is_first_non_filter()to, I don’t know,bdrv_recurse_can_be_replaced_by_mirror()?Why not.
It feels difficult to do in this series because this is a whole new can of worms.
In patch 35, I actually replace the mirror use case by is_filtered_child(). So it looks to me as if that should not be done, because I should instead fix bdrv_recurse_is_first_non_filter() (and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.
OTOH, there are other users of bdrv_is_first_non_filter(). Those are qmp_block_resize() and external_snapshot_prepare(), who throw an error if that returns false.
I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system). qmp_block_resize() should always work as long as it can get BLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
but also
bs->backingof a qcow2 image, whilebs->fileof qcow2 isn't. raw doesn't have any "filtered" child. What's the system behind this?“filtered” means: If the parent node returns data from this child, it won’t modify it, neither its content nor its position. COW and R/W filters differ in how they handle writes; R/W filters pass them through to the filtered child, COW filters copy them off to some other child node (and then the filtered child’s data will no longer be visible at that location).
Quoted 5 lines
But there is no reason why a node couldn't fulfill this condition for more than one child node.
bdrv_filtered_child()isn't well-defined then. Technically, the description "Return any filtered child" is correct because "any" can be interpreted as "an arbitrary", but obviously that makes the function useless.Which is why it currently returns NULL for Quorum.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Which is about the only possible choice that breaks the contract...
* Return any filtered child, independently of how it reacts to write
I don’t know if you’re serious about this proposition, because I don’t know whether that could be useful in any way. :-?
Huh? This is just quoting the contract from your code?
I see. I was thinking about “any of COW/RW, of which only one exists”.
There is an assertion for that (that only one filtered child exists at
a time) in the code. (And I consider assertions part of the contract.)
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
* accesses and whether data is copied onto this BDS through COR.
I meant the contract as “Return the single filtered child there is, or NULL”
Then that should probably be spelt out in the contract.Probably even explicitly "NULL if there is either no filtered child or multiple filtered children".
Well, it’s spelled out through the assertion, but not in the documentation, yes.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Maybe the documentation of
bdrv_filtered_child()needs to be rephrased?Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.
Well, it doesn’t. It isn’t an R/W filter.
What do I have to look at to see whether something is an R/W filter or not? qcow2 matches your criteria for an R/W filter.
No. Some qcow2 nodes match the criteria. But not all, which makes the qcow2 driver not a filter driver.
Quoted 3 lines
You say that it's
not useful, so it's not an R/W filter anyway. But where in the code could I get this information?
“Where in the code”? Do you want to add a comment to every BlockDriver structure on why it does or doesn’t set .is_filter?
Quoted 3 lines
This just doesn't make sense to me. If a driver matches the criteria for an R/W filter, then it should be one. If qcow2 should not be considered a R/W filter, then the criteria must be changed so that it isn't.
See below.
Quoted 8 lines
Quoted 4 lines
Maybe what we actually need to rephrase is the definition of .is_filter.
(Namely something along the lines of “Fulfills these guarantees (same
data, etc. pp.), *and* should be skipped for allocation information queries etc.”.Hm - does this imply that .is_filter == this is a R/W filter? Because this was never spelt out, neither in code comments nor in commit messages.
While I’m not a fan of comment-less code, I do think that it’s possible to read code. Which clearly stated this.
Quoted 3 lines
If we called R/W filters just "filters" (which makes it obvious how it relates to .is_filter) and COW nodes something that doesn't include the word "filter", things might become a lot clearer.
Because you apparently wrote this before reading that I agreed to your renaming proposal, I now feel free to argue that I could just as well rename .is_filter to .is_rw_filter.
Obviously I won’t because I prefer your proposal.
[...]
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?
Wait, where is there even a place where this could be declared?
The once thing I see that a driver even can declare is
drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.
Quoted 5 lines
Quoted 2 lines
That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.
I was thinking per-child, actually, because you declare one BdrvChild filtered and another not filtered.
Why don’t you say so from the start then?
(Sorry, but honestly about 30 % of this discussion to me feels like you’re playing games with me. Please don’t take this the wrong way, I mean it very neutrally. It’s just that I feel like I’m explaining things to you that you very much know, but you just want me to say them.
And that feels unproductive and sometimes indeed frustrating.)
One thing is that this wouldn’t make the quorum case any easier because it actually doesn’t know for which children it acts as a filter and for which it doesn’t.
Quoted 3 lines
But by now I think most of the confusion is really just a result of COW being considered a filter in some respects (mainly just the names of the child access functions), but not in others (like .is_filter).
I don’t quite see how it’s “by now” when in your first mail you already basically wrote that functionally, everything works (leaving out quorum), but that you’re confused (or claim to be confused, I have no idea what’s real and what’s pretended anymore) by the names.
We have come to two results, as far as I can see:
First, naming COW backing nodes “COW filtered children” clashes with our existing use of ”filter”. There is no point in forcing the ”filter” label on everything. We can just keep calling (R/W) filters filters and COW backing children COW children. The names are succinct enough.
In some cases, we don’t care whether something is a COW or filtered child, in such a case a caller can be bothered to use the slightly longer bdrv_cow_or_filtered_child().
Second, most of the time we want a filter node to have a clear and unique path to go down. This is the important property of filters: That you can skip them and go to the node that actually has the data.
Quorum breaks this by having multiple children, and nobody knows which of them has the data we will see on the next read operation.
All “filters” who could have multiple children would have this problem.
Hence a filter must always have a single unique data child. I think.
[...]
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
Either use a narrow definition, or use a broad one. But use only one and use it consistently.
I think the problem appears because you restrict the process to a single step where there’s actually two.
Drivers can be either
(1) R/W filters (e.g. throttle)
(2) COW filters (e.g. qcow2)
(3) None of the above (e.g. vhdx, curl)
Quoted 7 lines
Quoted 2 lines
This choice is made on the driver level, not on the node level (for good reason, see below*).
What prevents a driver from being
(4) COW filter and R/W filter (e.g. qcow2 if it were useful)?I mean, conceptually, not in the implementation.
An R/W filter always shows the same data as the filtered child. So the COW child‘s data can never be visible, and as such you couldn’t have a COW child at the same time.
Max
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Am 10.09.2019 um 13:36 hat Max Reitz geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 6 lines
Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:
Quoted 3 lines
Maybe we should stop declaring Quorum a filter and then rename the
bdrv_recurse_is_first_non_filter()to, I don’t know,bdrv_recurse_can_be_replaced_by_mirror()?Why not.
It feels difficult to do in this series because this is a whole new can of worms.
In patch 35, I actually replace the mirror use case by
is_filtered_child(). So it looks to me as if that should not be done, because I should instead fixbdrv_recurse_is_first_non_filter()(and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.OTOH, there are other users of
bdrv_is_first_non_filter(). Those areqmp_block_resize()andexternal_snapshot_prepare(), who throw an error if that returns false.
Quoted 5 lines
I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system).
qmp_block_resize()should always work as long as it can getBLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.
Hm, doesn't it make sense in a way for qmp_block_resize() at least? It means that you can't resize just a filter, but you need to resize the image that actually provides the data for the filter.
Of course, there is no reason for it to be the _first_ non-filter as long as BLK_PERM_RESIZE is shared, but just some non-filter node.
Two more random observations:
* quorum uses bdrv_filter_default_perms(), which allows BLK_PERM_RESIZE. I think this is wrong and quorum should make sure that all children are always the same size because otherwise it can't tell what its own size is. (Or vote on size...? :-/) Probably not a problem in practice as long as we check bdrv_is_first_non_filter().
* child_file and child_backing don't implement .resize. So if you resize a non-top-level image, parents (in particular filters) don't get their size adjusted. This is probably a bug, too, but one that isn't prevented by bdrv_is_first_non_filter() and should be visible today.
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Maybe the documentation of
bdrv_filtered_child()needs to be rephrased?Going back to qcow2, it's really not much different as it has multiple (two) filtered children, too.
Well, it doesn’t. It isn’t an R/W filter.
What do I have to look at to see whether something is an R/W filter or not? qcow2 matches your criteria for an R/W filter.
Quoted 8 lines
No. Some qcow2 nodes match the criteria. But not all, which makes the qcow2 driver not a filter driver.
Quoted 2 lines
You say that it's not useful, so it's not an R/W filter anyway. But where in the code could I get this information?
“Where in the code”? Do you want to add a comment to every BlockDriver structure on why it does or doesn’t set .is_filter?
Never mind, I just didn't understand that .is_filter is the thing that defines a R/W filter. In fact, I didn't really understand what .is_filter was supposed to mean at all because I was so confused. For some reason I was sure it had to mean any kind of filter, but that assumption just didn't match up with its use at all.
Quoted 12 lines
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
Specficially, according to your definition, qcow2 filters both the backing file (COW filter) and the external data file (R/W filter).
Not wrong. But the same question as for raw arises: Is there any use to declaring qcow2 an R/W filter driver just because it fits the definition?
Wait, where is there even a place where this could be declared?
The once thing I see that a driver even can declare is
drv->is_filter, which is about the whole driver and not about nodes. It is false for qcow2.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
That’s correct. But that’s not a fundamental problem, of course, we could make it a per-BDS attribute if that made sense.
I was thinking per-child, actually, because you declare one BdrvChild filtered and another not filtered.
Why don’t you say so from the start then?
Yes, I wrote "nodes", thought "child nodes" and should have said "children" because edges are not nodes. My bad, sorry.
Quoted 5 lines
(Sorry, but honestly about 30 % of this discussion to me feels like you’re playing games with me. Please don’t take this the wrong way, I mean it very neutrally. It’s just that I feel like I’m explaining things to you that you very much know, but you just want me to say them.
And that feels unproductive and sometimes indeed frustrating.)
No, certainly not. If my mails seemed confusing or pointless, it just shows how thoroughly confused I was.
Quoted 4 lines
One thing is that this wouldn’t make the quorum case any easier because it actually doesn’t know for which children it acts as a filter and for which it doesn’t.
Quoted 8 lines
Quoted 3 lines
But by now I think most of the confusion is really just a result of COW being considered a filter in some respects (mainly just the names of the child access functions), but not in others (like .is_filter).
I don’t quite see how it’s “by now” when in your first mail you already basically wrote that functionally, everything works (leaving out quorum), but that you’re confused (or claim to be confused, I have no idea what’s real and what’s pretended anymore) by the names.
Well, I saw that the special cases in the patches that I had reviewed so far seemed to be converted correctly, but I just didn't understand the whole concept behind it. It's possible to both understand that a transformation is correct and to fail to grasp the concept behind it.
And your first answer only confused me more because you gave definitions for R/W and COW filters that honestly ended up a bit misleading, possibly as a result of your endeavour to make R/W filters and COW sound like the same thing. (Which made me lose sight of basic facts like that R/W filters must forward _every_ request without exception to their filtered child even though COW doesn't.)
Quoted 10 lines
We have come to two results, as far as I can see:
First, naming COW backing nodes “COW filtered children” clashes with our existing use of ”filter”. There is no point in forcing the ”filter” label on everything. We can just keep calling (R/W) filters filters and COW backing children COW children. The names are succinct enough.
In some cases, we don’t care whether something is a COW or filtered child, in such a case a caller can be bothered to use the slightly longer
bdrv_cow_or_filtered_child().
Aye.
Quoted 9 lines
Second, most of the time we want a filter node to have a clear and unique path to go down. This is the important property of filters: That you can skip them and go to the node that actually has the data.
Quorum breaks this by having multiple children, and nobody knows which of them has the data we will see on the next read operation.
All “filters” who could have multiple children would have this problem.
Hence a filter must always have a single unique data child. I think.
I agree, and this is the condition that I mentioned somewhere above, but failed to actually find guaranteed somewhere. We should probably make this explicit.
Of course, quorum and similar things intend all their children to provide the same data, but the whole point of the driver is that this is not always guaranteed, so they aren't actually filters.
Kevin
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Quoted 28 lines · Kevin Wolf
Am 10.09.2019 um 13:36 hat Max Reitz geschrieben:
Quoted 21 lines · Kevin Wolf
Quoted 6 lines
Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:
Quoted 3 lines
Maybe we should stop declaring Quorum a filter and then rename the
bdrv_recurse_is_first_non_filter()to, I don’t know,bdrv_recurse_can_be_replaced_by_mirror()?Why not.
It feels difficult to do in this series because this is a whole new can of worms.
In patch 35, I actually replace the mirror use case by
is_filtered_child(). So it looks to me as if that should not be done, because I should instead fixbdrv_recurse_is_first_non_filter()(and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.OTOH, there are other users of
bdrv_is_first_non_filter(). Those areqmp_block_resize()andexternal_snapshot_prepare(), who throw an error if that returns false.Quoted 5 lines
I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system).
qmp_block_resize()should always work as long as it can getBLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.
Quoted 3 lines
Hm, doesn't it make sense in a way for
qmp_block_resize()at least? It means that you can't resize just a filter, but you need to resize the image that actually provides the data for the filter.
Filters generally implement .bdrv_truncate() by passing it through, so it should be fine.
Quoted 10 lines
Of course, there is no reason for it to be the _first_ non-filter as long as
BLK_PERM_RESIZEis shared, but just some non-filter node.Two more random observations:
* quorum uses
bdrv_filter_default_perms(), which allowsBLK_PERM_RESIZE. I think this is wrong and quorum should make sure that all children are always the same size because otherwise it can't tell what its own size is. (Or vote on size...? :-/) Probably not a problem in practice as long as we checkbdrv_is_first_non_filter().
(“Quorum is broken” seems to be a recurring observation.)
I agree, it shouldn’t share that permission.
Quoted 4 lines
*
child_fileandchild_backingdon't implement .resize. So if you resize a non-top-level image, parents (in particular filters) don't get their size adjusted. This is probably a bug, too, but one that isn't prevented bybdrv_is_first_non_filter()and should be visible today.
Hm. :-/
The good news is that I can try to fix this independently of this series.
[...]
Quoted 23 lines
Quoted 10 lines
We have come to two results, as far as I can see:
First, naming COW backing nodes “COW filtered children” clashes with our existing use of ”filter”. There is no point in forcing the ”filter” label on everything. We can just keep calling (R/W) filters filters and COW backing children COW children. The names are succinct enough.
In some cases, we don’t care whether something is a COW or filtered child, in such a case a caller can be bothered to use the slightly longer
bdrv_cow_or_filtered_child().Aye.
Quoted 9 lines
Second, most of the time we want a filter node to have a clear and unique path to go down. This is the important property of filters: That you can skip them and go to the node that actually has the data.
Quorum breaks this by having multiple children, and nobody knows which of them has the data we will see on the next read operation.
All “filters” who could have multiple children would have this problem.
Hence a filter must always have a single unique data child. I think.
Quoted 7 lines
I agree, and this is the condition that I mentioned somewhere above, but failed to actually find guaranteed somewhere. We should probably make this explicit.
Of course, quorum and similar things intend all their children to provide the same data, but the whole point of the driver is that this is not always guaranteed, so they aren't actually filters.
OK, great, I’ll get cracking then.
Max
Re: [Qemu-devel] [PATCH v6 04/42] block: Add child access functions
Am 10.09.2019 um 14:59 hat Max Reitz geschrieben:
Quoted 29 lines · Kevin Wolf
Quoted 28 lines
Am 10.09.2019 um 13:36 hat Max Reitz geschrieben:
Quoted 26 lines · Kevin Wolf
Quoted 6 lines
Am 10.09.2019 um 11:14 hat Max Reitz geschrieben:
Quoted 3 lines
Maybe we should stop declaring Quorum a filter and then rename the
bdrv_recurse_is_first_non_filter()to, I don’t know,bdrv_recurse_can_be_replaced_by_mirror()?Why not.
It feels difficult to do in this series because this is a whole new can of worms.
In patch 35, I actually replace the mirror use case by
is_filtered_child(). So it looks to me as if that should not be done, because I should instead fixbdrv_recurse_is_first_non_filter()(and rename it), because quorum does allow replacing its children by mirror, even if it does not act as a filter for them.OTOH, there are other users of
bdrv_is_first_non_filter(). Those areqmp_block_resize()andexternal_snapshot_prepare(), who throw an error if that returns false.I think that’s just wrong. First of all, I don’t even know why we have that restriction anymore (I can imagine why it used to make sense before the permission system).
qmp_block_resize()should always work as long as it can getBLK_PERM_RESIZE; and I don’t know why the parents of some node would care if you take a snapshot of their child.
Quoted 6 lines
Quoted 3 lines
Hm, doesn't it make sense in a way for
qmp_block_resize()at least? It means that you can't resize just a filter, but you need to resize the image that actually provides the data for the filter.Filters generally implement .bdrv_truncate() by passing it through, so it should be fine.
Good point.
Then checking bdrv_is_first_non_filter() probably just forbids the only command that would actually work correctly (resizing the top-level filter).
Kevin
PATCH v6 5/42 block: Add chain helper functions
In reply to Max ReitzAdd some helper functions for skipping filters in a chain of block nodes.
--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
@@ -1291,6 +1291,9 @@ BdrvChild *bdrv_filtered_child(BlockDriverState *bs);BdrvChild *bdrv_metadata_child (BlockDriverState *bs);BdrvChild *bdrv_storage_child (BlockDriverState *bs);BdrvChild *bdrv_primary_child (BlockDriverState *bs); +BlockDriverState *bdrv_skip_implicit_filters (BlockDriverState *bs); +BlockDriverState *bdrv_skip_rw_filters (BlockDriverState *bs); +BlockDriverState *bdrv_backing_chain_next (BlockDriverState *bs);static inline BlockDriverState *child_bs (BdrvChild *child) {
block.c +55 -0
58 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 05/42] block: Add chain helper functions
Quoted 9 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v6 6/42 qcow2: Implement .bdrv_storage_child()
In reply to Max Reitz--- block/qcow2.c | 9 +++++++++ 1 file changed, 9 insertions(+)
block/qcow2.c +9 -0
@@ -5086,6 +5086,13 @@ void qcow2_signal_corruption(BlockDriverState *bs, bool fatal, int64_t offset, s->signaled_corruption = true; } +static BdrvChild *qcow2_storage_child (BlockDriverState *bs) +{ +BDRVQcow2State *s = bs->opaque; + +return s->data_file; +} +static QemuOptsList qcow2_create_opts = { .name ="qcow2-create-opts" , .head =QTAILQ_HEAD_INITIALIZER (qcow2_create_opts.head),
@@ -5232,6 +5239,8 @@ BlockDriver bdrv_qcow2 = { .bdrv_reopen_bitmaps_rw = qcow2_reopen_bitmaps_rw, .bdrv_can_store_new_dirty_bitmap = qcow2_can_store_new_dirty_bitmap, .bdrv_remove_persistent_dirty_bitmap = qcow2_remove_persistent_dirty_bitmap, + + .bdrv_storage_child = qcow2_storage_child, };static void bdrv_qcow2_init (void )
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 06/42] qcow2: Implement .bdrv_storage_child()
Quoted 6 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/qcow2.c | 9 +++++++++ 1 file changed, 9 insertions(+)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
PATCH v6 7/42 block: *filtered_cow_child() for *has_zero_init()
In reply to Max Reitzbdrv_has_zero_init() and the related bdrv_unallocated_blocks_are_zero() should use bdrv_filtered_cow_child() if they want to check whether the given BDS has a COW backing file.
--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
@@ -5063,7 +5063,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)/* If BS is a copy on write image, it is initialized to the contents of the base image, which may not be zeroes. */ -if (bs->backing) { +if (bdrv_filtered_cow_child (bs)) {return 0 ; }if (bs->drv->bdrv_has_zero_init) {
@@ -5081,7 +5081,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs) {BlockDriverInfo bdi; -if (bs->backing) { +if (bdrv_filtered_cow_child (bs)) {return false; }
Signature · 1 line
2.21.0
PATCH v6 8/42 block: bdrv_set_backing_hd() is about bs->backing
In reply to Max Reitzbdrv_set_backing_hd() is a function that explicitly cares about the bs->backing child. Highlight that in its description and use child_bs(bs->backing) instead of backing_bs(bs) to make it more obvious.
--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
@@ -2539,7 +2539,7 @@ static bool bdrv_inherits_from_recursive(BlockDriverState *child, }/* -* Sets the backing file link of a BDS. A new reference is created; callers +* Sets the bs->backing link of a BDS. A new reference is created; callers * which don't need their own reference any more must call bdrv_unref(). */ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
@@ -2548,7 +2548,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,bool update_inherits_from =bdrv_chain_contains (bs, backing_hd) &&bdrv_inherits_from_recursive (backing_hd, bs); -if (bdrv_is_backing_chain_frozen (bs,backing_bs (bs), errp)) { +if (bdrv_is_backing_chain_frozen (bs,child_bs (bs->backing), errp)) {return ; }
Signature · 1 line
2.21.0
PATCH v6 9/42 block: Include filters when freezing backing chain
In reply to Max ReitzIn order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.
In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
--- include/block/block.h | 10 +++--- block.c | 81 +++++++++++++++++++++++++------------------ block/commit.c | 8 ++--- block/mirror.c | 4 +-- block/stream.c | 8 ++--- 5 files changed, 62 insertions(+), 49 deletions(-)
include/block/block.h +5 -5
@@ -364,11 +364,11 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs);BlockDriverState *bdrv_find_base (BlockDriverState *bs); -bool bdrv_is_backing_chain_frozen (BlockDriverState *bs,BlockDriverState *base, -Error **errp); -int bdrv_freeze_backing_chain (BlockDriverState *bs,BlockDriverState *base, -Error **errp); -void bdrv_unfreeze_backing_chain (BlockDriverState *bs,BlockDriverState *base); +bool bdrv_is_chain_frozen (BlockDriverState *bs,BlockDriverState *base, +Error **errp); +int bdrv_freeze_chain (BlockDriverState *bs,BlockDriverState *base, +Error **errp); +void bdrv_unfreeze_chain (BlockDriverState *bs,BlockDriverState *base); typedef struct BdrvCheckResult {
block.c +47 -34
186 lines not on this page · raw message
block/commit.c +4 -4
@@ -68,7 +68,7 @@ static int commit_prepare(Job *job) {CommitBlockJob *s =container_of (job, CommitBlockJob, common.job); -bdrv_unfreeze_backing_chain (s->commit_top_bs, s->base_bs); +bdrv_unfreeze_chain (s->commit_top_bs, s->base_bs); s->chain_frozen = false;/* Remove base node parent that still uses BLK_PERM_WRITE/RESIZE before
@@ -88,7 +88,7 @@ static void commit_abort(Job *job)BlockDriverState *top_bs =blk_bs (s->top);if (s->chain_frozen) { -bdrv_unfreeze_backing_chain (s->commit_top_bs, s->base_bs); +bdrv_unfreeze_chain (s->commit_top_bs, s->base_bs); }/* Make sure commit_top_bs and top stay around until bdrv_replace_node() */
@@ -331,7 +331,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, } } -if (bdrv_freeze_backing_chain (commit_top_bs, base, errp) <0 ) { +if (bdrv_freeze_chain (commit_top_bs, base, errp) <0 ) { goto fail; } s->chain_frozen = true;
@@ -372,7 +372,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, fail:if (s->chain_frozen) { -bdrv_unfreeze_backing_chain (commit_top_bs, base); +bdrv_unfreeze_chain (commit_top_bs, base); }if (s->base) {blk_unref (s->base);
block/mirror.c +2 -2
@@ -633,7 +633,7 @@ static int mirror_exit_common(Job *job) s->prepared = true;if (bdrv_chain_contains (src, target_bs)) { -bdrv_unfreeze_backing_chain (mirror_top_bs, target_bs); +bdrv_unfreeze_chain (mirror_top_bs, target_bs); }bdrv_release_dirty_bitmap (src, s->dirty_bitmap);
@@ -1707,7 +1707,7 @@ static BlockJob *mirror_start_job( } } -if (bdrv_freeze_backing_chain (mirror_top_bs, target, errp) <0 ) { +if (bdrv_freeze_chain (mirror_top_bs, target, errp) <0 ) { goto fail; } }
block/stream.c +4 -4
@@ -54,7 +54,7 @@ static void stream_abort(Job *job)if (s->chain_frozen) {BlockJob *bjob = &s->common; -bdrv_unfreeze_backing_chain (blk_bs (bjob->blk), s->bottom); +bdrv_unfreeze_chain (blk_bs (bjob->blk), s->bottom); } }
@@ -67,7 +67,7 @@ static int stream_prepare(Job *job)Error *local_err =NULL ;int ret =0 ; -bdrv_unfreeze_backing_chain (bs, s->bottom); +bdrv_unfreeze_chain (bs, s->bottom); s->chain_frozen = false;if (bs->backing) {
@@ -233,7 +233,7 @@ void stream_start(const char *job_id, BlockDriverState *bs,int basic_flags =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE_UNCHANGED ;BlockDriverState *bottom =bdrv_find_overlay (bs, base); -if (bdrv_freeze_backing_chain (bs, bottom, errp) <0 ) { +if (bdrv_freeze_chain (bs, bottom, errp) <0 ) {return ; }
@@ -284,5 +284,5 @@ fail:if (bs_read_only) {bdrv_reopen_set_read_only (bs, true,NULL ); } -bdrv_unfreeze_backing_chain (bs, bottom); +bdrv_unfreeze_chain (bs, bottom); }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain
Quoted 70 lines · Max Reitz
In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.
In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 10 +++--- block.c | 81 +++++++++++++++++++++++++------------------ block/commit.c | 8 ++--- block/mirror.c | 4 +-- block/stream.c | 8 ++--- 5 files changed, 62 insertions(+), 49 deletions(-)include/block/block.h +5
-5@@ -364,11 +364,11 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs);BlockDriverState *bdrv_find_base (BlockDriverState *bs); -bool bdrv_is_backing_chain_frozen (BlockDriverState *bs,BlockDriverState *base, -Error **errp); -int bdrv_freeze_backing_chain (BlockDriverState *bs,BlockDriverState *base, -Error **errp); -void bdrv_unfreeze_backing_chain (BlockDriverState *bs,BlockDriverState *base); +bool bdrv_is_chain_frozen (BlockDriverState *bs,BlockDriverState *base, +Error **errp); +int bdrv_freeze_chain (BlockDriverState *bs,BlockDriverState *base, +Error **errp); +void bdrv_unfreeze_chain (BlockDriverState *bs,BlockDriverState *base); typedef struct BdrvCheckResult {block.c +4
-0@@ -2303,12 +2303,15 @@ static void bdrv_replace_child_noperm(BdrvChild *child,* If @new_bs is not NULL, bdrv_check_perm() must be called beforehand, as this * function uses bdrv_set_perm() to update the permissions according to the new * reference that @new_bs gets. +* +* Callers must ensure that child->frozen is false. */ static void bdrv_replace_child (BdrvChild *child,BlockDriverState *new_bs) {BlockDriverState *old_bs = child->bs;uint64_t perm, shared_perm; +/* Asserts that child->frozen == false */ bdrv_replace_child_noperm (child, new_bs);/* @@ -2468,6 +2471,7 @@ static void bdrv_detach_child(BdrvChild *child)g_free (child); } +/* Callers must ensure that child->frozen is false. */ void bdrv_root_unref_child (BdrvChild *child) {BlockDriverState *child_bs;
block.c void bdrv_root_unref_child(BdrvChild *child) · line 2483
Quoted 7 lines
block.c +0
-4@@ -2477,10 +2481,6 @@ void bdrv_root_unref_child(BdrvChild *child)bdrv_unref (child_bs); } -/** -* Clear all inherits_from pointers from children and grandchildren of -* @root that point to @root, where necessary. -*/
Hmm, unrelated chunk? Without it:
Quoted 4 lines
block.c +0
-0static void bdrv_unset_inherits_from (BlockDriverState *root,BdrvChild *child) {BdrvChild *c;@@ -2505,6 +2505,7 @@ static void bdrv_unset_inherits_from(BlockDriverState *root, BdrvChild *child)
[..]
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain
Quoted 20 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines · Max Reitz
In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.
In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 10 +++--- block.c | 81 +++++++++++++++++++++++++------------------ block/commit.c | 8 ++--- block/mirror.c | 4 +-- block/stream.c | 8 ++--- 5 files changed, 62 insertions(+), 49 deletions(-)
[...]
Quoted 11 lines
block.c void bdrv_root_unref_child(BdrvChild *child) · line 2483
Quoted 8 lines
block.c +0
-4@@ -2477,10 +2481,6 @@ void bdrv_root_unref_child(BdrvChild *child)bdrv_unref (child_bs); } -/** -* Clear all inherits_from pointers from children and grandchildren of -* @root that point to @root, where necessary. -*/ Hmm, unrelated chunk? Without it:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
I don’t know how that slipped in, sorry...
Once again, thanks for reviewing!
Max
Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.
In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.
I don't think this is a good idea. The functions are still following the backing chain. A generic "chain" could mean following the bs->file links or any other children, so the new name is confusing because it doesn't really tell you any more what the function does. I'd prefer the name to stay specific.
Quoted 5 lines
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
Signed-off-by Max Reitz <[email protected]>
Kevin
Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain
Quoted 5 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 4 lines
In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.
Quoted 8 lines
Quoted 2 lines
In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.
I don't think this is a good idea. The functions are still following the backing chain. A generic "chain" could mean following the
bs->filelinks or any other children, so the new name is confusing because it doesn't really tell you any more what the function does. I'd prefer the name to stay specific.
They’re following backing chains, among others.
It would make sense to rename s/backing_chain/filter_chain/, that is, in case you don‘t find lumping COW and R/W filters together under “filter” too offensive.
(Naming things is hard. I’m open for suggestions, but I found the “filter” concept succinct, even if it does not fully align with our existing parlance.)
Max
Quoted 8 lines
Quoted 5 lines
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
Signed-off-by Max Reitz <[email protected]>Kevin
Re: [Qemu-devel] [PATCH v6 09/42] block: Include filters when freezing backing chain
Am 09.09.2019 um 10:02 hat Max Reitz geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 5 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 4 lines
In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze all filter links, be they COW or R/W filter links.
Quoted 8 lines
Quoted 2 lines
In the process, rename these functions to reflect that they now act on generalized chains of filter nodes instead of backing chains alone.
I don't think this is a good idea. The functions are still following the backing chain. A generic "chain" could mean following the
bs->filelinks or any other children, so the new name is confusing because it doesn't really tell you any more what the function does. I'd prefer the name to stay specific.They’re following backing chains, among others.
Quoted 7 lines
It would make sense to rename s/backing_chain/filter_chain/, that is, in case you don‘t find lumping COW and R/W filters together under “filter” too offensive.
(Naming things is hard. I’m open for suggestions, but I found the “filter” concept succinct, even if it does not fully align with our existing parlance.)
As you could see in my reply to patch 4, I didn't. :-)
I think it makes a lot more sense to just broaden the meaning of "backing chain" to be what you call a "filter chain" (following the backing file links, but accept filter nodes in between), because of how unspecific "filter chain" is. The primary thing we're interested in is still the backing files.
Kevin
PATCH v6 10/42 block: Drop bdrv_is_encrypted()
In reply to Max ReitzThe original purpose of bdrv_is_encrypted() was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.
Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.
So now we have to decide in which direction we want to fix bdrv_is_encrypted(): Recursively query all children, or drop it and just use bs->encrypted to get the current node's status?
Nowadays, its only purpose is to report through bdrv_query_image_info() whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).
--- include/block/block.h | 1 - block.c | 8 -------- block/qapi.c | 2 +- 3 files changed, 1 insertion(+), 10 deletions(-)
include/block/block.h +0 -1
@@ -487,7 +487,6 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);void bdrv_next_cleanup (BdrvNextIterator *it);BlockDriverState *bdrv_next_monitor_owned (BlockDriverState *bs); -bool bdrv_is_encrypted (BlockDriverState *bs);void bdrv_iterate_format (void (*it)(void *opaque,const char *name),void *opaque,bool read_only);const char *bdrv_get_node_name (const BlockDriverState *bs);
block.c +0 -8
@@ -4696,14 +4696,6 @@ bool bdrv_is_sg(BlockDriverState *bs)return bs->sg; } -bool bdrv_is_encrypted (BlockDriverState *bs) -{ -if (bs->backing && bs->backing->bs->encrypted) { -return true; - } -return bs->encrypted; -} -const char *bdrv_get_format_name (BlockDriverState *bs) {return bs->drv ? bs->drv->format_name :NULL ;
block/qapi.c +1 -1
@@ -281,7 +281,7 @@ void bdrv_query_image_info(BlockDriverState *bs, info->virtual_size = size; info->actual_size =bdrv_get_allocated_file_size (bs); info->has_actual_size = info->actual_size >=0 ; -if (bdrv_is_encrypted (bs)) { +if (bs->encrypted) { info->encrypted = true; info->has_encrypted = true; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 10/42] block: Drop bdrv_is_encrypted()
Quoted 19 lines · Max Reitz
The original purpose of
bdrv_is_encrypted()was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.
So now we have to decide in which direction we want to fix
bdrv_is_encrypted(): Recursively query all children, or drop it and just usebs->encryptedto get the current node's status?Nowadays, its only purpose is to report through
bdrv_query_image_info()whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).
Quoted 2 lines
Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v6 11/42 block: Add bdrv_supports_compressed_writes()
In reply to Max ReitzFilters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).
--- include/block/block.h | 1 + block.c | 22 ++++++++++++++++++++++ 2 files changed, 23 insertions(+)
include/block/block.h +1 -0
@@ -487,6 +487,7 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);void bdrv_next_cleanup (BdrvNextIterator *it);BlockDriverState *bdrv_next_monitor_owned (BlockDriverState *bs); +bool bdrv_supports_compressed_writes (BlockDriverState *bs);void bdrv_iterate_format (void (*it)(void *opaque,const char *name),void *opaque,bool read_only);const char *bdrv_get_node_name (const BlockDriverState *bs);
block.c +22 -0
@@ -4696,6 +4696,28 @@ bool bdrv_is_sg(BlockDriverState *bs)return bs->sg; } +/** +* Return whether the given node supports compressed writes. +*/ +bool bdrv_supports_compressed_writes (BlockDriverState *bs) +{ +BlockDriverState *filtered =bdrv_filtered_rw_bs (bs); + +if (!bs->drv || !bs->drv->bdrv_co_pwritev_compressed) { +return false; + } + +if (filtered) { +/* +* Filters can only forward compressed writes, so we have to +* check the child. +*/ +return bdrv_supports_compressed_writes (filtered); + } + +return true; +} +const char *bdrv_get_format_name (BlockDriverState *bs) {return bs->drv ? bs->drv->format_name :NULL ;
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 11/42] block: Add bdrv_supports_compressed_writes()
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 11 lines
Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether
bs->drv->bdrv_co_pwritev_compressedis non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Should patches 2 and 3 that add the .bdrv_co_pwritev_compressed() callback to filter drivers come only after this one? And should we also support it in the mirror filter?
Kevin
Re: [Qemu-devel] [PATCH v6 11/42] block: Add bdrv_supports_compressed_writes()
Quoted 10 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 9 lines
Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether
bs->drv->bdrv_co_pwritev_compressedis non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).
Quoted 5 lines
Quoted 2 lines
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Should patches 2 and 3 that add the .bdrv_co_pwritev_compressed() callback to filter drivers come only after this one?
Why not.
Quoted 2 lines
And should we also
support it in the mirror filter?
Hm. AFAIU, compressed writes have very limited use. You can basically only use them when writing to a new image (where you’d never write anywhere you’ve already written something to), i.e. for qemu-img convert or the backup target. It makes sense to blockdev-backup to throttle, so that’s why it should be implemented there. I don’t really see how it would make sense for mirror.
OTOH, it doesn’t make sense for COR either. And it isn’t that hard. Now I don’t have a strong preference for either dropping the COR patch or adding it to mirror as well...
Max
PATCH v6 12/42 block: Use bdrv_filtered_rw* where obvious
In reply to Max ReitzPlaces that use patterns like
if (bs->drv->is_filter && bs->file) {
... something about bs->file->bs ...
}should be
BlockDriverState *filtered = bdrv_filtered_rw_bs(bs);
if (filtered) {
... something about @filtered ...
}instead.
--- block.c | 23 +++++++++++++++-------- block/io.c | 5 +++-- 2 files changed, 18 insertions(+), 10 deletions(-)
block.c +15 -8
@@ -556,11 +556,12 @@ int bdrv_create_file(const char *filename, QemuOpts *opts, Error **errp)int bdrv_probe_blocksizes (BlockDriverState *bs,BlockSizes *bsz) {BlockDriver *drv = bs->drv; +BlockDriverState *filtered =bdrv_filtered_rw_bs (bs);if (drv && drv->bdrv_probe_blocksizes) {return drv->bdrv_probe_blocksizes (bs, bsz); - }else if (drv && drv->is_filter && bs->file) { -return bdrv_probe_blocksizes (bs->file->bs, bsz); + }else if (filtered) { +return bdrv_probe_blocksizes (filtered, bsz); }return -ENOTSUP ;
@@ -575,11 +576,12 @@ int bdrv_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)int bdrv_probe_geometry (BlockDriverState *bs,HDGeometry *geo) {BlockDriver *drv = bs->drv; +BlockDriverState *filtered =bdrv_filtered_rw_bs (bs);if (drv && drv->bdrv_probe_geometry) {return drv->bdrv_probe_geometry (bs, geo); - }else if (drv && drv->is_filter && bs->file) { -return bdrv_probe_geometry (bs->file->bs, geo); + }else if (filtered) { +return bdrv_probe_geometry (filtered, geo); }return -ENOTSUP ;
@@ -5084,6 +5086,8 @@ int bdrv_has_zero_init_1(BlockDriverState *bs)int bdrv_has_zero_init (BlockDriverState *bs) { +BlockDriverState *filtered; +if (!bs->drv) {return 0 ; }
@@ -5096,8 +5100,10 @@ int bdrv_has_zero_init(BlockDriverState *bs)if (bs->drv->bdrv_has_zero_init) {return bs->drv->bdrv_has_zero_init (bs); } -if (bs->file && bs->drv->is_filter) { -return bdrv_has_zero_init (bs->file->bs); + + filtered =bdrv_filtered_rw_bs (bs); +if (filtered) { +return bdrv_has_zero_init (filtered); }/* safe default */
@@ -5142,8 +5148,9 @@ int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)return -ENOMEDIUM ; }if (!drv->bdrv_get_info) { -if (bs->file && drv->is_filter) { -return bdrv_get_info (bs->file->bs, bdi); +BlockDriverState *filtered =bdrv_filtered_rw_bs (bs); +if (filtered) { +return bdrv_get_info (filtered, bdi); }return -ENOTSUP ; }
block/io.c +3 -2
@@ -3186,8 +3186,9 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, }if (!drv->bdrv_co_truncate) { -if (bs->file && drv->is_filter) { - ret =bdrv_co_truncate (bs->file, offset, prealloc, errp); +BdrvChild *filtered =bdrv_filtered_rw_child (bs); +if (filtered) { + ret =bdrv_co_truncate (filtered, offset, prealloc, errp); goto out; }error_setg (errp,"Image format driver does not support resize" );
Signature · 1 line
2.21.0
PATCH v6 13/42 block: Use CAFs in block status functions
In reply to Max ReitzUse the child access functions in the block status inquiry functions as appropriate.
--- block/io.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
block/io.c +10 -9
@@ -2180,11 +2180,12 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,if (ret & (BDRV_BLOCK_DATA |BDRV_BLOCK_ZERO )) { ret |=BDRV_BLOCK_ALLOCATED ; }else if (want_zero) { +BlockDriverState *cow_bs =bdrv_filtered_cow_bs (bs); +if (bdrv_unallocated_blocks_are_zero (bs)) { ret |=BDRV_BLOCK_ZERO ; - }else if (bs->backing) { -BlockDriverState *bs2 = bs->backing->bs; -int64_t size2 =bdrv_getlength (bs2); + }else if (cow_bs) { +int64_t size2 =bdrv_getlength (cow_bs);if (size2 >=0 && offset >= size2) { ret |=BDRV_BLOCK_ZERO ;
@@ -2250,7 +2251,7 @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,bool first = true;assert (bs != base); -for (p = bs; p != base; p =backing_bs (p)) { +for (p = bs; p != base; p =bdrv_filtered_bs (p)) { ret =bdrv_co_block_status (p, want_zero, offset, bytes, pnum, map, file);if (ret <0 ) {
@@ -2336,7 +2337,7 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,int bdrv_block_status (BlockDriverState *bs,int64_t offset,int64_t bytes,int64_t *pnum,int64_t *map,BlockDriverState **file) { -return bdrv_block_status_above (bs,backing_bs (bs), +return bdrv_block_status_above (bs,bdrv_filtered_bs (bs), offset, bytes, pnum, map, file); }
@@ -2346,9 +2347,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,int ret;int64_t dummy; - ret =bdrv_common_block_status_above (bs,backing_bs (bs), false, offset, - bytes, pnum ? pnum : &dummy,NULL , -NULL ); + ret =bdrv_common_block_status_above (bs,bdrv_filtered_bs (bs), false, + offset, bytes, pnum ? pnum : &dummy, +NULL ,NULL );if (ret <0 ) {return ret; }
@@ -2411,7 +2412,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,break ; } - intermediate =backing_bs (intermediate); + intermediate =bdrv_filtered_bs (intermediate); } *pnum = n;
Signature · 1 line
2.21.0
PATCH v6 14/42 block: Use CAFs when working with backing chains
In reply to Max ReitzUse child access functions when iterating through backing chains so filters do not break the chain.
--- block.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)
block.c +28 -12
@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file. * * active is the current topmost image. *
@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs) { -while (active && bs !=backing_bs (active)) { - active =backing_bs (active); + bs =bdrv_skip_rw_filters (bs); + active =bdrv_skip_rw_filters (active); + +while (active) { +BlockDriverState *next =bdrv_backing_chain_next (active); +if (bs == next) { +return active; + } + active = next; } -return active; +return NULL ; }/* Given a BDS, searches for the base layer. */
@@ -4544,9 +4552,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,* other intermediate nodes have been dropped. * If 'top' is an implicit node (e.g. "commit_top") we should skip * it because no one inherits from it. We use explicit_top for that. */ -while (explicit_top && explicit_top->implicit) { - explicit_top =backing_bs (explicit_top); - } + explicit_top =bdrv_skip_implicit_filters (explicit_top); update_inherits_from =bdrv_inherits_from_recursive (base, explicit_top);/* success - we can delete the intermediate states, and link top->base */
@@ -5014,7 +5020,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,bool bdrv_chain_contains (BlockDriverState *top,BlockDriverState *base) {while (top && top != base) { - top =backing_bs (top); + top =bdrv_filtered_bs (top); }return top !=NULL ;
@@ -5253,7 +5259,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, is_protocol =path_has_protocol (backing_file); -for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) { +/* +* Being largely a legacy function, skip any filters here +* (because filters do not have normal filenames, so they cannot +* match anyway; and allowing json:{} filenames is a bit out of +* scope). +*/ +for (curr_bs =bdrv_skip_rw_filters (bs); +bdrv_filtered_cow_child (curr_bs) !=NULL ; + curr_bs =bdrv_backing_chain_next (curr_bs)) + { +BlockDriverState *bs_below =bdrv_backing_chain_next (curr_bs);/* If either of the filename paths is actually a protocol, then * compare unmodified paths; otherwise make paths relative */
@@ -5261,7 +5277,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,char *backing_file_full_ret;if (strcmp (backing_file, curr_bs->backing_file) ==0 ) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; }/* Also check against the full backing filename for the image */
@@ -5271,7 +5287,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,bool equal =strcmp (backing_file, backing_file_full_ret) ==0 ;g_free (backing_file_full_ret);if (equal) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; } }
@@ -5297,7 +5313,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,g_free (filename_tmp);if (strcmp (backing_file_full, filename_full) ==0 ) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; } }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains
Quoted 4 lines · Max Reitz
Use child access functions when iterating through backing chains so filters do not break the chain.
Signed-off-by Max Reitz <[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 23 lines
Use child access functions when iterating through backing chains so filters do not break the chain.
Signed-off-by Max Reitz <[email protected]>--- block.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)block.c +2
-1@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file. * * active is the current topmost image. *
block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393
Quoted 7 lines
block.c +2
-2@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs) { -while (active && bs !=backing_bs (active)) { - active =backing_bs (active); + bs =bdrv_skip_rw_filters (bs); + active =bdrv_skip_rw_filters (active);
This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.
The block jobs seem to use it only for bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same. qmp_block_commit() will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.
All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.
Quoted 37 lines
block.c +9
-5+while (active) { +BlockDriverState *next =bdrv_backing_chain_next (active); +if (bs == next) { +return active; + } + active = next; } -return active; +return NULL ; }/* Given a BDS, searches for the base layer. */ @@ -4544,9 +4552,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,* other intermediate nodes have been dropped. * If 'top' is an implicit node (e.g. "commit_top") we should skip * it because no one inherits from it. We use explicit_top for that. */ -while (explicit_top && explicit_top->implicit) { - explicit_top =backing_bs (explicit_top); - } + explicit_top =bdrv_skip_implicit_filters (explicit_top); update_inherits_from =bdrv_inherits_from_recursive (base, explicit_top);/* success - we can delete the intermediate states, and link top->base */ @@ -5014,7 +5020,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,bool bdrv_chain_contains (BlockDriverState *top,BlockDriverState *base) {while (top && top != base) { - top =backing_bs (top); + top =bdrv_filtered_bs (top); }return top !=NULL ;@@ -5253,7 +5259,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, is_protocol =path_has_protocol (backing_file);
block.c BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, · line 5270
Quoted 10 lines
block.c +9
-1-for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) { +/* +* Being largely a legacy function, skip any filters here +* (because filters do not have normal filenames, so they cannot +* match anyway; and allowing json:{} filenames is a bit out of +* scope). +*/ +for (curr_bs =bdrv_skip_rw_filters (bs); +bdrv_filtered_cow_child (curr_bs) !=NULL ; + curr_bs =bdrv_backing_chain_next (curr_bs))
This could just use bs_below instead of recalculating the node if you moved the declaration of bs_below to the function scope.
Quoted 24 lines
block.c +4
-2+ { +BlockDriverState *bs_below =bdrv_backing_chain_next (curr_bs);/* If either of the filename paths is actually a protocol, then * compare unmodified paths; otherwise make paths relative */ @@ -5261,7 +5277,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,char *backing_file_full_ret;if (strcmp (backing_file, curr_bs->backing_file) ==0 ) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; }/* Also check against the full backing filename for the image */ @@ -5271,7 +5287,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,bool equal =strcmp (backing_file, backing_file_full_ret) ==0 ;g_free (backing_file_full_ret);if (equal) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; } }
block.c BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, · line 5319
Quoted 8 lines
block.c +1
-1@@ -5297,7 +5313,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,g_free (filename_tmp);if (strcmp (backing_file_full, filename_full) ==0 ) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; } }
Kevin
Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains
Quoted 28 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 27 lines
Use child access functions when iterating through backing chains so filters do not break the chain.
Signed-off-by Max Reitz <[email protected]>--- block.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)block.c +2
-2@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file. * * active is the current topmost image. * @@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs) { -while (active && bs !=backing_bs (active)) {
Quoted 8 lines
block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393
Quoted 3 lines
block.c +2
-1- active =backing_bs (active); + bs =bdrv_skip_rw_filters (bs); + active =bdrv_skip_rw_filters (active);This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.
Which is to say the overlay.
(I think we only ever use “overlay” in the COW sense.)
Quoted 5 lines
The block jobs seem to use it only for
bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same.qmp_block_commit()will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.
You mean that the op blocker could have been on a block job filter node before? I think that‘s pretty much the point of this fix; that that doesn’t make sense. (We didn’t have mirror_top_bs and the like at 058223a6e3b.)
Quoted 2 lines
All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.
I don’t know. The function was written when there were no filters. This change would have been a no-op then. The fact that it isn’t to me just means that introducing filters broke it.
So I don’t know what I would write. Maybe “bdrv_find_overlay() now actually finds the overlay, that is, it will not return a filter node. This is the behavior that all callers expect (because they work on COW backing chains).”
Quoted 37 lines
Quoted 37 lines
block.c +9
-5+while (active) { +BlockDriverState *next =bdrv_backing_chain_next (active); +if (bs == next) { +return active; + } + active = next; } -return active; +return NULL ; }/* Given a BDS, searches for the base layer. */ @@ -4544,9 +4552,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,* other intermediate nodes have been dropped. * If 'top' is an implicit node (e.g. "commit_top") we should skip * it because no one inherits from it. We use explicit_top for that. */ -while (explicit_top && explicit_top->implicit) { - explicit_top =backing_bs (explicit_top); - } + explicit_top =bdrv_skip_implicit_filters (explicit_top); update_inherits_from =bdrv_inherits_from_recursive (base, explicit_top);/* success - we can delete the intermediate states, and link top->base */ @@ -5014,7 +5020,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,bool bdrv_chain_contains (BlockDriverState *top,BlockDriverState *base) {while (top && top != base) { - top =backing_bs (top); + top =bdrv_filtered_bs (top); }return top !=NULL ;@@ -5253,7 +5259,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, is_protocol =path_has_protocol (backing_file);
Quoted 13 lines
block.c BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, · line 5270
Quoted 10 lines
block.c +9
-1-for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) { +/* +* Being largely a legacy function, skip any filters here +* (because filters do not have normal filenames, so they cannot +* match anyway; and allowing json:{} filenames is a bit out of +* scope). +*/ +for (curr_bs =bdrv_skip_rw_filters (bs); +bdrv_filtered_cow_child (curr_bs) !=NULL ; + curr_bs =bdrv_backing_chain_next (curr_bs))This could just use
bs_belowinstead of recalculating the node if you moved the declaration ofbs_belowto the function scope.
Indeed, thanks.
Max
Quoted 5 lines
Quoted 5 lines
block.c +2
-0+ { +BlockDriverState *bs_below =bdrv_backing_chain_next (curr_bs);/* If either of the filename paths is actually a protocol, then * compare unmodified paths; otherwise make paths relative */
Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains
Am 09.09.2019 um 10:25 hat Max Reitz geschrieben:
Quoted 38 lines · Kevin Wolf
Quoted 28 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 27 lines
Use child access functions when iterating through backing chains so filters do not break the chain.
Signed-off-by Max Reitz <[email protected]>--- block.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)block.c +2
-2@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file. * * active is the current topmost image. * @@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs) { -while (active && bs !=backing_bs (active)) {Quoted 8 lines
block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393
Quoted 3 lines
block.c +2
-1- active =backing_bs (active); + bs =bdrv_skip_rw_filters (bs); + active =bdrv_skip_rw_filters (active);This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.
Quoted 3 lines
Which is to say the overlay.
(I think we only ever use “overlay” in the COW sense.)
I think we do, but so far also only ever for immediate COW childs, not for skipping through intermediate node.
Quoted 10 lines
Quoted 5 lines
The block jobs seem to use it only for
bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same.qmp_block_commit()will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.You mean that the op blocker could have been on a block job filter node before? I think that‘s pretty much the point of this fix; that that doesn’t make sense. (We didn’t have
mirror_top_bsand the like at058223a6e3b.)
On the off chance that the op blocker actually works, it can't be a job filter. I was thinking more of throttling, blkdebug etc.
Quoted 4 lines
Quoted 2 lines
All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.
I don’t know. The function was written when there were no filters.
I doubt it. blkdebug is a really old filter.
Quoted 7 lines
This change would have been a no-op then. The fact that it isn’t to me just means that introducing filters broke it.
So I don’t know what I would write. Maybe “bdrv_find_overlay() now actually finds the overlay, that is, it will not return a filter node. This is the behavior that all callers expect (because they work on COW backing chains).”
Maybe just something like "In addition, filter nodes are not returned as overlays any more. Instead, the first non-filter node on top of bs is considered the overlay now."?
Kevin
Re: [Qemu-devel] [PATCH v6 14/42] block: Use CAFs when working with backing chains
Quoted 39 lines · Kevin Wolf
Am 09.09.2019 um 10:25 hat Max Reitz geschrieben:
Quoted 38 lines · Kevin Wolf
Quoted 36 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 23 lines
Use child access functions when iterating through backing chains so filters do not break the chain.
Signed-off-by Max Reitz <[email protected]>--- block.c | 40 ++++++++++++++++++++++++++++------------ 1 file changed, 28 insertions(+), 12 deletions(-)block.c +2
-1@@ -4376,7 +4376,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file. * * active is the current topmost image. * block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4393
Quoted 7 lines
block.c +2
-2@@ -4388,11 +4389,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs) { -while (active && bs !=backing_bs (active)) { - active =backing_bs (active); + bs =bdrv_skip_rw_filters (bs); + active =bdrv_skip_rw_filters (active);This does more than the commit message says. In addition to iterating through filters instead of stopping, it also changes the semantics of the function to return the next non-filter on top of bs instead of the next node.
Quoted 6 lines
Quoted 3 lines
Which is to say the overlay.
(I think we only ever use “overlay” in the COW sense.)
I think we do, but so far also only ever for immediate COW childs, not for skipping through intermediate node.
Yes, because it’s broken so far.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
The block jobs seem to use it only for
bdrv_is_allocated_above(), which should return the same thing in both cases, so the behaviour stays the same.qmp_block_commit()will check op blockers on a different node now, which could be a fix or a bug, I can't tell offhand. Probably the blocking doesn't really work anyway.
Quoted 7 lines
Quoted 4 lines
You mean that the op blocker could have been on a block job filter node before? I think that‘s pretty much the point of this fix; that that doesn’t make sense. (We didn’t have
mirror_top_bsand the like at058223a6e3b.)On the off chance that the op blocker actually works, it can't be a job filter. I was thinking more of throttling, blkdebug etc.
Well, that’s just broken altogether right now.
Quoted 10 lines
Quoted 4 lines
Quoted 2 lines
All of this should be mentioned in the commit message at least. Maybe it's also worth splitting in two patches.
I don’t know. The function was written when there were no filters.
I doubt it. blkdebug is a really old filter.
Quoted 3 lines
This change would have been a no-op then. The fact that it isn’t to me just means that introducing filters broke it.
Quoted 8 lines
Quoted 4 lines
So I don’t know what I would write. Maybe “bdrv_find_overlay() now actually finds the overlay, that is, it will not return a filter node. This is the behavior that all callers expect (because they work on COW backing chains).”
Maybe just something like "In addition, filter nodes are not returned as overlays any more. Instead, the first non-filter node on top of bs is considered the overlay now."?
Isn’t that basically the same? :-)
Max
PATCH v6 15/42 block: Re-evaluate backing file handling in reopen
In reply to Max ReitzReopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).
--- block.c | 45 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 38 insertions(+), 7 deletions(-)
block.c +38 -7
@@ -3660,25 +3660,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, } } +/* +* Ensure that @bs can really handle backing files, because we are +* about to give it one (or swap the existing one) +*/ +if (bs->drv->is_filter) { +/* Filters always have a file or a backing child */ +if (!bs->backing) { +error_setg (errp,"'%s' is a %s filter node that does not support a " +"backing child" , bs->node_name, bs->drv->format_name); +return -EINVAL ; + } + }else if (!bs->drv->supports_backing) { +error_setg (errp,"Driver '%s' of node '%s' does not support backing " +"files" , bs->drv->format_name, bs->node_name); +return -EINVAL ; + } +/* * Find the "actual" backing file by skipping all links that point * to an implicit node, if any (e.g. a commit filter node). +* We cannot use any of the bdrv_skip_*() functions here because +* those return the first explicit node, while we are looking for +* its overlay here. */ overlay_bs = bs; -while (backing_bs (overlay_bs) &&backing_bs (overlay_bs)->implicit) { - overlay_bs =backing_bs (overlay_bs); +while (bdrv_filtered_bs (overlay_bs) && +bdrv_filtered_bs (overlay_bs)->implicit) + { + overlay_bs =bdrv_filtered_bs (overlay_bs); }/* If we want to replace the backing file we need some extra checks */ -if (new_backing_bs !=backing_bs (overlay_bs)) { +if (new_backing_bs !=bdrv_filtered_bs (overlay_bs)) {/* Check for implicit nodes between bs and its backing file */ if (bs != overlay_bs) {error_setg (errp,"Cannot change backing link if '%s' has " "an implicit backing file" , bs->node_name);return -EPERM ; } -/* Check if the backing link that we want to replace is frozen */ -if (bdrv_is_chain_frozen (overlay_bs,backing_bs (overlay_bs), errp)) { +/* +* Check if the backing link that we want to replace is frozen. +* Note that +* bdrv_filtered_child(overlay_bs) == overlay_bs->backing, +* because we know that overlay_bs == bs, and that @bs +* either is an R/W filter that uses ->backing or a COW format +* with bs->drv->supports_backing == true. +*/ +if (bdrv_is_chain_frozen (overlay_bs,child_bs (overlay_bs->backing), + errp)) + {return -EPERM ; } reopen_state->replace_backing_bs = true;
@@ -3829,7 +3860,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,* its metadata. Otherwise the 'backing' option can be omitted. */ if (drv->supports_backing && reopen_state->backing_missing && - (backing_bs (reopen_state->bs) || reopen_state->bs->backing_file[0 ])) { + (reopen_state->bs->backing || reopen_state->bs->backing_file[0 ])) {error_setg (errp,"backing is missing for '%s'" , reopen_state->bs->node_name); ret = -EINVAL ;
@@ -3974,7 +4005,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)* from bdrv_set_backing_hd()) has the new values. */ if (reopen_state->replace_backing_bs) { -BlockDriverState *old_backing_bs =backing_bs (bs); +BlockDriverState *old_backing_bs =child_bs (bs->backing);assert (!old_backing_bs || !old_backing_bs->implicit);/* Abort the permission update on the backing bs we're detaching */ if (old_backing_bs) {
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 15/42] block: Re-evaluate backing file handling in reopen
Quoted 8 lines · Max Reitz
Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use
bs->backingbecause these functions are about the "backing" child (which may or may not be the COW backing file).Signed-off-by Max Reitz<[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v6 16/42 block: Flush all children in generic code
In reply to Max ReitzIf the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children.
In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.
--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +17 -6
@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;
@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {
@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */
@@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret =0 ; + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret; + + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + } + out:/* Notify any pending flushes that we have completed */ if (ret ==0 ) {
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code
Quoted 59 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)block/io.c +6
-5@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */ @@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent:
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2676
Quoted 10 lines
block/io.c +9
-1- ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret = 0; > + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret; + + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + }
Hmm, you said that we want to flush only children with write-access from parent.. Shouldn't we check it? Or we assume that it's always safe to call bdrv_co_flush on a node?
Quoted 5 lines
block/io.c +1
-0+ out:/* Notify any pending flushes that we have completed */ if (ret ==0 ) {
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code
Quoted 64 lines · Vladimir Sementsov-Ogievskiy
Quoted 63 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)block/io.c +9
-6@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */ @@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret = 0; > + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret;
Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677
Quoted 6 lines
block/io.c +6
-0+ + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + }Hmm, you said that we want to flush only children with write-access from parent..
Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)
Quoted 2 lines
Shouldn't we check it? Or we assume that it's always safe to call
bdrv_co_flushon a node?
I think it’s always safe. But checking it seems like a nice touch, yes.
Max
Quoted 7 lines
Quoted 5 lines
block/io.c +1
-0+ out:/* Notify any pending flushes that we have completed */ if (ret ==0 ) {
Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code
Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:
Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)block/io.c +9
-6@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */ @@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret = 0; > + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret;Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677
Quoted 6 lines
block/io.c +6
-0+ + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + }Hmm, you said that we want to flush only children with write-access from parent..
Quoted 7 lines
Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)
Quoted 2 lines
Shouldn't we check it? Or we assume that it's always safe to call
bdrv_co_flushon a node?I think it’s always safe. But checking it seems like a nice touch, yes.
I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.
But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe bs->write_gen saves us so that at least this doesn't result in an fdatasync() call for each, but still... Without a use case, I'd rather not do this.
Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?
Kevin
Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code
Quoted 95 lines · Kevin Wolf
Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:
Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)block/io.c +9
-6@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */ @@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret = 0; > + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret;Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677
Quoted 6 lines
block/io.c +6
-0+ + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + }Hmm, you said that we want to flush only children with write-access from parent..
Quoted 7 lines
Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)
Quoted 2 lines
Shouldn't we check it? Or we assume that it's always safe to call
bdrv_co_flushon a node?I think it’s always safe. But checking it seems like a nice touch, yes.
I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.
But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe
bs->write_gensaves us so that at least this doesn't result in anfdatasync()call for each, but still... Without a use case, I'd rather not do this.
Quoted 5 lines
Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?
And I should thus probably drop vmdk’s .bdrv_co_flush_to_disk() implementation.
I will indeed try to write a test, but to be completely honest, I feel like this series is long enough.
Max
Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code
Am 09.09.2019 um 10:31 hat Max Reitz geschrieben:
Quoted 102 lines · Kevin Wolf
Quoted 95 lines
Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:
Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)block/io.c +9
-6@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */ @@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret = 0; > + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret;Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677
Quoted 6 lines
block/io.c +6
-0+ + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + }Hmm, you said that we want to flush only children with write-access from parent..
Quoted 7 lines
Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)
Quoted 2 lines
Shouldn't we check it? Or we assume that it's always safe to call
bdrv_co_flushon a node?I think it’s always safe. But checking it seems like a nice touch, yes.
I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.
But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe
bs->write_gensaves us so that at least this doesn't result in anfdatasync()call for each, but still... Without a use case, I'd rather not do this.Quoted 5 lines
Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?
Quoted 5 lines
And I should thus probably drop vmdk’s .bdrv_co_flush_to_disk() implementation.
I will indeed try to write a test, but to be completely honest, I feel like this series is long enough.
I guess I could already merge patch 1 to give you space for another test patch. ;-)
Kevin
Re: [Qemu-devel] [PATCH v6 16/42] block: Flush all children in generic code
Quoted 103 lines · Kevin Wolf
Am 09.09.2019 um 10:31 hat Max Reitz geschrieben:
Quoted 102 lines · Kevin Wolf
Quoted 100 lines
Am 12.08.2019 um 14:58 hat Max Reitz geschrieben:
Quoted 74 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines
Quoted 63 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)block/io.c +9
-6@@ -2572,6 +2572,8 @@ static void coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;@@ -2601,7 +2603,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {@@ -2611,15 +2613,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */ @@ -2663,8 +2665,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret = 0; > + QLIST_FOREACH(child, &bs->children, next) { +int this_child_ret;Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2677
Quoted 6 lines
block/io.c +6
-0+ + this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + }Hmm, you said that we want to flush only children with write-access from parent..
Quoted 7 lines
Good that you remember it, I must have overlooked it (when reading the replies to the previous version). :-)
Quoted 2 lines
Shouldn't we check it? Or we assume that it's always safe to call
bdrv_co_flushon a node?I think it’s always safe. But checking it seems like a nice touch, yes.
I'm not sure why we would unconditionally flush all children anyway. The only drivers I can think of that really need to flush more than one child are blkverify and quorum, and both of them already implement this. blkverify implements .bdrv_co_flush, so it's not affected by the change anyway, but quorum children will be flushed twice now.
But more than this, I'm worried about the overhead of needlessly recursing through the whole backing chain and calling flush on every node there. Maybe
bs->write_gensaves us so that at least this doesn't result in anfdatasync()call for each, but still... Without a use case, I'd rather not do this.Oh, well, after having written all of this, I see that qcow2 with an external data file is buggy... This could be fixed in the qcow2 driver, but maybe restricting the recursion to read-only is actually good enough then. Can you mention this case in the commit message and maybe build a test for it?
Quoted 8 lines
Quoted 5 lines
And I should thus probably drop vmdk’s .bdrv_co_flush_to_disk() implementation.
I will indeed try to write a test, but to be completely honest, I feel like this series is long enough.
I guess I could already merge patch 1 to give you space for another test patch. ;-)
Don’t forget the mirror-top patch. And AFAIR, there was some comment from Vladimir that also required an additional patch. So it would need to be three!
(Or I just start squashing from the back?)
Max
PATCH v6 17/42 block: Use CAFs in bdrv_refresh_limits()
In reply to Max Reitz--- block/io.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
block/io.c +8 -6
@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)void bdrv_refresh_limits (BlockDriverState *bs,Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *storage_bs =bdrv_storage_bs (bs); +BlockDriverState *cow_bs =bdrv_filtered_cow_bs (bs);Error *local_err =NULL ;memset (&bs->bl,0 ,sizeof (bs->bl));
@@ -148,13 +150,13 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) drv->bdrv_aio_preadv) ? 1 : 512;/* Take some limits from the children as a default */ -if (bs->file) { -bdrv_refresh_limits (bs->file->bs, &local_err); +if (storage_bs) { +bdrv_refresh_limits (storage_bs, &local_err);if (local_err) {error_propagate (errp, local_err);return ; } -bdrv_merge_limits (&bs->bl, &bs->file->bs->bl); +bdrv_merge_limits (&bs->bl, &storage_bs->bl); }else { bs->bl.min_mem_alignment =512 ; bs->bl.opt_mem_alignment =getpagesize ();
@@ -163,13 +165,13 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) bs->bl.max_iov =IOV_MAX ; } -if (bs->backing) { -bdrv_refresh_limits (bs->backing->bs, &local_err); +if (cow_bs) { +bdrv_refresh_limits (cow_bs, &local_err);if (local_err) {error_propagate (errp, local_err);return ; } -bdrv_merge_limits (&bs->bl, &bs->backing->bs->bl); +bdrv_merge_limits (&bs->bl, &cow_bs->bl); }/* Then let the driver override it */
Signature · 1 line
2.21.0
PATCH v6 18/42 block: Use CAFs in bdrv_refresh_filename()
In reply to Max Reitzbdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.
--- block.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)
block.c +21 -8
@@ -6432,6 +6432,7 @@ void bdrv_refresh_filename(BlockDriverState *bs) {BlockDriver *drv = bs->drv;BdrvChild *child; +BlockDriverState *primary_child_bs;QDict *opts;bool backing_overridden;bool generate_json_filename;/* Whether our default implementation should
@@ -6500,20 +6501,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)qobject_unref (bs->full_open_options); bs->full_open_options = opts; + primary_child_bs =bdrv_primary_bs (bs); +if (drv->bdrv_refresh_filename) {/* Obsolete information is of no use here, so drop the old file name * information before refreshing it */ bs->exact_filename[0 ] ='\0' ; drv->bdrv_refresh_filename (bs); - }else if (bs->file) { -/* Try to reconstruct valid information from the underlying file */ + }else if (primary_child_bs) { +/* +* Try to reconstruct valid information from the underlying +* file -- this only works for format nodes (filter nodes +* cannot be probed and as such must be selected by the user +* either through an options dict, or through a special +* filename which the filter driver must construct in its +* .bdrv_refresh_filename() implementation). +*/ bs->exact_filename[0 ] ='\0' ;/* * We can use the underlying file's filename if: * - it has a filename, +* - the current BDS is not a filter, * - the file is a protocol BDS, and * - opening that file (as this BDS's format) will automatically create * the BDS tree we have right now, that is:
@@ -6522,11 +6533,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)* - no non-file child of this BDS has been overridden by the user * Both of these conditions are represented by generate_json_filename. */ -if (bs->file->bs->exact_filename[0 ] && - bs->file->bs->drv->bdrv_file_open && - !generate_json_filename) +if (primary_child_bs->exact_filename[0 ] && + primary_child_bs->drv->bdrv_file_open && + !drv->is_filter && !generate_json_filename) { -strcpy (bs->exact_filename, bs->file->bs->exact_filename); +strcpy (bs->exact_filename, primary_child_bs->exact_filename); } }
@@ -6543,6 +6554,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)char *bdrv_dirname (BlockDriverState *bs,Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *child_bs;if (!drv) {error_setg (errp,"Node '%s' is ejected" , bs->node_name);
@@ -6553,8 +6565,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp)return drv->bdrv_dirname (bs, errp); } -if (bs->file) { -return bdrv_dirname (bs->file->bs, errp); + child_bs =bdrv_primary_bs (bs); +if (child_bs) { +return bdrv_dirname (child_bs, errp); }bdrv_refresh_filename (bs);
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 18/42] block: Use CAFs in bdrv_refresh_filename()
Quoted 5 lines · Max Reitz
bdrv_refresh_filename()and the kind of relatedbdrv_dirname()should look to the primary child when they wish to copy the underlying file's filename.Signed-off-by Max Reitz <[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v6 19/42 block: Use CAF in bdrv_co_rw_vmstate()
In reply to Max ReitzIf a node whose driver does not provide VM state functions has a metadata child, the VM state should probably go there; if it is a filter, the VM state should probably go there. It follows that we should generally go down to the primary child.
--- block/io.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
block/io.c +3 -2
@@ -2434,6 +2434,7 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, bool is_read) {BlockDriver *drv = bs->drv; +BlockDriverState *child_bs =bdrv_primary_bs (bs);int ret = -ENOTSUP ;bdrv_inc_in_flight (bs);
@@ -2446,8 +2447,8 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, }else { ret = drv->bdrv_save_vmstate (bs, qiov, pos); } - }else if (bs->file) { - ret =bdrv_co_rw_vmstate (bs->file->bs, qiov, pos, is_read); + }else if (child_bs) { + ret =bdrv_co_rw_vmstate (child_bs, qiov, pos, is_read); }bdrv_dec_in_flight (bs);
Signature · 1 line
2.21.0
PATCH v6 20/42 block/snapshot: Fix fallback
In reply to Max ReitzIf the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto() becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child is bs->file or bs->backing (and then we have to find out which it is).
--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)
block/snapshot.c +79 -21
@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + } + +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + } + +return child_bs; +} +int bdrv_can_snapshot (BlockDriverState *bs) {BlockDriver *drv = bs->drv;
@@ -154,8 +180,9 @@ int bdrv_can_snapshot(BlockDriverState *bs) }if (!drv->bdrv_snapshot_create) { -if (bs->file !=NULL ) { -return bdrv_can_snapshot (bs->file->bs); +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +return bdrv_can_snapshot (fallback_bs); }return 0 ; }
@@ -167,14 +194,15 @@ int bdrv_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_create) {return drv->bdrv_snapshot_create (bs, sn_info); } -if (bs->file) { -return bdrv_snapshot_create (bs->file->bs, sn_info); +if (fallback_bs) { +return bdrv_snapshot_create (fallback_bs, sn_info); }return -ENOTSUP ; }
@@ -184,6 +212,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs;int ret, open_ret;if (!drv) {
@@ -204,39 +233,66 @@ int bdrv_snapshot_goto(BlockDriverState *bs,return ret; } -if (bs->file) { -BlockDriverState *file; -QDict *options =qdict_clone_shallow (bs->options); + fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +QDict *options;QDict *file_options;Error *local_err =NULL ; +bool is_backing_child; +BdrvChild **child_pointer; + +/* +* We need a pointer to the fallback child pointer, so let us +* see whether the child is referenced by a field in the BDS +* object. +*/ +if (fallback_bs == bs->file->bs) { + is_backing_child = false; + child_pointer = &bs->file; + }else if (fallback_bs == bs->backing->bs) { + is_backing_child = true; + child_pointer = &bs->backing; + }else { +/* +* The fallback child is not referenced by a field in the +* BDS object. We cannot go on then. +*/ +error_setg (errp,"Block driver does not support snapshots" ); +return -ENOTSUP ; + } + + options =qdict_clone_shallow (bs->options); - file = bs->file->bs;/* Prevent it from getting deleted when detached from bs */ -bdrv_ref (file); +bdrv_ref (fallback_bs); -qdict_extract_subqdict (options, &file_options,"file." ); +qdict_extract_subqdict (options, &file_options, + is_backing_child ?"backing." :"file." );qobject_unref (file_options); -qdict_put_str (options,"file" ,bdrv_get_node_name (file)); +qdict_put_str (options, is_backing_child ?"backing" :"file" , +bdrv_get_node_name (fallback_bs));if (drv->bdrv_close) { drv->bdrv_close (bs); } -bdrv_unref_child (bs, bs->file); - bs->file =NULL ; - ret =bdrv_snapshot_goto (file, snapshot_id, errp); +assert (fallback_bs == (*child_pointer)->bs); +bdrv_unref_child (bs, *child_pointer); + *child_pointer =NULL ; + + ret =bdrv_snapshot_goto (fallback_bs, snapshot_id, errp); open_ret = drv->bdrv_open (bs, options, bs->open_flags, &local_err);qobject_unref (options);if (open_ret <0 ) { -bdrv_unref (file); +bdrv_unref (fallback_bs); bs->drv =NULL ;/* A bdrv_snapshot_goto() error takes precedence */ error_propagate (errp, local_err);return ret <0 ? ret : open_ret; } -assert (bs->file->bs == file); -bdrv_unref (file); +assert (fallback_bs == (*child_pointer)->bs); +bdrv_unref (fallback_bs);return ret; }
@@ -272,6 +328,7 @@ int bdrv_snapshot_delete(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);int ret;if (!drv) {
@@ -288,8 +345,8 @@ int bdrv_snapshot_delete(BlockDriverState *bs,if (drv->bdrv_snapshot_delete) { ret = drv->bdrv_snapshot_delete (bs, snapshot_id, name, errp); - }else if (bs->file) { - ret =bdrv_snapshot_delete (bs->file->bs, snapshot_id, name, errp); + }else if (fallback_bs) { + ret =bdrv_snapshot_delete (fallback_bs, snapshot_id, name, errp); }else { error_setg(errp, "Block format '%s' used by device '%s' " "does not support internal snapshot deletion",
@@ -305,14 +362,15 @@ int bdrv_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_list) {return drv->bdrv_snapshot_list (bs, psn_info); } -if (bs->file) { -return bdrv_snapshot_list (bs->file->bs, psn_info); +if (fallback_bs) { +return bdrv_snapshot_list (fallback_bs, psn_info); }return -ENOTSUP ; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback
Quoted 115 lines · Max Reitz
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto()becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child isbs->fileorbs->backing(and then we have to find out which it is).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)block/snapshot.c +49
-7@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + } + +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + } + +return child_bs; +} +int bdrv_can_snapshot (BlockDriverState *bs) {BlockDriver *drv = bs->drv;@@ -154,8 +180,9 @@ int bdrv_can_snapshot(BlockDriverState *bs) }if (!drv->bdrv_snapshot_create) { -if (bs->file !=NULL ) { -return bdrv_can_snapshot (bs->file->bs); +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +return bdrv_can_snapshot (fallback_bs); }return 0 ; }@@ -167,14 +194,15 @@ int bdrv_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_create) {return drv->bdrv_snapshot_create (bs, sn_info); } -if (bs->file) { -return bdrv_snapshot_create (bs->file->bs, sn_info); +if (fallback_bs) { +return bdrv_snapshot_create (fallback_bs, sn_info); }return -ENOTSUP ; }@@ -184,6 +212,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs;int ret, open_ret;if (!drv) {@@ -204,39 +233,66 @@ int bdrv_snapshot_goto(BlockDriverState *bs,return ret; } -if (bs->file) { -BlockDriverState *file; -QDict *options =qdict_clone_shallow (bs->options); + fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +QDict *options;QDict *file_options;Error *local_err =NULL ; +bool is_backing_child; +BdrvChild **child_pointer; + +/* +* We need a pointer to the fallback child pointer, so let us +* see whether the child is referenced by a field in the BDS +* object. +*/ +if (fallback_bs == bs->file->bs) { + is_backing_child = false; + child_pointer = &bs->file; + }else if (fallback_bs == bs->backing->bs) { + is_backing_child = true;
block/snapshot.c int bdrv_snapshot_goto(BlockDriverState *bs, · line 263
Quoted 10 lines
block/snapshot.c +10
-0+ child_pointer = &bs->backing; + }else { +/* +* The fallback child is not referenced by a field in the +* BDS object. We cannot go on then. +*/ +error_setg (errp,"Block driver does not support snapshots" ); +return -ENOTSUP ; + } +
Hmm.. Should not this check be included into bdrv_snapshot_fallback(), to work only with file and backing?
And could we allow fallback only for filters? Is there real usecase except filters? Or may be, drop fallback at all?
Quoted 82 lines
block/snapshot.c +19
-14- file = bs->file->bs;/* Prevent it from getting deleted when detached from bs */ -bdrv_ref (file); +bdrv_ref (fallback_bs); -qdict_extract_subqdict (options, &file_options,"file." ); +qdict_extract_subqdict (options, &file_options, + is_backing_child ?"backing." :"file." );qobject_unref (file_options); -qdict_put_str (options,"file" ,bdrv_get_node_name (file)); +qdict_put_str (options, is_backing_child ?"backing" :"file" , +bdrv_get_node_name (fallback_bs));if (drv->bdrv_close) { drv->bdrv_close (bs); } -bdrv_unref_child (bs, bs->file); - bs->file =NULL ; - ret =bdrv_snapshot_goto (file, snapshot_id, errp); +assert (fallback_bs == (*child_pointer)->bs); +bdrv_unref_child (bs, *child_pointer); + *child_pointer =NULL ; + + ret =bdrv_snapshot_goto (fallback_bs, snapshot_id, errp); open_ret = drv->bdrv_open (bs, options, bs->open_flags, &local_err);qobject_unref (options);if (open_ret <0 ) { -bdrv_unref (file); +bdrv_unref (fallback_bs); bs->drv =NULL ;/* A bdrv_snapshot_goto() error takes precedence */ error_propagate (errp, local_err);return ret <0 ? ret : open_ret; } -assert (bs->file->bs == file); -bdrv_unref (file); +assert (fallback_bs == (*child_pointer)->bs); +bdrv_unref (fallback_bs);return ret; }@@ -272,6 +328,7 @@ int bdrv_snapshot_delete(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);int ret;if (!drv) {@@ -288,8 +345,8 @@ int bdrv_snapshot_delete(BlockDriverState *bs,if (drv->bdrv_snapshot_delete) { ret = drv->bdrv_snapshot_delete (bs, snapshot_id, name, errp); - }else if (bs->file) { - ret =bdrv_snapshot_delete (bs->file->bs, snapshot_id, name, errp); + }else if (fallback_bs) { + ret =bdrv_snapshot_delete (fallback_bs, snapshot_id, name, errp); }else { error_setg(errp, "Block format '%s' used by device '%s' " "does not support internal snapshot deletion",@@ -305,14 +362,15 @@ int bdrv_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_list) {return drv->bdrv_snapshot_list (bs, psn_info); } -if (bs->file) { -return bdrv_snapshot_list (bs->file->bs, psn_info); +if (fallback_bs) { +return bdrv_snapshot_list (fallback_bs, psn_info); }return -ENOTSUP ; }
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback
Quoted 121 lines · Vladimir Sementsov-Ogievskiy
Quoted 120 lines · Max Reitz
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto()becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child isbs->fileorbs->backing(and then we have to find out which it is).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)block/snapshot.c +54
-7@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + } + +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + } + +return child_bs; +} +int bdrv_can_snapshot (BlockDriverState *bs) {BlockDriver *drv = bs->drv;@@ -154,8 +180,9 @@ int bdrv_can_snapshot(BlockDriverState *bs) }if (!drv->bdrv_snapshot_create) { -if (bs->file !=NULL ) { -return bdrv_can_snapshot (bs->file->bs); +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +return bdrv_can_snapshot (fallback_bs); }return 0 ; }@@ -167,14 +194,15 @@ int bdrv_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_create) {return drv->bdrv_snapshot_create (bs, sn_info); } -if (bs->file) { -return bdrv_snapshot_create (bs->file->bs, sn_info); +if (fallback_bs) { +return bdrv_snapshot_create (fallback_bs, sn_info); }return -ENOTSUP ; }@@ -184,6 +212,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs;int ret, open_ret;if (!drv) {@@ -204,39 +233,66 @@ int bdrv_snapshot_goto(BlockDriverState *bs,return ret; } -if (bs->file) { -BlockDriverState *file; -QDict *options =qdict_clone_shallow (bs->options); + fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +QDict *options;QDict *file_options;Error *local_err =NULL ; +bool is_backing_child; +BdrvChild **child_pointer; + +/* +* We need a pointer to the fallback child pointer, so let us +* see whether the child is referenced by a field in the BDS +* object. +*/ +if (fallback_bs == bs->file->bs) { + is_backing_child = false; + child_pointer = &bs->file; + }else if (fallback_bs == bs->backing->bs) { + is_backing_child = true; + child_pointer = &bs->backing; + }else { +/* +* The fallback child is not referenced by a field in the +* BDS object. We cannot go on then.
Quoted 8 lines
block/snapshot.c int bdrv_snapshot_goto(BlockDriverState *bs, · line 263
Quoted 5 lines
block/snapshot.c +5
-0+*/ +error_setg (errp,"Block driver does not support snapshots" ); +return -ENOTSUP ; + } +Hmm.. Should not this check be included into
bdrv_snapshot_fallback(), to work only with file and backing?
I was under the impression that this was just special code for what turned out to be bdrv_snapshot_load_tmp() now, because it seems so weird. (So I thought just making the restriction here wouldn’t really be a limit.)
I was wrong. This is used when applying snapshots, so it is important.
If we make a restriction here, we should have it in all fallback code, yes.
Quoted 2 lines
And could we allow fallback only for filters? Is there real usecase except filters? Or may be, drop fallback at all?
raw isn’t a filter driver. And rbd as a protocol supports snapshotting.
Hence the fallback code, I presume.
Max
Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 38 lines
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto()becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child isbs->fileorbs->backing(and then we have to find out which it is).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)block/snapshot.c +15
-0@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + }
block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173
Quoted 10 lines
block/snapshot.c +10
-0+ +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + } + +return child_bs; +}
Why do we return child->bs here when bdrv_snapshot_goto() then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?
Kevin
Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback
Quoted 46 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 45 lines
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto()becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child isbs->fileorbs->backing(and then we have to find out which it is).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)block/snapshot.c +22
-0@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + } + +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + }
Quoted 8 lines
block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173
Quoted 3 lines
block/snapshot.c +3
-0+ +return child_bs; +}Why do we return
child->bshere whenbdrv_snapshot_goto()then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?
What would you return instead? &child doesn’t work.
We could limit ourselves to bs->file and bs->backing. It just seemed like a bit of an artificial limit to me, because we only really have it for bdrv_snapshot_goto().
Max
Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback
Am 10.09.2019 um 14:04 hat Max Reitz geschrieben:
Quoted 56 lines · Kevin Wolf
Quoted 46 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 45 lines
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto()becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child isbs->fileorbs->backing(and then we have to find out which it is).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)block/snapshot.c +22
-0@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + } + +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + }Quoted 8 lines
block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173
Quoted 3 lines
block/snapshot.c +3
-0+ +return child_bs; +}Why do we return
child->bshere whenbdrv_snapshot_goto()then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?
Quoted 1 line
What would you return instead? &child doesn’t work.
Oops, brain fart. :-)
Quoted 3 lines
We could limit ourselves to
bs->fileandbs->backing. It just seemed like a bit of an artificial limit to me, because we only really have it forbdrv_snapshot_goto().
Hm, but then, what use is supporting other children for creating a snapshot when you can't load it any more afterwards?
Kevin
Re: [Qemu-devel] [PATCH v6 20/42] block/snapshot: Fix fallback
Quoted 59 lines · Kevin Wolf
Am 10.09.2019 um 14:04 hat Max Reitz geschrieben:
Quoted 56 lines · Kevin Wolf
Quoted 46 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 45 lines
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child.
bdrv_snapshot_goto()becomes a bit weird because we may have to redirect the actual child pointer, so it only works if the fallback child isbs->fileorbs->backing(and then we have to find out which it is).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 100 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 79 insertions(+), 21 deletions(-)block/snapshot.c +22
-0@@ -146,6 +146,32 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return the child BDS to which we can fall back if the given BDS +* does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +*/ +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BlockDriverState *child_bs =NULL ; +BdrvChild *child; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore: COW children need not be included in snapshots */ +continue ; + } + +if (child_bs) { +/* Cannot fall back to a single child if there are multiple */ +return NULL ; + } + child_bs = child->bs; + }Quoted 8 lines
block/snapshot.c bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs, · line 173
Quoted 3 lines
block/snapshot.c +3
-0+ +return child_bs; +}Why do we return
child->bshere whenbdrv_snapshot_goto()then needs to reconstruct what the associated BdrvChild was? Wouldn't it make more sense to return BdrvChild** from here and maybe have a small wrapper for the other functions that only need a BDS?Quoted 1 line
What would you return instead? &child doesn’t work.
Quoted 8 lines
Oops, brain fart. :-)
Quoted 3 lines
We could limit ourselves to
bs->fileandbs->backing. It just seemed like a bit of an artificial limit to me, because we only really have it forbdrv_snapshot_goto().Hm, but then, what use is supporting other children for creating a snapshot when you can't load it any more afterwards?
Well, the snapshot is still there, it’s just on a different node. So in theory, you could take a snapshot in a live VM (where the snapshotting node is not at the top), and then later revert to it with qemu-img (by accessing the file with the snapshot directly).
Though in practice this is just a fallback anyway, and I don’t think we currently have anything where it would make sense to fall through some node to any child but .file or .backing. So why not. It would be shorter, too. (Well, plus the short comment why looking at .file and .backing is sufficient.)
Max
PATCH v6 21/42 block: Use CAFs for debug breakpoints
In reply to Max ReitzWhen looking for a blkdebug node (which implements debug breakpoints), use bdrv_primary_bs() to iterate through the graph, because that is where a blkdebug node would be.
--- block.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
block.c +4 -4
@@ -5218,7 +5218,7 @@ int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event, const char *tag) {while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {
@@ -5231,7 +5231,7 @@ int bdrv_debug_breakpoint(BlockDriverState *bs, const char *event,int bdrv_debug_remove_breakpoint (BlockDriverState *bs,const char *tag) {while (bs && bs->drv && !bs->drv->bdrv_debug_remove_breakpoint) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_remove_breakpoint) {
@@ -5244,7 +5244,7 @@ int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)int bdrv_debug_resume (BlockDriverState *bs,const char *tag) {while (bs && (!bs->drv || !bs->drv->bdrv_debug_resume)) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_resume) {
@@ -5257,7 +5257,7 @@ int bdrv_debug_resume(BlockDriverState *bs, const char *tag)bool bdrv_debug_is_suspended (BlockDriverState *bs,const char *tag) {while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
Signature · 1 line
2.21.0
PATCH v6 22/42 block: Fix bdrv_get_allocated_file_size's fallback
In reply to Max ReitzIf the driver does not implement bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of just bs->file.
--- block.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)
block.c +20 -2
@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (!QLIST_EMPTY (&bs->children)) { +BdrvChild *child; +int64_t child_size, total_size =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore COW backing files */ +continue ; + } + + child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size; + } + total_size += child_size; + } + +return total_size; } +return -ENOTSUP ; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 35 lines · Max Reitz
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)block.c +14
-2@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (!QLIST_EMPTY (&bs->children)) { +BdrvChild *child; +int64_t child_size, total_size =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore COW backing files */ +continue ; + } + + child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size;
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676
Quoted 10 lines
block.c +6
-0+ } + total_size += child_size; + } + +return total_size; } +return -ENOTSUP ; }
Hmm..
1. No children -> -ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUM
It's all arguable (the strictest way is -ENOTSUP in either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 35 lines · Max Reitz
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)block.c +14
-2@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (!QLIST_EMPTY (&bs->children)) { +BdrvChild *child; +int64_t child_size, total_size =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore COW backing files */ +continue ; + } + + child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size;block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676
Quoted 10 lines
block.c +6
-0+ } + total_size += child_size; + } + +return total_size; } +return -ENOTSUP ; }Hmm..
Quoted 7 lines
1. No children ->
-ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUMIt's all arguable (the strictest way is
-ENOTSUPin either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.
I don’t think 2 is possible at all. If you have a COW child, you need some other child to COW to.
And in the weird (and probably impossible) case where a node really only has a COW child, I’d say it’s correct that it has a disk size of 0 – because it hasn’t COWed anything yet. (Just like a new qcow2 image with a backing file only has its metadata as its disk size.)
Max
Vladimir Sementsov-Ogievskiy
Reviewed-byRe: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 58 lines · Max Reitz
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 35 lines · Max Reitz
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)block.c +14
-2@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (!QLIST_EMPTY (&bs->children)) { +BdrvChild *child; +int64_t child_size, total_size =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore COW backing files */ +continue ; + } + + child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size;block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676
Quoted 10 lines
block.c +6
-0+ } + total_size += child_size; + } + +return total_size; } +return -ENOTSUP ; }Hmm..
Quoted 7 lines
1. No children ->
-ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUMIt's all arguable (the strictest way is
-ENOTSUPin either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.
Quoted 8 lines
I don’t think 2 is possible at all. If you have a COW child, you need some other child to COW to.
And in the weird (and probably impossible) case where a node really only has a COW child, I’d say it’s correct that it has a disk size of 0 – because it hasn’t COWed anything yet. (Just like a new qcow2 image with a backing file only has its metadata as its disk size.)
Agreed. Then, why not return 0 on [1] ?
Also, another idea: shouldn't we return 0 for filters, i.e. skip filtered_rw_child too? [as filtered-child is more like backing child than file one, it's "less owned" by its parent]
with or without any of these suggestions:
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 62 lines · Vladimir Sementsov-Ogievskiy
Quoted 61 lines · Max Reitz
Quoted 49 lines · Vladimir Sementsov-Ogievskiy
Quoted 35 lines · Max Reitz
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block.c | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-)block.c +14
-2@@ -4650,9 +4650,27 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (!QLIST_EMPTY (&bs->children)) { +BdrvChild *child; +int64_t child_size, total_size =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child ==bdrv_filtered_cow_child (bs)) { +/* Ignore COW backing files */ +continue ; + } + + child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size;block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 4676
Quoted 10 lines
block.c +6
-0+ } + total_size += child_size; + } + +return total_size; } +return -ENOTSUP ; }Hmm..
Quoted 7 lines
1. No children ->
-ENOTSUP
2. Only cow child -> 0
3. Some non-cow children -> SUMIt's all arguable (the strictest way is
-ENOTSUPin either case), but if we want to fallback to SUM of non-cow children, 1. and 2. should return the same.I don’t think 2 is possible at all. If you have a COW child, you need some other child to COW to.
Quoted 7 lines
Quoted 5 lines
And in the weird (and probably impossible) case where a node really only has a COW child, I’d say it’s correct that it has a disk size of 0 – because it hasn’t COWed anything yet. (Just like a new qcow2 image with a backing file only has its metadata as its disk size.)
Agreed. Then, why not return 0 on [1] ?
(1) Because that’s the current behavior. :-)
(2) Nodes that have no children are protocol nodes. Protocol nodes (apart from null) still have to store their data somewhere. Therefore, they must implement .bdrv_get_allocated_file_size() to report that. If they don’t, that doesn’t mean they don’t store any data, but only that we don’t know how much data they store.
Quoted 2 lines
Also, another idea: shouldn't we return 0 for filters, i.e. skip
filtered_rw_childtoo? [as filtered-child is more like backing child than file one, it's "less owned" by its parent]
Why would we do that? If I have a block device with a throttle node attached to it and request how much space it uses, of course I will want to know how much space the whole tree below it uses.
(Otherwise, bdrv_get_allocated_file_size() should only report anything for protocol nodes, and 0 for everything else.)
Max
Quoted 2 lines
with or without any of these suggestions:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>
This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.
Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.
Or eventually the block job filter nodes.
Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.
And please adjust the comment for bdrv_get_allocated_file_size(), it only talks about a single file as if trees didn't exist. Actually, it doesn't even seem so easy to define. Maybe primary node + storage nodes? Then vmdk needs to expose its extents as storage nodes (plural!), but in the long run that might be needed anyway.
Kevin
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 13 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.
Quoted 4 lines
Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.
Or eventually the block job filter nodes.
I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.
If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?
Quoted 5 lines
Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in
block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.
I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.
Max
Quoted 5 lines
And please adjust the comment for
bdrv_get_allocated_file_size(), it only talks about a single file as if trees didn't exist. Actually, it doesn't even seem so easy to define. Maybe primary node + storage nodes? Then vmdk needs to expose its extents as storage nodes (plural!), but in the long run that might be needed anyway.
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:
Quoted 14 lines · Kevin Wolf
Quoted 13 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.
Quoted 7 lines
Quoted 4 lines
Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.
Or eventually the block job filter nodes.
I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.
Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.
If you include the block job filter nodes, I have to disagree, though. If mirror_top_bs (or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.
Quoted 5 lines
If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?
Do it's the quorum way: Just vote!
No, you're right, of course. -ENOTSUP is probably the only other thing you could do then.
Quoted 8 lines
Quoted 5 lines
Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in
block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.
Well, by default the primary child, which should cover like 90% of the drivers?
Kevin
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 29 lines · Kevin Wolf
Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:
Quoted 14 lines · Kevin Wolf
Quoted 13 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.
Quoted 7 lines
Quoted 4 lines
Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.
Or eventually the block job filter nodes.
I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.
Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.
Quoted 4 lines
If you include the block job filter nodes, I have to disagree, though. If
mirror_top_bs(or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.
Hm, I don’t care much either way. I think it makes complete sense to add the target size there, but OTOH it’s only temporary while the job runs, so it may be a bit confusing if it suddenly goes up and then down again.
But I think this is the special case, so this is what should be handled in a driver callback.
Quoted 7 lines
Quoted 5 lines
If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?
Do it's the quorum way: Just vote!
Add an option for it? Average, maximum, median, majority, sum? :-)
Quoted 9 lines
No, you're right, of course.
-ENOTSUPis probably the only other thing you could do then.Quoted 6 lines
Quoted 5 lines
Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in
block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.
Quoted 5 lines
Quoted 2 lines
I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.
Well, by default the primary child, which should cover like 90% of the drivers?
Hm, yes.
But I still think that the drivers that do not want to count every single non-COW child are the exception.
Max
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:
Quoted 35 lines · Kevin Wolf
Quoted 29 lines
Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:
Quoted 14 lines · Kevin Wolf
Quoted 13 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.
Quoted 7 lines
Quoted 4 lines
Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.
Or eventually the block job filter nodes.
I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.
Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.
Quoted 4 lines
If you include the block job filter nodes, I have to disagree, though. If
mirror_top_bs(or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.
Quoted 4 lines
Hm, I don’t care much either way. I think it makes complete sense to add the target size there, but OTOH it’s only temporary while the job runs, so it may be a bit confusing if it suddenly goes up and then down again.
I think the number that most users are interested in is knowing how much space the image for their /dev/vda takes up on the host.
I can see how they might be interested in not only that one image file, but all other image files connected to it, i.e. their /dev/vda with all of its snapshots. This would mean counting backing files. I think adding up the numbers for this should be done in the management layer.
I can possibly also imagine users wanting to count everything that's even loosely connected to their /dev/vda, like copies of it. I doubt, however, they want to count only copies that are currently being made, but not snapshots and copies that have been completed earlier. So this is clearly a management layer thing, too.
Quoted 2 lines
But I think this is the special case, so this is what should be handled in a driver callback.
It's a special case, yes. But see below.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?
Do it's the quorum way: Just vote!
Add an option for it? Average, maximum, median, majority, sum? :-)
We could also introduce a mode with an Electoral College so that sometimes an image that missed the majority has a chance to win anyway.
Quoted 12 lines
Quoted 12 lines
No, you're right, of course.
-ENOTSUPis probably the only other thing you could do then.Quoted 8 lines
Quoted 5 lines
Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in
block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.
Quoted 7 lines
Quoted 2 lines
Well, by default the primary child, which should cover like 90% of the drivers?
Hm, yes.
But I still think that the drivers that do not want to count every single non-COW child are the exception.
They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.
I really think we're likely to forget adding the callback where we need to disable the feature.
I can see two options that should address both of our views:
1. Just don't have a fallback at all, make the callback mandatory and provide implementations in block.c that can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).
2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).
Adding the size of storage children covers qcow2 and vmdk.
As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.
For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.
Kevin
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 41 lines · Kevin Wolf
Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:
Quoted 35 lines · Kevin Wolf
Quoted 29 lines
Am 11.09.2019 um 08:20 hat Max Reitz geschrieben:
Quoted 14 lines · Kevin Wolf
Quoted 13 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
If the driver does not implement
bdrv_get_allocated_file_size(), we should fall back to cumulating the allocated size of all non-COW children instead of justbs->file.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>This smells like an overgeneralisation, but if we want to count all vmdk extents, the qcow2 external data file, etc. it's an improvement anyway. A driver that has a child that should not be counted must just remember to implement the callback.
Quoted 7 lines
Quoted 4 lines
Let me think of an example... How about quorum, for a change? :-) Or the second blkverify child.
Or eventually the block job filter nodes.
I actually think it makes sense for all of these nodes to report the sum of all of their children’s allocated sizes.
Hm... Yes, in a way. But not much more than it would make sense to report the sum of the sizes of all images in the whole backing chain (this is a useful thing to ask for, just maybe not the right thing to return for a low-level interface). But I can accept that it's maybe a bit more expected for quorum and blkverify than for COW images.
Quoted 4 lines
If you include the block job filter nodes, I have to disagree, though. If
mirror_top_bs(or any other job filter) sits in the middle of the source chain, then I certainly don't want to see the target size added to it.Quoted 4 lines
Hm, I don’t care much either way. I think it makes complete sense to add the target size there, but OTOH it’s only temporary while the job runs, so it may be a bit confusing if it suddenly goes up and then down again.
Quoted 7 lines
I think the number that most users are interested in is knowing how much space the image for their
/dev/vdatakes up on the host.I can see how they might be interested in not only that one image file, but all other image files connected to it, i.e. their
/dev/vdawith all of its snapshots. This would mean counting backing files. I think adding up the numbers for this should be done in the management layer.
My main argument against counting backing files is that we’ve never done it.
(Whereas for quorum, I’d argue we just forgot to adjust bdrv_get_allocated_file_size() for it.)
Quoted 5 lines
I can possibly also imagine users wanting to count everything that's even loosely connected to their
/dev/vda, like copies of it. I doubt, however, they want to count only copies that are currently being made, but not snapshots and copies that have been completed earlier. So this is clearly a management layer thing, too.
OK.
Quoted 11 lines
Quoted 2 lines
But I think this is the special case, so this is what should be handled in a driver callback.
It's a special case, yes. But see below.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
If a quorum node has three children with allocated sizes of 3 MB, 1 MB, and 2 MB, respectively (totally possible if some have explicit zeroes and others don’t; it may also depend on the protocol, the filesystem, etc.), then I think it makes most sense to report indeed 6 MB for the quorum subtree as a whole. What would you report? 3 MB?
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Do it's the quorum way: Just vote!
Add an option for it? Average, maximum, median, majority, sum? :-)
We could also introduce a mode with an Electoral College so that sometimes an image that missed the majority has a chance to win anyway.
That’s actually a good idea for a quorum mode in general. Who says the majority is right? Better let someone with more authority cross-check the result.
Quoted 20 lines
Quoted 12 lines
Quoted 12 lines
No, you're right, of course.
-ENOTSUPis probably the only other thing you could do then.Quoted 8 lines
Quoted 5 lines
Ehm... Maybe I should just take back what I said first. It almost feels like it would be better if qcow2 and vmdk explicitly used a handler that counts all children (could still be a generic one in
block.c) rather than having to remember to disable the functionality everywhere where we don't want to have it.I don’t, because everywhere we don’t want this functionality, we still need to choose a child. This has to be done by the driver anyway.
Quoted 7 lines
Quoted 2 lines
Well, by default the primary child, which should cover like 90% of the drivers?
Hm, yes.
But I still think that the drivers that do not want to count every single non-COW child are the exception.
Quoted 7 lines
They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.
I really think we're likely to forget adding the callback where we need to disable the feature.
Well, I mean, we did forget adding it for qcow2.
Quoted 8 lines
I can see two options that should address both of our views:
1. Just don't have a fallback at all, make the callback mandatory and provide implementations in
block.cthat can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).
Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.
Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.
Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?
Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)
Quoted 5 lines
2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).
I didn’t answer that because I didn’t disagree.
Quoted 1 line
Adding the size of storage children covers qcow2 and vmdk.
That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.
Quoted 8 lines
As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.
For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.
I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.
The better example is replication, but that has a clear filtered child (the primary node).
So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.
Max
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 15 lines
Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:
Quoted 8 lines · Kevin Wolf
Quoted 2 lines
Well, by default the primary child, which should cover like 90% of the drivers?
Hm, yes.
But I still think that the drivers that do not want to count every single non-COW child are the exception.
They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.
Quoted 4 lines
Quoted 2 lines
I really think we're likely to forget adding the callback where we need to disable the feature.
Well, I mean, we did forget adding it for qcow2.
I'm afraid I have to agree. So the conclusion is that we won't get it right anyway?
Quoted 9 lines
Quoted 8 lines
I can see two options that should address both of our views:
1. Just don't have a fallback at all, make the callback mandatory and provide implementations in
block.cthat can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).
Quoted 7 lines
Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.
Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.
Yes, I think this is the same as I meant by "provide implementations in block.c".
Quoted 9 lines
Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?
Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)
The job filter nodes have only one child currently, which should be counted. We'll add other children that shouldn't be counted only later.
But we already have an idea of what possible extensions look like, so we can probably choose the right function from the start.
Quoted 6 lines
Quoted 5 lines
2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).
Quoted 8 lines
I didn’t answer that because I didn’t disagree.
Quoted 1 line
Adding the size of storage children covers qcow2 and vmdk.
That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.
Well, you added bdrv_storage_child(). I'd argue this interface is wrong because it assumes that only one storage child exists. You just didn't implement it for vmdk so that the problem didn't become apparent. It would have to return a list rather than a single child. So fixing the interface and then using it is what I was thinking.
Now that you mention a per-BdrvChild attribute, however, I start to wonder if the distinction between COW children, filter children, storage children, metadata children, etc. isn't really what BdrvChildRole was supposed to represent?
Maybe we want to split off child_storage from child_file, though it's not strictly necessary for this specific case because we want to treat both metadata and storage nodes the same. But it could be useful for other users of bdrv_storage_child(), if there are any.
Quoted 12 lines
Quoted 8 lines
As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.
For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.
I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.
Quoted 8 lines
The better example is replication, but that has a clear filtered child (the primary node).
So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.
Yes, that should work.
We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.
Kevin
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Quoted 55 lines · Kevin Wolf
Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 15 lines
Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:
Quoted 8 lines · Kevin Wolf
Quoted 2 lines
Well, by default the primary child, which should cover like 90% of the drivers?
Hm, yes.
But I still think that the drivers that do not want to count every single non-COW child are the exception.
They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.
Quoted 4 lines
Quoted 2 lines
I really think we're likely to forget adding the callback where we need to disable the feature.
Well, I mean, we did forget adding it for qcow2.
I'm afraid I have to agree. So the conclusion is that we won't get it right anyway?
Quoted 9 lines
Quoted 8 lines
I can see two options that should address both of our views:
1. Just don't have a fallback at all, make the callback mandatory and provide implementations in
block.cthat can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).Quoted 7 lines
Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.
Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.
Yes, I think this is the same as I meant by "provide implementations in
block.c".Quoted 9 lines
Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?
Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)
Quoted 5 lines
The job filter nodes have only one child currently, which should be counted. We'll add other children that shouldn't be counted only later.
But we already have an idea of what possible extensions look like, so we can probably choose the right function from the start.
Yep.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).
I didn’t answer that because I didn’t disagree.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Adding the size of storage children covers qcow2 and vmdk.
That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.
Well, you added bdrv_storage_child().I'd argue this interface is wrong
Yes, it probably is.
Quoted 9 lines
because it assumes that only one storage child exists. You just didn't implement it for vmdk so that the problem didn't become apparent. It would have to return a list rather than a single child. So fixing the interface and then using it is what I was thinking.
Now that you mention a per-BdrvChild attribute, however, I start to wonder if the distinction between COW children, filter children, storage children, metadata children, etc. isn't really what BdrvChildRole was supposed to represent?
That’s a good point.
Quoted 4 lines
Maybe we want to split off
child_storagefromchild_file, though it's not strictly necessary for this specific case because we want to treat both metadata and storage nodes the same. But it could be useful for other users ofbdrv_storage_child(), if there are any.
Possible. Maybe it turns out that at least for this series I don’t need bdrv_storage_child() at all.
Quoted 21 lines
Quoted 12 lines
Quoted 8 lines
As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.
For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.
I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.
Quoted 8 lines
The better example is replication, but that has a clear filtered child (the primary node).
So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.
Quoted 5 lines
Yes, that should work.
We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.
Hm. It feels like something that should be done before this series, actually.
So I think we should add at least a child role per child access function so that they match? And then maybe in bdrv_attach_child() assert that a BDS never has more than one primary or filtered child (a filtered child acts as a primary child, too), or more than one COW child. (And that these are always in bs->file or bs->backing so the child access functions do work.)
Max
Re: [Qemu-devel] [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
Am 11.09.2019 um 13:00 hat Max Reitz geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 11 lines
Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:
Quoted 4 lines
So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.
Yes, that should work.
We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.
Hm. It feels like something that should be done before this series, actually.
Quoted 6 lines
So I think we should add at least a child role per child access function so that they match? And then maybe in
bdrv_attach_child()assert that a BDS never has more than one primary or filtered child (a filtered child acts as a primary child, too), or more than one COW child. (And that these are always inbs->fileorbs->backingso the child access functions do work.)
Makes sense to me.
Kevin
Re: [PATCH v6 22/42] block: Fix bdrv_get_allocated_file_size's fallback
In reply to Max ReitzQuoted 133 lines · Max Reitz
Quoted 55 lines · Kevin Wolf
Am 11.09.2019 um 12:00 hat Max Reitz geschrieben:
Quoted 16 lines · Kevin Wolf
Quoted 15 lines
Am 11.09.2019 um 09:37 hat Max Reitz geschrieben:
Quoted 8 lines · Kevin Wolf
Quoted 2 lines
Well, by default the primary child, which should cover like 90% of the drivers?
Hm, yes.
But I still think that the drivers that do not want to count every single non-COW child are the exception.
They are, but drivers that want to count more than their primary node are exceptions, too. And I think you're more likely to remember adding the callback when you want to have a certain feature, not when you don't want to have it.
Quoted 4 lines
Quoted 2 lines
I really think we're likely to forget adding the callback where we need to disable the feature.
Well, I mean, we did forget adding it for qcow2.
I'm afraid I have to agree. So the conclusion is that we won't get it right anyway?
Quoted 9 lines
Quoted 8 lines
I can see two options that should address both of our views:
1. Just don't have a fallback at all, make the callback mandatory and provide implementations in
block.cthat can be referred to in BlockDriver. Not specifying the callback causes an assertion failure, so we'd hopefully notice it quite early (assuming that we run either 'qemu-img info' or 'query-block' on a configuration with the block driver, but I think that's faily safe to assume).Quoted 7 lines
Hm. Seems a bit much, but if we can’t agree on what’s a good general implementation that works for everything, this is probably the only thing that would actually keep us from forgetting to add special cases.
Though I actually don’t know. I’d probably add two globally available helpers, one that returns the sum of everything but the backing node, and one that just returns the primary node.
Yes, I think this is the same as I meant by "provide implementations in
block.c".Quoted 9 lines
Now if I were to make qcow2 use the primary node helper function, would we have remembered changing it once we added a data file?
Hmm. Maybe not, but it should be OK to just make everything use the sum helper, except the drivers that want the primary node. That should work for all cases. (I think that whenever a format driver suddenly gains more child nodes, we probably will want to count them. OTOH, everything that has nodes that shouldn’t be counted probably always wants to use the primary node helper function from the start.)
Quoted 5 lines
The job filter nodes have only one child currently, which should be counted. We'll add other children that shouldn't be counted only later.
But we already have an idea of what possible extensions look like, so we can probably choose the right function from the start.
Yep.
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
2. Make the 90% solution a 100% solution: Allow drivers to have multiple storage children (for vmdk) and then have the fallback add up the primary child plus all storage children. This is what I suggested as the documented semantics in my initial reply to this patch (that you chose not to answer).
I didn’t answer that because I didn’t disagree.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Adding the size of storage children covers qcow2 and vmdk.
That’s of course exactly what we’re trying to do, but the question is, how do we figure out that storage children? Make it a per-BdrvChild attribute? That seems rather heavy-handed, because I think we’d need it only here.
Well, you added bdrv_storage_child().I'd argue this interface is wrong
Yes, it probably is.
Quoted 9 lines
because it assumes that only one storage child exists. You just didn't implement it for vmdk so that the problem didn't become apparent. It would have to return a list rather than a single child. So fixing the interface and then using it is what I was thinking.
Now that you mention a per-BdrvChild attribute, however, I start to wonder if the distinction between COW children, filter children, storage children, metadata children, etc. isn't really what BdrvChildRole was supposed to represent?
That’s a good point.
Quoted 4 lines
Maybe we want to split off
child_storagefromchild_file, though it's not strictly necessary for this specific case because we want to treat both metadata and storage nodes the same. But it could be useful for other users ofbdrv_storage_child(), if there are any.Possible. Maybe it turns out that at least for this series I don’t need
bdrv_storage_child()at all.Quoted 21 lines
Quoted 12 lines
Quoted 8 lines
As the job filter won't declare the target or any other involved nodes their storage nodes (I hope), this will do the right thing for them, too.
For quorum and blkverify both ways could be justifiable. I think they probably shouldn't declare their children as storage nodes. They are more like filters that don't have a single filtered node. So some kind of almost-filters.
I don’t think quorum is a filter, and blkverify can only be justified to be a filter because it quits qemu when there is a mismatch.
Quoted 8 lines
The better example is replication, but that has a clear filtered child (the primary node).
So all in all I think it’s best to make the callback mandatory and add two global helper functions. That’s simple enough and should prevent us from making mistakes by forgetting to adjust something in the future.
Quoted 5 lines
Yes, that should work.
We should probably still figure out what the relationship between the child access functions and child roles is, even if we don't need it for this solution. But it feels like an important part of the design.
Hm. It feels like something that should be done before this series, actually.
Quoted 6 lines
So I think we should add at least a child role per child access function so that they match? And then maybe in
bdrv_attach_child()assert that a BDS never has more than one primary or filtered child (a filtered child acts as a primary child, too), or more than one COW child. (And that these are always inbs->fileorbs->backingso the child access functions do work.)
I’ve been trying to make this work, but I don’t think it does. It just feels all wrong and I need up with things like “child_metadata_and_data”. The last straw was that blkverify should have the raw file be the filtered child (because, well, it’s bs->file), but then the format file would need to be a non-filtered child, and those would default to BDRV_O_PROTOCOL (which we decidedly don’t want).
Anyway, I’m currently attempting to solve this differently:
BdrvChildRole isn’t suitable for the job, I think. The name is completely what we want, but it actually doesn’t look like something that describes the child role to me.
Instead, I’m introducing a new BdrvChildRole enum mask that describes how the child is going to be used: stay-at-node, cow, metadata, data, etc.
I’m going to rename the current BdrvChildRole structure to BdrvChildParent (in want of a better name), because really most of what it does is describe the parent, but precisely not the child. I’m moving .stay_as_node to the new BdrvChildRole enum.
I hope this lets me unify child_file, child_backing, and child_format into a child_of_bds object. The callbacks should then decide the particularities based on the BdrvChildRole enum.
Hope that makes sense. (? :S)
At least I feel much happier implementing it this way, which I suppose is a good sign.
Max
PATCH v6 23/42 blockdev: Use CAF in external_snapshot_prepare()
In reply to Max ReitzThis allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
--- blockdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +6 -1
@@ -1664,7 +1664,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } -if (state->new_bs->backing !=NULL ) { +if (state->new_bs->drv->is_filter) { +error_setg (errp,"Filters cannot be used as overlays" ); + goto out; + } + +if (bdrv_filtered_cow_child (state->new_bs)) {error_setg (errp,"The overlay already has a backing image" ); goto out; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 23/42] blockdev: Use CAF in external_snapshot_prepare()
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
Didn't we occasionally advertise blockdev-snapshot as the way to insert filters on top at runtime? Though it seems it has always only worked for filters that use bs->backing, among which I think there aren't any user-creatable ones. So we're probably good.
Kevin
Quoted 24 lines
blockdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)blockdev.c +6
-1@@ -1664,7 +1664,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } -if (state->new_bs->backing !=NULL ) { +if (state->new_bs->drv->is_filter) { +error_setg (errp,"Filters cannot be used as overlays" ); + goto out; + } + +if (bdrv_filtered_cow_child (state->new_bs)) {error_setg (errp,"The overlay already has a backing image" ); goto out; }Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 23/42] blockdev: Use CAF in external_snapshot_prepare()
Quoted 10 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 6 lines
This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>Didn't we occasionally advertise blockdev-snapshot as the way to insert filters on top at runtime?
I can only remember advertising for it as the only graph manipulation tool we had, and maybe saying “We’d want something like blockdev-snapshot for filters, too”.
Max
Quoted 30 lines
Though it seems it has always only worked for
filters that usebs->backing, among which I think there aren't any user-creatable ones. So we're probably good.Kevin
Quoted 24 lines
blockdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)blockdev.c +6
-1@@ -1664,7 +1664,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } -if (state->new_bs->backing !=NULL ) { +if (state->new_bs->drv->is_filter) { +error_setg (errp,"Filters cannot be used as overlays" ); + goto out; + } + +if (bdrv_filtered_cow_child (state->new_bs)) {error_setg (errp,"The overlay already has a backing image" ); goto out; }Signature · 1 line
2.21.0
PATCH v6 24/42 block: Use child access functions for QAPI queries
In reply to Max Reitzquery-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just bs->backing or COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.
--- block/qapi.c | 39 +++++++++++++++++++++++--------------- tests/qemu-iotests/184.out | 7 ++++++- 2 files changed, 30 insertions(+), 16 deletions(-)
block/qapi.c +24 -15
@@ -156,9 +156,13 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,return NULL ; } -if (bs0->drv && bs0->backing) { +if (bs0->drv &&bdrv_filtered_child (bs0)) { +/* +* Put any filtered child here (for backwards compatibility to when +* we put bs0->backing here, which might be any filtered child). +*/ info->backing_file_depth++; - bs0 = bs0->backing->bs; + bs0 =bdrv_filtered_bs (bs0); (*p_image_info)->has_backing_image = true; p_image_info = &((*p_image_info)->backing_image); }else {
@@ -167,9 +171,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,/* Skip automatically inserted nodes that the user isn't aware of for * query-block (blk != NULL), but not for query-named-block-nodes */ -while (blk && bs0->drv && bs0->implicit) { - bs0 =backing_bs (bs0); -assert (bs0); +if (blk) { + bs0 =bdrv_skip_implicit_filters (bs0); } }
@@ -354,9 +357,9 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,BlockDriverState *bs =blk_bs (blk);char *qdev; -/* Skip automatically inserted nodes that the user isn't aware of */ -while (bs && bs->drv && bs->implicit) { - bs =backing_bs (bs); +if (bs) { +/* Skip automatically inserted nodes that the user isn't aware of */ + bs =bdrv_skip_implicit_filters (bs); } info->device =g_strdup (blk_name (blk));
@@ -513,6 +516,7 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk)static BlockStats *bdrv_query_bds_stats (BlockDriverState *bs,bool blk_level) { +BlockDriverState *storage_bs, *filtered_bs;BlockStats *s =NULL ; s =g_malloc0 (sizeof (*s));
@@ -525,9 +529,8 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,/* Skip automatically inserted nodes that the user isn't aware of in * a BlockBackend-level command. Stay at the exact node for a node-level * command. */ -while (blk_level && bs->drv && bs->implicit) { - bs =backing_bs (bs); -assert (bs); +if (blk_level) { + bs =bdrv_skip_implicit_filters (bs); }if (bdrv_get_node_name (bs)[0 ]) {
@@ -537,14 +540,20 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, s->stats->wr_highest_offset =stat64_get (&bs->wr_highest_offset); -if (bs->file) { + storage_bs =bdrv_storage_bs (bs); +if (storage_bs) { s->has_parent = true; - s->parent =bdrv_query_bds_stats (bs->file->bs, blk_level); + s->parent =bdrv_query_bds_stats (storage_bs, blk_level); } -if (blk_level && bs->backing) { + filtered_bs =bdrv_filtered_bs (bs); +if (blk_level && filtered_bs) { +/* +* Put any filtered child here (for backwards compatibility to when +* we put bs0->backing here, which might be any filtered child). +*/ s->has_backing = true; - s->backing =bdrv_query_bds_stats (bs->backing->bs, blk_level); + s->backing =bdrv_query_bds_stats (filtered_bs, blk_level); }return s;
tests/qemu-iotests/184.out +6 -1
@@ -27,6 +27,11 @@ Testing: "iops_rd": 0, "detect_zeroes": "off", "image": { + "backing-image": { + "virtual-size": 1073741824, + "filename": "null-co://", + "format": "null-co" + }, "virtual-size": 1073741824, "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}", "format": "throttle"
@@ -34,7 +39,7 @@ Testing: "iops_wr": 0, "ro": false, "node-name": "throttle0", - "backing_file_depth": 0, + "backing_file_depth": 1, "drv": "throttle", "iops": 0, "bps_wr": 0,
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 24/42] block: Use child access functions for QAPI queries
Quoted 16 lines · Max Reitz
query-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just
bs->backingor COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.Signed-off-by Max Reitz <[email protected]>--- block/qapi.c | 39 +++++++++++++++++++++++--------------- tests/qemu-iotests/184.out | 7 ++++++- 2 files changed, 30 insertions(+), 16 deletions(-)block/qapi.c +0
-0
[..]
block/qapi.c static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info, · line 363
Quoted 11 lines
block/qapi.c +3
-3@@ -354,9 +357,9 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,BlockDriverState *bs =blk_bs (blk);char *qdev; -/* Skip automatically inserted nodes that the user isn't aware of */ -while (bs && bs->drv && bs->implicit) { - bs =backing_bs (bs); +if (bs) { +/* Skip automatically inserted nodes that the user isn't aware of */ + bs =bdrv_skip_implicit_filters (bs); }
bdrv_skip_implicit_filters supports NULL, so it may be written without "if"
Anyway:
[..]
Signature · 2 lines
Best regards, Vladimir
PATCH v6 25/42 mirror: Deal with filters
In reply to Max ReitzThis includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)
block/mirror.c +91 -26
@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;
@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;
@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does
@@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }
@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);
@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret; -assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target)); + ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }
@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(MirrorBlockJob *s;MirrorBDSOpaque *bs_opaque;BlockDriverState *mirror_top_bs; -bool target_graph_mod;bool target_is_backing; +uint64_t target_perms, target_shared_perms;Error *local_err =NULL ;int ret;
@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job( buf_size =DEFAULT_MIRROR_BUF_SIZE ; } -if (bs == target) { +if (bdrv_skip_rw_filters (bs) ==bdrv_skip_rw_filters (target)) {error_setg (errp,"Can't mirror node into itself" );return NULL ; }
@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(* In the case of active commit, things look a bit different, though, * because the target is an already populated backing file in active use. * We can allow anything except resize there.*/ + + target_perms =BLK_PERM_WRITE ; + target_shared_perms =BLK_PERM_WRITE_UNCHANGED ; + target_is_backing =bdrv_chain_contains (bs, target); - target_graph_mod = (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ); +if (target_is_backing) { +int64_t bs_size, target_size; + bs_size =bdrv_getlength (bs); +if (bs_size <0 ) { +error_setg_errno (errp, -bs_size, +"Could not inquire top image size" ); + goto fail; + } + + target_size =bdrv_getlength (target); +if (target_size <0 ) { +error_setg_errno (errp, -target_size, +"Could not inquire base image size" ); + goto fail; + } + +if (target_size < bs_size) { + target_perms |=BLK_PERM_RESIZE ; + } + + target_shared_perms |=BLK_PERM_CONSISTENT_READ + |BLK_PERM_WRITE + |BLK_PERM_GRAPH_MOD ; + }else if (bdrv_chain_contains (bs,bdrv_skip_rw_filters (target))) { +/* +* We may want to allow this in the future, but it would +* require taking some extra care. +*/ +error_setg (errp,"Cannot mirror to a filter on top of a node in the " +"source's backing chain" ); + goto fail; + } + +if (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ) { + target_perms |=BLK_PERM_GRAPH_MOD ; + } + s->target =blk_new (s->common.job.aio_context, -BLK_PERM_WRITE |BLK_PERM_RESIZE | - (target_graph_mod ?BLK_PERM_GRAPH_MOD :0 ), -BLK_PERM_WRITE_UNCHANGED | - (target_is_backing ?BLK_PERM_CONSISTENT_READ | -BLK_PERM_WRITE | -BLK_PERM_GRAPH_MOD :0 )); + target_perms, target_shared_perms); ret =blk_insert_bs (s->target, target, errp);if (ret <0 ) { goto fail;
@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job( s->backing_mode = backing_mode; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;
@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }
@@ -1737,7 +1802,7 @@ fail: bs_opaque->stop = true;bdrv_child_refresh_perms (mirror_top_bs, mirror_top_bs->backing, &error_abort); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);bdrv_unref (mirror_top_bs);
@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,return ; } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; - base = mode ==MIRROR_SYNC_MODE_TOP ?backing_bs (bs) :NULL ; + base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; mirror_start_job(job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, on_source_error, on_target_error, unmap, NULL, NULL,
blockdev.c +40 -7
114 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 14 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +0
-0
[..]
Quoted 29 lines
block/mirror.c +19
-6@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ;
block/mirror.c static BlockJob *mirror_start_job( · line 1765
Quoted 10 lines
block/mirror.c +10
-0+ +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ;
Hmm, I don't understand, why read from upper nodes is not shared?
Quoted 9 lines
block/mirror.c +3
-2+ } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }
[..]
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +0
-0[..]
Quoted 34 lines
block/mirror.c +24
-6@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) {
Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1765
Quoted 5 lines
block/mirror.c +5
-0+/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ;Hmm, I don't understand, why read from upper nodes is not shared?
Because they don’t represent a consistent disk state during the commit.
Please don’t ask me details about CONSISTENT_READ, because I always pretend I understand it, but I never really do, actually.
(My problem is that I do understand why the intermediate nodes shouldn’t share CONSISTENT_READ: It’s because they only read garbage, effectively.
But I don’t understand how any block job target (like our base here)
can have CONSISTENT_READ. Block job targets are mostly written front to back (except with sync=none), so they too don’t “[represent] the contents of a disk at a specific point.”
But that is how it was, so that is how it should be kept.)
If it makes you any happier, BLK_PERM_CONSISTENT_READ’s description explicitly notes that it will not be shared on intermediate nodes of a commit job.
Max
Quoted 12 lines
Quoted 9 lines
block/mirror.c +3
-2+ } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }[..]
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 65 lines · Max Reitz
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +0
-0[..]
Quoted 34 lines
block/mirror.c +24
-6@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) {Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1765
Quoted 5 lines
block/mirror.c +5
-0+/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ;Hmm, I don't understand, why read from upper nodes is not shared?
Because they don’t represent a consistent disk state during the commit.
Quoted 7 lines
Please don’t ask me details about
CONSISTENT_READ, because I always pretend I understand it, but I never really do, actually.(My problem is that I do understand why the intermediate nodes shouldn’t share
CONSISTENT_READ: It’s because they only read garbage, effectively.
But I don’t understand how any block job target (like our base here)
can haveCONSISTENT_READ.
I know such example: it's image fleecing scheme, when for backup job source is a backing for target. If serialization of requests works well target represents consistent state of disk ate backup-start point in time.
But yes, it's not about mirror or commit.
Quoted 25 lines
Block job targets are mostly written front to
back (except with sync=none), so they too don’t “[represent] the contents of a disk at a specific point.”
But that is how it was, so that is how it should be kept.)If it makes you any happier, BLK_PERM_CONSISTENT_READ’s description explicitly notes that it will not be shared on intermediate nodes of a commit job.
Max
Quoted 12 lines
Quoted 9 lines
block/mirror.c +3
-2+ } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }[..]
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
In reply to Max ReitzQuoted 64 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +9
-5@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does @@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);
block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095
Quoted 6 lines
block/mirror.c +1
-2@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret; -assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target));
Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.
Quoted 26 lines
block/mirror.c +4
-2+ ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(MirrorBlockJob *s;MirrorBDSOpaque *bs_opaque;BlockDriverState *mirror_top_bs; -bool target_graph_mod;bool target_is_backing; +uint64_t target_perms, target_shared_perms;Error *local_err =NULL ;int ret;@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job( buf_size =DEFAULT_MIRROR_BUF_SIZE ; } -if (bs == target) { +if (bdrv_skip_rw_filters (bs) ==bdrv_skip_rw_filters (target)) {error_setg (errp,"Can't mirror node into itself" );return NULL ; }@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(* In the case of active commit, things look a bit different, though,
block/mirror.c static BlockJob *mirror_start_job( · line 1629
Quoted 10 lines
block/mirror.c +6
-1* because the target is an already populated backing file in active use. * We can allow anything except resize there.*/ + + target_perms =BLK_PERM_WRITE ; + target_shared_perms =BLK_PERM_WRITE_UNCHANGED ; + target_is_backing =bdrv_chain_contains (bs, target); - target_graph_mod = (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ); +if (target_is_backing) { +int64_t bs_size, target_size;
<empty after definitions>
Quoted 81 lines
block/mirror.c +54
-12+ bs_size =bdrv_getlength (bs); +if (bs_size <0 ) { +error_setg_errno (errp, -bs_size, +"Could not inquire top image size" ); + goto fail; + } + + target_size =bdrv_getlength (target); +if (target_size <0 ) { +error_setg_errno (errp, -target_size, +"Could not inquire base image size" ); + goto fail; + } + +if (target_size < bs_size) { + target_perms |=BLK_PERM_RESIZE ; + } + + target_shared_perms |=BLK_PERM_CONSISTENT_READ + |BLK_PERM_WRITE + |BLK_PERM_GRAPH_MOD ; + }else if (bdrv_chain_contains (bs,bdrv_skip_rw_filters (target))) { +/* +* We may want to allow this in the future, but it would +* require taking some extra care. +*/ +error_setg (errp,"Cannot mirror to a filter on top of a node in the " +"source's backing chain" ); + goto fail; + } + +if (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ) { + target_perms |=BLK_PERM_GRAPH_MOD ; + } + s->target =blk_new (s->common.job.aio_context, -BLK_PERM_WRITE |BLK_PERM_RESIZE | - (target_graph_mod ?BLK_PERM_GRAPH_MOD :0 ), -BLK_PERM_WRITE_UNCHANGED | - (target_is_backing ?BLK_PERM_CONSISTENT_READ | -BLK_PERM_WRITE | -BLK_PERM_GRAPH_MOD :0 )); + target_perms, target_shared_perms); ret =blk_insert_bs (s->target, target, errp);if (ret <0 ) { goto fail;@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job( s->backing_mode = backing_mode; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target).
block/mirror.c static BlockJob *mirror_start_job( · line 1763
Quoted 10 lines
block/mirror.c +10
-0+*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ.
I'd prefere to add something like: "because we share it on target (see target BlockBackend creation and corresponding comment above)".
blockdev.c
Quoted 31 lines
block/mirror.c +7
-4+*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -1737,7 +1802,7 @@ fail: bs_opaque->stop = true;bdrv_child_refresh_perms (mirror_top_bs, mirror_top_bs->backing, &error_abort); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);bdrv_unref (mirror_top_bs);@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,return ; } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; - base = mode ==MIRROR_SYNC_MODE_TOP ?backing_bs (bs) :NULL ; + base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; mirror_start_job(job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, on_source_error, on_target_error, unmap, NULL, NULL,blockdev.c +0
-0
block/mirroc.c is OK for me. Continue with blockdev.c...
Quoted 46 lines
blockdev.c +13
-2@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } -if (!bs->backing && sync ==MIRROR_SYNC_MODE_TOP ) { +if (!bdrv_backing_chain_next (bs) && sync ==MIRROR_SYNC_MODE_TOP ) { sync =MIRROR_SYNC_MODE_FULL ; }@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,void qmp_drive_mirror (DriveMirror *arg,Error **errp) { -BlockDriverState *bs; +BlockDriverState *bs, *unfiltered_bs;BlockDriverState *source, *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode;@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)int flags;int64_t size;const char *format = arg->format; +const char *replaces_node_name =NULL ;int ret; bs =qmp_get_root_bs (arg->device, errp);@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)return ; } +/* +* If the user has not instructed us otherwise, we should let the +* block job run from @bs (thus taking into account all filters on +* it) but replace @unfiltered_bs when it finishes (thus not +* removing those filters). +* (And if there are any explicit filters, we should assume the +* user knows how to use the @replaces option.) +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs); + aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3949
Quoted 6 lines
blockdev.c +1
-1@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } flags = bs->open_flags |BDRV_O_RDWR ; - source =backing_bs (bs); + source =bdrv_filtered_cow_bs (unfiltered_bs);if (!source && arg->sync ==MIRROR_SYNC_MODE_TOP ) {
Hmm, you handle this case a bit differently here and in blockdev_mirror_common.. Can we handle it only in blockdev_mirror_common, to be consistent with qmp_blockdev_mirror?
Quoted 59 lines
blockdev.c +16
-4+if (bdrv_filtered_bs (unfiltered_bs)) { +/* @unfiltered_bs is an explicit filter */ +error_setg (errp,"Cannot perform sync=top mirror through an " +"explicitly added filter node on the source" ); + goto out; + } arg->sync =MIRROR_SYNC_MODE_FULL ; }if (arg->sync ==MIRROR_SYNC_MODE_NONE ) {@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces; + }else if (unfiltered_bs != bs) { + replaces_node_name = unfiltered_bs->node_name; }if (arg->mode ==NEW_IMAGE_MODE_ABSOLUTE_PATHS ) {@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)bdrv_img_create (arg->target, format,NULL ,NULL ,NULL , size, flags, false, &local_err); }else { +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing =bdrv_skip_implicit_filters (source); +switch (arg->mode) {case NEW_IMAGE_MODE_EXISTING :break ;@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)/* create new image with backing file */ bdrv_refresh_filename (source);bdrv_img_create (arg->target, format, - source->filename, - source->drv->format_name, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err);break ;default :@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs, - arg->has_replaces, arg->replaces, arg->sync, + !!replaces_node_name, replaces_node_name, arg->sync, backing_mode, arg->has_speed, arg->speed, arg->has_granularity, arg->granularity, arg->has_buf_size, arg->buf_size,@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { - BlockDriverState *bs; + BlockDriverState *bs, *unfiltered_bs;BlockDriverState *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode =MIRROR_LEAVE_BACKING_CHAIN ;
blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4092
Quoted 5 lines
blockdev.c +2
-0@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,return ; } +/* +* Same as in qmp_drive_mirror():
Then, may be better do it in blockdev_mirror_common ?
Quoted 13 lines
blockdev.c +8
-0We want to run the job from @bs, +* but we want to replace @unfiltered_bs on completion. +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs); +if (!has_replaces && unfiltered_bs != bs) { + replaces = unfiltered_bs->node_name; + has_replaces = true; + } + target_bs =bdrv_lookup_bs (target, target, errp);if (!target_bs) {return;
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +9
-5@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does @@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret;
Quoted 7 lines
block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095
Quoted 3 lines
block/mirror.c +1
-2-assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target));Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.
The idea is that MIRROR_OPEN_BACKING_CHAIN is set only when using drive-mirror with mode=existing. In this case, we also set BDRV_O_NO_BACKING for the target.
You’re right that a user could add a backing chain to the target during the operation. They really have to make an effort to shoot themselves in the foot for this because the target must have an auto-generated node name.
I suppose the best would be not to open the backing chain if the target node already has a backing child?
Quoted 30 lines
Quoted 30 lines
block/mirror.c +6
-2+ ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(MirrorBlockJob *s;MirrorBDSOpaque *bs_opaque;BlockDriverState *mirror_top_bs; -bool target_graph_mod;bool target_is_backing; +uint64_t target_perms, target_shared_perms;Error *local_err =NULL ;int ret;@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job( buf_size =DEFAULT_MIRROR_BUF_SIZE ; } -if (bs == target) { +if (bdrv_skip_rw_filters (bs) ==bdrv_skip_rw_filters (target)) {error_setg (errp,"Can't mirror node into itself" );return NULL ; }@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(* In the case of active commit, things look a bit different, though, * because the target is an already populated backing file in active use. * We can allow anything except resize there.*/ + + target_perms =BLK_PERM_WRITE ;
Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1629
Quoted 6 lines
block/mirror.c +4
-1+ target_shared_perms =BLK_PERM_WRITE_UNCHANGED ; + target_is_backing =bdrv_chain_contains (bs, target); - target_graph_mod = (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ); +if (target_is_backing) { +int64_t bs_size, target_size;<empty after definitions>
Is that part of any of our guidelines? :-)
Sure, will add.
Quoted 86 lines
Quoted 86 lines
block/mirror.c +59
-12+ bs_size =bdrv_getlength (bs); +if (bs_size <0 ) { +error_setg_errno (errp, -bs_size, +"Could not inquire top image size" ); + goto fail; + } + + target_size =bdrv_getlength (target); +if (target_size <0 ) { +error_setg_errno (errp, -target_size, +"Could not inquire base image size" ); + goto fail; + } + +if (target_size < bs_size) { + target_perms |=BLK_PERM_RESIZE ; + } + + target_shared_perms |=BLK_PERM_CONSISTENT_READ + |BLK_PERM_WRITE + |BLK_PERM_GRAPH_MOD ; + }else if (bdrv_chain_contains (bs,bdrv_skip_rw_filters (target))) { +/* +* We may want to allow this in the future, but it would +* require taking some extra care. +*/ +error_setg (errp,"Cannot mirror to a filter on top of a node in the " +"source's backing chain" ); + goto fail; + } + +if (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ) { + target_perms |=BLK_PERM_GRAPH_MOD ; + } + s->target =blk_new (s->common.job.aio_context, -BLK_PERM_WRITE |BLK_PERM_RESIZE | - (target_graph_mod ?BLK_PERM_GRAPH_MOD :0 ), -BLK_PERM_WRITE_UNCHANGED | - (target_is_backing ?BLK_PERM_CONSISTENT_READ | -BLK_PERM_WRITE | -BLK_PERM_GRAPH_MOD :0 )); + target_perms, target_shared_perms); ret =blk_insert_bs (s->target, target, errp);if (ret <0 ) { goto fail;@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job( s->backing_mode = backing_mode; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter))
Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1763
Quoted 5 lines
block/mirror.c +5
-0+ { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. I'd prefere to add something like: "because we share it on target (see target BlockBackend creation and corresponding comment above)".
I’d rather not refer to other comments in case they change… Maybe just “This allows us to share BLK_PERM_CONSISTENT_READ, as we do on the target.”? I think if someone is interested, they will scan the file for what permissions are shared on the target anyway.
Quoted 83 lines
blockdev.c
Quoted 31 lines
block/mirror.c +7
-4+*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -1737,7 +1802,7 @@ fail: bs_opaque->stop = true;bdrv_child_refresh_perms (mirror_top_bs, mirror_top_bs->backing, &error_abort); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);bdrv_unref (mirror_top_bs);@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,return ; } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; - base = mode ==MIRROR_SYNC_MODE_TOP ?backing_bs (bs) :NULL ; + base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; mirror_start_job(job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, on_source_error, on_target_error, unmap, NULL, NULL,blockdev.c +0
-0
block/mirroc.cis OK for me. Continue withblockdev.c...Quoted 48 lines
blockdev.c +13
-2@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } -if (!bs->backing && sync ==MIRROR_SYNC_MODE_TOP ) { +if (!bdrv_backing_chain_next (bs) && sync ==MIRROR_SYNC_MODE_TOP ) { sync =MIRROR_SYNC_MODE_FULL ; }@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,void qmp_drive_mirror (DriveMirror *arg,Error **errp) { -BlockDriverState *bs; +BlockDriverState *bs, *unfiltered_bs;BlockDriverState *source, *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode;@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)int flags;int64_t size;const char *format = arg->format; +const char *replaces_node_name =NULL ;int ret; bs =qmp_get_root_bs (arg->device, errp);@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)return ; } +/* +* If the user has not instructed us otherwise, we should let the +* block job run from @bs (thus taking into account all filters on +* it) but replace @unfiltered_bs when it finishes (thus not +* removing those filters). +* (And if there are any explicit filters, we should assume the +* user knows how to use the @replaces option.) +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs); + aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) }
Quoted 8 lines
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3949
Quoted 4 lines
blockdev.c +1
-1flags = bs->open_flags |BDRV_O_RDWR ; - source =backing_bs (bs); + source =bdrv_filtered_cow_bs (unfiltered_bs);if (!source && arg->sync ==MIRROR_SYNC_MODE_TOP ) {Hmm, you handle this case a bit differently here and in
blockdev_mirror_common.. Can we handle it only inblockdev_mirror_common, to be consistent withqmp_blockdev_mirror?
What exactly do you mean? The difference between skipping all filters and just skipping implicit filters? Hm.
First, the check in blockdev_mirror_common() should actually be unnecessary. In qmp_{blockdev,drive}_mirror(), we do nearly the same check anyway (and then force sync=full if there is no backing file). So if all three functions did the same check, we wouldn’t need it in blockdev_mirror_common().
Second, let’s look at the difference in an example: One where blockdev_mirror_common() would not decide to enforce mode=full, but qmp_{blockdev,drive}_mirror() would.
This happens when @bs is an explicit filter over some overlay with a backing file, e.g.:
throttle --file--> qcow2 --backing--> raw
It’s correct to run the mirror job from the throttle node; but @source should be bdrv_backing_chain_next() so it will point to the raw node. Currently, it is NULL (because the throttle node does not have a COW child).
But then again, I’ve made qmp_{blockdev,drive}_mirror() throw an error in such a case:
Quoted 6 lines
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3955
Quoted 6 lines
blockdev.c +6
-0+if (bdrv_filtered_bs (unfiltered_bs)) { +/* @unfiltered_bs is an explicit filter */ +error_setg (errp,"Cannot perform sync=top mirror through an " +"explicitly added filter node on the source" ); + goto out; + }
So it isn’t really a problem. Still, does the error make sense? Should we just allow that case by letting source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs))?
(BTW, I just noticed that @base seems to be pretty much unused in block/mirror.c. It only really uses @base_overlay now. So I suppose it makes sense to remove it in v7.)
Quoted 56 lines
Quoted 56 lines
blockdev.c +10
-4arg->sync =MIRROR_SYNC_MODE_FULL ; }if (arg->sync ==MIRROR_SYNC_MODE_NONE ) {@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces; + }else if (unfiltered_bs != bs) { + replaces_node_name = unfiltered_bs->node_name; }if (arg->mode ==NEW_IMAGE_MODE_ABSOLUTE_PATHS ) {@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)bdrv_img_create (arg->target, format,NULL ,NULL ,NULL , size, flags, false, &local_err); }else { +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing =bdrv_skip_implicit_filters (source); +switch (arg->mode) {case NEW_IMAGE_MODE_EXISTING :break ;@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)/* create new image with backing file */ bdrv_refresh_filename (source);bdrv_img_create (arg->target, format, - source->filename, - source->drv->format_name, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err);break ;default :@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs, - arg->has_replaces, arg->replaces, arg->sync, + !!replaces_node_name, replaces_node_name, arg->sync, backing_mode, arg->has_speed, arg->speed, arg->has_granularity, arg->granularity, arg->has_buf_size, arg->buf_size,@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { - BlockDriverState *bs; + BlockDriverState *bs, *unfiltered_bs;BlockDriverState *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode =MIRROR_LEAVE_BACKING_CHAIN ;@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,return ; }
Quoted 4 lines
blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4092
Quoted 2 lines
blockdev.c +2
-0+/* +* Same as in qmp_drive_mirror(): Then, may be better do it in
blockdev_mirror_common?
Hm, maybe. Should we decide to let @source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs)) in qmp_drive_mirror(), I don’t think we need @unfiltered_bs there to determine @source.
Max
Quoted 15 lines
Quoted 13 lines
blockdev.c +8
-0We want to run the job from @bs, +* but we want to replace @unfiltered_bs on completion. +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs); +if (!has_replaces && unfiltered_bs != bs) { + replaces = unfiltered_bs->node_name; + has_replaces = true; + } + target_bs =bdrv_lookup_bs (target, target, errp);if (!target_bs) {return;
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 81 lines · Max Reitz
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +9
-5@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does @@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret;Quoted 7 lines
block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095
Quoted 3 lines
block/mirror.c +1
-2-assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target));Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.
The idea is that
MIRROR_OPEN_BACKING_CHAINis set only when using drive-mirror with mode=existing. In this case, we also setBDRV_O_NO_BACKINGfor the target.
Quoted 7 lines
You’re right that a user could add a backing chain to the target during the operation. They really have to make an effort to shoot themselves in the foot for this because the target must have an auto-generated node name.
I suppose the best would be not to open the backing chain if the target node already has a backing child?
Hmm, but we still should generate an error, as we can't do what was requested.
Quoted 40 lines
Quoted 31 lines
Quoted 31 lines
block/mirror.c +6
-2+ ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(MirrorBlockJob *s;MirrorBDSOpaque *bs_opaque;BlockDriverState *mirror_top_bs; -bool target_graph_mod;bool target_is_backing; +uint64_t target_perms, target_shared_perms;Error *local_err =NULL ;int ret;@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job( buf_size =DEFAULT_MIRROR_BUF_SIZE ; } -if (bs == target) { +if (bdrv_skip_rw_filters (bs) ==bdrv_skip_rw_filters (target)) {error_setg (errp,"Can't mirror node into itself" );return NULL ; }@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(* In the case of active commit, things look a bit different, though, * because the target is an already populated backing file in active use. * We can allow anything except resize there.*/ + + target_perms =BLK_PERM_WRITE ;Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1629
Quoted 6 lines
block/mirror.c +4
-1+ target_shared_perms =BLK_PERM_WRITE_UNCHANGED ; + target_is_backing =bdrv_chain_contains (bs, target); - target_graph_mod = (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ); +if (target_is_backing) { +int64_t bs_size, target_size;<empty after definitions>
Quoted 3 lines
Is that part of any of our guidelines? :-)
Sure, will add.
Not sure. Someone asked me about it on list in past and I'm used to.
Quoted 96 lines
Quoted 87 lines
Quoted 87 lines
block/mirror.c +59
-12+ bs_size =bdrv_getlength (bs); +if (bs_size <0 ) { +error_setg_errno (errp, -bs_size, +"Could not inquire top image size" ); + goto fail; + } + + target_size =bdrv_getlength (target); +if (target_size <0 ) { +error_setg_errno (errp, -target_size, +"Could not inquire base image size" ); + goto fail; + } + +if (target_size < bs_size) { + target_perms |=BLK_PERM_RESIZE ; + } + + target_shared_perms |=BLK_PERM_CONSISTENT_READ + |BLK_PERM_WRITE + |BLK_PERM_GRAPH_MOD ; + }else if (bdrv_chain_contains (bs,bdrv_skip_rw_filters (target))) { +/* +* We may want to allow this in the future, but it would +* require taking some extra care. +*/ +error_setg (errp,"Cannot mirror to a filter on top of a node in the " +"source's backing chain" ); + goto fail; + } + +if (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ) { + target_perms |=BLK_PERM_GRAPH_MOD ; + } + s->target =blk_new (s->common.job.aio_context, -BLK_PERM_WRITE |BLK_PERM_RESIZE | - (target_graph_mod ?BLK_PERM_GRAPH_MOD :0 ), -BLK_PERM_WRITE_UNCHANGED | - (target_is_backing ?BLK_PERM_CONSISTENT_READ | -BLK_PERM_WRITE | -BLK_PERM_GRAPH_MOD :0 )); + target_perms, target_shared_perms); ret =blk_insert_bs (s->target, target, errp);if (ret <0 ) { goto fail;@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job( s->backing_mode = backing_mode; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter))Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1763
Quoted 5 lines
block/mirror.c +5
-0+ { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. I'd prefere to add something like: "because we share it on target (see target BlockBackend creation and corresponding comment above)".
Quoted 4 lines
I’d rather not refer to other comments in case they change… Maybe just “This allows us to share
BLK_PERM_CONSISTENT_READ, as we do on the target.”? I think if someone is interested, they will scan the file for what permissions are shared on the target anyway.
OK. Yes I just wanted to stress that we just duplicate behavior about target, as it helped me to understand.
Quoted 93 lines
Quoted 84 lines
blockdev.c
Quoted 32 lines
block/mirror.c +7
-4+*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -1737,7 +1802,7 @@ fail: bs_opaque->stop = true;bdrv_child_refresh_perms (mirror_top_bs, mirror_top_bs->backing, &error_abort); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);bdrv_unref (mirror_top_bs);@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,return ; } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; - base = mode ==MIRROR_SYNC_MODE_TOP ?backing_bs (bs) :NULL ; + base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; mirror_start_job(job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, on_source_error, on_target_error, unmap, NULL, NULL,blockdev.c +0
-0
block/mirroc.cis OK for me. Continue withblockdev.c...Quoted 48 lines
blockdev.c +13
-2@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } -if (!bs->backing && sync ==MIRROR_SYNC_MODE_TOP ) { +if (!bdrv_backing_chain_next (bs) && sync ==MIRROR_SYNC_MODE_TOP ) { sync =MIRROR_SYNC_MODE_FULL ; }@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,void qmp_drive_mirror (DriveMirror *arg,Error **errp) { -BlockDriverState *bs; +BlockDriverState *bs, *unfiltered_bs;BlockDriverState *source, *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode;@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)int flags;int64_t size;const char *format = arg->format; +const char *replaces_node_name =NULL ;int ret; bs =qmp_get_root_bs (arg->device, errp);@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)return ; } +/* +* If the user has not instructed us otherwise, we should let the +* block job run from @bs (thus taking into account all filters on +* it) but replace @unfiltered_bs when it finishes (thus not +* removing those filters). +* (And if there are any explicit filters, we should assume the +* user knows how to use the @replaces option.) +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs); + aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) }Quoted 8 lines
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3949
Quoted 4 lines
blockdev.c +1
-1flags = bs->open_flags |BDRV_O_RDWR ; - source =backing_bs (bs); + source =bdrv_filtered_cow_bs (unfiltered_bs);if (!source && arg->sync ==MIRROR_SYNC_MODE_TOP ) {Hmm, you handle this case a bit differently here and in
blockdev_mirror_common.. Can we handle it only inblockdev_mirror_common, to be consistent withqmp_blockdev_mirror?
Quoted 6 lines
What exactly do you mean? The difference between skipping all filters and just skipping implicit filters? Hm.
First, the check in
blockdev_mirror_common()should actually be unnecessary. In qmp_{blockdev,drive}_mirror(), we do nearly the same check anyway (and then force sync=full if there is no backing file).
Hmm, I see it only in qmp_drive_mirror, not in _blockdev_
Quoted 3 lines
So
if all three functions did the same check, we wouldn’t need it inblockdev_mirror_common().
And if it was so, better have one check in _common than two duplicated.
Quoted 23 lines
Second, let’s look at the difference in an example: One where
blockdev_mirror_common()would not decide to enforce mode=full, but qmp_{blockdev,drive}_mirror() would.
This happens when @bs is an explicit filter over some overlay with a backing file, e.g.:throttle --file--> qcow2 --backing--> raw
It’s correct to run the mirror job from the throttle node; but @source should be
bdrv_backing_chain_next()so it will point to the raw node. Currently, it is NULL (because the throttle node does not have a COW child).But then again, I’ve made qmp_{blockdev,drive}_mirror() throw an error in such a case:
Quoted 6 lines
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3955
Quoted 6 lines
blockdev.c +6
-0+if (bdrv_filtered_bs (unfiltered_bs)) { +/* @unfiltered_bs is an explicit filter */ +error_setg (errp,"Cannot perform sync=top mirror through an " +"explicitly added filter node on the source" ); + goto out; + }
Quoted 3 lines
So it isn’t really a problem. Still, does the error make sense? Should we just allow that case by letting source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs))?
Looks good. As I understand, you have the test (40/42) for this case and it works for _blockdev_ and for for _drive_ version of command. Of course, they'd better behave in same manner.
Quoted 89 lines
(BTW, I just noticed that @base seems to be pretty much unused in
block/mirror.c. It only really uses @base_overlay now. So I suppose it makes sense to remove it in v7.)Quoted 56 lines
Quoted 56 lines
blockdev.c +10
-4arg->sync =MIRROR_SYNC_MODE_FULL ; }if (arg->sync ==MIRROR_SYNC_MODE_NONE ) {@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces; + }else if (unfiltered_bs != bs) { + replaces_node_name = unfiltered_bs->node_name; }if (arg->mode ==NEW_IMAGE_MODE_ABSOLUTE_PATHS ) {@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)bdrv_img_create (arg->target, format,NULL ,NULL ,NULL , size, flags, false, &local_err); }else { +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing =bdrv_skip_implicit_filters (source); +switch (arg->mode) {case NEW_IMAGE_MODE_EXISTING :break ;@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)/* create new image with backing file */ bdrv_refresh_filename (source);bdrv_img_create (arg->target, format, - source->filename, - source->drv->format_name, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err);break ;default :@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs, - arg->has_replaces, arg->replaces, arg->sync, + !!replaces_node_name, replaces_node_name, arg->sync, backing_mode, arg->has_speed, arg->speed, arg->has_granularity, arg->granularity, arg->has_buf_size, arg->buf_size,@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { - BlockDriverState *bs; + BlockDriverState *bs, *unfiltered_bs;BlockDriverState *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode =MIRROR_LEAVE_BACKING_CHAIN ;@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,return ; }Quoted 4 lines
blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4092
Quoted 2 lines
blockdev.c +2
-0+/* +* Same as in qmp_drive_mirror(): Then, may be better do it in
blockdev_mirror_common?Hm, maybe. Should we decide to let @source be bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs)) in
qmp_drive_mirror(), I don’t think we need @unfiltered_bs there to determine @source.Max
Quoted 15 lines
Quoted 13 lines
blockdev.c +8
-0We want to run the job from @bs, +* but we want to replace @unfiltered_bs on completion. +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs); +if (!has_replaces && unfiltered_bs != bs) { + replaces = unfiltered_bs->node_name; + has_replaces = true; + } + target_bs =bdrv_lookup_bs (target, target, errp);if (!target_bs) {return;
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 68 lines · Vladimir Sementsov-Ogievskiy
Quoted 67 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +9
-5@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does @@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret;Quoted 7 lines
block/mirror.c static void mirror_complete(Job *job, Error **errp) · line 1095
Quoted 3 lines
block/mirror.c +1
-2-assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target));Preexisting, but seems we may crash here, I don't see where it is checked before, to return error if there is some backing. And even if we do so, we don't prevent appearing of target backing during mirror operation.
The idea is that
MIRROR_OPEN_BACKING_CHAINis set only when using drive-mirror with mode=existing. In this case, we also setBDRV_O_NO_BACKINGfor the target.You’re right that a user could add a backing chain to the target during the operation. They really have to make an effort to shoot themselves in the foot for this because the target must have an auto-generated node name.
Quoted 4 lines
Quoted 2 lines
I suppose the best would be not to open the backing chain if the target node already has a backing child?
Hmm, but we still should generate an error, as we can't do what was requested.
But the user didn’t request anything. They just specified an existing file as the target with mode=existing, then (for whatever reason) made a real effort to add a backing node to it (i.e. they had to look up the target’s auto-generated node name), and then the job finishes.
The user didn’t request us to open the backing chain of the target. They just requested to mirror to an existing file. If they manually override that existing file’s backing chain, I think it makes sense to keep it that way.
Max
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
block/mirror.c +2
-0+ ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 130 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +44
-10@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does @@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret; -assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target)); + ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(MirrorBlockJob *s;MirrorBDSOpaque *bs_opaque;BlockDriverState *mirror_top_bs; -bool target_graph_mod;bool target_is_backing; +uint64_t target_perms, target_shared_perms;Error *local_err =NULL ;int ret;@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job( buf_size =DEFAULT_MIRROR_BUF_SIZE ; } -if (bs == target) { +if (bdrv_skip_rw_filters (bs) ==bdrv_skip_rw_filters (target)) {error_setg (errp,"Can't mirror node into itself" );return NULL ; }@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(* In the case of active commit, things look a bit different, though, * because the target is an already populated backing file in active use. * We can allow anything except resize there.*/ + + target_perms =BLK_PERM_WRITE ; + target_shared_perms =BLK_PERM_WRITE_UNCHANGED ; + target_is_backing =bdrv_chain_contains (bs, target); - target_graph_mod = (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ); +if (target_is_backing) { +int64_t bs_size, target_size; + bs_size =bdrv_getlength (bs); +if (bs_size <0 ) { +error_setg_errno (errp, -bs_size, +"Could not inquire top image size" ); + goto fail; + } + + target_size =bdrv_getlength (target); +if (target_size <0 ) { +error_setg_errno (errp, -target_size, +"Could not inquire base image size" ); + goto fail; + } + +if (target_size < bs_size) { + target_perms |=BLK_PERM_RESIZE ; + } + + target_shared_perms |=BLK_PERM_CONSISTENT_READ + |BLK_PERM_WRITE + |BLK_PERM_GRAPH_MOD ; + }else if (bdrv_chain_contains (bs,bdrv_skip_rw_filters (target))) { +/* +* We may want to allow this in the future, but it would
block/mirror.c static BlockJob *mirror_start_job( · line 1663
Quoted 10 lines
block/mirror.c +10
-0+* require taking some extra care. +*/ +error_setg (errp,"Cannot mirror to a filter on top of a node in the " +"source's backing chain" ); + goto fail; + } + +if (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ) { + target_perms |=BLK_PERM_GRAPH_MOD ; + }
This is getting absurd. We keep moving GRAPH_MOD around, but still nobody knows what it's actually supposed to mean. Maybe it would be better to just remove it finally?
Of course, not a reason to stop this patch, after all it's moving the nonsensical piece of code correctly...
Quoted 118 lines
block/mirror.c +36
-16s->target =blk_new (s->common.job.aio_context, -BLK_PERM_WRITE |BLK_PERM_RESIZE | - (target_graph_mod ?BLK_PERM_GRAPH_MOD :0 ), -BLK_PERM_WRITE_UNCHANGED | - (target_is_backing ?BLK_PERM_CONSISTENT_READ | -BLK_PERM_WRITE | -BLK_PERM_GRAPH_MOD :0 )); + target_perms, target_shared_perms); ret =blk_insert_bs (s->target, target, errp);if (ret <0 ) { goto fail;@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job( s->backing_mode = backing_mode; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -1737,7 +1802,7 @@ fail: bs_opaque->stop = true;bdrv_child_refresh_perms (mirror_top_bs, mirror_top_bs->backing, &error_abort); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);bdrv_unref (mirror_top_bs);@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,return ; } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; - base = mode ==MIRROR_SYNC_MODE_TOP ?backing_bs (bs) :NULL ; + base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; mirror_start_job(job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, on_source_error, on_target_error, unmap, NULL, NULL,blockdev.c +3
-2@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } -if (!bs->backing && sync ==MIRROR_SYNC_MODE_TOP ) { +if (!bdrv_backing_chain_next (bs) && sync ==MIRROR_SYNC_MODE_TOP ) { sync =MIRROR_SYNC_MODE_FULL ; }@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,void qmp_drive_mirror (DriveMirror *arg,Error **errp) { -BlockDriverState *bs; +BlockDriverState *bs, *unfiltered_bs;BlockDriverState *source, *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode;@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)int flags;int64_t size;const char *format = arg->format; +const char *replaces_node_name =NULL ;int ret; bs =qmp_get_root_bs (arg->device, errp);@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)return ; }
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3933
Quoted 10 lines
blockdev.c +9
-0+/* +* If the user has not instructed us otherwise, we should let the +* block job run from @bs (thus taking into account all filters on +* it) but replace @unfiltered_bs when it finishes (thus not +* removing those filters). +* (And if there are any explicit filters, we should assume the +* user knows how to use the @replaces option.) +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs);
Should this behaviour be documented in the QAPI schema for drive-mirror?
Quoted 69 lines
blockdev.c +17
-5aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } flags = bs->open_flags |BDRV_O_RDWR ; - source =backing_bs (bs); + source =bdrv_filtered_cow_bs (unfiltered_bs);if (!source && arg->sync ==MIRROR_SYNC_MODE_TOP ) { +if (bdrv_filtered_bs (unfiltered_bs)) { +/* @unfiltered_bs is an explicit filter */ +error_setg (errp,"Cannot perform sync=top mirror through an " +"explicitly added filter node on the source" ); + goto out; + } arg->sync =MIRROR_SYNC_MODE_FULL ; }if (arg->sync ==MIRROR_SYNC_MODE_NONE ) {@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces; + }else if (unfiltered_bs != bs) { + replaces_node_name = unfiltered_bs->node_name; }if (arg->mode ==NEW_IMAGE_MODE_ABSOLUTE_PATHS ) {@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)bdrv_img_create (arg->target, format,NULL ,NULL ,NULL , size, flags, false, &local_err); }else { +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing =bdrv_skip_implicit_filters (source); +switch (arg->mode) {case NEW_IMAGE_MODE_EXISTING :break ;@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)/* create new image with backing file */ bdrv_refresh_filename (source);bdrv_img_create (arg->target, format, - source->filename, - source->drv->format_name, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err);break ;default :@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs, - arg->has_replaces, arg->replaces, arg->sync, + !!replaces_node_name, replaces_node_name, arg->sync, backing_mode, arg->has_speed, arg->speed, arg->has_granularity, arg->granularity, arg->has_buf_size, arg->buf_size,@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { - BlockDriverState *bs; + BlockDriverState *bs, *unfiltered_bs;BlockDriverState *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode =MIRROR_LEAVE_BACKING_CHAIN ;
blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4095
Quoted 8 lines
blockdev.c +5
-0@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,return ; } +/* +* Same as in qmp_drive_mirror(): We want to run the job from @bs, +* but we want to replace @unfiltered_bs on completion. +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs);
Do we? I thought the idea with blockdev-mirror was that the client tells us the exact node it is interested in, without any magic skipping nodes.
Skipping implicit nodes is a feature for compatibility with legacy clients, but a client using blockdev-mirror isn't a legacy client.
blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4103
Quoted 8 lines
blockdev.c +5
-0+if (!has_replaces && unfiltered_bs != bs) { + replaces = unfiltered_bs->node_name; + has_replaces = true; + } + target_bs =bdrv_lookup_bs (target, target, errp);if (!target_bs) {return ;
Kevin
Re: [Qemu-devel] [PATCH v6 25/42] mirror: Deal with filters
Quoted 137 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 136 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 117 ++++++++++++++++++++++++++++++++++++++----------- blockdev.c | 47 +++++++++++++++++--- 2 files changed, 131 insertions(+), 33 deletions(-)block/mirror.c +50
-10@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;@@ -665,8 +666,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_rw_filters (target_bs); + +if (bdrv_filtered_cow_bs (unfiltered_target) != backing) { +bdrv_set_backing_hd (unfiltered_target, backing, &local_err);if (local_err) {error_report_err (local_err); ret = -EPERM ;@@ -715,7 +718,7 @@ static int mirror_exit_common(Job *job)* valid. */ block_job_remove_all_bdrv (bjob); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);/* We just changed the BDS the job BB refers to (with either or both of the * bdrv_replace_node() calls), so switch the BB back so the cleanup does @@ -812,7 +815,8 @@ static int coroutine_fn mirror_dirty_init(MirrorBlockJob *s)return 0 ; } - ret =bdrv_is_allocated_above (bs, base, false, offset, bytes, &count); + ret =bdrv_is_allocated_above (bs, s->base_overlay, true, offset, bytes, + &count);if (ret <0 ) {return ret; }@@ -908,7 +912,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }else { s->target_cluster_size =BDRV_SECTOR_SIZE ; } -if (backing_filename[0 ] && !target_bs->backing && +if (backing_filename[0 ] && !bdrv_backing_chain_next (target_bs) && s->granularity < s->target_cluster_size) { s->buf_size =MAX (s->buf_size, s->target_cluster_size); s->cow_bitmap =bitmap_new (length);@@ -1088,8 +1092,9 @@ static void mirror_complete(Job *job, Error **errp)if (s->backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) {int ret; -assert (!target->backing); - ret =bdrv_open_backing_file (target,NULL ,"backing" , errp); +assert (!bdrv_backing_chain_next (target)); + ret =bdrv_open_backing_file (bdrv_skip_rw_filters (target),NULL , +"backing" , errp);if (ret <0 ) {return ; }@@ -1531,8 +1536,8 @@ static BlockJob *mirror_start_job(MirrorBlockJob *s;MirrorBDSOpaque *bs_opaque;BlockDriverState *mirror_top_bs; -bool target_graph_mod;bool target_is_backing; +uint64_t target_perms, target_shared_perms;Error *local_err =NULL ;int ret;@@ -1551,7 +1556,7 @@ static BlockJob *mirror_start_job( buf_size =DEFAULT_MIRROR_BUF_SIZE ; } -if (bs == target) { +if (bdrv_skip_rw_filters (bs) ==bdrv_skip_rw_filters (target)) {error_setg (errp,"Can't mirror node into itself" );return NULL ; }@@ -1615,15 +1620,50 @@ static BlockJob *mirror_start_job(* In the case of active commit, things look a bit different, though, * because the target is an already populated backing file in active use. * We can allow anything except resize there.*/ + + target_perms =BLK_PERM_WRITE ; + target_shared_perms =BLK_PERM_WRITE_UNCHANGED ; + target_is_backing =bdrv_chain_contains (bs, target); - target_graph_mod = (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ); +if (target_is_backing) { +int64_t bs_size, target_size; + bs_size =bdrv_getlength (bs); +if (bs_size <0 ) { +error_setg_errno (errp, -bs_size, +"Could not inquire top image size" ); + goto fail; + } + + target_size =bdrv_getlength (target); +if (target_size <0 ) { +error_setg_errno (errp, -target_size, +"Could not inquire base image size" ); + goto fail; + } + +if (target_size < bs_size) { + target_perms |=BLK_PERM_RESIZE ; + } + + target_shared_perms |=BLK_PERM_CONSISTENT_READ + |BLK_PERM_WRITE + |BLK_PERM_GRAPH_MOD ; + }else if (bdrv_chain_contains (bs,bdrv_skip_rw_filters (target))) { +/* +* We may want to allow this in the future, but it would +* require taking some extra care. +*/ +error_setg (errp,"Cannot mirror to a filter on top of a node in the " +"source's backing chain" ); + goto fail; + }
Quoted 8 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1663
Quoted 4 lines
block/mirror.c +4
-0+ +if (backing_mode !=MIRROR_LEAVE_BACKING_CHAIN ) { + target_perms |=BLK_PERM_GRAPH_MOD ; + }This is getting absurd. We keep moving
GRAPH_MODaround, but still nobody knows what it's actually supposed to mean. Maybe it would be better to just remove it finally?
I suppose even if we ever needed it, we no longer do now with .freeze.
Quoted 122 lines
Of course, not a reason to stop this patch, after all it's moving the nonsensical piece of code correctly...
Quoted 119 lines
block/mirror.c +36
-16s->target =blk_new (s->common.job.aio_context, -BLK_PERM_WRITE |BLK_PERM_RESIZE | - (target_graph_mod ?BLK_PERM_GRAPH_MOD :0 ), -BLK_PERM_WRITE_UNCHANGED | - (target_is_backing ?BLK_PERM_CONSISTENT_READ | -BLK_PERM_WRITE | -BLK_PERM_GRAPH_MOD :0 )); + target_perms, target_shared_perms); ret =blk_insert_bs (s->target, target, errp);if (ret <0 ) { goto fail;@@ -1647,6 +1687,7 @@ static BlockJob *mirror_start_job( s->backing_mode = backing_mode; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;@@ -1693,15 +1734,39 @@ static BlockJob *mirror_start_job(/* In commit_active_start() all intermediate nodes disappear, so * any jobs in them must be blocked */ if (target_is_backing) { -BlockDriverState *iter; -for (iter =backing_bs (bs); iter != target; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block -* ourselves at s->base (if writes are blocked for a node, they are -* also blocked for its backing file). The other options would be a -* second filter driver above s->base (== target). */ +BlockDriverState *iter, *filtered_target; +uint64_t iter_shared_perms; + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_target) == bdrv_skip_rw_filters(target) +*/ + filtered_target =bdrv_filtered_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_rw_filters (filtered_target) == +bdrv_skip_rw_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filtered_bs (bs); iter != target; + iter =bdrv_filtered_bs (iter)) + { +if (iter == filtered_target) { +/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -1737,7 +1802,7 @@ fail: bs_opaque->stop = true;bdrv_child_refresh_perms (mirror_top_bs, mirror_top_bs->backing, &error_abort); -bdrv_replace_node (mirror_top_bs,backing_bs (mirror_top_bs), &error_abort); +bdrv_replace_node (mirror_top_bs, mirror_top_bs->backing->bs, &error_abort);bdrv_unref (mirror_top_bs);@@ -1764,7 +1829,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,return ; } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; - base = mode ==MIRROR_SYNC_MODE_TOP ?backing_bs (bs) :NULL ; + base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; mirror_start_job(job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, on_source_error, on_target_error, unmap, NULL, NULL,blockdev.c +3
-2@@ -3851,7 +3851,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } -if (!bs->backing && sync ==MIRROR_SYNC_MODE_TOP ) { +if (!bdrv_backing_chain_next (bs) && sync ==MIRROR_SYNC_MODE_TOP ) { sync =MIRROR_SYNC_MODE_FULL ; }@@ -3900,7 +3900,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,void qmp_drive_mirror (DriveMirror *arg,Error **errp) { -BlockDriverState *bs; +BlockDriverState *bs, *unfiltered_bs;BlockDriverState *source, *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode;@@ -3909,6 +3909,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)int flags;int64_t size;const char *format = arg->format; +const char *replaces_node_name =NULL ;int ret; bs =qmp_get_root_bs (arg->device, errp);@@ -3921,6 +3922,16 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)return ; }
Quoted 11 lines
blockdev.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3933
Quoted 9 lines
blockdev.c +9
-0+/* +* If the user has not instructed us otherwise, we should let the +* block job run from @bs (thus taking into account all filters on +* it) but replace @unfiltered_bs when it finishes (thus not +* removing those filters). +* (And if there are any explicit filters, we should assume the +* user knows how to use the @replaces option.) +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs);Should this behaviour be documented in the QAPI schema for drive-mirror?
Hm. I’d document it for @replaces. But what would I write? “By default, @device is replaced, though implicitly created nodes on it are kept”?
I feel bad about referencing implicit nodes, especially if our plan is to remove them anyway. OTOH, if I dropped special handling of implicit nodes here, I should probably drop it in the whole series. And I don’t feel like that’s right as long as we haven’t actually removed implicit nodes.
So I suppose I’ll go for the documentation addendum? :-/
Quoted 81 lines
Quoted 69 lines
blockdev.c +17
-5aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);@@ -3934,8 +3945,14 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } flags = bs->open_flags |BDRV_O_RDWR ; - source =backing_bs (bs); + source =bdrv_filtered_cow_bs (unfiltered_bs);if (!source && arg->sync ==MIRROR_SYNC_MODE_TOP ) { +if (bdrv_filtered_bs (unfiltered_bs)) { +/* @unfiltered_bs is an explicit filter */ +error_setg (errp,"Cannot perform sync=top mirror through an " +"explicitly added filter node on the source" ); + goto out; + } arg->sync =MIRROR_SYNC_MODE_FULL ; }if (arg->sync ==MIRROR_SYNC_MODE_NONE ) {@@ -3954,6 +3971,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces; + }else if (unfiltered_bs != bs) { + replaces_node_name = unfiltered_bs->node_name; }if (arg->mode ==NEW_IMAGE_MODE_ABSOLUTE_PATHS ) {@@ -3973,6 +3993,9 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)bdrv_img_create (arg->target, format,NULL ,NULL ,NULL , size, flags, false, &local_err); }else { +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing =bdrv_skip_implicit_filters (source); +switch (arg->mode) {case NEW_IMAGE_MODE_EXISTING :break ;@@ -3980,8 +4003,8 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)/* create new image with backing file */ bdrv_refresh_filename (source);bdrv_img_create (arg->target, format, - source->filename, - source->drv->format_name, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err);break ;default :@@ -4017,7 +4040,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs, - arg->has_replaces, arg->replaces, arg->sync, + !!replaces_node_name, replaces_node_name, arg->sync, backing_mode, arg->has_speed, arg->speed, arg->has_granularity, arg->granularity, arg->has_buf_size, arg->buf_size,@@ -4053,7 +4076,7 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { - BlockDriverState *bs; + BlockDriverState *bs, *unfiltered_bs;BlockDriverState *target_bs;AioContext *aio_context;BlockMirrorBackingMode backing_mode =MIRROR_LEAVE_BACKING_CHAIN ;blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4095
Quoted 8 lines
blockdev.c +5
-0@@ -4065,6 +4088,16 @@ void qmp_blockdev_mirror(bool has_job_id, const char *job_id,return ; } +/* +* Same as in qmp_drive_mirror(): We want to run the job from @bs, +* but we want to replace @unfiltered_bs on completion. +*/ + unfiltered_bs =bdrv_skip_implicit_filters (bs);Do we? I thought the idea with blockdev-mirror was that the client tells us the exact node it is interested in, without any magic skipping nodes.
Quoted 2 lines
Skipping implicit nodes is a feature for compatibility with legacy clients, but a client using blockdev-mirror isn't a legacy client.
And I thought legacy clients didn’t let implicit nodes be created.
We could return an error if @bs is an implicit node. But I don’t think that would help anyone, and it’d be more complicated in my opinion.
Again, the best thing (IMO) is to remove the concept of implicit nodes altogether and then we can drop this piece of code anyway.
Max
Quoted 11 lines
blockdev.c void qmp_blockdev_mirror(bool has_job_id, const char *job_id, · line 4103
Quoted 8 lines
blockdev.c +5
-0+if (!has_replaces && unfiltered_bs != bs) { + replaces = unfiltered_bs->node_name; + has_replaces = true; + } + target_bs =bdrv_lookup_bs (target, target, errp);if (!target_bs) {return ;Kevin
PATCH v6 26/42 backup: Deal with filters
In reply to Max Reitz--- block/backup.c | 9 +++++---- blockdev.c | 19 +++++++++++++++---- 2 files changed, 20 insertions(+), 8 deletions(-)
block/backup.c +5 -4
@@ -611,6 +611,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target, {int ret;BlockDriverInfo bdi; +bool target_does_cow =bdrv_backing_chain_next (target);/* * If there is no backing file on the target, we cannot rely on COW if our
@@ -618,7 +619,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,* targets with a backing file, try to avoid COW if possible. */ ret =bdrv_get_info (target, &bdi); -if (ret == -ENOTSUP && !target->backing) { +if (ret == -ENOTSUP && !target_does_cow) {/* Cluster size is not defined */ warn_report ("The target block device doesn't provide " "information about the block size and it doesn't have a "
@@ -627,14 +628,14 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,"this default, the backup may be unusable" , BACKUP_CLUSTER_SIZE_DEFAULT);return BACKUP_CLUSTER_SIZE_DEFAULT ; - }else if (ret <0 && !target->backing) { + }else if (ret <0 && !target_does_cow) {error_setg_errno (errp, -ret,"Couldn't determine the cluster size of the target image, " "which has no backing file" );error_append_hint (errp,"Aborting, since this may create an unusable destination image\n" );return ret; - }else if (ret <0 && target->backing) { + }else if (ret <0 && target_does_cow) {/* Not fatal; just trudge on ahead. */ return BACKUP_CLUSTER_SIZE_DEFAULT ; }
@@ -683,7 +684,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,return NULL ; } -if (compress && target->drv->bdrv_co_pwritev_compressed ==NULL ) { +if (compress && !bdrv_supports_compressed_writes (target)) {error_setg (errp,"Compression is not supported for this drive %s" ,bdrv_get_device_name (target));return NULL ;
blockdev.c +15 -4
@@ -3656,7 +3656,13 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,/* See if we have a backing HD we can use to create our new image * on top of. */ if (backup->sync ==MIRROR_SYNC_MODE_TOP ) { - source =backing_bs (bs); +/* +* Backup will not replace the source by the target, so none +* of the filters skipped here will be removed (in contrast to +* mirror). Therefore, we can skip all of them when looking +* for the first COW relationship. +*/ + source =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (bs));if (!source) { backup->sync =MIRROR_SYNC_MODE_FULL ; }
@@ -3676,9 +3682,14 @@ static BlockJob *do_drive_backup(DriveBackup *backup, JobTxn *txn,if (backup->mode !=NEW_IMAGE_MODE_EXISTING ) {assert (backup->format);if (source) { -bdrv_refresh_filename (source); -bdrv_img_create (backup->target, backup->format, source->filename, - source->drv->format_name,NULL , +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing = +bdrv_skip_implicit_filters (source); + +bdrv_refresh_filename (explicit_backing); +bdrv_img_create (backup->target, backup->format, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err); }else { bdrv_img_create(backup->target, backup->format, NULL, NULL, NULL,
Signature · 1 line
2.21.0
PATCH v6 27/42 commit: Deal with filters
In reply to Max ReitzThis includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
--- block/block-backend.c | 16 +++++--- block/commit.c | 96 +++++++++++++++++++++++++++++++------------ blockdev.c | 6 ++- 3 files changed, 85 insertions(+), 33 deletions(-)
block/block-backend.c +11 -5
@@ -2180,11 +2180,17 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs); -if (ret <0 ) { -aio_context_release (aio_context); -return ret; +if (blk_is_inserted (blk)) { +BlockDriverState *non_filter; + +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); +if (bdrv_filtered_cow_child (non_filter)) { +int ret =bdrv_commit (non_filter); +if (ret <0 ) { +aio_context_release (aio_context); +return ret; + } } }aio_context_release (aio_context);
block/commit.c +70 -26
173 lines not on this page · raw message
blockdev.c +4 -2
@@ -1094,7 +1094,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)return ; } - bs =blk_bs (blk); + bs =bdrv_skip_implicit_filters (blk_bs (blk)); aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);
@@ -3454,7 +3454,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,assert (bdrv_get_aio_context (base_bs) == aio_context); -for (iter = top_bs; iter !=backing_bs (base_bs); iter =backing_bs (iter)) { +for (iter = top_bs; iter !=bdrv_filtered_bs (base_bs); + iter =bdrv_filtered_bs (iter)) + {if (bdrv_op_is_blocked (iter,BLOCK_OP_TYPE_COMMIT_TARGET , errp)) { goto out; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 27/42] commit: Deal with filters
Quoted 25 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/block-backend.c | 16 +++++--- block/commit.c | 96 +++++++++++++++++++++++++++++++------------ blockdev.c | 6 ++- 3 files changed, 85 insertions(+), 33 deletions(-)block/block-backend.c +2
-5@@ -2180,11 +2180,17 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs); -if (ret <0 ) { -aio_context_release (aio_context); -return ret; +if (blk_is_inserted (blk)) { +BlockDriverState *non_filter;
block/block-backend.c int blk_commit_all(void) · line 2194
Quoted 10 lines
block/block-backend.c +9
-0+ +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); +if (bdrv_filtered_cow_child (non_filter)) { +int ret =bdrv_commit (non_filter); +if (ret <0 ) { +aio_context_release (aio_context); +return ret; + } }
and if non_filter is explicit filter we just skip it. I think we'd better return error in this case. For example, just drop if (bdrv_filtered_cow_child) and get ENOTSUP from bdrv_commit in this case.
And with at least this fixed I'm OK with this patch:
However some comments below:
block/commit.c typedef struct CommitBlockJob { · line 40
Quoted 11 lines
block/block-backend.c +0
-0}aio_context_release (aio_context);block/commit.c +1
-0@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {BlockBackend *top;BlockBackend *base;BlockDriverState *base_bs; +BlockDriverState *above_base;
you called it base_overlay in mirror, seems better to keep same naming
Quoted 60 lines
block/commit.c +23
-3BlockdevOnError on_error;bool base_read_only;bool chain_frozen;@@ -110,7 +111,7 @@ static void commit_abort(Job *job)* XXX Can (or should) we somehow keep 'consistent read' blocked even * after the failed/cancelled commit job is gone? If we already wrote * something to base, the intermediate images aren't valid any more. */ -bdrv_replace_node (s->commit_top_bs,backing_bs (s->commit_top_bs), +bdrv_replace_node (s->commit_top_bs, s->commit_top_bs->backing->bs, &error_abort);bdrv_unref (s->commit_top_bs);@@ -174,7 +175,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)break ; }/* Copy if allocated above the base */ - ret =bdrv_is_allocated_above (blk_bs (s->top),blk_bs (s->base), false, + ret =bdrv_is_allocated_above (blk_bs (s->top), s->above_base, true, offset,COMMIT_BUFFER_SIZE , &n); copy = (ret ==1 );trace_commit_one_iteration (s, offset, n, ret);@@ -267,15 +268,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,CommitBlockJob *s;BlockDriverState *iter;BlockDriverState *commit_top_bs =NULL ; +BlockDriverState *filtered_base;Error *local_err =NULL ; +int64_t base_size, top_size; +uint64_t perms, iter_shared_perms;int ret;assert (top != bs); -if (top == base) { +if (bdrv_skip_rw_filters (top) ==bdrv_skip_rw_filters (base)) {error_setg (errp,"Invalid files for merge: top and base are the same" );return ; } + base_size =bdrv_getlength (base); +if (base_size <0 ) { +error_setg_errno (errp, -base_size,"Could not inquire base image size" ); +return ; + } + + top_size =bdrv_getlength (top); +if (top_size <0 ) { +error_setg_errno (errp, -top_size,"Could not inquire top image size" ); +return ; + } + + perms =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE ; +if (base_size < top_size) { + perms |=BLK_PERM_RESIZE ; + } + s =block_job_create (job_id, &commit_job_driver,NULL , bs,0 ,BLK_PERM_ALL , speed, creation_flags,NULL ,NULL , errp);if (!s) {@@ -315,17 +336,43 @@ void commit_start(const char *job_id, BlockDriverState *bs,
block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 325
Quoted 10 lines
block/commit.c +0
-8s->commit_top_bs = commit_top_bs; -/* Block all nodes between top and base, because they will -* disappear from the chain after this operation. */ -assert (bdrv_chain_contains (top, base)); -for (iter = top; iter != base; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves -* at s->base (if writes are blocked for a node, they are also blocked -* for its backing file). The other options would be a second filter -* driver above s->base. */
This code part is absolutely equal to corresponding in block/mirror.c.. It would be great to put it into a function and reuse. However its not about these series.
Quoted 54 lines
block/commit.c +37
-5+/* +* Block all nodes between top and base, because they will +* disappear from the chain after this operation. +* Note that this assumes that the user is fine with removing all +* nodes (including R/W filters) between top and base. Assuring +* this is the responsibility of the interface (i.e. whoever calls +* commit_start()). +*/ + s->above_base =bdrv_find_overlay (top, base); +assert (s->above_base); + +/* +* The topmost node with +* bdrv_skip_rw_filters(filtered_base) == bdrv_skip_rw_filters(base) +*/ + filtered_base =bdrv_filtered_cow_bs (s->above_base); +assert (bdrv_skip_rw_filters (filtered_base) ==bdrv_skip_rw_filters (base)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves +* at s->base (if writes are blocked for a node, they are also blocked +* for its backing file). The other options would be a second filter +* driver above s->base. +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter = top; iter != base; iter =bdrv_filtered_bs (iter)) { +if (iter == filtered_base) { +/* +* From here on, all nodes are filters on the base. This +* allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -342,9 +389,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, } s->base =blk_new (s->common.job.aio_context, -BLK_PERM_CONSISTENT_READ - |BLK_PERM_WRITE - |BLK_PERM_RESIZE , + perms,BLK_PERM_CONSISTENT_READ |BLK_PERM_GRAPH_MOD |BLK_PERM_WRITE_UNCHANGED );
block/commit.c int bdrv_commit(BlockDriverState *bs) · line 460
Quoted 5 lines
block/commit.c +1
-1@@ -412,19 +457,22 @@ int bdrv_commit(BlockDriverState *bs)if (!drv)return -ENOMEDIUM ; -if (!bs->backing) { + backing_file_bs =bdrv_filtered_cow_bs (bs);
Hmm just note: if in future we'll have cow child which is not bs->backing, a lot of code will fail, as we always assume that cow child is bs->backing. May be, this should be commented in bdrv_filtered_cow_child implementation.
Quoted 31 lines
block/commit.c +6
-5+ +if (!backing_file_bs) {return -ENOTSUP ; }if (bdrv_op_is_blocked (bs,BLOCK_OP_TYPE_COMMIT_SOURCE ,NULL ) || -bdrv_op_is_blocked (bs->backing->bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) { +bdrv_op_is_blocked (backing_file_bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) + {return -EBUSY ; } - ro = bs->backing->bs->read_only; + ro = backing_file_bs->read_only;if (ro) { -if (bdrv_reopen_set_read_only (bs->backing->bs, false,NULL )) { +if (bdrv_reopen_set_read_only (backing_file_bs, false,NULL )) {return -EACCES ; } }@@ -440,8 +488,6 @@ int bdrv_commit(BlockDriverState *bs) }/* Insert commit_top block node above backing, so we can write to it */ - backing_file_bs =backing_bs (bs); - commit_top_bs =bdrv_new_open_driver (&bdrv_commit_top,NULL ,BDRV_O_RDWR , &local_err);if (commit_top_bs ==NULL ) {
block/commit.c ro_cleanup: · line 575
Quoted 7 lines
block/commit.c +1
-3@@ -526,15 +572,13 @@ ro_cleanup:qemu_vfree (buf);blk_unref (backing); -if (backing_file_bs) { -bdrv_set_backing_hd (bs, backing_file_bs, &error_abort); - } +bdrv_set_backing_hd (bs, backing_file_bs, &error_abort);
Preexisting, but we should not drop filter if we didn't added it (if we failed above filter insertion). You increased amount of such cases. No damage still.
Quoted 35 lines
block/commit.c +1
-1bdrv_unref (commit_top_bs);blk_unref (src);if (ro) {/* ignoring error return here */ -bdrv_reopen_set_read_only (bs->backing->bs, true,NULL ); +bdrv_reopen_set_read_only (backing_file_bs, true,NULL ); }return ret;blockdev.c +4
-2@@ -1094,7 +1094,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)return ; } - bs =blk_bs (blk); + bs =bdrv_skip_implicit_filters (blk_bs (blk)); aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);@@ -3454,7 +3454,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,assert (bdrv_get_aio_context (base_bs) == aio_context); -for (iter = top_bs; iter !=backing_bs (base_bs); iter =backing_bs (iter)) { +for (iter = top_bs; iter !=bdrv_filtered_bs (base_bs); + iter =bdrv_filtered_bs (iter)) + {if (bdrv_op_is_blocked (iter,BLOCK_OP_TYPE_COMMIT_TARGET , errp)) { goto out; }
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 27/42] commit: Deal with filters
Quoted 32 lines · Vladimir Sementsov-Ogievskiy
Quoted 31 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/block-backend.c | 16 +++++--- block/commit.c | 96 +++++++++++++++++++++++++++++++------------ blockdev.c | 6 ++- 3 files changed, 85 insertions(+), 33 deletions(-)block/block-backend.c +8
-5@@ -2180,11 +2180,17 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs); -if (ret <0 ) { -aio_context_release (aio_context); -return ret; +if (blk_is_inserted (blk)) { +BlockDriverState *non_filter; + +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); +if (bdrv_filtered_cow_child (non_filter)) { +int ret =bdrv_commit (non_filter); +if (ret <0 ) {
Quoted 8 lines
block/block-backend.c int blk_commit_all(void) · line 2194
Quoted 4 lines
block/block-backend.c +3
-0+aio_context_release (aio_context); +return ret; + } }and if
non_filteris explicit filter we just skip it. I think we'd better return error in this case. For example, just drop if (bdrv_filtered_cow_child) and get ENOTSUP frombdrv_commitin this case.
Sounds good, yes.
Quoted 6 lines
And with at least this fixed I'm OK with this patch:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>However some comments below:
Quoted 13 lines
block/commit.c typedef struct CommitBlockJob { · line 40
Quoted 11 lines
block/block-backend.c +0
-0}aio_context_release (aio_context);block/commit.c +1
-0@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {BlockBackend *top;BlockBackend *base;BlockDriverState *base_bs; +BlockDriverState *above_base;you called it
base_overlayin mirror, seems better to keep same naming
Indeed.
[...]
Quoted 4 lines
Quoted 4 lines
block/commit.c +0
-0@@ -315,17 +336,43 @@ void commit_start(const char *job_id, BlockDriverState *bs, s->commit_top_bs = commit_top_bs;
Quoted 11 lines
block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 325
Quoted 8 lines
block/commit.c +0
-8-/* Block all nodes between top and base, because they will -* disappear from the chain after this operation. */ -assert (bdrv_chain_contains (top, base)); -for (iter = top; iter != base; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves -* at s->base (if writes are blocked for a node, they are also blocked -* for its backing file). The other options would be a second filter -* driver above s->base. */ This code part is absolutely equal to corresponding in
block/mirror.c.. It would be great to put it into a function and reuse. However its not about these series.
It would probably be great to just drop block/commit.c altogether and fully merge it into block/mirror.c at some point.
(I suppose we’d just have to check whether there’s any parent who’s taken the WRITE permission on the top node, and if so, emit READY (and if not, skip to COMPLETED).)
[...]
Quoted 10 lines
block/commit.c int bdrv_commit(BlockDriverState *bs) · line 460
Quoted 6 lines
block/commit.c +1
-1@@ -412,19 +457,22 @@ int bdrv_commit(BlockDriverState *bs)if (!drv)return -ENOMEDIUM ; -if (!bs->backing) { + backing_file_bs =bdrv_filtered_cow_bs (bs);Hmm just note: if in future we'll have cow child which is not
bs->backing, a lot of code will fail, as we always assume that cow child isbs->backing. May be, this should be commented inbdrv_filtered_cow_childimplementation.
I couldn’t see why we’d ever do this. I hope we never do.
(Aside from just removing bs->file and bs->backing altogether.)
Max
PATCH v6 28/42 stream: Deal with filters
In reply to Max ReitzBecause of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)
qapi/block-core.json +4 -0
@@ -2516,6 +2516,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +33 -19
126 lines not on this page · raw message
blockdev.c +1 -1
@@ -3297,7 +3297,7 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, }/* Check for op blockers in the whole chain between bs and base */ -for (iter = bs; iter && iter != base_bs; iter =backing_bs (iter)) { +for (iter = bs; iter && iter != base_bs; iter =bdrv_filtered_bs (iter)) {if (bdrv_op_is_blocked (iter,BLOCK_OP_TYPE_STREAM , errp)) { goto out; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters
Quoted 6 lines · Max Reitz
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters
In reply to Max ReitzAm 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 32 lines
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)qapi/block-core.json +4
-0@@ -2516,6 +2516,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #block/stream.c +0
-0
block/stream.c enum { · line 35
Quoted 6 lines
block/stream.c +2
-1@@ -31,7 +31,8 @@ enum { typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *bottom_cow_node; + BlockDriverState *above_base;
Confusing naming, especially because in commit you used above_base for what is bottom_cow_node here. Vladimir already suggested using base_overlay consistently, so we should do this here too (for bottom_cow_node). above_base can keep its name because the different above_base in commit is going to be renamed).
Quoted 79 lines
block/stream.c +13
-10BlockdevOnError on_error;char *backing_file_str;bool bs_read_only;@@ -54,7 +55,7 @@ static void stream_abort(Job *job)if (s->chain_frozen) {BlockJob *bjob = &s->common; -bdrv_unfreeze_chain (blk_bs (bjob->blk), s->bottom); +bdrv_unfreeze_chain (blk_bs (bjob->blk), s->above_base); } }@@ -63,14 +64,15 @@ static int stream_prepare(Job *job)StreamBlockJob *s =container_of (job, StreamBlockJob, common.job);BlockJob *bjob = &s->common;BlockDriverState *bs =blk_bs (bjob->blk); -BlockDriverState *base =backing_bs (s->bottom); +BlockDriverState *unfiltered_bs =bdrv_skip_rw_filters (bs); +BlockDriverState *base =bdrv_filtered_bs (s->above_base);Error *local_err =NULL ;int ret =0 ; -bdrv_unfreeze_chain (bs, s->bottom); +bdrv_unfreeze_chain (bs, s->above_base); s->chain_frozen = false; -if (bs->backing) { +if (bdrv_filtered_cow_child (unfiltered_bs)) {const char *base_id =NULL , *base_fmt =NULL ;if (base) { base_id = s->backing_file_str;@@ -78,8 +80,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } -bdrv_set_backing_hd (bs, base, &local_err); - ret =bdrv_change_backing_file (bs, base_id, base_fmt); +bdrv_set_backing_hd (unfiltered_bs, base, &local_err); + ret =bdrv_change_backing_file (unfiltered_bs, base_id, base_fmt);if (local_err) {error_report_err (local_err);return -EPERM ;@@ -110,7 +112,8 @@ static int coroutine_fn stream_run(Job *job, Error **errp)StreamBlockJob *s =container_of (job, StreamBlockJob, common.job);BlockBackend *blk = s->common.blk;BlockDriverState *bs =blk_bs (blk); -bool enable_cor = !backing_bs (s->bottom); +BlockDriverState *unfiltered_bs =bdrv_skip_rw_filters (bs); +bool enable_cor = !bdrv_filtered_bs (s->above_base);int64_t len;int64_t offset =0 ;uint64_t delay_ns =0 ;@@ -119,7 +122,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)int64_t n =0 ;/* bytes */ void *buf; -if (bs == s->bottom) { +if (unfiltered_bs == s->bottom_cow_node) {/* Nothing to stream */ return 0 ; }@@ -154,13 +157,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret =bdrv_is_allocated (bs, offset,STREAM_BUFFER_SIZE , &n); + ret =bdrv_is_allocated (unfiltered_bs, offset,STREAM_BUFFER_SIZE , &n);if (ret ==1 ) {/* Allocated in the top, no need to copy. */ }else if (ret >=0 ) {/* Copy if allocated in the intermediate images. Limit to the * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE). */ - ret =bdrv_is_allocated_above (backing_bs (bs), s->bottom, true, + ret =bdrv_is_allocated_above (bdrv_filtered_cow_bs (unfiltered_bs), + s->bottom_cow_node, true, offset, n, &n);/* Finish early if end of backing file has been reached */ if (ret ==0 && n ==0 ) {
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 239
Quoted 6 lines
block/stream.c +2
-1@@ -231,9 +235,16 @@ void stream_start(const char *job_id, BlockDriverState *bs,BlockDriverState *iter;bool bs_read_only;int basic_flags =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE_UNCHANGED ; -BlockDriverState *bottom =bdrv_find_overlay (bs, base); +BlockDriverState *bottom_cow_node =bdrv_find_overlay (bs, base); +BlockDriverState *above_base;
Do we need to check for bottom_cow_node == NULL?
I think you could get a bs that is a filter of bottom_cow_node, and then bdrv_find_overlay() returns NULL and...
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 245
Quoted 6 lines
block/stream.c +5
-1-if (bdrv_freeze_chain (bs, bottom, errp) <0 ) { +/* Find the node directly above @base */ +for (above_base = bottom_cow_node; +bdrv_filtered_bs (above_base) != base; + above_base =bdrv_filtered_bs (above_base)) + {}
...bottom_cow_node == NULL turns this into an endless loop.
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 249
Quoted 3 lines
block/stream.c +1
-0+if (bdrv_freeze_chain (bs, above_base, errp) <0 ) {return ; }
Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at base_overlay, like we (intend to) do since commit c624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.
So should this use bottom_cow_node/base_overlay instead of above_base?
You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 278
Quoted 12 lines
block/stream.c +3
-4@@ -261,16 +272,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,* disappear from the chain after this operation. The streaming job reads * every block only once, assuming that it doesn't change, so forbid writes * and resizes. Reassign the base node pointer because the backing BS of the -* bottom node might change after the call to bdrv_reopen_set_read_only() -* due to parallel block jobs running. +* above_base node might change after the call to +* bdrv_reopen_set_read_only() due to parallel block jobs running. */ - base =backing_bs (bottom); -for (iter =backing_bs (bs); iter && iter != base; iter =backing_bs (iter)) { + base =bdrv_filtered_bs (above_base);
We just calculated above_base such that it's the parent of base. Why would base not already have the value we're assigning it again here?
Quoted 14 lines
block/stream.c +5
-1+for (iter =bdrv_filtered_bs (bs); iter && iter != base; + iter =bdrv_filtered_bs (iter)) + {block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , basic_flags, &error_abort); } - s->bottom = bottom; + s->bottom_cow_node = bottom_cow_node; + s->above_base = above_base; s->backing_file_str =g_strdup (backing_file_str); s->bs_read_only = bs_read_only; s->chain_frozen = true;
block/stream.c fail: · line 302
Quoted 6 lines
block/stream.c +1
-1@@ -284,5 +298,5 @@ fail:if (bs_read_only) {bdrv_reopen_set_read_only (bs, true,NULL ); } -bdrv_unfreeze_chain (bs, bottom); +bdrv_unfreeze_chain (bs, above_base); }
Kevin
Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters
Quoted 34 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 33 lines
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)qapi/block-core.json +4
-0@@ -2516,6 +2516,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #block/stream.c +0
-0@@ -31,7 +31,8 @@ enum {
Quoted 11 lines
block/stream.c enum { · line 35
Quoted 5 lines
block/stream.c +2
-1typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *bottom_cow_node; + BlockDriverState *above_base;Confusing naming, especially because in commit you used
above_basefor what isbottom_cow_nodehere. Vladimir already suggested usingbase_overlayconsistently, so we should do this here too (forbottom_cow_node).above_basecan keep its name because the differentabove_basein commit is going to be renamed).
Sure.
Quoted 88 lines
Quoted 79 lines
block/stream.c +13
-10BlockdevOnError on_error;char *backing_file_str;bool bs_read_only;@@ -54,7 +55,7 @@ static void stream_abort(Job *job)if (s->chain_frozen) {BlockJob *bjob = &s->common; -bdrv_unfreeze_chain (blk_bs (bjob->blk), s->bottom); +bdrv_unfreeze_chain (blk_bs (bjob->blk), s->above_base); } }@@ -63,14 +64,15 @@ static int stream_prepare(Job *job)StreamBlockJob *s =container_of (job, StreamBlockJob, common.job);BlockJob *bjob = &s->common;BlockDriverState *bs =blk_bs (bjob->blk); -BlockDriverState *base =backing_bs (s->bottom); +BlockDriverState *unfiltered_bs =bdrv_skip_rw_filters (bs); +BlockDriverState *base =bdrv_filtered_bs (s->above_base);Error *local_err =NULL ;int ret =0 ; -bdrv_unfreeze_chain (bs, s->bottom); +bdrv_unfreeze_chain (bs, s->above_base); s->chain_frozen = false; -if (bs->backing) { +if (bdrv_filtered_cow_child (unfiltered_bs)) {const char *base_id =NULL , *base_fmt =NULL ;if (base) { base_id = s->backing_file_str;@@ -78,8 +80,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } -bdrv_set_backing_hd (bs, base, &local_err); - ret =bdrv_change_backing_file (bs, base_id, base_fmt); +bdrv_set_backing_hd (unfiltered_bs, base, &local_err); + ret =bdrv_change_backing_file (unfiltered_bs, base_id, base_fmt);if (local_err) {error_report_err (local_err);return -EPERM ;@@ -110,7 +112,8 @@ static int coroutine_fn stream_run(Job *job, Error **errp)StreamBlockJob *s =container_of (job, StreamBlockJob, common.job);BlockBackend *blk = s->common.blk;BlockDriverState *bs =blk_bs (blk); -bool enable_cor = !backing_bs (s->bottom); +BlockDriverState *unfiltered_bs =bdrv_skip_rw_filters (bs); +bool enable_cor = !bdrv_filtered_bs (s->above_base);int64_t len;int64_t offset =0 ;uint64_t delay_ns =0 ;@@ -119,7 +122,7 @@ static int coroutine_fn stream_run(Job *job, Error **errp)int64_t n =0 ;/* bytes */ void *buf; -if (bs == s->bottom) { +if (unfiltered_bs == s->bottom_cow_node) {/* Nothing to stream */ return 0 ; }@@ -154,13 +157,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret =bdrv_is_allocated (bs, offset,STREAM_BUFFER_SIZE , &n); + ret =bdrv_is_allocated (unfiltered_bs, offset,STREAM_BUFFER_SIZE , &n);if (ret ==1 ) {/* Allocated in the top, no need to copy. */ }else if (ret >=0 ) {/* Copy if allocated in the intermediate images. Limit to the * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE). */ - ret =bdrv_is_allocated_above (backing_bs (bs), s->bottom, true, + ret =bdrv_is_allocated_above (bdrv_filtered_cow_bs (unfiltered_bs), + s->bottom_cow_node, true, offset, n, &n);/* Finish early if end of backing file has been reached */ if (ret ==0 && n ==0 ) {block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 239
Quoted 6 lines
block/stream.c +2
-1@@ -231,9 +235,16 @@ void stream_start(const char *job_id, BlockDriverState *bs,BlockDriverState *iter;bool bs_read_only;int basic_flags =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE_UNCHANGED ; -BlockDriverState *bottom =bdrv_find_overlay (bs, base); +BlockDriverState *bottom_cow_node =bdrv_find_overlay (bs, base); +BlockDriverState *above_base;Do we need to check for
bottom_cow_node== NULL?
Quoted 2 lines
I think you could get a bs that is a filter of
bottom_cow_node, and thenbdrv_find_overlay()returns NULL and...
Ah, yes. It isn’t even about the infinite loop, it’s just a case of “Nothing to stream” (if @bs is just a filter chain away from @base).
Also, I just noticed that bdrv_find_overlay() in the version of this series won’t work if the BDS passed to it (@base here) is a filter, so that’s something else to be fixed.
Quoted 13 lines
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 245
Quoted 6 lines
block/stream.c +5
-1-if (bdrv_freeze_chain (bs, bottom, errp) <0 ) { +/* Find the node directly above @base */ +for (above_base = bottom_cow_node; +bdrv_filtered_bs (above_base) != base; + above_base =bdrv_filtered_bs (above_base)) + {}...bottom_cow_node == NULL turns this into an endless loop.
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 249
Quoted 3 lines
block/stream.c +1
-0+if (bdrv_freeze_chain (bs, above_base, errp) <0 ) {return ; }
Quoted 7 lines
Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at
base_overlay, like we (intend to) do since commitc624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.So should this use bottom_cow_node/base_overlay instead of
above_base?
I suppose I thought “Better be safe than sorry”.
Quoted 5 lines
You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find
bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.
Great, I’ll see to it.
Quoted 7 lines
Quoted 7 lines
block/stream.c +1
-2@@ -261,16 +272,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,* disappear from the chain after this operation. The streaming job reads * every block only once, assuming that it doesn't change, so forbid writes * and resizes. Reassign the base node pointer because the backing BS of the -* bottom node might change after the call to bdrv_reopen_set_read_only() -* due to parallel block jobs running. +* above_base node might change after the call to
Quoted 8 lines
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 278
Quoted 5 lines
block/stream.c +2
-2+* bdrv_reopen_set_read_only() due to parallel block jobs running. */ - base =backing_bs (bottom); -for (iter =backing_bs (bs); iter && iter != base; iter =backing_bs (iter)) { + base =bdrv_filtered_bs (above_base);We just calculated
above_basesuch that it's the parent of base. Why would base not already have the value we're assigning it again here?
That’s no change to existing code, whose reasoning is explained in the comment above: bdrv_reopen_set_read_only() can yield, which might lead to children of the bottom node changing.
If you feel like either that’s superfluous, or that if something like that were to happen we’d have much bigger problems, be my guest to drop both.
But in this series I’d rather just not change it.
Max
Re: [Qemu-devel] [PATCH v6 28/42] stream: Deal with filters
Am 16.09.2019 um 11:52 hat Max Reitz geschrieben:
Quoted 22 lines · Kevin Wolf
Quoted 16 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 278
Quoted 12 lines
block/stream.c +3
-4@@ -261,16 +272,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,* disappear from the chain after this operation. The streaming job reads * every block only once, assuming that it doesn't change, so forbid writes * and resizes. Reassign the base node pointer because the backing BS of the -* bottom node might change after the call to bdrv_reopen_set_read_only() -* due to parallel block jobs running. +* above_base node might change after the call to +* bdrv_reopen_set_read_only() due to parallel block jobs running. */ - base =backing_bs (bottom); -for (iter =backing_bs (bs); iter && iter != base; iter =backing_bs (iter)) { + base =bdrv_filtered_bs (above_base);We just calculated
above_basesuch that it's the parent of base. Why would base not already have the value we're assigning it again here?That’s no change to existing code, whose reasoning is explained in the comment above:
bdrv_reopen_set_read_only()can yield, which might lead to children of the bottom node changing.
Quoted 5 lines
If you feel like either that’s superfluous, or that if something like that were to happen we’d have much bigger problems, be my guest to drop both.
But in this series I’d rather just not change it.
Ah, you mean comments are there to be read?
But actually, I think iterating down to base is too much anyway. The reasoning in the comment for block_job_add_bdrv() is that the nodes will be dropped at the end. But base with all of its filter will be kept after this patch.
So I think the for loop should stop after bs->base_overlay. And then concurrently changing links aren't even a problem any more because that's exactly the place up to which we've frozen the chain.
Kevin
Re: [PATCH v6 28/42] stream: Deal with filters
In reply to Max ReitzQuoted 13 lines · Max Reitz
Quoted 12 lines · Kevin Wolf
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 11 lines
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)
[...]
Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+ if (bdrv_freeze_chain(bs, above_base, errp) < 0) { return; }Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at
base_overlay, like we (intend to) do since commitc624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.So should this use bottom_cow_node/base_overlay instead of
above_base?I suppose I thought “Better be safe than sorry”.
Quoted 7 lines
Quoted 5 lines
You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find
bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.Great, I’ll see to it.
On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.
If @base is a filter, then bdrv_backing_chain_next(s->base_overlay) will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.
(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)
So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?
What this patch did in this version worked because the graph was frozen until @above_base.
Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between s->base_overlay and bdrv_backing_chain_next(s->base_overlay) that matches that at the end of streaming.
Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.
The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of base_overlay with that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.
But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...
Max
Re: [PATCH v6 28/42] stream: Deal with filters
Am 11.12.2019 um 13:52 hat Max Reitz geschrieben:
Quoted 43 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
Quoted 12 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 11 lines
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)[...]
Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+ if (bdrv_freeze_chain(bs, above_base, errp) < 0) { return; }Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at
base_overlay, like we (intend to) do since commitc624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.So should this use bottom_cow_node/base_overlay instead of
above_base?I suppose I thought “Better be safe than sorry”.
Quoted 7 lines
Quoted 5 lines
You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find
bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.Great, I’ll see to it.
On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.
Quoted 6 lines
If @base is a filter, then
bdrv_backing_chain_next(s->base_overlay)will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)
Hm... Okay, let's try an example:
... <- backing <- filter1 <- filter2 <- filter3 <- top
| | |
@base above_base @device
base_overlay
The expected result after the job has completed is:
... <- backing <- filter1 <- top
This means that filter1 must not go away until the job has completed. In other words, we would need to freeze the link between @base and above_base.
If we use backing_bs(above_base) as we currently do, we wouldn't necessarily get filter1 as the new backing child of top (as the documentation promises), but whatever is the backing child of filter2 when the job completes. In other words, documentation and code don't match currently.
Let's look at a few more examples to see which of the options makes more sense:
1. Assume we drop filter1 while the stream job is running, i.e. backing is now the backing child of filter 2. I think it's obvious that when the stream job completes, you want backing to be the new backing child of top and not add filter1 back to the chain.
2. If we add filter1b between filter1 and filter2, do we want to drop it when the job completes? It's not entirely clear, but there are probably cases where keeping it makes sense. The user can always drop it manually again, but if we delete a node, some requests will go through unfiltered.
3. Imagine we replace filter1 wholesale, for example because a concurrently running mirror job performs a storage migration for everything up to filter1. Do we then want to reinstate the whole old subtree when the stream job completes? Certainly not.
So from this I would probably conclude that the bug is the documentation, not necessarily in the code.
Quoted 6 lines
So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?
What this patch did in this version worked because the graph was frozen until @above_base.
No, it didn't provide the promised semantics, because "unless the user changed the graph" isn't part of the documentation. But the promised semantics are probably not what we want, so it's okay.
Quoted 15 lines
Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between
s->base_overlayandbdrv_backing_chain_next(s->base_overlay)that matches that at the end of streaming.Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.
The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of
base_overlaywith that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.
Quoted 4 lines
But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below
base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...
Using node names feels completely wrong to me. If we can't keep a pointer (with a reference) because we want the user to be able to delete the node, then we certainly can't keep the node name either because that could refer to an entirely different node when the job completes.
I don't think it's useful to waste too much thought on how to implement the behaviour required by the documentation. The requirement seems to be just wrong.
So in the end, maybe your current code is fine, and the way to address the "doesn't make obvious sense" part is having a comment that explains why above_base is where we stop freezing.
Kevin
Re: [PATCH v6 28/42] stream: Deal with filters
Quoted 67 lines · Kevin Wolf
Am 11.12.2019 um 13:52 hat Max Reitz geschrieben:
Quoted 43 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
Quoted 12 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 11 lines
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)[...]
Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+ if (bdrv_freeze_chain(bs, above_base, errp) < 0) { return; }Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at
base_overlay, like we (intend to) do since commitc624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.So should this use bottom_cow_node/base_overlay instead of
above_base?I suppose I thought “Better be safe than sorry”.
Quoted 7 lines
Quoted 5 lines
You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find
bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.Great, I’ll see to it.
On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.
Quoted 6 lines
If @base is a filter, then
bdrv_backing_chain_next(s->base_overlay)will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)
Hm... Okay, let's try an example:
... <- backing <- filter1 <- filter2 <- filter3 <- top
| | |
@baseabove_base@device
base_overlayThe expected result after the job has completed is:
... <- backing <- filter1 <- topThis means that filter1 must not go away until the job has completed. In other words, we would need to freeze the link between @base and
above_base.
Quoted 5 lines
If we use
backing_bs(above_base)as we currently do, we wouldn't necessarily get filter1 as the new backing child of top (as the documentation promises), but whatever is the backing child of filter2 when the job completes. In other words, documentation and code don't match currently.
Correct.
Quoted 8 lines
Let's look at a few more examples to see which of the options makes more sense:
1. Assume we drop filter1 while the stream job is running, i.e. backing is now the backing child of filter 2. I think it's obvious that when the stream job completes, you want backing to be the new backing child of top and not add filter1 back to the chain.
Quoted 5 lines
2. If we add filter1b between filter1 and filter2, do we want to drop it when the job completes? It's not entirely clear, but there are probably cases where keeping it makes sense. The user can always drop it manually again, but if we delete a node, some requests will go through unfiltered.
Depends. If the base-node was given to be literally "backing", then I’d say the user wants us to use "backing" as the new base still.
Quoted 4 lines
3. Imagine we replace filter1 wholesale, for example because a concurrently running mirror job performs a storage migration for everything up to filter1. Do we then want to reinstate the whole old subtree when the stream job completes? Certainly not.
I’m not sure. Again, if I as the user specified the "backing" node-name as the base, I’d expect that to be the new backing file in all cases.
Quoted 2 lines
So from this I would probably conclude that the bug is the documentation, not necessarily in the code.
It certainly is true that it does not address what happens when you meddle with base.
(Except it can be argued (and I suppose I did argue that) that as it is it does say the base file should be the backing file after stream, so if the base file is still there, I do interpret that to mean that the stream job always uses that as the backing node. Of course even if it says that, that doesn’t mean that it makes sense. And if it doesn’t make sense, then that’s the definition of a bug in the documentation, yes.)
Quoted 10 lines
Quoted 6 lines
So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?
What this patch did in this version worked because the graph was frozen until @above_base.
No, it didn't provide the promised semantics, because "unless the user changed the graph" isn't part of the documentation.But the promised semantics are probably not what we want, so it's okay.
I would have said that no semantics are promised for when the node is no longer a valid base, so we can basically do what we think makes sense.
Quoted 20 lines
Quoted 15 lines
Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between
s->base_overlayandbdrv_backing_chain_next(s->base_overlay)that matches that at the end of streaming.Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.
The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of
base_overlaywith that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.Quoted 4 lines
But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below
base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...
Quoted 5 lines
Using node names feels completely wrong to me. If we can't keep a pointer (with a reference) because we want the user to be able to delete the node, then we certainly can't keep the node name either because that could refer to an entirely different node when the job completes.
I was thinking that if the user does graph manipulation, we can expect them to have given base-node. So they actually did refer to a node name and it makes sense to me to keep looking it up.
(As a side note: I originally intended to say “storing a pointer or the node-name are the only things that come to mind, but obviously both are stupid” – but then I changed my mind and realized that the node name maybe actually isn’t that stupid.)
Quoted 7 lines
I don't think it's useful to waste too much thought on how to implement the behaviour required by the documentation. The requirement seems to be just wrong.
So in the end, maybe your current code is fine, and the way to address the "doesn't make obvious sense" part is having a comment that explains why
above_baseis where we stop freezing.
Let’s maybe talk about it tomorrow. :-)
Max
Re: [PATCH v6 28/42] stream: Deal with filters
Am 11.12.2019 um 17:12 hat Max Reitz geschrieben:
Quoted 98 lines · Kevin Wolf
Quoted 67 lines
Am 11.12.2019 um 13:52 hat Max Reitz geschrieben:
Quoted 43 lines · Max Reitz
Quoted 13 lines · Kevin Wolf
Quoted 12 lines
Am 09.08.2019 um 18:13 hat Max Reitz geschrieben:
Quoted 11 lines
Because of the recent changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 ++++ block/stream.c | 52 ++++++++++++++++++++++++++++---------------- blockdev.c | 2 +- 3 files changed, 38 insertions(+), 20 deletions(-)[...]
Quoted 14 lines
Quoted 11 lines
Quoted 3 lines
+ if (bdrv_freeze_chain(bs, above_base, errp) < 0) { return; }Hm... This feels odd. There are two places where stopping to freeze the chain would make obvious sense: At base, like we originally did; or at
base_overlay, like we (intend to) do since commitc624b015, because we say that we don't actually mind if the user replaces the base image. I don't see how stopping at the first filter above base makes sense.So should this use bottom_cow_node/base_overlay instead of
above_base?I suppose I thought “Better be safe than sorry”.
Quoted 7 lines
Quoted 5 lines
You couldn't use StreamBlockJob.above_base any more then because it could change, but you also don't really need it anywhere. It's only used for unfreezing (which would change) and for finding the base (you can still find
bdrv_backing_chain_next(s->base_overlay)). I guess this would even be a code simplification.Great, I’ll see to it.
On second thought (yes, I know, it’s been a couple of months...) I’m not so sure.
Quoted 6 lines
If @base is a filter, then
bdrv_backing_chain_next(s->base_overlay)will not return it. So then the filter will be dropped, but that probably isn’t what the user intended.(In fact, the block-stream doc says “When streaming completes the image file will have the base file as its backing file.”)
Hm... Okay, let's try an example:
... <- backing <- filter1 <- filter2 <- filter3 <- top
| | |
@baseabove_base@device
base_overlayThe expected result after the job has completed is:
... <- backing <- filter1 <- topThis means that filter1 must not go away until the job has completed. In other words, we would need to freeze the link between @base and
above_base.Quoted 5 lines
If we use
backing_bs(above_base)as we currently do, we wouldn't necessarily get filter1 as the new backing child of top (as the documentation promises), but whatever is the backing child of filter2 when the job completes. In other words, documentation and code don't match currently.Correct.
Quoted 8 lines
Let's look at a few more examples to see which of the options makes more sense:
1. Assume we drop filter1 while the stream job is running, i.e. backing is now the backing child of filter 2. I think it's obvious that when the stream job completes, you want backing to be the new backing child of top and not add filter1 back to the chain.
Quoted 5 lines
2. If we add filter1b between filter1 and filter2, do we want to drop it when the job completes? It's not entirely clear, but there are probably cases where keeping it makes sense. The user can always drop it manually again, but if we delete a node, some requests will go through unfiltered.
Depends. If the base-node was given to be literally "backing", then I’d say the user wants us to use "backing" as the new base still.
Quoted 4 lines
3. Imagine we replace filter1 wholesale, for example because a concurrently running mirror job performs a storage migration for everything up to filter1. Do we then want to reinstate the whole old subtree when the stream job completes? Certainly not.
Quoted 8 lines
I’m not sure. Again, if I as the user specified the "backing" node-name as the base, I’d expect that to be the new backing file in all cases.
Quoted 2 lines
So from this I would probably conclude that the bug is the documentation, not necessarily in the code.
It certainly is true that it does not address what happens when you meddle with base.
Which is because it wasn't relevant then: You were not allowed to meddle with base when block-stream documentation was written. It's only a possible case since commit c624b015.
If we really want to guarantee that it's exactly base that will be used at the end, then we need to freeze all links down to base, essentially reverting commit c624b015 and disallowing some operations that would be possible if we didn't make that promise. I don't think that's the right thing to do, but at least it would be self-consistent.
Promising base, but not freezing the links, leads to weird results in some of the cases mentioned above.
Quoted 14 lines
(Except it can be argued (and I suppose I did argue that) that as it is it does say the base file should be the backing file after stream, so if the base file is still there, I do interpret that to mean that the stream job always uses that as the backing node. Of course even if it says that, that doesn’t mean that it makes sense. And if it doesn’t make sense, then that’s the definition of a bug in the documentation, yes.)
Quoted 7 lines
Quoted 6 lines
So now this gets hairy. We do want exactly @base as the backing file unless the user changed the graph. But how can we detect that and if it hasn’t changed find @base again?
What this patch did in this version worked because the graph was frozen until @above_base.
Quoted 6 lines
Quoted 3 lines
No, it didn't provide the promised semantics, because "unless the user changed the graph" isn't part of the documentation.But the promised semantics are probably not what we want, so it's okay.
I would have said that no semantics are promised for when the node is no longer a valid base, so we can basically do what we think makes sense.
Like undefined behaviour in C?
Sounds great, let's reboot the guest into some 512-byte demo then.
Quoted 26 lines
Quoted 20 lines
Quoted 15 lines
Alternatively, we could store a pointer to @base directly (or its node nmae) and then see whether there is some node between
s->base_overlayandbdrv_backing_chain_next(s->base_overlay)that matches that at the end of streaming.Well, actually, a pointer won’t work because of course maybe that node was deleted and the area is now used for an unrelated node that the user doesn’t want as the new backing file.
The node name could actually work, though. I mean, if there is some node in the immediate backing filter chain of
base_overlaywith that node name after streaming, it does make sense to assume that the user wants this to be the backing file; regardless of whether that’s exactly the same node as it was at the start of streaming.Quoted 4 lines
But we now also have to think about what to do when there is no node with such a node name. Should we keep all filters below
base_overlay? Or should we drop all of them? I actually think keeping them is the safer choice...Quoted 5 lines
Using node names feels completely wrong to me. If we can't keep a pointer (with a reference) because we want the user to be able to delete the node, then we certainly can't keep the node name either because that could refer to an entirely different node when the job completes.
Quoted 8 lines
I was thinking that if the user does graph manipulation, we can expect them to have given base-node. So they actually did refer to a node name and it makes sense to me to keep looking it up.
(As a side note: I originally intended to say “storing a pointer or the node-name are the only things that come to mind, but obviously both are stupid” – but then I changed my mind and realized that the node name maybe actually isn’t that stupid.)
I think it is. A node name refers to the node with that name at the time of the QMP command. What node was referenced in that command doesn't change later because of graph manipulations.
Quoted 9 lines
Quoted 7 lines
I don't think it's useful to waste too much thought on how to implement the behaviour required by the documentation. The requirement seems to be just wrong.
So in the end, maybe your current code is fine, and the way to address the "doesn't make obvious sense" part is having a comment that explains why
above_baseis where we stop freezing.Let’s maybe talk about it tomorrow. :-)
Okay, added to my preliminary agenda.
Kevin
PATCH v6 29/42 nbd: Use CAF when looking for dirty bitmap
In reply to Max ReitzWhen looking for a dirty bitmap to share, we should handle filters by just including them in the search (so they do not break backing chains).
--- nbd/server.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
nbd/server.c +3 -3
@@ -1511,13 +1511,13 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,if (bitmap) {BdrvDirtyBitmap *bm =NULL ; -while (true) { +while (bs) { bm =bdrv_find_dirty_bitmap (bs, bitmap); -if (bm !=NULL || bs->backing ==NULL ) { +if (bm !=NULL ) {break ; } - bs = bs->backing->bs; + bs =bdrv_filtered_bs (bs); }if (bm ==NULL ) {
Signature · 1 line
2.21.0
PATCH v6 30/42 qemu-img: Use child access functions
In reply to Max ReitzThis changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
--- qemu-img.c | 33 ++++++++++++++++++++------------- tests/qemu-iotests/204.out | 1 + 2 files changed, 21 insertions(+), 13 deletions(-)
qemu-img.c +20 -13
@@ -1012,7 +1012,7 @@ static int img_commit(int argc, char **argv)/* This is different from QMP, which by default uses the deepest file in * the backing chain (i.e., the very base); however, the traditional * behavior of qemu-img commit is using the immediate backing file. */ - base_bs =backing_bs (bs); + base_bs =bdrv_backing_chain_next (bs);if (!base_bs) {error_setg (&local_err,"Image does not have a backing file" ); goto done;
@@ -1632,18 +1632,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)if (s->sector_next_status <= sector_num) {uint64_t offset = (sector_num - src_cur_offset) *BDRV_SECTOR_SIZE ;int64_t count; +BlockDriverState *src_bs =blk_bs (s->src[src_cur]); +BlockDriverState *base; + +if (s->target_has_backing) { + base =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (src_bs)); + }else { + base =NULL ; + } do { count = n * BDRV_SECTOR_SIZE; - if (s->target_has_backing) { - ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset, - count, &count, NULL, NULL); - } else { - ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL, - offset, count, &count, NULL, - NULL); - } + ret = bdrv_block_status_above(src_bs, base, offset, count, &count, + NULL, NULL); if (ret < 0) { if (s->salvage) {
@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)* s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only * for optimizations, so such a case is not fatal. */ - s.target_backing_sectors =bdrv_nb_sectors (out_bs->backing->bs); + s.target_backing_sectors = +bdrv_nb_sectors (bdrv_filtered_cow_bs (out_bs)); }else { s.target_backing_sectors =-1 ; }
@@ -2853,6 +2856,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, depth =0 ;for (;;) { + bs =bdrv_skip_rw_filters (bs); ret =bdrv_block_status (bs, offset, bytes, &bytes, &map, &file);if (ret <0 ) {return ret;
@@ -2861,7 +2865,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,if (ret & (BDRV_BLOCK_ZERO |BDRV_BLOCK_DATA )) {break ; } - bs =backing_bs (bs); + bs =bdrv_filtered_cow_bs (bs);if (bs ==NULL ) { ret =0 ;break ;
@@ -3216,6 +3220,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;
@@ -3350,6 +3355,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_rw_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);
@@ -3361,7 +3368,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_filtered_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3517,7 +3524,7 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need * to take action */ - ret =bdrv_is_allocated_above (backing_bs (bs), prefix_chain_bs, + ret =bdrv_is_allocated_above (unfiltered_bs, prefix_chain_bs, false, offset, n, &n);if (ret <0 ) { error_report("error while reading image metadata: %s",
tests/qemu-iotests/204.out +1 -0
@@ -59,5 +59,6 @@ Offset Length File 0x900000 0x2400000 TEST_DIR/t.IMGFMT 0x3c00000 0x1100000 TEST_DIR/t.IMGFMT 0x6a00000 0x400000 TEST_DIR/t.IMGFMT +0x6e00000 0x1200000 TEST_DIR/t.IMGFMT.base No errors were found on the image. *** done
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 30/42] qemu-img: Use child access functions
Quoted 54 lines · Max Reitz
This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
Signed-off-by Max Reitz <[email protected]>--- qemu-img.c | 33 ++++++++++++++++++++------------- tests/qemu-iotests/204.out | 1 + 2 files changed, 21 insertions(+), 13 deletions(-)qemu-img.c +11
-9@@ -1012,7 +1012,7 @@ static int img_commit(int argc, char **argv)/* This is different from QMP, which by default uses the deepest file in * the backing chain (i.e., the very base); however, the traditional * behavior of qemu-img commit is using the immediate backing file. */ - base_bs =backing_bs (bs); + base_bs =bdrv_backing_chain_next (bs);if (!base_bs) {error_setg (&local_err,"Image does not have a backing file" ); goto done;@@ -1632,18 +1632,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)if (s->sector_next_status <= sector_num) {uint64_t offset = (sector_num - src_cur_offset) *BDRV_SECTOR_SIZE ;int64_t count; +BlockDriverState *src_bs =blk_bs (s->src[src_cur]); +BlockDriverState *base; + +if (s->target_has_backing) { + base =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (src_bs)); + }else { + base =NULL ; + } do { count = n * BDRV_SECTOR_SIZE; - if (s->target_has_backing) { - ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset, - count, &count, NULL, NULL); - } else { - ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL, - offset, count, &count, NULL, - NULL); - } + ret = bdrv_block_status_above(src_bs, base, offset, count, &count, + NULL, NULL); if (ret < 0) { if (s->salvage) {
qemu-img.c static int img_convert(int argc, char **argv) · line 2496
Quoted 6 lines
qemu-img.c +2
-1@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)* s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only * for optimizations, so such a case is not fatal. */ - s.target_backing_sectors =bdrv_nb_sectors (out_bs->backing->bs); + s.target_backing_sectors = +bdrv_nb_sectors (bdrv_filtered_cow_bs (out_bs));
why not skip_rw_filters? It will fail if out_bs is filter..
Quoted 67 lines
qemu-img.c +7
-3}else { s.target_backing_sectors =-1 ; }@@ -2853,6 +2856,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, depth =0 ;for (;;) { + bs =bdrv_skip_rw_filters (bs); ret =bdrv_block_status (bs, offset, bytes, &bytes, &map, &file);if (ret <0 ) {return ret;@@ -2861,7 +2865,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,if (ret & (BDRV_BLOCK_ZERO |BDRV_BLOCK_DATA )) {break ; } - bs =backing_bs (bs); + bs =bdrv_filtered_cow_bs (bs);if (bs ==NULL ) { ret =0 ;break ;@@ -3216,6 +3220,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;@@ -3350,6 +3355,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_rw_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);@@ -3361,7 +3368,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_filtered_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),@@ -3517,7 +3524,7 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need * to take action */ - ret =bdrv_is_allocated_above (backing_bs (bs), prefix_chain_bs, + ret =bdrv_is_allocated_above (unfiltered_bs, prefix_chain_bs, false, offset, n, &n);if (ret <0 ) { error_report("error while reading image metadata: %s",tests/qemu-iotests/204.out +1
-0@@ -59,5 +59,6 @@ Offset Length File 0x900000 0x2400000 TEST_DIR/t.IMGFMT 0x3c00000 0x1100000 TEST_DIR/t.IMGFMT 0x6a00000 0x400000 TEST_DIR/t.IMGFMT +0x6e00000 0x1200000 TEST_DIR/t.IMGFMT.base No errors were found on the image. *** done
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 30/42] qemu-img: Use child access functions
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
Quoted 15 lines · Max Reitz
This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
Signed-off-by Max Reitz <[email protected]>--- qemu-img.c | 33 ++++++++++++++++++++------------- tests/qemu-iotests/204.out | 1 + 2 files changed, 21 insertions(+), 13 deletions(-)qemu-img.c +0
-0
[...]
Quoted 9 lines
qemu-img.c static int img_convert(int argc, char **argv) · line 2496
Quoted 7 lines
qemu-img.c +2
-1@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)* s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only * for optimizations, so such a case is not fatal. */ - s.target_backing_sectors =bdrv_nb_sectors (out_bs->backing->bs); + s.target_backing_sectors = +bdrv_nb_sectors (bdrv_filtered_cow_bs (out_bs));why not
skip_rw_filters? It will fail ifout_bsis filter..
Because I forgot this place. :-) Although backing_chain_next() would be simpler here and do the same, effectively.
Max
Re: [Qemu-devel] [PATCH v6 30/42] qemu-img: Use child access functions
In reply to Max ReitzQuoted 54 lines · Max Reitz
This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
Signed-off-by Max Reitz <[email protected]>--- qemu-img.c | 33 ++++++++++++++++++++------------- tests/qemu-iotests/204.out | 1 + 2 files changed, 21 insertions(+), 13 deletions(-)qemu-img.c +11
-9@@ -1012,7 +1012,7 @@ static int img_commit(int argc, char **argv)/* This is different from QMP, which by default uses the deepest file in * the backing chain (i.e., the very base); however, the traditional * behavior of qemu-img commit is using the immediate backing file. */ - base_bs =backing_bs (bs); + base_bs =bdrv_backing_chain_next (bs);if (!base_bs) {error_setg (&local_err,"Image does not have a backing file" ); goto done;@@ -1632,18 +1632,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)if (s->sector_next_status <= sector_num) {uint64_t offset = (sector_num - src_cur_offset) *BDRV_SECTOR_SIZE ;int64_t count; +BlockDriverState *src_bs =blk_bs (s->src[src_cur]); +BlockDriverState *base; + +if (s->target_has_backing) { + base =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (src_bs)); + }else { + base =NULL ; + } do { count = n * BDRV_SECTOR_SIZE; - if (s->target_has_backing) { - ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset, - count, &count, NULL, NULL); - } else { - ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL, - offset, count, &count, NULL, - NULL); - } + ret = bdrv_block_status_above(src_bs, base, offset, count, &count, + NULL, NULL); if (ret < 0) { if (s->salvage) {
qemu-img.c static int img_convert(int argc, char **argv) · line 2496
Quoted 6 lines
qemu-img.c +2
-1@@ -2490,7 +2492,8 @@ static int img_convert(int argc, char **argv)* s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only * for optimizations, so such a case is not fatal. */ - s.target_backing_sectors =bdrv_nb_sectors (out_bs->backing->bs); + s.target_backing_sectors = +bdrv_nb_sectors (bdrv_filtered_cow_bs (out_bs));
bdrv_nb_sectors(bdrv_backing_chain_next(out_bs))
Quoted 47 lines
qemu-img.c +6
-2}else { s.target_backing_sectors =-1 ; }@@ -2853,6 +2856,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, depth =0 ;for (;;) { + bs =bdrv_skip_rw_filters (bs); ret =bdrv_block_status (bs, offset, bytes, &bytes, &map, &file);if (ret <0 ) {return ret;@@ -2861,7 +2865,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,if (ret & (BDRV_BLOCK_ZERO |BDRV_BLOCK_DATA )) {break ; } - bs =backing_bs (bs); + bs =bdrv_filtered_cow_bs (bs);if (bs ==NULL ) { ret =0 ;break ;@@ -3216,6 +3220,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;@@ -3350,6 +3355,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_rw_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);@@ -3361,7 +3368,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_filtered_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),
qemu-img.c static int img_rebase(int argc, char **argv) · line 3527
Quoted 5 lines
qemu-img.c +1
-1@@ -3517,7 +3524,7 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need * to take action */ - ret = bdrv_is_allocated_above(backing_bs(bs), prefix_chain_bs, + ret = bdrv_is_allocated_above(unfiltered_bs, prefix_chain_bs,
s/unfiltered_bs/bdrv_filtered_cow_bs(unfiltered_bs)/
Quoted 9 lines
qemu-img.c +0
-0false, offset, n, &n);if (ret <0 ) { error_report("error while reading image metadata: %s",tests/qemu-iotests/204.out +0
-0
tests/qemu-iotests/204.out Offset Length File · line 64
Quoted 6 lines
tests/qemu-iotests/204.out +1
-0@@ -59,5 +59,6 @@ Offset Length File 0x900000 0x2400000 TEST_DIR/t.IMGFMT 0x3c00000 0x1100000 TEST_DIR/t.IMGFMT 0x6a00000 0x400000 TEST_DIR/t.IMGFMT +0x6e00000 0x1200000 TEST_DIR/t.IMGFMT.base No errors were found on the image. *** done
With two fixes:
Signature · 2 lines
Best regards, Vladimir
PATCH v6 31/42 block: Drop backing_bs()
In reply to Max ReitzWe want to make it explicit where bs->backing is used, and we have done so. The old role of backing_bs() is now effectively taken by bdrv_filtered_cow_bs().
--- include/block/block_int.h | 5 ----- 1 file changed, 5 deletions(-)
include/block/block_int.h +0 -5
@@ -932,11 +932,6 @@ typedef enum BlockMirrorBackingMode { MIRROR_LEAVE_BACKING_CHAIN, } BlockMirrorBackingMode; -static inline BlockDriverState *backing_bs (BlockDriverState *bs) -{ -return bs->backing ? bs->backing->bs :NULL ; -} -/* Essential block drivers which must always be statically linked into qemu, and * which therefore can be accessed without using bdrv_find_format() */
Signature · 1 line
2.21.0
PATCH v6 32/42 block: Make bdrv_get_cumulative_perm() public
In reply to Max ReitzThis is useful in other files like blockdev.c to determine for example whether a node can be written to or not.
--- include/block/block_int.h | 3 +++ block.c | 6 ++---- 2 files changed, 5 insertions(+), 4 deletions(-)
include/block/block_int.h +3 -0
@@ -1205,6 +1205,9 @@ int bdrv_child_try_set_perm(BdrvChild *c, uint64_t perm, uint64_t shared,*/ int bdrv_child_refresh_perms (BlockDriverState *bs,BdrvChild *c,Error **errp); +void bdrv_get_cumulative_perm (BlockDriverState *bs, +uint64_t *perm,uint64_t *shared_perm); +/* Default implementation for BlockDriver.bdrv_child_perm() that can be used by * block filters: Forward CONSISTENT_READ, WRITE, WRITE_UNCHANGED and RESIZE to * all children */
block.c +2 -4
@@ -1713,8 +1713,6 @@ static int bdrv_child_check_perm(BdrvChild *c, BlockReopenQueue *q, bool *tighten_restrictions, Error **errp);static void bdrv_child_abort_perm_update (BdrvChild *c);static void bdrv_child_set_perm (BdrvChild *c,uint64_t perm,uint64_t shared); -static void bdrv_get_cumulative_perm (BlockDriverState *bs,uint64_t *perm, -uint64_t *shared_perm); typedef struct BlockReopenQueueEntry { bool prepared;
@@ -1938,8 +1936,8 @@ static void bdrv_set_perm(BlockDriverState *bs, uint64_t cumulative_perms, } } -static void bdrv_get_cumulative_perm (BlockDriverState *bs,uint64_t *perm, -uint64_t *shared_perm) +void bdrv_get_cumulative_perm (BlockDriverState *bs, +uint64_t *perm,uint64_t *shared_perm) {BdrvChild *c;uint64_t cumulative_perms =0 ;
Signature · 1 line
2.21.0
PATCH v6 33/42 blockdev: Fix active commit choice
In reply to Max ReitzWe have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.
--- blockdev.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +21 -3
@@ -3356,6 +3356,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,*/ BlockdevOnError on_error =BLOCKDEV_ON_ERROR_REPORT ;int job_flags =JOB_DEFAULT ; +uint64_t top_perm, top_shared;if (!has_speed) { speed =0 ;
@@ -3468,14 +3469,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, goto out; } -if (top_bs == bs) { +/* +* Active commit is required if and only if someone has taken a +* WRITE permission on the top node. Historically, we have always +* used active commit for top nodes, so continue that practice. +* (Active commit is never really wrong.) +*/ +bdrv_get_cumulative_perm (top_bs, &top_perm, &top_shared); +if (top_perm &BLK_PERM_WRITE || +bdrv_skip_rw_filters (top_bs) ==bdrv_skip_rw_filters (bs)) + {if (has_backing_file) {error_setg (errp,"'backing-file' specified," " but 'top' is the active layer" ); goto out; } -commit_active_start (has_job_id ? job_id :NULL , bs, base_bs, - job_flags, speed, on_error, +if (!has_job_id) { +/* +* Emulate here what block_job_create() does, because it +* is possible that @bs != @top_bs (the block job should +* be named after @bs, even if @top_bs is the actual +* source) +*/ + job_id =bdrv_get_device_name (bs); + } +commit_active_start (job_id, top_bs, base_bs, job_flags, speed, on_error, filter_node_name,NULL ,NULL , false, &local_err); }else {BlockDriverState *overlay_bs =bdrv_find_overlay (bs, top_bs);
Signature · 1 line
2.21.0
PATCH v6 34/42 block: Inline bdrv_co_block_status_from_*()
In reply to Max ReitzWith bdrv_filtered_rw_bs(), we can easily handle this default filter behavior in bdrv_co_block_status().
blkdebug wants to have an additional assertion, so it keeps its own implementation, except bdrv_co_block_status_from_file() needs to be inlined there.
--- include/block/block_int.h | 22 ----------------- block/blkdebug.c | 7 ++++-- block/blklogwrites.c | 1 - block/commit.c | 1 - block/copy-on-read.c | 2 -- block/io.c | 51 +++++++++++++-------------------------- block/mirror.c | 1 - block/throttle.c | 1 - 8 files changed, 22 insertions(+), 64 deletions(-)
include/block/block_int.h +0 -22
@@ -1227,28 +1227,6 @@ void bdrv_format_default_perms(BlockDriverState *bs, BdrvChild *c, uint64_t perm, uint64_t shared, uint64_t *nperm, uint64_t *nshared); -/* -* Default implementation for drivers to pass bdrv_co_block_status() to -* their file. -*/ -int coroutine_fn bdrv_co_block_status_from_file(BlockDriverState *bs, -bool want_zero, -int64_t offset, -int64_t bytes, -int64_t *pnum, -int64_t *map, -BlockDriverState **file); -/* -* Default implementation for drivers to pass bdrv_co_block_status() to -* their backing file. -*/ -int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs, -bool want_zero, -int64_t offset, -int64_t bytes, -int64_t *pnum, -int64_t *map, -BlockDriverState **file);const char *bdrv_get_parent_name (const BlockDriverState *bs);void blk_dev_change_media_cb (BlockBackend *blk,bool load,Error **errp);bool blk_dev_has_removable_media (BlockBackend *blk);
block/blkdebug.c +5 -2
@@ -679,8 +679,11 @@ static int coroutine_fn blkdebug_co_block_status(BlockDriverState *bs,return err; } -return bdrv_co_block_status_from_file (bs, want_zero, offset, bytes, - pnum, map, file); +assert (bs->file && bs->file->bs); + *pnum = bytes; + *map = offset; + *file = bs->file->bs; +return BDRV_BLOCK_RAW |BDRV_BLOCK_OFFSET_VALID ; }static void blkdebug_close (BlockDriverState *bs)
block/blklogwrites.c +0 -1
@@ -519,7 +519,6 @@ static BlockDriver bdrv_blk_log_writes = { .bdrv_co_pwrite_zeroes = blk_log_writes_co_pwrite_zeroes, .bdrv_co_flush_to_disk = blk_log_writes_co_flush_to_disk, .bdrv_co_pdiscard = blk_log_writes_co_pdiscard, - .bdrv_co_block_status = bdrv_co_block_status_from_file, .is_filter = true, .strong_runtime_opts = blk_log_writes_strong_runtime_opts,
block/commit.c +0 -1
@@ -252,7 +252,6 @@ static void bdrv_commit_top_child_perm(BlockDriverState *bs, BdrvChild *c,static BlockDriver bdrv_commit_top = { .format_name ="commit_top" , .bdrv_co_preadv = bdrv_commit_top_preadv, - .bdrv_co_block_status = bdrv_co_block_status_from_backing, .bdrv_refresh_filename = bdrv_commit_top_refresh_filename, .bdrv_child_perm = bdrv_commit_top_child_perm,
block/copy-on-read.c +0 -2
@@ -160,8 +160,6 @@ static BlockDriver bdrv_copy_on_read = { .bdrv_eject = cor_eject, .bdrv_lock_medium = cor_lock_medium, - .bdrv_co_block_status = bdrv_co_block_status_from_file, - .bdrv_recurse_is_first_non_filter = cor_recurse_is_first_non_filter, .has_variable_length = true,
block/io.c +17 -34
75 lines not on this page · raw message
block/mirror.c +0 -1
@@ -1508,7 +1508,6 @@ static BlockDriver bdrv_mirror_top = { .bdrv_co_pwrite_zeroes = bdrv_mirror_top_pwrite_zeroes, .bdrv_co_pdiscard = bdrv_mirror_top_pdiscard, .bdrv_co_flush = bdrv_mirror_top_flush, - .bdrv_co_block_status = bdrv_co_block_status_from_backing, .bdrv_refresh_filename = bdrv_mirror_top_refresh_filename, .bdrv_child_perm = bdrv_mirror_top_child_perm, .bdrv_refresh_limits = bdrv_mirror_top_refresh_limits,
block/throttle.c +0 -1
@@ -270,7 +270,6 @@ static BlockDriver bdrv_throttle = { .bdrv_reopen_prepare = throttle_reopen_prepare, .bdrv_reopen_commit = throttle_reopen_commit, .bdrv_reopen_abort = throttle_reopen_abort, - .bdrv_co_block_status = bdrv_co_block_status_from_file, .bdrv_co_drain_begin = throttle_co_drain_begin, .bdrv_co_drain_end = throttle_co_drain_end,
Signature · 1 line
2.21.0
PATCH v6 35/42 block: Fix check_to_replace_node()
In reply to Max ReitzCurrently, check_to_replace_node() only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.
This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.
This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:
$ qemu-img create -f qcow2 /tmp/source.qcow2 64M
$ qemu-system-x86_64 -qmp stdio
{"execute": "qmp_capabilities"}
{"execute": "object-add",
"arguments": {"qom-type": "throttle-group", "id": "tg0"}}
{"execute": "blockdev-add",
"arguments": {
"node-name": "source",
"driver": "throttle",
"throttle-group": "tg0",
"file": {
"node-name": "filtered",
"driver": "qcow2",
"file": {
"driver": "file",
"filename": "/tmp/source.qcow2"
} } } }
{"execute": "drive-mirror",
"arguments": {
"job-id": "mirror",
"device": "source",
"target": "/tmp/target.qcow2",
"format": "qcow2",
"node-name": "target",
"sync" :"none",
"replaces": "filtered"
} }
{"execute": "block-job-complete", "arguments": {"device": "mirror"}}And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).
(blockdev-mirror can be broken similarly.)
So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.
--- include/block/block.h | 1 + block.c | 83 +++++++++++++++++++++++++++++++++++++++---- blockdev.c | 34 ++++++++++++++++-- 3 files changed, 110 insertions(+), 8 deletions(-)
include/block/block.h +1 -0
@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);/* check if a named node can be replaced when doing drive-mirror */ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp);/* async block I/O */
block.c +77 -6
97 lines not on this page · raw message
blockdev.c +32 -2
@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, }if (has_replaces) { -BlockDriverState *to_replace_bs; +BlockDriverState *to_replace_bs, *backing_bs;AioContext *replace_aio_context;int64_t bs_size, replace_size;
@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } - to_replace_bs =check_to_replace_node (bs, replaces, errp); +if (backing_mode ==MIRROR_SOURCE_BACKING_CHAIN || + backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) + { +/* +* While we do not quite know what OPEN_BACKING_CHAIN +* (used for mode=existing) will yield, it is probably +* best to restrict it exactly like SOURCE_BACKING_CHAIN, +* because that is our best guess. +*/ +switch (sync) { +case MIRROR_SYNC_MODE_FULL : + backing_bs =NULL ; +break ; + +case MIRROR_SYNC_MODE_TOP : + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (bs)); +break ; + +case MIRROR_SYNC_MODE_NONE : + backing_bs = bs; +break ; + +default : +abort (); + } + }else { +assert (backing_mode ==MIRROR_LEAVE_BACKING_CHAIN ); + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (target)); + } + + to_replace_bs =check_to_replace_node (bs, backing_bs, replaces, errp);if (!to_replace_bs) {return ; }
Signature · 1 line
2.21.0
Re: [Qemu-devel] [PATCH v6 35/42] block: Fix check_to_replace_node()
Quoted 92 lines · Max Reitz
Currently,
check_to_replace_node()only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:$ qemu-img create -f qcow2 /tmp/source.qcow2 64M $ qemu-system-x86_64 -qmp stdio {"execute": "qmp_capabilities"} {"execute": "object-add", "arguments": {"qom-type": "throttle-group", "id": "tg0"}} {"execute": "blockdev-add", "arguments": { "node-name": "source", "driver": "throttle", "throttle-group": "tg0", "file": { "node-name": "filtered", "driver": "qcow2", "file": { "driver": "file", "filename": "/tmp/source.qcow2" } } } } {"execute": "drive-mirror", "arguments": { "job-id": "mirror", "device": "source", "target": "/tmp/target.qcow2", "format": "qcow2", "node-name": "target", "sync" :"none", "replaces": "filtered" } } {"execute": "block-job-complete", "arguments": {"device": "mirror"}}And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).
(blockdev-mirror can be broken similarly.)
So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.
Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 1 + block.c | 83 +++++++++++++++++++++++++++++++++++++++---- blockdev.c | 34 ++++++++++++++++-- 3 files changed, 110 insertions(+), 8 deletions(-)include/block/block.h +1
-0@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);/* check if a named node can be replaced when doing drive-mirror */ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp);/* async block I/O */ block.c +11
-0@@ -6290,7 +6290,59 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate)return false; } +static bool is_child_of (BlockDriverState *child,BlockDriverState *parent) +{ +BdrvChild *c; + +if (!parent) { +return false; + } + + QLIST_FOREACH(c, &parent->children, next) { +if (c->bs == child ||is_child_of (child, c->bs)) { +return true;
block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6313
Quoted 10 lines
block.c +10
-0+ } + } + +return false; +} + +/* +* Return true if there are only filters in [@top, @base). Note that +* this may include quorum (which bdrv_chain_contains() cannot +* handle).
More presizely: return true if exists chain of filters from top to base or if top == base.
I keep in mind backup-top filter:
[backup-top]
| \target
|backing -------->[target]
V /
[source] <---------/backing
Quoted 12 lines
block.c +12
-0+*/ +static bool is_filtered_child (BlockDriverState *top,BlockDriverState *base) +{ +BdrvChild *c; + +if (!top) { +return false; + } + +if (top == base) { +return true; + }
block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6335
Quoted 10 lines
block.c +10
-0+ +if (!top->drv->is_filter) { +return false; + } + + QLIST_FOREACH(c, &top->children, next) { +if (is_filtered_child (c->bs, base)) { +return true; + } + }
interesting, how much is it better to somehow reuse DFS search written in should_update_child().. [just note, don't do it in these series please]
Quoted 18 lines
block.c +9
-1+ +return false; +} + +/* +* @parent_bs is mirror's source BDS, @backing_bs is the BDS which +* will be attached to the target when mirror completes. +*/ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp) {BlockDriverState *to_replace_bs =bdrv_find_node (node_name);@@ -6309,13 +6361,32 @@ BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, goto out; } -/* We don't want arbitrary node of the BDS chain to be replaced only the top
block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6368
Quoted 10 lines
block.c +4
-5-* most non filter in order to prevent data corruption. -* Another benefit is that this tests exclude backing files which are -* blocked by the backing blockers. +/* +* If to_replace_bs is (recursively) a child of backing_bs, +* replacing it may create a loop. We cannot allow that. */ -if (!bdrv_recurse_is_first_non_filter (parent_bs, to_replace_bs)) { -error_setg (errp,"Only top most non filter can be replaced" ); +if (to_replace_bs == backing_bs ||is_child_of (to_replace_bs, backing_bs)) {
first condition is covered by second, so first may be omitted.
block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
Quoted 8 lines
block.c +8
-0+error_setg (errp,"Replacing this node would result in a loop" ); + to_replace_bs =NULL ; + goto out; + } + +/* +* Mirror is designed in such a way that when it completes, the +* source BDS is seamlessly replaced.
Not source but to_replace_bs is replaced?
Quoted 4 lines
It is therefore not allowed
block.c +3
-0+* to replace a BDS where this condition would be violated, as that +* would defeat the purpose of mirror and could lead to data +* corruption.
block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6389
Quoted 10 lines
block.c +10
-0+* Therefore, between parent_bs and to_replace_bs there may be +* only filters (and the one on top must be a filter, too), so +* their data always stays in sync and mirror can complete and +* replace to_replace_bs without any possible corruptions. +*/ +if (!is_filtered_child (parent_bs, to_replace_bs) && + !is_filtered_child (to_replace_bs, parent_bs)) + { +error_setg (errp,"The node to be replaced must be connected to the " +"source through filter nodes only" );
"and the one on top must be a filter, too" not mentioned in the error..
Quoted 27 lines
to_replace_bs = NULL; goto out; }blockdev.c +7
-2@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, }if (has_replaces) { -BlockDriverState *to_replace_bs; +BlockDriverState *to_replace_bs, *backing_bs;AioContext *replace_aio_context;int64_t bs_size, replace_size;@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } - to_replace_bs =check_to_replace_node (bs, replaces, errp); +if (backing_mode ==MIRROR_SOURCE_BACKING_CHAIN || + backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) + { +/* +* While we do not quite know what OPEN_BACKING_CHAIN +* (used for mode=existing) will yield, it is probably
blockdev.c static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, · line 3915
Quoted 10 lines
blockdev.c +10
-0+* best to restrict it exactly like SOURCE_BACKING_CHAIN, +* because that is our best guess. +*/ +switch (sync) { +case MIRROR_SYNC_MODE_FULL : + backing_bs =NULL ; +break ; + +case MIRROR_SYNC_MODE_TOP : + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (bs));
why not bdrv_backing_chain_next(bs) like in mirror_start?
Quoted 19 lines
blockdev.c +15
-0+break ; + +case MIRROR_SYNC_MODE_NONE : + backing_bs = bs; +break ; + +default : +abort (); + } + }else { +assert (backing_mode ==MIRROR_LEAVE_BACKING_CHAIN ); + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (target)); + } + + to_replace_bs =check_to_replace_node (bs, backing_bs, replaces, errp);if (!to_replace_bs) {return ; }
Signature · 2 lines
Best regards, Vladimir
Re: [Qemu-devel] [PATCH v6 35/42] block: Fix check_to_replace_node()
Quoted 107 lines · Vladimir Sementsov-Ogievskiy
Quoted 92 lines · Max Reitz
Currently,
check_to_replace_node()only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:$ qemu-img create -f qcow2 /tmp/source.qcow2 64M $ qemu-system-x86_64 -qmp stdio {"execute": "qmp_capabilities"} {"execute": "object-add", "arguments": {"qom-type": "throttle-group", "id": "tg0"}} {"execute": "blockdev-add", "arguments": { "node-name": "source", "driver": "throttle", "throttle-group": "tg0", "file": { "node-name": "filtered", "driver": "qcow2", "file": { "driver": "file", "filename": "/tmp/source.qcow2" } } } } {"execute": "drive-mirror", "arguments": { "job-id": "mirror", "device": "source", "target": "/tmp/target.qcow2", "format": "qcow2", "node-name": "target", "sync" :"none", "replaces": "filtered" } } {"execute": "block-job-complete", "arguments": {"device": "mirror"}}And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).
(blockdev-mirror can be broken similarly.)
So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.
Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 1 + block.c | 83 +++++++++++++++++++++++++++++++++++++++---- blockdev.c | 34 ++++++++++++++++-- 3 files changed, 110 insertions(+), 8 deletions(-)include/block/block.h +1
-0@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);/* check if a named node can be replaced when doing drive-mirror */ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp);/* async block I/O */ block.c +11
-0@@ -6290,7 +6290,59 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate)return false; } +static bool is_child_of (BlockDriverState *child,BlockDriverState *parent) +{ +BdrvChild *c; + +if (!parent) { +return false; + } + + QLIST_FOREACH(c, &parent->children, next) { +if (c->bs == child ||is_child_of (child, c->bs)) { +return true;block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6313
Quoted 10 lines
block.c +10
-0+ } + } + +return false; +} + +/* +* Return true if there are only filters in [@top, @base). Note that +* this may include quorum (which bdrv_chain_contains() cannot +* handle). More presizely: return true if exists chain of filters from top to base or if top == base.
Quoted 4 lines
I keep in mind backup-top filter:
[backup-top]
| \target
backup-top can’t be a filter if it has two children with different contents, though.
(commit-top and mirror-top aren’t filters either.)
That’s why there must be a unique chain [@top, @base).
I should probably not that it will return true if top == base, though, yes.
Quoted 47 lines
|backing -------->[target]
V /
[source] <---------/backingQuoted 12 lines
block.c +12
-0+*/ +static bool is_filtered_child (BlockDriverState *top,BlockDriverState *base) +{ +BdrvChild *c; + +if (!top) { +return false; + } + +if (top == base) { +return true; + }block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6335
Quoted 10 lines
block.c +10
-0+ +if (!top->drv->is_filter) { +return false; + } + + QLIST_FOREACH(c, &top->children, next) { +if (is_filtered_child (c->bs, base)) { +return true; + } + }interesting, how much is it better to somehow reuse DFS search written in
should_update_child().. [just note, don't do it in these series please]Quoted 17 lines
block.c +9
-0+ +return false; +} + +/* +* @parent_bs is mirror's source BDS, @backing_bs is the BDS which +* will be attached to the target when mirror completes. +*/ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp) {BlockDriverState *to_replace_bs =bdrv_find_node (node_name);@@ -6309,13 +6361,32 @@ BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, goto out; }
Quoted 13 lines
block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6368
Quoted 11 lines
block.c +4
-6-/* We don't want arbitrary node of the BDS chain to be replaced only the top -* most non filter in order to prevent data corruption. -* Another benefit is that this tests exclude backing files which are -* blocked by the backing blockers. +/* +* If to_replace_bs is (recursively) a child of backing_bs, +* replacing it may create a loop. We cannot allow that. */ -if (!bdrv_recurse_is_first_non_filter (parent_bs, to_replace_bs)) { -error_setg (errp,"Only top most non filter can be replaced" ); +if (to_replace_bs == backing_bs ||is_child_of (to_replace_bs, backing_bs)) {first condition is covered by second, so first may be omitted.
It is not. is_child_of() does not return true if child == parent.
Quoted 10 lines
block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs,
Quoted 8 lines
block.c +8
-0+error_setg (errp,"Replacing this node would result in a loop" ); + to_replace_bs =NULL ; + goto out; + } + +/* +* Mirror is designed in such a way that when it completes, the +* source BDS is seamlessly replaced. Not source but
to_replace_bsis replaced?
It has originally been designed to replace the source. If it could replace any arbitrary BDS, all of this would be moot.
Quoted 15 lines
Quoted 4 lines
It is therefore not allowed
block.c +3
-0+* to replace a BDS where this condition would be violated, as that +* would defeat the purpose of mirror and could lead to data +* corruption. block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6389
Quoted 10 lines
block.c +10
-0+* Therefore, between parent_bs and to_replace_bs there may be +* only filters (and the one on top must be a filter, too), so +* their data always stays in sync and mirror can complete and +* replace to_replace_bs without any possible corruptions. +*/ +if (!is_filtered_child (parent_bs, to_replace_bs) && + !is_filtered_child (to_replace_bs, parent_bs)) + { +error_setg (errp,"The node to be replaced must be connected to the " +"source through filter nodes only" );
Quoted 1 line
"and the one on top must be a filter, too" not mentioned in the error..
Well, unless the source node is the node to be replaced. Hm... This gets very hard to express. I think I’d prefer to keep this as it is, even though it is not quite correct, unless you have a better suggestion of what to report. :-/
Quoted 31 lines
Quoted 31 lines
to_replace_bs = NULL; goto out; }blockdev.c +11
-2@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, }if (has_replaces) { -BlockDriverState *to_replace_bs; +BlockDriverState *to_replace_bs, *backing_bs;AioContext *replace_aio_context;int64_t bs_size, replace_size;@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } - to_replace_bs =check_to_replace_node (bs, replaces, errp); +if (backing_mode ==MIRROR_SOURCE_BACKING_CHAIN || + backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) + { +/* +* While we do not quite know what OPEN_BACKING_CHAIN +* (used for mode=existing) will yield, it is probably +* best to restrict it exactly like SOURCE_BACKING_CHAIN, +* because that is our best guess. +*/ +switch (sync) {
Quoted 8 lines
blockdev.c static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, · line 3915
Quoted 6 lines
blockdev.c +6
-0+case MIRROR_SYNC_MODE_FULL : + backing_bs =NULL ; +break ; + +case MIRROR_SYNC_MODE_TOP : + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (bs));why not
bdrv_backing_chain_next(bs)like inmirror_start?
Good question. I suppose it should be bdrv_filtered_cow_bs(bdrv_backing_chain_next(bs)) in mirror_start()? Because with sync=top, we just want to remove the topmost COW node (and filters on top), but keep filters behind it.
Max
Quoted 21 lines
Quoted 19 lines
blockdev.c +15
-0+break ; + +case MIRROR_SYNC_MODE_NONE : + backing_bs = bs; +break ; + +default : +abort (); + } + }else { +assert (backing_mode ==MIRROR_LEAVE_BACKING_CHAIN ); + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (target)); + } + + to_replace_bs =check_to_replace_node (bs, backing_bs, replaces, errp);if (!to_replace_bs) {return ; }
Re: [Qemu-devel] [PATCH v6 35/42] block: Fix check_to_replace_node()
Quoted 108 lines · Max Reitz
Quoted 107 lines · Vladimir Sementsov-Ogievskiy
Quoted 92 lines · Max Reitz
Currently,
check_to_replace_node()only allows mirror to replace a node in the chain of the source node, and only if it is the first non-filter node below the source. Well, technically, the idea is that you can exactly replace a quorum child by mirroring from quorum.This has (probably) two reasons:
(1) We do not want to create loops.
(2) @replaces and @device should have exactly the same content so replacing them does not cause visible data to change.This has two issues:
(1) It is overly restrictive. It is completely fine for @replaces to be a filter.
(2) It is not restrictive enough. You can create loops with this as follows:$ qemu-img create -f qcow2 /tmp/source.qcow2 64M $ qemu-system-x86_64 -qmp stdio {"execute": "qmp_capabilities"} {"execute": "object-add", "arguments": {"qom-type": "throttle-group", "id": "tg0"}} {"execute": "blockdev-add", "arguments": { "node-name": "source", "driver": "throttle", "throttle-group": "tg0", "file": { "node-name": "filtered", "driver": "qcow2", "file": { "driver": "file", "filename": "/tmp/source.qcow2" } } } } {"execute": "drive-mirror", "arguments": { "job-id": "mirror", "device": "source", "target": "/tmp/target.qcow2", "format": "qcow2", "node-name": "target", "sync" :"none", "replaces": "filtered" } } {"execute": "block-job-complete", "arguments": {"device": "mirror"}}And qemu crashes because of a stack overflow due to the loop being created (target's backing file is source, so when it replaces filtered, it points to itself through source).
(blockdev-mirror can be broken similarly.)
So let us make the checks for the two conditions above explicit, which makes the whole function exactly as restrictive as it needs to be.
Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 1 + block.c | 83 +++++++++++++++++++++++++++++++++++++++---- blockdev.c | 34 ++++++++++++++++-- 3 files changed, 110 insertions(+), 8 deletions(-)include/block/block.h +1
-0@@ -404,6 +404,7 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate);/* check if a named node can be replaced when doing drive-mirror */ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp);/* async block I/O */ block.c +11
-0@@ -6290,7 +6290,59 @@ bool bdrv_is_first_non_filter(BlockDriverState *candidate)return false; } +static bool is_child_of (BlockDriverState *child,BlockDriverState *parent) +{ +BdrvChild *c; + +if (!parent) { +return false; + } + + QLIST_FOREACH(c, &parent->children, next) { +if (c->bs == child ||is_child_of (child, c->bs)) { +return true;block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6313
Quoted 10 lines
block.c +10
-0+ } + } + +return false; +} + +/* +* Return true if there are only filters in [@top, @base). Note that +* this may include quorum (which bdrv_chain_contains() cannot +* handle). More presizely: return true if exists chain of filters from top to base or if top == base.
Quoted 7 lines
Quoted 4 lines
I keep in mind backup-top filter:
[backup-top]
| \targetbackup-top can’t be a filter if it has two children with different contents, though.
Why? target is special child, unrelated to what is read/written over backup-top. It's an own business of backup-top.
Quoted 2 lines
(commit-top and mirror-top aren’t filters either.)
Ahm, I missed something. They have is_filter = true and their children considered to be filtered-rw children in your series? And than, who they are? Format nodes? And how they appears in backing chains than?
Quoted 77 lines
That’s why there must be a unique chain [@top, @base).
I should probably not that it will return true if top == base, though, yes.
Quoted 47 lines
|backing -------->[target]
V /
[source] <---------/backingQuoted 12 lines
block.c +12
-0+*/ +static bool is_filtered_child (BlockDriverState *top,BlockDriverState *base) +{ +BdrvChild *c; + +if (!top) { +return false; + } + +if (top == base) { +return true; + }block.c bool bdrv_is_first_non_filter(BlockDriverState *candidate) · line 6335
Quoted 10 lines
block.c +10
-0+ +if (!top->drv->is_filter) { +return false; + } + + QLIST_FOREACH(c, &top->children, next) { +if (is_filtered_child (c->bs, base)) { +return true; + } + }interesting, how much is it better to somehow reuse DFS search written in
should_update_child().. [just note, don't do it in these series please]Quoted 17 lines
block.c +9
-0+ +return false; +} + +/* +* @parent_bs is mirror's source BDS, @backing_bs is the BDS which +* will be attached to the target when mirror completes. +*/ BlockDriverState *check_to_replace_node (BlockDriverState *parent_bs, +BlockDriverState *backing_bs,const char *node_name,Error **errp) {BlockDriverState *to_replace_bs =bdrv_find_node (node_name);@@ -6309,13 +6361,32 @@ BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, goto out; }Quoted 13 lines
block.c BlockDriverState *check_to_replace_node(BlockDriverState *parent_bs, · line 6368
Quoted 11 lines
block.c +4
-6-/* We don't want arbitrary node of the BDS chain to be replaced only the top -* most non filter in order to prevent data corruption. -* Another benefit is that this tests exclude backing files which are -* blocked by the backing blockers. +/* +* If to_replace_bs is (recursively) a child of backing_bs, +* replacing it may create a loop. We cannot allow that. */ -if (!bdrv_recurse_is_first_non_filter (parent_bs, to_replace_bs)) { -error_setg (errp,"Only top most non filter can be replaced" ); +if (to_replace_bs == backing_bs ||is_child_of (to_replace_bs, backing_bs)) {first condition is covered by second, so first may be omitted.
It is not.
is_child_of()does not return true if child == parent.Quoted 9 lines
Quoted 8 lines
block.c +8
-0+error_setg (errp,"Replacing this node would result in a loop" ); + to_replace_bs =NULL ; + goto out; + } + +/* +* Mirror is designed in such a way that when it completes, the +* source BDS is seamlessly replaced.
Quoted 4 lines
Quoted 1 line
Not source but
to_replace_bsis replaced?It has originally been designed to replace the source. If it could replace any arbitrary BDS, all of this would be moot.
quorum child, you saying about in commit message?
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
It is therefore not allowed
block.c +13
-0+* to replace a BDS where this condition would be violated, as that +* would defeat the purpose of mirror and could lead to data +* corruption. +* Therefore, between parent_bs and to_replace_bs there may be +* only filters (and the one on top must be a filter, too), so +* their data always stays in sync and mirror can complete and +* replace to_replace_bs without any possible corruptions. +*/ +if (!is_filtered_child (parent_bs, to_replace_bs) && + !is_filtered_child (to_replace_bs, parent_bs)) + { +error_setg (errp,"The node to be replaced must be connected to the " +"source through filter nodes only" );
Quoted 6 lines
Quoted 1 line
"and the one on top must be a filter, too" not mentioned in the error..
Well, unless the source node is the node to be replaced. Hm... This gets very hard to express. I think I’d prefer to keep this as it is, even though it is not quite correct, unless you have a better suggestion of what to report. :-/
I can't imaging something better than just add "(and the one on top must be a filter, too)"
Quoted 41 lines
Quoted 32 lines
Quoted 32 lines
to_replace_bs = NULL; goto out; }blockdev.c +11
-2@@ -3887,7 +3887,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, }if (has_replaces) { -BlockDriverState *to_replace_bs; +BlockDriverState *to_replace_bs, *backing_bs;AioContext *replace_aio_context;int64_t bs_size, replace_size;@@ -3897,7 +3897,37 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } - to_replace_bs =check_to_replace_node (bs, replaces, errp); +if (backing_mode ==MIRROR_SOURCE_BACKING_CHAIN || + backing_mode ==MIRROR_OPEN_BACKING_CHAIN ) + { +/* +* While we do not quite know what OPEN_BACKING_CHAIN +* (used for mode=existing) will yield, it is probably +* best to restrict it exactly like SOURCE_BACKING_CHAIN, +* because that is our best guess. +*/ +switch (sync) {Quoted 8 lines
blockdev.c static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, · line 3915
Quoted 6 lines
blockdev.c +6
-0+case MIRROR_SYNC_MODE_FULL : + backing_bs =NULL ; +break ; + +case MIRROR_SYNC_MODE_TOP : + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (bs));why not
bdrv_backing_chain_next(bs)like inmirror_start?
Quoted 2 lines
Good question. I suppose it should be bdrv_filtered_cow_bs(bdrv_backing_chain_next(bs)) in
mirror_start()?
You mean bdrv_filtered_cow_bs(bdrv_skip_rw_filters(bs)), I hope)
Quoted 3 lines
Because with sync=top, we just want to remove the topmost COW node (and filters on top), but keep filters behind it.
Agreed.
Quoted 24 lines
Quoted 22 lines
Quoted 20 lines
blockdev.c +15
-0+break ; + +case MIRROR_SYNC_MODE_NONE : + backing_bs = bs; +break ; + +default : +abort (); + } + }else { +assert (backing_mode ==MIRROR_LEAVE_BACKING_CHAIN ); + backing_bs =bdrv_filtered_cow_bs (bdrv_skip_rw_filters (target)); + } + + to_replace_bs =check_to_replace_node (bs, backing_bs, replaces, errp);if (!to_replace_bs) {return ; }
Signature · 2 lines
Best regards, Vladimir