block: Deal with filters
v7 cover block: Deal with filters
v6: https://lists.nongnu.org/archive/html/qemu-devel/2019-08/msg01715.html
Branch: https://github.com/XanClic/qemu.git child-access-functions-v7
Branch: https://git.xanclic.moe/XanClic/qemu.git child-access-functions-v7
Hello!
This is v7. Conceptually, not much has changed, so please follow the above link to v6’s cover letter if you’re looking for an introduction to this series.
I did say that conceptually, not much has changed, but from a diff standpoint, a lot has changed all over this series.
Changes from v6:
- Patch 1:
- More elaborate explanation of .is_filter
- Changed function names
- Dropped bdrv_storage_child() and bdrv_metadata_child()
- Some checking of the BdrvChildRole
- Patch 2:
- Mostly changes resulting from the different naming scheme
- Patch 3: New
- Patch 5:
- Don’t rename those functions
- Don’t drop a comment that shouldn’t be dropped
- Patch 7:
- Use block_driver_can_compress()
- Move setting @filtered down where it’s needed
- Patches 10 and 11: New (extension of 8 and 9)
- Patch 12:
- Function name changes
- More cases:
- bdrv_recurse_can_replace()
- init_dirty_bitmap_migration()
- bdrv_co_truncate() has changed
- Patch 13:
- Function name changes
- Patch 14:
- Variables renamed to be more consistent with the rest of this series
- Function name changes
- The freeze backing chain functions haven’t been renamed
- STREAM_BUFFER_SIZE is STREAM_CHUNK as of some point last year
- Fix overlay finding (e.g. handle when @base is not in the device’s backing chain)
- Patch 15:
- Added note to the commit message that bdrv_find_overlay()’s behavior changes a bit
- Function name changes
- Restructured bdrv_find_backing_image() loop a bit
- Patch 16: New (became necessary because of truncate having to look at
the backing file as of 955c7d6687fefcd903900)
- Patch 17:
- Function name changes
- The freeze backing chain functions haven’t been renamed
- Patch 18:
- Only flush children for which the parent has taken the WRITE permission
- Mention that this is a bug fix for qcow2
- Patch 19: New
- Patch 20: New, replaces “block: Use CAFs in bdrv_refresh_limits()”
- Patch 23:
- We can only really fall back to bs->file or bs->backing, so stop pretending otherwise
- Patch 24:
- Rebase conflicts
- Patches 25, 26, 27, and 28: New, they replace “block: Fix bdrv_get_allocated_file_size's fallback”
- Patch 29:
- Function name changes
- Patch 30: New, split out from the next patch
- Patch 31:
- Function name changes
- bdrv_skip_implicit_filters() can deal with NULL arguments, so don’t wrap it in an “if (bs) {}” block
- For bdrv_query_bds_stats(), the bs->file part has been split into the preceding patch (patch 30)
- Addional actual-size line in the iotest output thanks to patch 28
- Patch 32: New
- Patch 33:
- Additional note in the QAPI documentation concerning @replaces (that by default, the first non-implicit node on @device is replaced)
- Move that skippage of implicit nodes to blockdev_mirror_common() (from qmp_drive_mirror() and qmp_blockdev_mirror())
- Function name changes
- Rename s/source/target_backing_bs/ in qmp_drive_mirror(), because that’s better
- Don’t disallow mirroring through filters with sync=top
- Patch 34:
- Function name changes
- There is backup-top.c to care about, too, now
- Patch 35:
- Function name changes
- Call bdrv_commit() even for nodes that do not have backing files so we get an error
- s/above_base/base_overlay/
- Patch 36:
- Function name changes
- Patch 37:
- Function name changes
- In img_convert(), when inquiring target_backing_sectors, use bdrv_backing_chain_next() instead of bdrv_cow_bs() (because @out_bs may be a filter)
- Forgot to use the backing file of @unfiltered_bs for bdrv_is_allocated_above() in img_rebase() (instead of @unfiltered_bs itself), fixed
- Patch 39:
- Function name changes
- Patch 40:
- Function name changes
- There are backup-top and filter-compress now
- Patch 41:
- Make bdrv_backing_overridden() globally available, so block/qapi.c can use it to determine whether we can inquire bs->backing’s format to get the backing_format
- Function name changes
- Patch 42: New
- Patch 43:
- Rebase conflicts
- Patch 44:
- Create a dedicated do_test_io() function
- Don’t unnecessarily clear and pass has_quit
- Drop assert_no_active_block_jobs() that does very little
- Additional graph constraint check
- Rebase conflict in the reference output
- Patch 45:
- Rebase conflict in the reference output
- Patch 46:
- Use _rm_test_img rather than rm -f
- Skip one of the test cases when IMGOPTS asks for a data_file
- Patch 47:
- Rebase conflict in the reference output
Patches removed:
- The whole check_to_replace_node() stuff for mirror (was its own series)
- Making bdrv_get_cumulative_perm() public, because it already was
- bdrv_storage_child() (was replaced by child roles)
git-backport-diff against v6:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/47:[0126] [FC] 'block: Add child access functions' 002/47:[0032] [FC] 'block: Add chain helper functions' 003/47:[down] 'block: bdrv_cow_child() for bdrv_has_zero_init()' 004/47:[----] [-C] 'block: bdrv_set_backing_hd() is about bs->backing' 005/47:[0067] [FC] 'block: Include filters when freezing backing chain' 006/47:[----] [--] 'block: Drop bdrv_is_encrypted()' 007/47:[0005] [FC] 'block: Add bdrv_supports_compressed_writes()' 008/47:[----] [-C] 'throttle: Support compressed writes' 009/47:[----] [--] 'copy-on-read: Support compressed writes' 010/47:[down] 'mirror-top: Support compressed writes' 011/47:[down] 'backup-top: Support compressed writes' 012/47:[0036] [FC] 'block: Use bdrv_filtered_rw* where obvious' 013/47:[0010] [FC] 'block: Use CAFs in block status functions' 014/47:[0079] [FC] 'stream: Deal with filters'
015/47:[0015] [FC] 'block: Use CAFs when working with backing chains' 016/47:[down] 'block: Use bdrv_cow_child() in bdrv_co_truncate()' 017/47:[0019] [FC] 'block: Re-evaluate backing file handling in reopen' 018/47:[0010] [FC] 'block: Flush all children in generic code' 019/47:[down] 'vmdk: Drop vmdk_co_flush()'
020/47:[down] 'block: Iterate over children in refresh_limits' 021/47:[----] [--] 'block: Use CAFs in bdrv_refresh_filename()' 022/47:[----] [--] 'block: Use CAF in bdrv_co_rw_vmstate()' 023/47:[0094] [FC] 'block/snapshot: Fix fallback' 024/47:[0014] [FC] 'block: Use CAFs for debug breakpoints' 025/47:[down] 'block: Def. impl.s for get_allocated_file_size' 026/47:[down] 'block: Improve get_allocated_file_size's default' 027/47:[down] 'blkverify: Use bdrv_sum_allocated_file_size()' 028/47:[down] 'block/null: Implement bdrv_get_allocated_file_size' 029/47:[0002] [FC] 'blockdev: Use CAF in external_snapshot_prepare()' 030/47:[down] 'block: Report data child for query-blockstats' 031/47:[0031] [FC] 'block: Use child access functions for QAPI queries' 032/47:[down] 'block-copy: Use CAF to find sync=top base' 033/47:[0086] [FC] 'mirror: Deal with filters'
034/47:[0006] [FC] 'backup: Deal with filters'
035/47:[0035] [FC] 'commit: Deal with filters'
036/47:[0002] [FC] 'nbd: Use CAF when looking for dirty bitmap' 037/47:[0017] [FC] 'qemu-img: Use child access functions' 038/47:[----] [--] 'block: Drop backing_bs()'
039/47:[0002] [FC] 'blockdev: Fix active commit choice' 040/47:[0009] [FC] 'block: Inline bdrv_co_block_status_from_*()' 041/47:[0019] [FC] 'block: Leave BDS.backing_file constant' 042/47:[down] 'iotests: Test that qcow2's data-file is flushed' 043/47:[0016] [FC] 'iotests: Let complete_and_wait() work with commit' 044/47:[0042] [FC] 'iotests: Add filter commit test cases' 045/47:[0008] [FC] 'iotests: Add filter mirror test cases' 046/47:[0018] [FC] 'iotests: Add test for commit in sub directory' 047/47:[0008] [FC] 'iotests: Test committing to overridden backing'
Max Reitz (47):
block: Add child access functions
block: Add chain helper functions
block: bdrv_cow_child() for bdrv_has_zero_init()
block: bdrv_set_backing_hd() is about bs->backing
block: Include filters when freezing backing chain
block: Drop bdrv_is_encrypted()
block: Add bdrv_supports_compressed_writes()
throttle: Support compressed writes
copy-on-read: Support compressed writes
mirror-top: Support compressed writes
backup-top: Support compressed writes
block: Use bdrv_filter_(bs|child) where obvious
block: Use CAFs in block status functions
stream: Deal with filters
block: Use CAFs when working with backing chains
block: Use bdrv_cow_child() in bdrv_co_truncate()
block: Re-evaluate backing file handling in reopen
block: Flush all children in generic code
vmdk: Drop vmdk_co_flush()
block: Iterate over children in refresh_limits
block: Use CAFs in bdrv_refresh_filename()
block: Use CAF in bdrv_co_rw_vmstate() block/snapshot: Fix fallback
block: Use CAFs for debug breakpoints
block: Def. impl.s for get_allocated_file_size
block: Improve get_allocated_file_size's default
blkverify: Use bdrv_sum_allocated_file_size() block/null: Implement bdrv_get_allocated_file_size
blockdev: Use CAF in external_snapshot_prepare()
block: Report data child for query-blockstats
block: Use child access functions for QAPI queries
block-copy: Use CAF to find sync=top base
mirror: Deal with filters
backup: Deal with filters
commit: Deal with filters
nbd: Use CAF when looking for dirty bitmap
qemu-img: Use child access functions
block: Drop backing_bs()
blockdev: Fix active commit choice
block: Inline bdrv_co_block_status_from_*()
block: Leave BDS.backing_file constant
iotests: Test that qcow2's data-file is flushed
iotests: Let complete_and_wait() work with commit
iotests: Add filter commit test cases
iotests: Add filter mirror test cases
iotests: Add test for commit in sub directory
iotests: Test committing to overridden backing
qapi/block-core.json | 10 +- include/block/block.h | 2 +- include/block/block_int.h | 99 ++++--- block.c | 500 +++++++++++++++++++++++++++------ block/backup-top.c | 14 +- block/backup.c | 9 +- block/blkdebug.c | 7 +- block/blklogwrites.c | 1 - block/blkverify.c | 1 + block/block-backend.c | 9 +- block/block-copy.c | 4 +- block/commit.c | 97 +++++-- block/copy-on-read.c | 13 +- block/filter-compress.c | 2 - block/io.c | 142 +++++----- block/mirror.c | 129 +++++++-- block/monitor/block-hmp-cmds.c | 2 +- block/null.c | 7 + block/qapi.c | 83 ++++-- block/snapshot.c | 104 +++++-- block/stream.c | 63 +++-- block/throttle.c | 11 +- block/vmdk.c | 16 -- blockdev.c | 94 +++++-- migration/block-dirty-bitmap.c | 8 +- nbd/server.c | 6 +- qemu-img.c | 36 ++- tests/qemu-iotests/020 | 44 +++ tests/qemu-iotests/020.out | 10 + tests/qemu-iotests/040 | 238 ++++++++++++++++ tests/qemu-iotests/040.out | 4 +- tests/qemu-iotests/041 | 146 +++++++++- tests/qemu-iotests/041.out | 4 +- tests/qemu-iotests/153.out | 2 +- tests/qemu-iotests/184.out | 14 +- tests/qemu-iotests/204.out | 1 + tests/qemu-iotests/228 | 6 +- tests/qemu-iotests/228.out | 6 +- tests/qemu-iotests/244 | 49 ++++ tests/qemu-iotests/244.out | 7 + tests/qemu-iotests/245 | 4 +- tests/qemu-iotests/iotests.py | 10 +- 42 files changed, 1602 insertions(+), 412 deletions(-)
Signature · 1 line
2.26.2
PATCH v7 1/47 block: Add child access functions
There are BDS children that the general block layer code can access, namely bs->file and bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear. bs->backing can be a COW source, or it can be a filtered child; bs->file can be a filtered child, it can be data and metadata storage, or it can be just metadata storage.
This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
--- include/block/block_int.h | 44 +++++++++++++++++-- block.c | 90 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+), 3 deletions(-)
include/block/block_int.h +41 -3
@@ -90,9 +90,17 @@ struct BlockDriver {int instance_size;/* set to true if the BlockDriver is a block filter. Block filters pass -* certain callbacks that refer to data (see block.c) to their bs->file if -* the driver doesn't implement them. Drivers that do not wish to forward -* must implement them and return -ENOTSUP. +* certain callbacks that refer to data (see block.c) to their bs->file +* or bs->backing (whichever one exists) if the driver doesn't implement +* them. Drivers that do not wish to forward must implement them and return +* -ENOTSUP. +* Note that filters are not allowed to modify data. +* +* Filters generally cannot have more than a single filtered child, +* because the data they present must at all times be the same as +* that on their filtered child. That would be impossible to +* achieve for multiple filtered children. +* (And this filtered child must then be bs->file or bs->backing.) */ bool is_filter;/*
@@ -1370,4 +1378,34 @@ BdrvDirtyBitmap *block_dirty_bitmap_remove(const char *node, const char *name, BlockDriverState **bitmap_bs, Error **errp); +BdrvChild *bdrv_cow_child (BlockDriverState *bs); +BdrvChild *bdrv_filter_child (BlockDriverState *bs); +BdrvChild *bdrv_filter_or_cow_child (BlockDriverState *bs); +BdrvChild *bdrv_primary_child (BlockDriverState *bs); + +static inline BlockDriverState *child_bs (BdrvChild *child) +{ +return child ? child->bs :NULL ; +} + +static inline BlockDriverState *bdrv_cow_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_cow_child (bs)); +} + +static inline BlockDriverState *bdrv_filter_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_filter_child (bs)); +} + +static inline BlockDriverState *bdrv_filter_or_cow_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_filter_or_cow_child (bs)); +} + +static inline BlockDriverState *bdrv_primary_bs (BlockDriverState *bs) +{ +return child_bs (bdrv_primary_child (bs)); +} +#endif /* BLOCK_INT_H */
block.c +90 -0
93 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v7 01/47] block: Add child access functions
Quoted 23 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be a filtered child;bs->filecan be a filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 44 +++++++++++++++++-- block.c | 90 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+), 3 deletions(-)include/block/block_int.h +0
-0
include/block/block_int.h struct BlockDriver { · line 91
Quoted 3 lines
include/block/block_int.h +0
-0@@ -90,9 +90,17 @@ struct BlockDriver {int instance_size;
Re: [PATCH v7 01/47] block: Add child access functions
In reply to Max ReitzQuoted 14 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be a filtered child;bs->filecan be a filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
Signed-off-by Max Reitz <[email protected]>---
[..]
Quoted 13 lines
(quoted diff) +12
-0+/* + * Return the primary child of this node: For filters, that is the + * filtered child. For other nodes, that is usually the child storing + * metadata. + * (A generally more helpful description is that this is (usually) the + * child that has the same filename as @bs.) + * + * Drivers do not necessarily have a primary child; for example quorum + * does not. + */ +BdrvChild *bdrv_primary_child(BlockDriverState *bs) +{
Review
Quoted 10 lines
(quoted diff) +10
-0+ BdrvChild *c; + + QLIST_FOREACH(c, &bs->children, next) { + if (c->role & BDRV_CHILD_PRIMARY) { + return c; + } + } + + return NULL; +}
Suggest squash-in to also assert that not more than one primary child:
block.c +4 -3
@@ -6998,13 +6998,14 @@ BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs)*/ BdrvChild *bdrv_primary_child (BlockDriverState *bs) { -BdrvChild *c; +BdrvChild *c, *found =NULL ; QLIST_FOREACH(c, &bs->children, next) {if (c->role &BDRV_CHILD_PRIMARY ) { -return c; +assert (!found); + found = c; } } -return NULL ; +return c; }
with or without:
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 01/47] block: Add child access functions
Quoted 52 lines · Vladimir Sementsov-Ogievskiy
Quoted 14 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileand bs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be a filtered child;bs->filecan be a filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
Signed-off-by Max Reitz <[email protected]>---
[..]
Quoted 13 lines
(quoted diff) +12
-0+/* + * Return the primary child of this node: For filters, that is the + * filtered child. For other nodes, that is usually the child storing + * metadata. + * (A generally more helpful description is that this is (usually) the + * child that has the same filename as @bs.) + * + * Drivers do not necessarily have a primary child; for example quorum + * does not. + */ +BdrvChild *bdrv_primary_child(BlockDriverState *bs) +{Review
Quoted 10 lines
(quoted diff) +10
-0+ BdrvChild *c; + + QLIST_FOREACH(c, &bs->children, next) { + if (c->role & BDRV_CHILD_PRIMARY) { + return c; + } + } + + return NULL; +}Suggest squash-in to also assert that not more than one primary child:
block.c +1
-1@@ -6998,13 +6998,14 @@ BdrvChild*bdrv_filter_or_cow_child(BlockDriverState *bs) */ BdrvChild *bdrv_primary_child(BlockDriverState *bs) { - BdrvChild *c; + BdrvChild *c, *found = NULL;
Quoted 5 lines
QLIST_FOREACH(c, &bs->children, next){
if (c->role&BDRV_CHILD_PRIMARY) {
- return c;
+ assert(!found);
Hm, why not.
Quoted 11 lines
+ found = c;
}
}
- return NULL;
+ return c;
}with or without:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
In any case, thanks for reviewing!
Max
Re: [PATCH v7 01/47] block: Add child access functions
In reply to Vladimir Sementsov-OgievskiyOn 2020-07-13 at 11:06 CEST, Vladimir Sementsov-Ogievskiy wrote...
Quoted 52 lines
Quoted 14 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be a filtered child;bs->filecan be a filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
Signed-off-by Max Reitz <[email protected]>---
[..]
Quoted 13 lines
(quoted diff) +12
-0+/* + * Return the primary child of this node: For filters, that is the + * filtered child. For other nodes, that is usually the child storing + * metadata. + * (A generally more helpful description is that this is (usually) the + * child that has the same filename as @bs.) + * + * Drivers do not necessarily have a primary child; for example quorum + * does not. + */ +BdrvChild *bdrv_primary_child(BlockDriverState *bs) +{Review
Quoted 10 lines
(quoted diff) +10
-0+ BdrvChild *c; + + QLIST_FOREACH(c, &bs->children, next) { + if (c->role & BDRV_CHILD_PRIMARY) { + return c; + } + } + + return NULL; +}Suggest squash-in to also assert that not more than one primary child:
block.c +1
-1@@ -6998,13 +6998,14 @@ BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs)*/ BdrvChild *bdrv_primary_child (BlockDriverState *bs) { -BdrvChild *c; +BdrvChild *c, *found =NULL ;
block.c BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs) · line 7010
Quoted 10 lines
block.c +3
-2QLIST_FOREACH(c, &bs->children, next) {if (c->role &BDRV_CHILD_PRIMARY ) { -return c; +assert (!found); + found = c; } } -return NULL ; +return c;
Shouldn't that be "return found"?
Quoted 5 lines
block.c +0
-0}with or without:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>
--
Cheers,
Christophe de Dinechin (IRC c3d)
Re: [PATCH v7 01/47] block: Add child access functions
Quoted 62 lines · Christophe de Dinechin
On 2020-07-13 at 11:06 CEST, Vladimir Sementsov-Ogievskiy wrote...
Quoted 60 lines
Quoted 14 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be a filtered child;bs->filecan be a filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
Signed-off-by Max Reitz <[email protected]>---
[..]
Quoted 13 lines
(quoted diff) +12
-0+/* + * Return the primary child of this node: For filters, that is the + * filtered child. For other nodes, that is usually the child storing + * metadata. + * (A generally more helpful description is that this is (usually) the + * child that has the same filename as @bs.) + * + * Drivers do not necessarily have a primary child; for example quorum + * does not. + */ +BdrvChild *bdrv_primary_child(BlockDriverState *bs) +{Review
Quoted 10 lines
(quoted diff) +10
-0+ BdrvChild *c; + + QLIST_FOREACH(c, &bs->children, next) { + if (c->role & BDRV_CHILD_PRIMARY) { + return c; + } + } + + return NULL; +}Suggest squash-in to also assert that not more than one primary child:
block.c +3
-2@@ -6998,13 +6998,14 @@ BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs)*/ BdrvChild *bdrv_primary_child (BlockDriverState *bs) { -BdrvChild *c; +BdrvChild *c, *found =NULL ; QLIST_FOREACH(c, &bs->children, next) {if (c->role &BDRV_CHILD_PRIMARY ) { -return c; +assert (!found); + found = c; } }
Quoted 4 lines
block.c BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs) · line 7010
Quoted 2 lines
block.c +1
-1-return NULL ; +return c;Shouldn't that be "return found"?
Oops, you are right of course!
Quoted 11 lines
Quoted 5 lines
block.c +0
-0}with or without:
Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--
Cheers,
Christophe de Dinechin (IRC c3d)
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 01/47] block: Add child access functions
In reply to Max ReitzQuoted 22 lines · Max Reitz
There are BDS children that the general block layer code can access, namely
bs->fileandbs->backing. Since the introduction of filters and external data files, their meaning is not quite clear.bs->backingcan be a COW source, or it can be a filtered child;bs->filecan be a filtered child, it can be data and metadata storage, or it can be just metadata storage.This overloading really is not helpful. This patch adds functions that retrieve the correct child for each exact purpose. Later patches in this series will make use of them. Doing so will allow us to handle filter nodes in a meaningful way.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 44 +++++++++++++++++-- block.c | 90 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+), 3 deletions(-)include/block/block_int.h +0
-0
[..]
include/block/block_int.h
Quoted 5 lines
include/block/block_int.h +5
-0+/* +* If @bs acts as a filter for exactly one of its children, return +* that child. +*/ +BdrvChild *bdrv_filter_child (BlockDriverState *bs)
Hmm you called it filter_child instead of filterED_child..
Quoted 10 lines
include/block/block_int.h +10
-0+{ +BdrvChild *c; + +if (!bs || !bs->drv) { +return NULL ; + } + +if (!bs->drv->is_filter) { +return NULL ; + }
include/block/block_int.h
Quoted 10 lines
include/block/block_int.h +10
-0+ +/* Only one of @backing or @file may be used */ +assert (!(bs->backing && bs->file)); + + c = bs->backing ?: bs->file; +if (!c) { +return NULL ; + } + +assert (c->role &BDRV_CHILD_FILTERED );
But the role is still called CHILD_FILTERED
include/block/block_int.h
Quoted 2 lines
include/block/block_int.h +2
-0+return c; +}
(just note that it's a bit inconsistent, keep my r-b anyway)
Signature · 2 lines
Best regards, Vladimir
PATCH v7 2/47 block: Add chain helper functions
In reply to Max ReitzAdd some helper functions for skipping filters in a chain of block nodes.
--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)
include/block/block_int.h +3 -0
@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs);BdrvChild *bdrv_filter_child (BlockDriverState *bs);BdrvChild *bdrv_filter_or_cow_child (BlockDriverState *bs);BdrvChild *bdrv_primary_child (BlockDriverState *bs); +BlockDriverState *bdrv_skip_implicit_filters (BlockDriverState *bs); +BlockDriverState *bdrv_skip_filters (BlockDriverState *bs); +BlockDriverState *bdrv_backing_chain_next (BlockDriverState *bs);static inline BlockDriverState *child_bs (BdrvChild *child) {
block.c +55 -0
58 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v7 02/47] block: Add chain helper functions
Quoted 37 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)include/block/block_int.h +3
-0@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs);BdrvChild *bdrv_filter_child (BlockDriverState *bs);BdrvChild *bdrv_filter_or_cow_child (BlockDriverState *bs);BdrvChild *bdrv_primary_child (BlockDriverState *bs); +BlockDriverState *bdrv_skip_implicit_filters (BlockDriverState *bs); +BlockDriverState *bdrv_skip_filters (BlockDriverState *bs); +BlockDriverState *bdrv_backing_chain_next (BlockDriverState *bs);static inline BlockDriverState *child_bs (BdrvChild *child) {block.c +6
-0@@ -7008,3 +7008,58 @@ BdrvChild *bdrv_primary_child(BlockDriverState *bs)return NULL ; } + +static BlockDriverState *bdrv_do_skip_filters (BlockDriverState *bs, +bool stop_on_explicit_filter) +{ +BdrvChild *c; +
block.c BdrvChild *bdrv_primary_child(BlockDriverState *bs) · line 7026
Quoted 10 lines
block.c +10
-0+if (!bs) { +return NULL ; + } + +while (!(stop_on_explicit_filter && !bs->implicit)) { + c =bdrv_filter_child (bs); +if (!c) { +break ; + } + bs = c->bs;
Could it be child_bs(bs) ?
Andrey
block.c BdrvChild *bdrv_primary_child(BlockDriverState *bs) · line 7027
Quoted 1 line
block.c +1
-0+ }
Re: [PATCH v7 02/47] block: Add chain helper functions
Quoted 42 lines · Andrey Shinkevich
Quoted 41 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>---
include/block/block_int.h| 3 +++
block.c | 55 +++++++++++++++++++++++++++++++++++++++2 files changed, 58 insertions(+)include/block/block_int.h +3
-0@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs); BdrvChild *bdrv_filter_child(BlockDriverState *bs); BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs); BdrvChild *bdrv_primary_child(BlockDriverState *bs); +BlockDriverState *bdrv_skip_implicit_filters (BlockDriverState *bs); +BlockDriverState *bdrv_skip_filters (BlockDriverState *bs); +BlockDriverState *bdrv_backing_chain_next (BlockDriverState *bs); static inline BlockDriverState *child_bs(BdrvChild *child) {block.c +3
-0@@ -7008,3 +7008,58 @@ BdrvChild *bdrv_primary_child(BlockDriverState *bs) return NULL; } + +static BlockDriverState *bdrv_do_skip_filters (BlockDriverState *bs, + bool
stop_on_explicit_filter)block.c +7
-0+{ + BdrvChild *c; + + if (!bs) { + return NULL; + } +
Quoted 8 lines
Quoted 6 lines
+ while (!(stop_on_explicit_filter && !bs->implicit)) { + c =
bdrv_filter_child(bs);
+ if (!c) {
+ break;
+ }
+ bs =c->bs;Could it be
child_bs(bs)?
Well, in a sense, but not really. We need to check whether there is a child before overwriting @bs (because @bs must stay a non-NULL pointer), so we wouldn’t have fewer lines of code if we replaced “BdrvChild *c” by “BlockDriverState *child_bs”, and then used bdrv_child() to set child_bs.
(And because we have to check whether @c is NULL anyway, there is no real reason to use child_bs(c) instead of c->bs afterwards.)
Quoted 2 lines
Quoted 1 line
+ }
Reviewed-by Andrey Shinkevich <[email protected]>
Thanks a lot for reviewing!
Re: [PATCH v7 02/47] block: Add chain helper functions
Quoted 50 lines · Max Reitz
Quoted 41 lines · Andrey Shinkevich
Quoted 40 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)include/block/block_int.h +3
-0@@ -1382,6 +1382,9 @@ BdrvChild *bdrv_cow_child(BlockDriverState *bs); BdrvChild *bdrv_filter_child(BlockDriverState *bs); BdrvChild *bdrv_filter_or_cow_child(BlockDriverState *bs); BdrvChild *bdrv_primary_child(BlockDriverState *bs); +BlockDriverState *bdrv_skip_implicit_filters (BlockDriverState *bs); +BlockDriverState *bdrv_skip_filters (BlockDriverState *bs); +BlockDriverState *bdrv_backing_chain_next (BlockDriverState *bs); static inline BlockDriverState *child_bs(BdrvChild *child) {block.c +3
-0@@ -7008,3 +7008,58 @@ BdrvChild *bdrv_primary_child(BlockDriverState *bs) return NULL; } + +static BlockDriverState *bdrv_do_skip_filters (BlockDriverState *bs, + bool
stop_on_explicit_filter)block.c +6
-0+{ + BdrvChild *c; + + if (!bs) { + return NULL; + }Quoted 8 lines
Quoted 7 lines
+
+ while (!(stop_on_explicit_filter && !bs->implicit)) { + c =bdrv_filter_child(bs);
+ if (!c) {
+ break;
+ }
+ bs =c->bs;Could it be
child_bs(bs)?
Quoted 7 lines
Well, in a sense, but not really. We need to check whether there is a child before overwriting @bs (because @bs must stay a non-NULL pointer), so we wouldn’t have fewer lines of code if we replaced “BdrvChild *c” by “BlockDriverState *child_bs”, and then used
bdrv_child()to setchild_bs.(And because we have to check whether @c is NULL anyway, there is no real reason to use
child_bs(c)instead ofc->bsafterwards.)
Got it, thanks.
Andrey
Quoted 3 lines
Quoted 2 lines
Quoted 1 line
+ }
Reviewed-by Andrey Shinkevich <[email protected]>Thanks a lot for reviewing!
Pleasure!
Andrey
Re: [PATCH v7 02/47] block: Add chain helper functions
In reply to Max Reitzinclude/block/block_int.h
Quoted 11 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)include/block/block_int.h +0
-0
This patch raises two questions:
1. How to treat filters at the end of the backing chain?
- child-access function will return no filter child for such nodes, it's correct of course
- filer skipping functions will return this filter.. How much is it correct - I don't know.
Consider a chain
top --- backing ---> filter-with-no-child
if bdrv_backing_chain_next(top) returns NULL, it's incorrect, because top actually have backing, and on read it will read from it for unallocated clusters (and this should crash). So, probably, returning filter as a backing-chain-next is a valid thing to do. Or we should assert that we are not in such situation (which may crash more often than trying to really read from nonexistent child).
so, returning NULL, may even less correct than returning a filter..
2. How to tread nodes with drv=NULL, but with filter child (with BDRV_CHILD_FILTERED role).
- child-access functions returns no filtered child for such nodes
- filter skipping functions will stop on it..
=======
Isn't it better to drop drv->is_filter at all? And call filter nodes with a bs->file or bs->backing child in BDRV_CHILD_FILTERED role? This automatically closes the two questions:
- node without a child in BDRV_CHILD_FILTERED is automatically non-filter. So, filter driver is responsible for having such child.
- node without a drv may still be a filter if it have BDRV_CHILD_FILTERED.. Still, not very useful.
Anyway, is_filter and BDRV_CHILD_FILTERED are in contradiction, and it seems good to get rid of is_filter. But I may miss something.
[..]
Quoted 13 lines
include/block/block_int.h +13
-0+ +static BlockDriverState *bdrv_do_skip_filters (BlockDriverState *bs, +bool stop_on_explicit_filter) +{ +BdrvChild *c; + +if (!bs) { +return NULL ; + } + +while (!(stop_on_explicit_filter && !bs->implicit)) { + c =bdrv_filter_child (bs); +if (!c) {
include/block/block_int.h
Quoted 10 lines
include/block/block_int.h +10
-0+break ; + } + bs = c->bs; + } +/* +* Note that this treats nodes with bs->drv == NULL as not being +* filters (bs->drv == NULL should be replaced by something else +* anyway). +* The advantage of this behavior is that this function will thus +* always return a non-NULL value (given a non-NULL @bs).
I don't see, how it is follows from first sentence? We can skip nodes with a child of BDRV_CHILD_FILTERED and drv=NULL as well, and still return non-NULL bs at the end...
Didn't you mean "treat nodes without filter child as not being filters, even if they have drv->is_filter == true"? This is a real reason for the second sentence.
... and the disadvantage is that we may return filter node, which may be not expected by caller ...
include/block/block_int.h
Quoted 4 lines
include/block/block_int.h +4
-0+*/ + +return bs; +}
I think, I can live with it as is for now. If we don't drop is_filter, I think it worth documenting these corner cases somewhere (may be near .is_filter definition).
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 02/47] block: Add chain helper functions
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>---
include/block/block_int.h| 3 +++
block.c | 55 +++++++++++++++++++++++++++++++++++++++2 files changed, 58 insertions(+)
Quoted 7 lines
include/block/block_int.h
Quoted 2 lines
include/block/block_int.h +0
-0This patch raises two questions:
1. How to treat filters at the end of the backing chain?
It was my understanding that this configuration would be impossible.
Quoted 9 lines
- child-access function will return no filter child for such nodes, it's correct of course
- filer skipping functions will return this filter.. How much is it correct - I don't know.Consider a chain
top --- backing ---> filter-with-no-child
How would it be possible to have filter without a child?
Quoted 7 lines
if
bdrv_backing_chain_next(top)returns NULL, it's incorrect, because top actually have backing, and on read it will read from it for unallocated clusters (and this should crash). So, probably, returning filter as a backing-chain-next is a valid thing to do. Or we should assert that we are not in such situation (which may crash more often than trying to really read from nonexistent child).
Quoted 5 lines
so, returning NULL, may even less correct than returning a filter..
2. How to tread nodes with drv=NULL, but with filter child (with
BDRV_CHILD_FILTEREDrole).
drv=NULL is a bug on its own that should be fixed... (The idea we had at some point was to introduce a special driver that just always returns -EIO on everything, and to replace corrupt qcow2 nodes by that. Or, well, we might just return -EIO from the qcow2 driver, actually, if we never use drv=NULL anywhere else.)
In any case, drv=NULL is an edge case that I think never has anything to do with filters.
Quoted 40 lines
- child-access functions returns no filtered child for such nodes
- filter skipping functions will stop on it..=======
Isn't it better to drop
drv->is_filterat all? And call filter nodes with abs->fileorbs->backing
child inBDRV_CHILD_FILTEREDrole? This automatically closes the two questions:- node without a child in
BDRV_CHILD_FILTEREDis automatically non-filter. So, filter driver is responsible for having such child.
- node without a drv may still be a filter if it haveBDRV_CHILD_FILTERED.. Still, not very useful.Anyway,
is_filterandBDRV_CHILD_FILTEREDare in contradiction, and it seems good to get rid ofis_filter. But I may miss something.[..]
Quoted 20 lines
include/block/block_int.h +3
-0+ +static BlockDriverState *bdrv_do_skip_filters (BlockDriverState *bs, + bool
stop_on_explicit_filter)include/block/block_int.h +16
-0+{ + BdrvChild *c; + + if (!bs) { + return NULL; + } + + while (!(stop_on_explicit_filter && !bs->implicit)) { + c = bdrv_filter_child(bs); + if (!c) { + break; + } + bs = c->bs; + } + /* +* Note that this treats nodes with bs->drv == NULL as not being
Quoted 8 lines
Quoted 4 lines
+ * filters (
bs->drv== NULL should be replaced by something else + * anyway).
+ * The advantage of this behavior is that this function will thus + * always return a non-NULL value (given a non-NULL @bs).I don't see, how it is follows from first sentence? We can skip nodes with a child of
BDRV_CHILD_FILTEREDand drv=NULL as well, and still return non-NULL bs at the end...
My idea was that nodes with bs->drv == NULL might not even have children. If we treated them like filters and skipped through them, we would have to return NULL if there is no child.
Quoted 3 lines
Didn't you mean "treat nodes without filter child as not being filters, even if they have
drv->is_filter== true"? This is a real reason for the second sentence.
Hm. I implicitly always assume that filters always have a filter child, so I tend to not even question that part.
Max
Vladimir Sementsov-Ogievskiy
Reviewed-byRe: [PATCH v7 02/47] block: Add chain helper functions
Quoted 17 lines · Max Reitz
Quoted 16 lines · Vladimir Sementsov-Ogievskiy
include/block/block_int.h
Quoted 11 lines · Max Reitz
Add some helper functions for skipping filters in a chain of block nodes.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 3 +++ block.c | 55 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 58 insertions(+)include/block/block_int.h +0
-0This patch raises two questions:
Quoted 3 lines
Quoted 1 line
1. How to treat filters at the end of the backing chain?
It was my understanding that this configuration would be impossible.
OK for me, but I'd prefer to have a comment and assertions.
Quoted 26 lines
Quoted 10 lines
- child-access function will return no filter child for such nodes, it's correct of course
- filer skipping functions will return this filter.. How much is it correct - I don't know.Consider a chain
top --- backing ---> filter-with-no-child
How would it be possible to have filter without a child?
Quoted 7 lines
if
bdrv_backing_chain_next(top)returns NULL, it's incorrect, because top actually have backing, and on read it will read from it for unallocated clusters (and this should crash). So, probably, returning filter as a backing-chain-next is a valid thing to do. Or we should assert that we are not in such situation (which may crash more often than trying to really read from nonexistent child).Quoted 5 lines
so, returning NULL, may even less correct than returning a filter..
2. How to tread nodes with drv=NULL, but with filter child (with
BDRV_CHILD_FILTEREDrole).
Quoted 8 lines
drv=NULL is a bug on its own that should be fixed... (The idea we had at some point was to introduce a special driver that just always returns
-EIOon everything, and to replace corrupt qcow2 nodes by that. Or, well, we might just return-EIOfrom the qcow2 driver, actually, if we never use drv=NULL anywhere else.)In any case, drv=NULL is an edge case that I think never has anything to do with filters.
So, again some good comment and assertion won't hurt.
Quoted 54 lines
Quoted 41 lines
- child-access functions returns no filtered child for such nodes
- filter skipping functions will stop on it..=======
Isn't it better to drop
drv->is_filterat all? And call filter nodes with abs->fileorbs->backing
child inBDRV_CHILD_FILTEREDrole? This automatically closes the two questions:- node without a child in
BDRV_CHILD_FILTEREDis automatically non-filter. So, filter driver is responsible for having such child.
- node without a drv may still be a filter if it haveBDRV_CHILD_FILTERED.. Still, not very useful.Anyway,
is_filterandBDRV_CHILD_FILTEREDare in contradiction, and it seems good to get rid ofis_filter. But I may miss something.[..]
Quoted 20 lines
include/block/block_int.h +3
-0+ +static BlockDriverState *bdrv_do_skip_filters (BlockDriverState *bs, + bool
stop_on_explicit_filter)include/block/block_int.h +16
-0+{ + BdrvChild *c; + + if (!bs) { + return NULL; + } + + while (!(stop_on_explicit_filter && !bs->implicit)) { + c = bdrv_filter_child(bs); + if (!c) { + break; + } + bs = c->bs; + } + /* +* Note that this treats nodes with bs->drv == NULL as not being Quoted 8 lines
Quoted 4 lines
+ * filters (
bs->drv== NULL should be replaced by something else + * anyway).
+ * The advantage of this behavior is that this function will thus + * always return a non-NULL value (given a non-NULL @bs).I don't see, how it is follows from first sentence? We can skip nodes with a child of
BDRV_CHILD_FILTEREDand drv=NULL as well, and still return non-NULL bs at the end...My idea was that nodes with
bs->drv== NULL might not even have children. If we treated them like filters and skipped through them, we would have to return NULL if there is no child.
Quoted 7 lines
Quoted 3 lines
Didn't you mean "treat nodes without filter child as not being filters, even if they have
drv->is_filter== true"? This is a real reason for the second sentence.Hm. I implicitly always assume that filters always have a filter child, so I tend to not even question that part.
Hmm. Still, the relationship in the comment seems unclear to me, the code itself is simpler :)
Okay, I'm actually OK with this all. I'd prefer to have assertions and comments on corner-cases I mentioned, but patch is OK as is:
Signature · 2 lines
Best regards, Vladimir
PATCH v7 3/47 block: bdrv_cow_child() for bdrv_has_zero_init()
In reply to Max Reitzbdrv_has_zero_init() and the related bdrv_unallocated_blocks_are_zero() should use bdrv_cow_child() if they want to check whether the given BDS has a COW backing file.
--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
@@ -5394,7 +5394,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)/* If BS is a copy on write image, it is initialized to the contents of the base image, which may not be zeroes. */ -if (bs->backing) { +if (bdrv_cow_child (bs)) {return 0 ; }if (bs->drv->bdrv_has_zero_init) {
@@ -5412,7 +5412,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs) {BlockDriverInfo bdi; -if (bs->backing) { +if (bdrv_cow_child (bs)) {return false; }
Signature · 1 line
2.26.2
Re: [PATCH v7 03/47] block: bdrv_cow_child() for bdrv_has_zero_init()
Quoted 23 lines · Max Reitz
bdrv_has_zero_init()and the relatedbdrv_unallocated_blocks_are_zero()should usebdrv_cow_child()if they want to check whether the given BDS has a COW backing file.Signed-off-by Max Reitz <[email protected]>--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)block.c +1
-1@@ -5394,7 +5394,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)/* If BS is a copy on write image, it is initialized to the contents of the base image, which may not be zeroes. */ -if (bs->backing) { +if (bdrv_cow_child (bs)) {return 0 ; }if (bs->drv->bdrv_has_zero_init) {
block.c bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs) · line 5418
Quoted 8 lines
block.c +1
-1@@ -5412,7 +5412,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs) {BlockDriverInfo bdi; -if (bs->backing) { +if (bdrv_cow_child (bs)) {return false; }
Re: [PATCH v7 03/47] block: bdrv_cow_child() for bdrv_has_zero_init()
In reply to Max Reitzblock.c bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)
Quoted 23 lines · Max Reitz
bdrv_has_zero_init()and the relatedbdrv_unallocated_blocks_are_zero()should usebdrv_cow_child()if they want to check whether the given BDS has a COW backing file.Signed-off-by Max Reitz <[email protected]>--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)block.c +1
-1@@ -5394,7 +5394,7 @@ int bdrv_has_zero_init(BlockDriverState *bs)/* If BS is a copy on write image, it is initialized to the contents of the base image, which may not be zeroes. */ -if (bs->backing) { +if (bdrv_cow_child (bs)) {return 0 ; }if (bs->drv->bdrv_has_zero_init) {@@ -5412,7 +5412,7 @@ bool bdrv_unallocated_blocks_are_zero(BlockDriverState *bs)
the function is removed, so, this chunk should be dropped or rebased..
Quoted 9 lines
block.c +1
-1{BlockDriverInfo bdi; -if (bs->backing) { +if (bdrv_cow_child (bs)) {return false; }
Signature · 2 lines
Best regards, Vladimir
PATCH v7 4/47 block: bdrv_set_backing_hd() is about bs->backing
In reply to Max Reitzbdrv_set_backing_hd() is a function that explicitly cares about the bs->backing child. Highlight that in its description and use child_bs(bs->backing) instead of backing_bs(bs) to make it more obvious.
--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block.c +2 -2
@@ -2846,7 +2846,7 @@ static BdrvChildRole bdrv_backing_role(BlockDriverState *bs) }/* -* Sets the backing file link of a BDS. A new reference is created; callers +* Sets the bs->backing link of a BDS. A new reference is created; callers * which don't need their own reference any more must call bdrv_unref(). */ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
@@ -2855,7 +2855,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,bool update_inherits_from =bdrv_chain_contains (bs, backing_hd) &&bdrv_inherits_from_recursive (backing_hd, bs); -if (bdrv_is_backing_chain_frozen (bs,backing_bs (bs), errp)) { +if (bdrv_is_backing_chain_frozen (bs,child_bs (bs->backing), errp)) {return ; }
Signature · 1 line
2.26.2
Re: [PATCH v7 04/47] block: bdrv_set_backing_hd() is about bs->backing
Quoted 24 lines · Max Reitz
bdrv_set_backing_hd()is a function that explicitly cares about thebs->backingchild. Highlight that in its description and usechild_bs(bs->backing)instead ofbacking_bs(bs)to make it more obvious.Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)block.c +1
-1@@ -2846,7 +2846,7 @@ static BdrvChildRole bdrv_backing_role(BlockDriverState *bs) }/* -* Sets the backing file link of a BDS. A new reference is created; callers +* Sets the bs->backing link of a BDS. A new reference is created; callers * which don't need their own reference any more must call bdrv_unref(). */ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,
block.c void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd, · line 2861
Quoted 8 lines
block.c +1
-1@@ -2855,7 +2855,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd,bool update_inherits_from =bdrv_chain_contains (bs, backing_hd) &&bdrv_inherits_from_recursive (backing_hd, bs); -if (bdrv_is_backing_chain_frozen (bs,backing_bs (bs), errp)) { +if (bdrv_is_backing_chain_frozen (bs,child_bs (bs->backing), errp)) {return ; }
PATCH v7 5/47 block: Include filters when freezing backing chain
In reply to Max ReitzIn order to make filters work in backing chains, the associated functions must be able to deal with them and freeze both COW and filter child links.
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
--- block.c | 60 +++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 21 deletions(-)
block.c +39 -21
@@ -2595,12 +2595,15 @@ static void bdrv_replace_child_noperm(BdrvChild *child,* If @new_bs is not NULL, bdrv_check_perm() must be called beforehand, as this * function uses bdrv_set_perm() to update the permissions according to the new * reference that @new_bs gets. +* +* Callers must ensure that child->frozen is false. */ static void bdrv_replace_child (BdrvChild *child,BlockDriverState *new_bs) {BlockDriverState *old_bs = child->bs;uint64_t perm, shared_perm; +/* Asserts that child->frozen == false */ bdrv_replace_child_noperm (child, new_bs);/*
@@ -2761,6 +2764,7 @@ static void bdrv_detach_child(BdrvChild *child)g_free (child); } +/* Callers must ensure that child->frozen is false. */ void bdrv_root_unref_child (BdrvChild *child) {BlockDriverState *child_bs;
@@ -2798,6 +2802,7 @@ static void bdrv_unset_inherits_from(BlockDriverState *root, BdrvChild *child) } } +/* Callers must ensure that child->frozen is false. */ void bdrv_unref_child (BlockDriverState *parent,BdrvChild *child) {if (child ==NULL ) {
@@ -2864,6 +2869,7 @@ void bdrv_set_backing_hd(BlockDriverState *bs, BlockDriverState *backing_hd, }if (bs->backing) { +/* Cannot be frozen, we checked that above */ bdrv_unref_child (bs, bs->backing); bs->backing =NULL ; }
@@ -4372,6 +4378,7 @@ static void bdrv_close(BlockDriverState *bs)if (bs->drv) {if (bs->drv->bdrv_close) { +/* Must unfreeze all children, so bdrv_unref_child() works */ bs->drv->bdrv_close (bs); } bs->drv =NULL ;
@@ -4741,20 +4748,22 @@ BlockDriverState *bdrv_find_base(BlockDriverState *bs) }/* -* Return true if at least one of the backing links between @bs and -* @base is frozen. @errp is set if that's the case. +* Return true if at least one of the COW (backing) and filter links +* between @bs and @base is frozen. @errp is set if that's the case. * @base must be reachable from @bs, or NULL. */ bool bdrv_is_backing_chain_frozen (BlockDriverState *bs,BlockDriverState *base,Error **errp) {BlockDriverState *i; +BdrvChild *child; -for (i = bs; i != base; i =backing_bs (i)) { -if (i->backing && i->backing->frozen) { +for (i = bs; i != base; i =child_bs (child)) { + child =bdrv_filter_or_cow_child (i); + +if (child && child->frozen) {error_setg (errp,"Cannot change '%s' link from '%s' to '%s'" , - i->backing->name, i->node_name, -backing_bs (i)->node_name); + child->name, i->node_name, child->bs->node_name);return true; } }
@@ -4763,7 +4772,7 @@ bool bdrv_is_backing_chain_frozen(BlockDriverState *bs, BlockDriverState *base, }/* -* Freeze all backing links between @bs and @base. +* Freeze all COW (backing) and filter links between @bs and @base. * If any of the links is already frozen the operation is aborted and * none of the links are modified. * @base must be reachable from @bs, or NULL.
@@ -4773,22 +4782,25 @@ int bdrv_freeze_backing_chain(BlockDriverState *bs, BlockDriverState *base, Error **errp) {BlockDriverState *i; +BdrvChild *child;if (bdrv_is_backing_chain_frozen (bs, base, errp)) {return -EPERM ; } -for (i = bs; i != base; i =backing_bs (i)) { -if (i->backing &&backing_bs (i)->never_freeze) { +for (i = bs; i != base; i =child_bs (child)) { + child =bdrv_filter_or_cow_child (i); +if (child && child->bs->never_freeze) {error_setg (errp,"Cannot freeze '%s' link to '%s'" , - i->backing->name,backing_bs (i)->node_name); + child->name, child->bs->node_name);return -EPERM ; } } -for (i = bs; i != base; i =backing_bs (i)) { -if (i->backing) { - i->backing->frozen = true; +for (i = bs; i != base; i =child_bs (child)) { + child =bdrv_filter_or_cow_child (i); +if (child) { + child->frozen = true; } }
@@ -4796,18 +4808,21 @@ int bdrv_freeze_backing_chain(BlockDriverState *bs, BlockDriverState *base, }/* -* Unfreeze all backing links between @bs and @base. The caller must -* ensure that all links are frozen before using this function. +* Unfreeze all COW (backing) and filter links between @bs and @base. +* The caller must ensure that all links are frozen before using this +* function. * @base must be reachable from @bs, or NULL. */ void bdrv_unfreeze_backing_chain (BlockDriverState *bs,BlockDriverState *base) {BlockDriverState *i; +BdrvChild *child; -for (i = bs; i != base; i =backing_bs (i)) { -if (i->backing) { -assert (i->backing->frozen); - i->backing->frozen = false; +for (i = bs; i != base; i =child_bs (child)) { + child =bdrv_filter_or_cow_child (i); +if (child) { +assert (child->frozen); + child->frozen = false; } } }
@@ -4910,8 +4925,11 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base, } } -/* Do the actual switch in the in-memory graph. -* Completes bdrv_check_update_perm() transaction internally. */ +/* +* Do the actual switch in the in-memory graph. +* Completes bdrv_check_update_perm() transaction internally. +* c->frozen is false, we have checked that above. +*/ bdrv_ref (base);bdrv_replace_child (c, base);bdrv_unref (top);
Signature · 1 line
2.26.2
Re: [PATCH v7 05/47] block: Include filters when freezing backing chain
block.c
Quoted 17 lines · Max Reitz
In order to make filters work in backing chains, the associated functions must be able to deal with them and freeze both COW and filter child links.
While at it, add some comments that note which functions require their caller to ensure that a given child link is not frozen, and how the callers do so.
Signed-off-by Max Reitz <[email protected]>--- block.c | 60 +++++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 21 deletions(-)block.c +0
-0
PATCH v7 6/47 block: Drop bdrv_is_encrypted()
In reply to Max ReitzThe original purpose of bdrv_is_encrypted() was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.
Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.
So now we have to decide in which direction we want to fix bdrv_is_encrypted(): Recursively query all children, or drop it and just use bs->encrypted to get the current node's status?
Nowadays, its only purpose is to report through bdrv_query_image_info() whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).
--- include/block/block.h | 1 - block.c | 8 -------- block/qapi.c | 2 +- 3 files changed, 1 insertion(+), 10 deletions(-)
include/block/block.h +0 -1
@@ -538,7 +538,6 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);void bdrv_next_cleanup (BdrvNextIterator *it);BlockDriverState *bdrv_next_monitor_owned (BlockDriverState *bs); -bool bdrv_is_encrypted (BlockDriverState *bs);void bdrv_iterate_format (void (*it)(void *opaque,const char *name),void *opaque,bool read_only);const char *bdrv_get_node_name (const BlockDriverState *bs);
block.c +0 -8
@@ -5044,14 +5044,6 @@ bool bdrv_is_sg(BlockDriverState *bs)return bs->sg; } -bool bdrv_is_encrypted (BlockDriverState *bs) -{ -if (bs->backing && bs->backing->bs->encrypted) { -return true; - } -return bs->encrypted; -} -const char *bdrv_get_format_name (BlockDriverState *bs) {return bs->drv ? bs->drv->format_name :NULL ;
block/qapi.c +1 -1
@@ -288,7 +288,7 @@ void bdrv_query_image_info(BlockDriverState *bs, info->virtual_size = size; info->actual_size =bdrv_get_allocated_file_size (bs); info->has_actual_size = info->actual_size >=0 ; -if (bdrv_is_encrypted (bs)) { +if (bs->encrypted) { info->encrypted = true; info->has_encrypted = true; }
Signature · 1 line
2.26.2
Re: [PATCH v7 06/47] block: Drop bdrv_is_encrypted()
Quoted 63 lines · Max Reitz
The original purpose of
bdrv_is_encrypted()was to inquire whether a BDS can be used without the user entering a password or not. It has not been used for that purpose for quite some time.Actually, it is not even fit for that purpose, because to answer that question, it would have recursively query all of the given node's children.
So now we have to decide in which direction we want to fix
bdrv_is_encrypted(): Recursively query all children, or drop it and just usebs->encryptedto get the current node's status?Nowadays, its only purpose is to report through
bdrv_query_image_info()whether the given image is encrypted or not. For this purpose, it is probably more interesting to see whether a given node itself is encrypted or not (otherwise, a management application cannot discern for certain which nodes are really encrypted and which just have encrypted children).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 1 - block.c | 8 -------- block/qapi.c | 2 +- 3 files changed, 1 insertion(+), 10 deletions(-)include/block/block.h +0
-1@@ -538,7 +538,6 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);void bdrv_next_cleanup (BdrvNextIterator *it);BlockDriverState *bdrv_next_monitor_owned (BlockDriverState *bs); -bool bdrv_is_encrypted (BlockDriverState *bs);void bdrv_iterate_format (void (*it)(void *opaque,const char *name),void *opaque,bool read_only);const char *bdrv_get_node_name (const BlockDriverState *bs);block.c +0
-8@@ -5044,14 +5044,6 @@ bool bdrv_is_sg(BlockDriverState *bs)return bs->sg; } -bool bdrv_is_encrypted (BlockDriverState *bs) -{ -if (bs->backing && bs->backing->bs->encrypted) { -return true; - } -return bs->encrypted; -} -const char *bdrv_get_format_name (BlockDriverState *bs) {return bs->drv ? bs->drv->format_name :NULL ;block/qapi.c +0
-0
block/qapi.c void bdrv_query_image_info(BlockDriverState *bs, · line 294
Quoted 8 lines
block/qapi.c +1
-1@@ -288,7 +288,7 @@ void bdrv_query_image_info(BlockDriverState *bs, info->virtual_size = size; info->actual_size =bdrv_get_allocated_file_size (bs); info->has_actual_size = info->actual_size >=0 ; -if (bdrv_is_encrypted (bs)) { +if (bs->encrypted) { info->encrypted = true; info->has_encrypted = true; }
PATCH v7 7/47 block: Add bdrv_supports_compressed_writes()
In reply to Max ReitzFilters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether bs->drv->bdrv_co_pwritev_compressed is non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).
--- include/block/block.h | 1 + block.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)
include/block/block.h +1 -0
@@ -538,6 +538,7 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);void bdrv_next_cleanup (BdrvNextIterator *it);BlockDriverState *bdrv_next_monitor_owned (BlockDriverState *bs); +bool bdrv_supports_compressed_writes (BlockDriverState *bs);void bdrv_iterate_format (void (*it)(void *opaque,const char *name),void *opaque,bool read_only);const char *bdrv_get_node_name (const BlockDriverState *bs);
block.c +23 -0
@@ -5044,6 +5044,29 @@ bool bdrv_is_sg(BlockDriverState *bs)return bs->sg; } +/** +* Return whether the given node supports compressed writes. +*/ +bool bdrv_supports_compressed_writes (BlockDriverState *bs) +{ +BlockDriverState *filtered; + +if (!bs->drv || !block_driver_can_compress (bs->drv)) { +return false; + } + + filtered =bdrv_filter_bs (bs); +if (filtered) { +/* +* Filters can only forward compressed writes, so we have to +* check the child. +*/ +return bdrv_supports_compressed_writes (filtered); + } + +return true; +} +const char *bdrv_get_format_name (BlockDriverState *bs) {return bs->drv ? bs->drv->format_name :NULL ;
Signature · 1 line
2.26.2
Re: [PATCH v7 07/47] block: Add bdrv_supports_compressed_writes()
Quoted 51 lines · Max Reitz
Filters cannot compress data themselves but they have to implement .bdrv_co_pwritev_compressed() still (or they cannot forward compressed writes). Therefore, checking whether
bs->drv->bdrv_co_pwritev_compressedis non-NULL is not sufficient to know whether the node can actually handle compressed writes. This function looks down the filter chain to see whether there is a non-filter that can actually convert the compressed writes into compressed data (and thus normal writes).Signed-off-by Max Reitz <[email protected]>--- include/block/block.h | 1 + block.c | 23 +++++++++++++++++++++++ 2 files changed, 24 insertions(+)include/block/block.h +1
-0@@ -538,6 +538,7 @@ BlockDriverState *bdrv_next(BdrvNextIterator *it);void bdrv_next_cleanup (BdrvNextIterator *it);BlockDriverState *bdrv_next_monitor_owned (BlockDriverState *bs); +bool bdrv_supports_compressed_writes (BlockDriverState *bs);void bdrv_iterate_format (void (*it)(void *opaque,const char *name),void *opaque,bool read_only);const char *bdrv_get_node_name (const BlockDriverState *bs);block.c +16
-0@@ -5044,6 +5044,29 @@ bool bdrv_is_sg(BlockDriverState *bs)return bs->sg; } +/** +* Return whether the given node supports compressed writes. +*/ +bool bdrv_supports_compressed_writes (BlockDriverState *bs) +{ +BlockDriverState *filtered; + +if (!bs->drv || !block_driver_can_compress (bs->drv)) { +return false; + } + + filtered =bdrv_filter_bs (bs); +if (filtered) { +/* +* Filters can only forward compressed writes, so we have to +* check the child.
block.c bool bdrv_is_sg(BlockDriverState *bs) · line 5072
Quoted 10 lines
block.c +7
-0+*/ +return bdrv_supports_compressed_writes (filtered); + } + +return true; +} +const char *bdrv_get_format_name (BlockDriverState *bs) {return bs->drv ? bs->drv->format_name :NULL ;
PATCH v7 8/47 throttle: Support compressed writes
In reply to Max Reitz--- block/throttle.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
block/throttle.c +10 -0
@@ -154,6 +154,15 @@ static int coroutine_fn throttle_co_pdiscard(BlockDriverState *bs,return bdrv_co_pdiscard (bs->file, offset, bytes); } +static int coroutine_fn throttle_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return throttle_co_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} +static int throttle_co_flush (BlockDriverState *bs) {return bdrv_co_flush (bs->file->bs);
@@ -246,6 +255,7 @@ static BlockDriver bdrv_throttle = { .bdrv_co_pwrite_zeroes = throttle_co_pwrite_zeroes, .bdrv_co_pdiscard = throttle_co_pdiscard, + .bdrv_co_pwritev_compressed = throttle_co_pwritev_compressed, .bdrv_attach_aio_context = throttle_attach_aio_context, .bdrv_detach_aio_context = throttle_detach_aio_context,
Signature · 1 line
2.26.2
Re: [PATCH v7 08/47] throttle: Support compressed writes
Quoted 27 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/throttle.c | 10 ++++++++++ 1 file changed, 10 insertions(+)block/throttle.c +9
-0@@ -154,6 +154,15 @@ static int coroutine_fn throttle_co_pdiscard(BlockDriverState *bs,return bdrv_co_pdiscard (bs->file, offset, bytes); } +static int coroutine_fn throttle_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return throttle_co_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} +static int throttle_co_flush (BlockDriverState *bs) {return bdrv_co_flush (bs->file->bs);
block/throttle.c static BlockDriver bdrv_throttle = { · line 261
Quoted 7 lines
block/throttle.c +1
-0@@ -246,6 +255,7 @@ static BlockDriver bdrv_throttle = { .bdrv_co_pwrite_zeroes = throttle_co_pwrite_zeroes, .bdrv_co_pdiscard = throttle_co_pdiscard, + .bdrv_co_pwritev_compressed = throttle_co_pwritev_compressed, .bdrv_attach_aio_context = throttle_attach_aio_context, .bdrv_detach_aio_context = throttle_detach_aio_context,
PATCH v7 9/47 copy-on-read: Support compressed writes
In reply to Max Reitz--- block/copy-on-read.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
block/copy-on-read.c +11 -0
@@ -107,6 +107,16 @@ static int coroutine_fn cor_co_pdiscard(BlockDriverState *bs, } +static int coroutine_fn cor_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_co_pwritev (bs->file, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} + +static void cor_eject (BlockDriverState *bs,bool eject_flag) {bdrv_eject (bs->file->bs, eject_flag);
@@ -131,6 +141,7 @@ static BlockDriver bdrv_copy_on_read = { .bdrv_co_pwritev = cor_co_pwritev, .bdrv_co_pwrite_zeroes = cor_co_pwrite_zeroes, .bdrv_co_pdiscard = cor_co_pdiscard, + .bdrv_co_pwritev_compressed = cor_co_pwritev_compressed, .bdrv_eject = cor_eject, .bdrv_lock_medium = cor_lock_medium,
Signature · 1 line
2.26.2
Re: [PATCH v7 09/47] copy-on-read: Support compressed writes
Quoted 27 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/copy-on-read.c | 11 +++++++++++ 1 file changed, 11 insertions(+)block/copy-on-read.c +10
-0@@ -107,6 +107,16 @@ static int coroutine_fn cor_co_pdiscard(BlockDriverState *bs, } +static int coroutine_fn cor_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_co_pwritev (bs->file, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} + +static void cor_eject (BlockDriverState *bs,bool eject_flag) {bdrv_eject (bs->file->bs, eject_flag);
block/copy-on-read.c static BlockDriver bdrv_copy_on_read = { · line 147
Quoted 7 lines
block/copy-on-read.c +1
-0@@ -131,6 +141,7 @@ static BlockDriver bdrv_copy_on_read = { .bdrv_co_pwritev = cor_co_pwritev, .bdrv_co_pwrite_zeroes = cor_co_pwrite_zeroes, .bdrv_co_pdiscard = cor_co_pdiscard, + .bdrv_co_pwritev_compressed = cor_co_pwritev_compressed, .bdrv_eject = cor_eject, .bdrv_lock_medium = cor_lock_medium,
PATCH v7 10/47 mirror-top: Support compressed writes
In reply to Max Reitz--- block/mirror.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
block/mirror.c +10 -0
@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, NULL, 0); } +static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_mirror_top_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} +static void bdrv_mirror_top_refresh_filename (BlockDriverState *bs) {if (bs->backing ==NULL ) {
@@ -1526,6 +1535,7 @@ static BlockDriver bdrv_mirror_top = { .bdrv_co_pwritev = bdrv_mirror_top_pwritev, .bdrv_co_pwrite_zeroes = bdrv_mirror_top_pwrite_zeroes, .bdrv_co_pdiscard = bdrv_mirror_top_pdiscard, + .bdrv_co_pwritev_compressed = bdrv_mirror_top_pwritev_compressed, .bdrv_co_flush = bdrv_mirror_top_flush, .bdrv_co_block_status = bdrv_co_block_status_from_backing, .bdrv_refresh_filename = bdrv_mirror_top_refresh_filename,
Signature · 1 line
2.26.2
Re: [PATCH v7 10/47] mirror-top: Support compressed writes
Quoted 26 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 10 ++++++++++ 1 file changed, 10 insertions(+)block/mirror.c +9
-0@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, NULL, 0); } +static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_mirror_top_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} +static void bdrv_mirror_top_refresh_filename (BlockDriverState *bs) {if (bs->backing ==NULL ) {
block/mirror.c static BlockDriver bdrv_mirror_top = { · line 1541
Quoted 7 lines
block/mirror.c +1
-0@@ -1526,6 +1535,7 @@ static BlockDriver bdrv_mirror_top = { .bdrv_co_pwritev = bdrv_mirror_top_pwritev, .bdrv_co_pwrite_zeroes = bdrv_mirror_top_pwrite_zeroes, .bdrv_co_pdiscard = bdrv_mirror_top_pdiscard, + .bdrv_co_pwritev_compressed = bdrv_mirror_top_pwritev_compressed, .bdrv_co_flush = bdrv_mirror_top_flush, .bdrv_co_block_status = bdrv_co_block_status_from_backing, .bdrv_refresh_filename = bdrv_mirror_top_refresh_filename,
Re: [PATCH v7 10/47] mirror-top: Support compressed writes
In reply to Max ReitzAm 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 11 lines
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 10 ++++++++++ 1 file changed, 10 insertions(+)block/mirror.c +0
-0@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, NULL, 0);
block/mirror.c static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, · line 1490
Quoted 10 lines
block/mirror.c +8
-0} +static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_mirror_top_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +}
Hm, not sure if it's a problem, but bdrv_supports_compressed_writes() will now return true for mirror-top. However, with an active mirror to a target that doesn't support compression, trying to actually do a compressed write will always return -ENOTSUP.
So I guess the set of nodes patch 7 looks at still isn't quite complete. However, it's not obvious how to make it more complete without delegating to the driver.
Maybe we need to use bs->supported_write_flags, which is set by the driver, instead of looking at the presence of callbacks.
Of course, in the general case, we also should make sure that graph changes will be reflected in bs->supported_write_flags, but we already fail to do this in raw-format, so I guess ignoring it for now is good enough here, too...
Kevin
Re: [PATCH v7 10/47] mirror-top: Support compressed writes
Quoted 14 lines · Kevin Wolf
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 13 lines
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 10 ++++++++++ 1 file changed, 10 insertions(+)block/mirror.c +0
-0@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, NULL, 0); }
Quoted 13 lines
block/mirror.c static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, · line 1490
Quoted 8 lines
block/mirror.c +8
-0+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_mirror_top_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +}Hm, not sure if it's a problem, but
bdrv_supports_compressed_writes()will now return true for mirror-top. However, with an active mirror to a target that doesn't support compression, trying to actually do a compressed write will always return-ENOTSUP.
Right.
Quoted 6 lines
So I guess the set of nodes patch 7 looks at still isn't quite complete. However, it's not obvious how to make it more complete without delegating to the driver.
Maybe we need to use
bs->supported_write_flags, which is set by the driver, instead of looking at the presence of callbacks.
Hm, yes, that would work better. Not sure if it’s worth it for this series.
The only problem we’d have is late failure when trying to do a compressed backup to a target that’s running an active mirror. (Late as in “first write fails without explanation”, as opposed to “job fails during set-up”.)
Which I hope is not a case anyone would ever encounter, and even if they do, the failure doesn’t seem catastrophic to me. So I don’t think it’s really a problem.
Max
Re: [PATCH v7 10/47] mirror-top: Support compressed writes
Am 19.08.2020 um 17:35 hat Max Reitz geschrieben:
Quoted 35 lines · Kevin Wolf
Quoted 14 lines
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 13 lines
Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 10 ++++++++++ 1 file changed, 10 insertions(+)block/mirror.c +0
-0@@ -1480,6 +1480,15 @@ static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, NULL, 0); }Quoted 13 lines
block/mirror.c static int coroutine_fn bdrv_mirror_top_pdiscard(BlockDriverState *bs, · line 1490
Quoted 8 lines
block/mirror.c +8
-0+static int coroutine_fn bdrv_mirror_top_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return bdrv_mirror_top_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +}Hm, not sure if it's a problem, but
bdrv_supports_compressed_writes()will now return true for mirror-top. However, with an active mirror to a target that doesn't support compression, trying to actually do a compressed write will always return-ENOTSUP.Right.
Quoted 4 lines
So I guess the set of nodes patch 7 looks at still isn't quite complete. However, it's not obvious how to make it more complete without delegating to the driver.
Quoted 5 lines
Quoted 2 lines
Maybe we need to use
bs->supported_write_flags, which is set by the driver, instead of looking at the presence of callbacks.Hm, yes, that would work better. Not sure if it’s worth it for this series.
This patch looks like a feature addition that is only marginally related to the goal of the series anyway. Maybe it should be a separate small series on top?
The other compression related patches in the series don't seem to have this problem, so they could stay there anyway.
Quoted 8 lines
The only problem we’d have is late failure when trying to do a compressed backup to a target that’s running an active mirror. (Late as in “first write fails without explanation”, as opposed to “job fails during set-up”.)
Which I hope is not a case anyone would ever encounter, and even if they do, the failure doesn’t seem catastrophic to me. So I don’t think it’s really a problem.
Yeah, it's just a bit unfortunate to add a new function that we know doesn't do what it promises.
Kevin
PATCH v7 11/47 backup-top: Support compressed writes
In reply to Max Reitz--- block/backup-top.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
block/backup-top.c +10 -0
@@ -99,6 +99,15 @@ static coroutine_fn int backup_top_co_pwritev(BlockDriverState *bs,return bdrv_co_pwritev (bs->backing, offset, bytes, qiov, flags); } +static coroutine_fn int backup_top_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return backup_top_co_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} + static int coroutine_fn backup_top_co_flush(BlockDriverState *bs) {if (!bs->backing) {
@@ -173,6 +182,7 @@ BlockDriver bdrv_backup_top_filter = { .bdrv_co_pwritev = backup_top_co_pwritev, .bdrv_co_pwrite_zeroes = backup_top_co_pwrite_zeroes, .bdrv_co_pdiscard = backup_top_co_pdiscard, + .bdrv_co_pwritev_compressed = backup_top_co_pwritev_compressed, .bdrv_co_flush = backup_top_co_flush, .bdrv_co_block_status = bdrv_co_block_status_from_backing,
Signature · 1 line
2.26.2
Re: [PATCH v7 11/47] backup-top: Support compressed writes
Quoted 26 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/backup-top.c | 10 ++++++++++ 1 file changed, 10 insertions(+)block/backup-top.c +9
-0@@ -99,6 +99,15 @@ static coroutine_fn int backup_top_co_pwritev(BlockDriverState *bs,return bdrv_co_pwritev (bs->backing, offset, bytes, qiov, flags); } +static coroutine_fn int backup_top_co_pwritev_compressed(BlockDriverState *bs, +uint64_t offset, +uint64_t bytes, +QEMUIOVector *qiov) +{ +return backup_top_co_pwritev (bs, offset, bytes, qiov, +BDRV_REQ_WRITE_COMPRESSED ); +} + static int coroutine_fn backup_top_co_flush(BlockDriverState *bs) {if (!bs->backing) {
block/backup-top.c BlockDriver bdrv_backup_top_filter = { · line 188
Quoted 7 lines
block/backup-top.c +1
-0@@ -173,6 +182,7 @@ BlockDriver bdrv_backup_top_filter = { .bdrv_co_pwritev = backup_top_co_pwritev, .bdrv_co_pwrite_zeroes = backup_top_co_pwrite_zeroes, .bdrv_co_pdiscard = backup_top_co_pdiscard, + .bdrv_co_pwritev_compressed = backup_top_co_pwritev_compressed, .bdrv_co_flush = backup_top_co_flush, .bdrv_co_block_status = bdrv_co_block_status_from_backing,
PATCH v7 12/47 block: Use bdrv_filter_(bs|child) where obvious
In reply to Max ReitzPlaces that use patterns like
if (bs->drv->is_filter && bs->file) {
... something about bs->file->bs ...
}should be
BlockDriverState *filtered = bdrv_filter_bs(bs);
if (filtered) {
... something about @filtered ...
}instead.
--- block.c | 31 ++++++++++++++++++++----------- block/io.c | 7 +++++-- migration/block-dirty-bitmap.c | 8 +------- 3 files changed, 26 insertions(+), 20 deletions(-)
block.c +20 -11
@@ -710,11 +710,12 @@ int coroutine_fn bdrv_co_delete_file(BlockDriverState *bs, Error **errp)int bdrv_probe_blocksizes (BlockDriverState *bs,BlockSizes *bsz) {BlockDriver *drv = bs->drv; +BlockDriverState *filtered =bdrv_filter_bs (bs);if (drv && drv->bdrv_probe_blocksizes) {return drv->bdrv_probe_blocksizes (bs, bsz); - }else if (drv && drv->is_filter && bs->file) { -return bdrv_probe_blocksizes (bs->file->bs, bsz); + }else if (filtered) { +return bdrv_probe_blocksizes (filtered, bsz); }return -ENOTSUP ;
@@ -729,11 +730,12 @@ int bdrv_probe_blocksizes(BlockDriverState *bs, BlockSizes *bsz)int bdrv_probe_geometry (BlockDriverState *bs,HDGeometry *geo) {BlockDriver *drv = bs->drv; +BlockDriverState *filtered =bdrv_filter_bs (bs);if (drv && drv->bdrv_probe_geometry) {return drv->bdrv_probe_geometry (bs, geo); - }else if (drv && drv->is_filter && bs->file) { -return bdrv_probe_geometry (bs->file->bs, geo); + }else if (filtered) { +return bdrv_probe_geometry (filtered, geo); }return -ENOTSUP ;
@@ -5421,6 +5423,8 @@ int bdrv_has_zero_init_1(BlockDriverState *bs)int bdrv_has_zero_init (BlockDriverState *bs) { +BlockDriverState *filtered; +if (!bs->drv) {return 0 ; }
@@ -5433,8 +5437,10 @@ int bdrv_has_zero_init(BlockDriverState *bs)if (bs->drv->bdrv_has_zero_init) {return bs->drv->bdrv_has_zero_init (bs); } -if (bs->file && bs->drv->is_filter) { -return bdrv_has_zero_init (bs->file->bs); + + filtered =bdrv_filter_bs (bs); +if (filtered) { +return bdrv_has_zero_init (filtered); }/* safe default */
@@ -5479,8 +5485,9 @@ int bdrv_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)return -ENOMEDIUM ; }if (!drv->bdrv_get_info) { -if (bs->file && drv->is_filter) { -return bdrv_get_info (bs->file->bs, bdi); +BlockDriverState *filtered =bdrv_filter_bs (bs); +if (filtered) { +return bdrv_get_info (filtered, bdi); }return -ENOTSUP ; }
@@ -6546,6 +6553,8 @@ int bdrv_amend_options(BlockDriverState *bs, QemuOpts *opts,bool bdrv_recurse_can_replace (BlockDriverState *bs,BlockDriverState *to_replace) { +BlockDriverState *filtered; +if (!bs || !bs->drv) {return false; }
@@ -6560,9 +6569,9 @@ bool bdrv_recurse_can_replace(BlockDriverState *bs, }/* For filters without an own implementation, we can recurse on our own */ -if (bs->drv->is_filter) { -BdrvChild *child = bs->file ?: bs->backing; -return bdrv_recurse_can_replace (child->bs, to_replace); + filtered =bdrv_filter_bs (bs); +if (filtered) { +return bdrv_recurse_can_replace (filtered, to_replace); }/* Safe default */
block/io.c +5 -2
@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, Error **errp) {BlockDriverState *bs = child->bs; +BdrvChild *filtered;BlockDriver *drv = bs->drv;BdrvTrackedRequest req;int64_t old_size, new_bytes;
@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } + filtered =bdrv_filter_child (bs); +/* * If the image has a backing file that is large enough that it would * provide data for the new area, we cannot leave it unallocated because
@@ -3390,8 +3393,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } ret = drv->bdrv_co_truncate (bs, offset, exact, prealloc, flags, errp); - }else if (bs->file && drv->is_filter) { - ret =bdrv_co_truncate (bs->file, offset, exact, prealloc, flags, errp); + }else if (filtered) { + ret =bdrv_co_truncate (filtered, offset, exact, prealloc, flags, errp); }else {error_setg (errp,"Image format driver does not support resize" ); ret = -ENOTSUP ;
migration/block-dirty-bitmap.c +1 -7
@@ -356,13 +356,7 @@ static int init_dirty_bitmap_migration(void)while (bs && bs->drv && bs->drv->is_filter && !bdrv_has_named_bitmaps (bs)) { -if (bs->backing) { - bs = bs->backing->bs; - }else if (bs->file) { - bs = bs->file->bs; - }else { - bs =NULL ; - } + bs =bdrv_filter_bs (bs); }if (bs && bs->drv && !bs->drv->is_filter) {
Signature · 1 line
2.26.2
Re: [PATCH v7 12/47] block: Use bdrv_filter_(bs|child) where obvious
Quoted 22 lines · Max Reitz
Places that use patterns like
if (bs->drv->is_filter && bs->file) { ... something about bs->file->bs ... }should be
BlockDriverState *filtered = bdrv_filter_bs(bs); if (filtered) { ... something about @filtered ... }instead.
Signed-off-by Max Reitz <[email protected]>--- block.c | 31 ++++++++++++++++++++----------- block/io.c | 7 +++++-- migration/block-dirty-bitmap.c | 8 +------- 3 files changed, 26 insertions(+), 20 deletions(-)
...
Quoted 13 lines
block/io.c +1
-0@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, Error **errp) {BlockDriverState *bs = child->bs; +BdrvChild *filtered;BlockDriver *drv = bs->drv;BdrvTrackedRequest req;int64_t old_size, new_bytes;
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3363
Quoted 5 lines
block/io.c +2
-0@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } + filtered =bdrv_filter_child (bs); +
Isn't better to have this initialization right before the relevant if/else block?
Andrey
Quoted 14 lines
block/io.c +2
-2/* * If the image has a backing file that is large enough that it would * provide data for the new area, we cannot leave it unallocated because @@ -3390,8 +3393,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } ret = drv->bdrv_co_truncate (bs, offset, exact, prealloc, flags, errp); - }else if (bs->file && drv->is_filter) { - ret =bdrv_co_truncate (bs->file, offset, exact, prealloc, flags, errp); + }else if (filtered) { + ret =bdrv_co_truncate (filtered, offset, exact, prealloc, flags, errp); }else {error_setg (errp,"Image format driver does not support resize" ); ret = -ENOTSUP ;
...
Re: [PATCH v7 12/47] block: Use bdrv_filter_(bs|child) where obvious
Quoted 38 lines · Andrey Shinkevich
Quoted 22 lines · Max Reitz
Places that use patterns like
if (
bs->drv->is_filter&&bs->file) {
... something aboutbs->file->bs...
}should be
BlockDriverState *filtered =
bdrv_filter_bs(bs); if (filtered) {
... something about @filtered ...
}instead.
Signed-off-by Max Reitz <[email protected]>---
block.c | 31 ++++++++++++++++++++----------- block/io.c | 7 +++++--migration/block-dirty-bitmap.c| 8 +-------3 files changed, 26 insertions(+), 20 deletions(-)...
Quoted 14 lines
block/io.c +1
-0@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, Error **errp) { BlockDriverState *bs = child->bs; + BdrvChild *filtered; BlockDriver *drv = bs->drv; BdrvTrackedRequest req;
int64_told_size,new_bytes;block/io.c +0
-0@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild
Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3363
Quoted 5 lines
block/io.c +1
-0*child, int64_t offset, bool exact, goto out; } + filtered = bdrv_filter_child(bs); +Isn't better to have this initialization right before the relevant if/else block?
Hm, well, yes. In this case, though, maybe not. Patch 16 will add another BdrvChild to be initialized here (@backing), and we need to initialize that one here. So I felt it made sense to group them together.
They got split up when I decided to put @filtered into this patch and @backing into its own. So now it may look a bit weird, but I feel like after patch 16 it makes sense.
(I’m indifferent, basically.)
Max
Re: [PATCH v7 12/47] block: Use bdrv_filter_(bs|child) where obvious
Quoted 50 lines · Max Reitz
Quoted 37 lines · Andrey Shinkevich
Quoted 22 lines · Max Reitz
Places that use patterns like
if (
bs->drv->is_filter&&bs->file) {
... something aboutbs->file->bs...
}should be
BlockDriverState *filtered =
bdrv_filter_bs(bs); if (filtered) {
... something about @filtered ...
}instead.
Signed-off-by Max Reitz <[email protected]>--- block.c | 31 ++++++++++++++++++++----------- block/io.c | 7 +++++-- migration/block-dirty-bitmap.c | 8 +------- 3 files changed, 26 insertions(+), 20 deletions(-)...
Quoted 13 lines
block/io.c +1
-0@@ -3307,6 +3307,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, Error **errp) { BlockDriverState *bs = child->bs; + BdrvChild *filtered; BlockDriver *drv = bs->drv; BdrvTrackedRequest req;
int64_told_size,new_bytes;Quoted 8 lines
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild · line 3363
Quoted 6 lines
block/io.c +1
-0@@ -3358,6 +3359,8 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, goto out; } + filtered = bdrv_filter_child(bs); +Isn't better to have this initialization right before the relevant if/else block?
Hm, well, yes. In this case, though, maybe not. Patch 16 will add another BdrvChild to be initialized here (@backing), and we need to initialize that one here. So I felt it made sense to group them together.
Quoted 7 lines
They got split up when I decided to put @filtered into this patch and @backing into its own. So now it may look a bit weird, but I feel like after patch 16 it makes sense.
(I’m indifferent, basically.)
Max
Yes, it makes a sence. I am on the way to reviewing further and have not reached the 16th yet.
It is a minor thing anyway )) Thank you for your response.
Andrey
PATCH v7 13/47 block: Use CAFs in block status functions
In reply to Max ReitzUse the child access functions in the block status inquiry functions as appropriate.
--- block/io.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)
block/io.c +10 -9
@@ -2407,11 +2407,12 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,if (ret & (BDRV_BLOCK_DATA |BDRV_BLOCK_ZERO )) { ret |=BDRV_BLOCK_ALLOCATED ; }else if (want_zero) { +BlockDriverState *cow_bs =bdrv_cow_bs (bs); +if (bdrv_unallocated_blocks_are_zero (bs)) { ret |=BDRV_BLOCK_ZERO ; - }else if (bs->backing) { -BlockDriverState *bs2 = bs->backing->bs; -int64_t size2 =bdrv_getlength (bs2); + }else if (cow_bs) { +int64_t size2 =bdrv_getlength (cow_bs);if (size2 >=0 && offset >= size2) { ret |=BDRV_BLOCK_ZERO ;
@@ -2477,7 +2478,7 @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,bool first = true;assert (bs != base); -for (p = bs; p != base; p =backing_bs (p)) { +for (p = bs; p != base; p =bdrv_filter_or_cow_bs (p)) { ret =bdrv_co_block_status (p, want_zero, offset, bytes, pnum, map, file);if (ret <0 ) {
@@ -2551,7 +2552,7 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,int bdrv_block_status (BlockDriverState *bs,int64_t offset,int64_t bytes,int64_t *pnum,int64_t *map,BlockDriverState **file) { -return bdrv_block_status_above (bs,backing_bs (bs), +return bdrv_block_status_above (bs,bdrv_filter_or_cow_bs (bs), offset, bytes, pnum, map, file); }
@@ -2561,9 +2562,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,int ret;int64_t dummy; - ret =bdrv_common_block_status_above (bs,backing_bs (bs), false, offset, - bytes, pnum ? pnum : &dummy,NULL , -NULL ); + ret =bdrv_common_block_status_above (bs,bdrv_filter_or_cow_bs (bs), false, + offset, bytes, pnum ? pnum : &dummy, +NULL ,NULL );if (ret <0 ) {return ret; }
@@ -2626,7 +2627,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,break ; } - intermediate =backing_bs (intermediate); + intermediate =bdrv_filter_or_cow_bs (intermediate); } *pnum = n;
Signature · 1 line
2.26.2
Re: [PATCH v7 13/47] block: Use CAFs in block status functions
Quoted 60 lines · Max Reitz
Use the child access functions in the block status inquiry functions as appropriate.
Signed-off-by Max Reitz <[email protected]>--- block/io.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-)block/io.c +9
-8@@ -2407,11 +2407,12 @@ static int coroutine_fn bdrv_co_block_status(BlockDriverState *bs,if (ret & (BDRV_BLOCK_DATA |BDRV_BLOCK_ZERO )) { ret |=BDRV_BLOCK_ALLOCATED ; }else if (want_zero) { +BlockDriverState *cow_bs =bdrv_cow_bs (bs); +if (bdrv_unallocated_blocks_are_zero (bs)) { ret |=BDRV_BLOCK_ZERO ; - }else if (bs->backing) { -BlockDriverState *bs2 = bs->backing->bs; -int64_t size2 =bdrv_getlength (bs2); + }else if (cow_bs) { +int64_t size2 =bdrv_getlength (cow_bs);if (size2 >=0 && offset >= size2) { ret |=BDRV_BLOCK_ZERO ;@@ -2477,7 +2478,7 @@ static int coroutine_fn bdrv_co_block_status_above(BlockDriverState *bs,bool first = true;assert (bs != base); -for (p = bs; p != base; p =backing_bs (p)) { +for (p = bs; p != base; p =bdrv_filter_or_cow_bs (p)) { ret =bdrv_co_block_status (p, want_zero, offset, bytes, pnum, map, file);if (ret <0 ) {@@ -2551,7 +2552,7 @@ int bdrv_block_status_above(BlockDriverState *bs, BlockDriverState *base,int bdrv_block_status (BlockDriverState *bs,int64_t offset,int64_t bytes,int64_t *pnum,int64_t *map,BlockDriverState **file) { -return bdrv_block_status_above (bs,backing_bs (bs), +return bdrv_block_status_above (bs,bdrv_filter_or_cow_bs (bs), offset, bytes, pnum, map, file); }@@ -2561,9 +2562,9 @@ int coroutine_fn bdrv_is_allocated(BlockDriverState *bs, int64_t offset,int ret;int64_t dummy; - ret =bdrv_common_block_status_above (bs,backing_bs (bs), false, offset, - bytes, pnum ? pnum : &dummy,NULL , -NULL ); + ret =bdrv_common_block_status_above (bs,bdrv_filter_or_cow_bs (bs), false, + offset, bytes, pnum ? pnum : &dummy, +NULL ,NULL );if (ret <0 ) {return ret; }
block/io.c int bdrv_is_allocated_above(BlockDriverState *top, · line 2633
Quoted 8 lines
block/io.c +1
-1@@ -2626,7 +2627,7 @@ int bdrv_is_allocated_above(BlockDriverState *top,break ; } - intermediate =backing_bs (intermediate); + intermediate =bdrv_filter_or_cow_bs (intermediate); } *pnum = n;
PATCH v7 14/47 stream: Deal with filters
In reply to Max ReitzBecause of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)
qapi/block-core.json +4 -0
@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +46 -17
137 lines not on this page · raw message
blockdev.c +3 -1
@@ -2515,7 +2515,9 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, }/* Check for op blockers in the whole chain between bs and base */ -for (iter = bs; iter && iter != base_bs; iter =backing_bs (iter)) { +for (iter = bs; iter && iter != base_bs; + iter =bdrv_filter_or_cow_bs (iter)) + {if (bdrv_op_is_blocked (iter,BLOCK_OP_TYPE_STREAM , errp)) { goto out; }
Signature · 1 line
2.26.2
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 33 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +4
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #block/stream.c +0
-0
block/stream.c enum { · line 35
Quoted 6 lines
block/stream.c +2
-1@@ -31,7 +31,8 @@ enum { typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream from this) */ + BlockDriverState *above_base; /* Node directly above the base */
Keeping the base_overlay is enough to complete the stream job.
The above_base may disappear during the job and we can't rely on it.
Quoted 13 lines
block/stream.c +1
-1BlockdevOnError on_error;char *backing_file_str;bool bs_read_only;@@ -53,7 +54,7 @@ static void stream_abort(Job *job)if (s->chain_frozen) {BlockJob *bjob = &s->common; -bdrv_unfreeze_backing_chain (blk_bs (bjob->blk), s->bottom); +bdrv_unfreeze_backing_chain (blk_bs (bjob->blk), s->above_base); } }
block/stream.c static int stream_prepare(Job *job) · line 67
Quoted 6 lines
block/stream.c +2
-1@@ -62,14 +63,15 @@ static int stream_prepare(Job *job)StreamBlockJob *s =container_of (job, StreamBlockJob, common.job);BlockJob *bjob = &s->common;BlockDriverState *bs =blk_bs (bjob->blk); -BlockDriverState *base =backing_bs (s->bottom); +BlockDriverState *unfiltered_bs =bdrv_skip_filters (bs); +BlockDriverState *base =bdrv_filter_or_cow_bs (s->above_base);
The initial base node may be a top node for a concurrent commit job and
may disappear. It is true for the above_base as well.
base = bdrv_filter_or_cow_bs(s->base_overlay) is more reliable.
Quoted 60 lines
block/stream.c +10
-8Error *local_err =NULL ;int ret =0 ; -bdrv_unfreeze_backing_chain (bs, s->bottom); +bdrv_unfreeze_backing_chain (bs, s->above_base); s->chain_frozen = false; -if (bs->backing) { +if (bdrv_cow_child (unfiltered_bs)) {const char *base_id =NULL , *base_fmt =NULL ;if (base) { base_id = s->backing_file_str;@@ -77,8 +79,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } -bdrv_set_backing_hd (bs, base, &local_err); - ret =bdrv_change_backing_file (bs, base_id, base_fmt); +bdrv_set_backing_hd (unfiltered_bs, base, &local_err); + ret =bdrv_change_backing_file (unfiltered_bs, base_id, base_fmt);if (local_err) {error_report_err (local_err);return -EPERM ;@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job, Error **errp)StreamBlockJob *s =container_of (job, StreamBlockJob, common.job);BlockBackend *blk = s->common.blk;BlockDriverState *bs =blk_bs (blk); -bool enable_cor = !backing_bs (s->bottom); +BlockDriverState *unfiltered_bs =bdrv_skip_filters (bs); +bool enable_cor = !bdrv_cow_child (s->base_overlay);int64_t len;int64_t offset =0 ;uint64_t delay_ns =0 ;int error =0 ;int64_t n =0 ;/* bytes */ -if (bs == s->bottom) { +if (unfiltered_bs == s->base_overlay) {/* Nothing to stream */ return 0 ; }@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret =bdrv_is_allocated (bs, offset,STREAM_CHUNK , &n); + ret =bdrv_is_allocated (unfiltered_bs, offset,STREAM_CHUNK , &n);if (ret ==1 ) {/* Allocated in the top, no need to copy. */ }else if (ret >=0 ) {/* Copy if allocated in the intermediate images. Limit to the * known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE). */ - ret =bdrv_is_allocated_above (backing_bs (bs), s->bottom, true, + ret =bdrv_is_allocated_above (bdrv_cow_bs (unfiltered_bs), + s->base_overlay, true, offset, n, &n);/* Finish early if end of backing file has been reached */ if (ret ==0 && n ==0 ) {@@ -223,9 +227,29 @@ void stream_start(const char *job_id, BlockDriverState *bs,BlockDriverState *iter;
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 235
Quoted 10 lines
block/stream.c +5
-2bool bs_read_only;int basic_flags =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE_UNCHANGED ; -BlockDriverState *bottom =bdrv_find_overlay (bs, base); +BlockDriverState *base_overlay =bdrv_find_overlay (bs, base); +BlockDriverState *above_base; -if (bdrv_freeze_backing_chain (bs, bottom, errp) <0 ) { +if (!base_overlay) { +error_setg (errp,"'%s' is not in the backing chain of '%s'" , + base->node_name, bs->node_name);
Sorry, I am not clear with the error message.
In this case, there is no an intermediate COW node but the base, if not NULL, is
in the backing chain of bs, isn't it?
Quoted 7 lines
block/stream.c +7
-0+return ; + } + +/* +* Find the node directly above @base. @base_overlay is a COW overlay, so +* it must have a bdrv_cow_child(), but it is the immediate overlay of +* @base, so between the two there can only be filters.
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 252
Quoted 10 lines
block/stream.c +10
-0+*/ + above_base = base_overlay; +if (bdrv_cow_bs (above_base) != base) { + above_base =bdrv_cow_bs (above_base); +while (bdrv_filter_bs (above_base) != base) { + above_base =bdrv_filter_bs (above_base); + } + } + +if (bdrv_freeze_backing_chain (bs, above_base, errp) <0 ) {
When a concurrent stream job tries to freeze or remove the above_base node,
we will encounter the frozen node error. The above_base node is a part of the
concurrent job frozen chain.
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 282
Quoted 8 lines
block/stream.c +1
-0return ; }@@ -255,14 +279,19 @@ void stream_start(const char *job_id, BlockDriverState *bs,* and resizes. Reassign the base node pointer because the backing BS of the * bottom node might change after the call to bdrv_reopen_set_read_only() * due to parallel block jobs running. +* above_base node might change after the call to
Yes, if not frozen.
Quoted 5 lines
block/stream.c +2
-2+* bdrv_reopen_set_read_only() due to parallel block jobs running. */ - base =backing_bs (bottom); -for (iter =backing_bs (bs); iter && iter != base; iter =backing_bs (iter)) { + base =bdrv_filter_or_cow_bs (above_base);
block/stream.c void stream_start(const char *job_id, BlockDriverState *bs, · line 294
Quoted 10 lines
block/stream.c +5
-1+for (iter =bdrv_filter_or_cow_bs (bs); iter != base; + iter =bdrv_filter_or_cow_bs (iter)) + {block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , basic_flags, &error_abort); } - s->bottom = bottom; + s->base_overlay = base_overlay; + s->above_base = above_base;
Generally, being the filter for a concurrent job, the above_base node may be deleted any time
and we will keep the dangling pointer. It may happen even earlier if above_base is not frozen.
If it is, as it here, we may get the frozen link error then.
Andrey
Quoted 25 lines
block/stream.c +1
-1s->backing_file_str =g_strdup (backing_file_str); s->bs_read_only = bs_read_only; s->chain_frozen = true;@@ -276,5 +305,5 @@ fail:if (bs_read_only) {bdrv_reopen_set_read_only (bs, true,NULL ); } -bdrv_unfreeze_backing_chain (bs, bottom); +bdrv_unfreeze_backing_chain (bs, above_base); }blockdev.c +3
-1@@ -2515,7 +2515,9 @@ void qmp_block_stream(bool has_job_id, const char *job_id, const char *device, }/* Check for op blockers in the whole chain between bs and base */ -for (iter = bs; iter && iter != base_bs; iter =backing_bs (iter)) { +for (iter = bs; iter && iter != base_bs; + iter =bdrv_filter_or_cow_bs (iter)) + {if (bdrv_op_is_blocked (iter,BLOCK_OP_TYPE_STREAM , errp)) { goto out; }
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 14 lines · Andrey Shinkevich
Quoted 13 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)
...
Quoted 7 lines
Quoted 6 lines
+ BlockDriverState
*base_overlay=bdrv_find_overlay(bs, base); + BlockDriverState*above_base;
- if (bdrv_freeze_backing_chain(bs, bottom, errp)< 0) { + if (!base_overlay) {
+ error_setg(errp, "'%s' is not in the backing chain of '%s'", +base->node_name,bs->node_name);
Quoted 6 lines
Sorry, I am not clear with the error message.
In this case, there is no an intermediate COW node but the base, if not NULL, is
in the backing chain of bs, isn't it?
I am discarding this question. No need to answer.
Andrey
Quoted 3 lines
Quoted 3 lines
+ return;
+ }
+
Re: [PATCH v7 14/47] stream: Deal with filters
In reply to Andrey ShinkevichQuoted 36 lines · Andrey Shinkevich
Quoted 35 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +## @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +0
-0
Quoted 8 lines
block/stream.c enum { · line 34
Quoted 6 lines
block/stream.c +2
-1@@ -31,7 +31,8 @@ enum { typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream from this) */ + BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.
In this version of this series, it may not, because the chain is frozen.
So the above_base cannot disappear.
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we keep above_base and use its immediate child.
If we don’t keep above_base, then we’re basically left guessing as to what should be the backing file after the stream job.
Quoted 20 lines
Quoted 12 lines
block/stream.c +1
-1BlockdevOnError on_error; char *backing_file_str; bool bs_read_only;@@ -53,7 +54,7 @@ static void stream_abort(Job *job) if (s->chain_frozen) { BlockJob *bjob = &s->common; - bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom); + bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base); } } @@ -62,14 +63,15 @@ static int stream_prepare(Job *job) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);Quoted 5 lines
BlockJob *bjob =
&s->common;
BlockDriverState *bs =blk_bs(bjob->blk);
- BlockDriverState *base =backing_bs(s->bottom);
+ BlockDriverState*unfiltered_bs=bdrv_skip_filters(bs); + BlockDriverState *base =bdrv_filter_or_cow_bs(s->above_base);The initial base node may be a top node for a concurrent commit job and
Quoted 1 line
may disappear.
Then it would just be replaced by another node, though, so above_base keeps a child. The @base here is not necessarily the initial @base, and that’s intentional.
Quoted 1 line
base =
bdrv_filter_or_cow_bs(s->base_overlay)is more reliable.
But also wrong. The point of keeping above_base around is to get its child here to use that child as the new backing child of the top node.
Quoted 103 lines
Quoted 65 lines
Error
*local_err= NULL;
int ret = 0;
-bdrv_unfreeze_backing_chain(bs, s->bottom); +bdrv_unfreeze_backing_chain(bs, s->above_base);s->chain_frozen= false;
- if (bs->backing) {
+ if (bdrv_cow_child(unfiltered_bs)) { const char*base_id= NULL,*base_fmt= NULL; if (base) {
base_id=s->backing_file_str;block/stream.c +2
-2@@ -77,8 +79,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } - bdrv_set_backing_hd(bs, base, &local_err); - ret = bdrv_change_backing_file(bs, base_id, base_fmt); + bdrv_set_backing_hd(unfiltered_bs, base, &local_err); + ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt); if (local_err) { error_report_err(local_err);return
-EPERM;block/stream.c +2
-1@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job, Error **errp) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockBackend *blk = s->common.blk; BlockDriverState *bs = blk_bs(blk); - bool enable_cor = !backing_bs(s->bottom); + BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); + bool enable_cor = !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset = 0; uint64_t delay_ns = 0; int error = 0;
int64_tn = 0; /* bytes */
- if (bs ==s->bottom) {
+ if (unfiltered_bs==s->base_overlay) {
/* Nothing to stream */
return 0;
}block/stream.c +1
-0@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n); + ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK, &n); if (ret == 1) { /* Allocated in the top, no need to copy. */ } else if (ret >= 0) { /* Copy if allocated in the intermediate images. Limitto the
* known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE). */
- ret = bdrv_is_allocated_above(backing_bs(bs),s->bottom, true,
+ ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +s->base_overlay, true, offset, n, &n); /* Finish early if end of backing file has been reached */ if (ret == 0 && n == 0) {block/stream.c +0
-0@@ -223,9 +227,29 @@ void stream_start(const char *job_id,BlockDriverState *bs, BlockDriverState *iter; bool bs_read_only;int
basic_flags=BLK_PERM_CONSISTENT_READ|Quoted 8 lines
BLK_PERM_WRITE_UNCHANGED;
- BlockDriverState *bottom =bdrv_find_overlay(bs, base); + BlockDriverState*base_overlay=bdrv_find_overlay(bs, base); + BlockDriverState*above_base;
- if (bdrv_freeze_backing_chain(bs, bottom, errp)< 0) { + if (!base_overlay) {
+ error_setg(errp, "'%s' is not in the backing chain of '%s'", +base->node_name,bs->node_name);Sorry, I am not clear with the error message.
In this case, there is no an intermediate COW node but the base, if not NULL, is
in the backing chain of bs, isn't it?
Quoted 11 lines
+ return;
+ }
+
+ /*
+ * Find the node directly above @base. @base_overlay is a COW overlay, so
+ * it must have abdrv_cow_child(), but it is the immediate overlay of
+ * @base, so between the two there can only be filters. + */
+above_base=base_overlay;Quoted 8 lines
+ if (
bdrv_cow_bs(above_base)!= base) {
+above_base=bdrv_cow_bs(above_base);
+ while (bdrv_filter_bs(above_base)!= base) { +above_base=bdrv_filter_bs(above_base); + }
+ }
+
+ if (bdrv_freeze_backing_chain(bs, above_base, errp)< 0) {When a concurrent stream job tries to freeze or remove the
above_basenode,
Quoted 4 lines
we will encounter the frozen node error. The
above_basenode is a part of theconcurrent job frozen chain.
Correct.
Quoted 27 lines
Quoted 10 lines
return;
}
@@ -255,14 +279,19 @@ void stream_start(const char*job_id, BlockDriverState *bs,
* and resizes. Reassign the base node pointer because the backing BS of the
* bottom node might change after the call tobdrv_reopen_set_read_only()
* due to parallel block jobs running.
+ *above_basenode might change after the call toYes, if not frozen.
Quoted 7 lines
+ *
bdrv_reopen_set_read_only()due to parallel block jobs running. */
- base =backing_bs(bottom);
- for (iter =backing_bs(bs); iter && iter != base; iter =backing_bs(iter)) {
+ base =bdrv_filter_or_cow_bs(above_base);
+ for (iter =bdrv_filter_or_cow_bs(bs); iter != base;Quoted 8 lines
+ iter =
bdrv_filter_or_cow_bs(iter))
+ {
block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
}
-s->bottom= bottom;
+s->base_overlay=base_overlay;
+s->above_base=above_base;
Quoted 7 lines
Generally, being the filter for a concurrent job, the
above_basenode may be deleted any timeand we will keep the dangling pointer. It may happen even earlier if
above_baseis not frozen.If it is, as it here, we may get the frozen link error then.
I’m not sure what you mean here. Freezing it was absolutely intentional. A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.
Max
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 44 lines · Max Reitz
Quoted 35 lines · Andrey Shinkevich
Quoted 34 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #block/stream.c +0
-1--- a/block/stream.cQuoted 8 lines
block/stream.c enum { · line 34
Quoted 7 lines
+++
b/block/stream.cblock/stream.c +2
-1@@ -31,7 +31,8 @@ enum { typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream from this) */ + BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.
Quoted 6 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen.
So theabove_basecannot disappear.
Once we insert a filter above the top bs of the stream job, the parallel jobs in
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
Quoted 6 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we keep
above_baseand use its immediate child.
We can do the same with the base_overlay.
If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
Quoted 23 lines
If we don’t keep
above_base, then we’re basically left guessing as to what should be the backing file after the stream job.Quoted 19 lines
Quoted 12 lines
block/stream.c +1
-1BlockdevOnError on_error; char *backing_file_str; bool bs_read_only;@@ -53,7 +54,7 @@ static void stream_abort(Job *job) if (s->chain_frozen) { BlockJob *bjob = &s->common; - bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom); + bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base); } } @@ -62,14 +63,15 @@ static int stream_prepare(Job *job) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job);Quoted 5 lines
BlockJob *bjob =
&s->common;
BlockDriverState *bs =blk_bs(bjob->blk);
- BlockDriverState *base =backing_bs(s->bottom);
+ BlockDriverState*unfiltered_bs=bdrv_skip_filters(bs); + BlockDriverState *base =bdrv_filter_or_cow_bs(s->above_base);The initial base node may be a top node for a concurrent commit job and
Quoted 4 lines
Quoted 1 line
may disappear.
Then it would just be replaced by another node, though, so
above_basekeeps a child. The @base here is not necessarily the initial @base, and that’s intentional.
Not really. In my example, above_base becomes a dangling
pointer because after the commit job finishes, its filter that should belong to the
commit job frozen chain will be deleted. If we freeze the link to the above_base
for this job, the iotests #30 will not pass.
Quoted 143 lines
Quoted 1 line
base =
bdrv_filter_or_cow_bs(s->base_overlay)is more reliable.But also wrong. The point of keeping
above_basearound is to get its child here to use that child as the new backing child of the top node.Quoted 101 lines
Quoted 63 lines
Error
*local_err= NULL;
int ret = 0;
-bdrv_unfreeze_backing_chain(bs, s->bottom);
+bdrv_unfreeze_backing_chain(bs, s->above_base);
s->chain_frozen= false;
- if (bs->backing) {
+ if (bdrv_cow_child(unfiltered_bs)) {
const char*base_id= NULL,*base_fmt= NULL; if (base) {
base_id=s->backing_file_str;block/stream.c +2
-2@@ -77,8 +79,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } - bdrv_set_backing_hd(bs, base, &local_err); - ret = bdrv_change_backing_file(bs, base_id, base_fmt); + bdrv_set_backing_hd(unfiltered_bs, base, &local_err); + ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt); if (local_err) { error_report_err(local_err);return
-EPERM;block/stream.c +4
-1@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job, Error **errp) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockBackend *blk = s->common.blk; BlockDriverState *bs = blk_bs(blk); - bool enable_cor = !backing_bs(s->bottom); + BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); + bool enable_cor = !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset = 0; uint64_t delay_ns = 0; int error = 0; int64_t n = 0; /* bytes */ - if (bs == s->bottom) { + if (unfiltered_bs == s->base_overlay) { /* Nothing to stream */ return 0; }@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n); + ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK, &n); if (ret == 1) { /* Allocated in the top, no need to copy. */ } else if (ret >= 0) { /* Copy if allocated in the intermediate images. Limitto the
* known-unallocated area [offset,
offset+n*BDRV_SECTOR_SIZE). */
- ret = bdrv_is_allocated_above(backing_bs(bs),s->bottom, true,
+ ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +s->base_overlay, true,
offset, n, &n); /* Finish early if end of backing file has been reached */ if (ret == 0 && n == 0) {block/stream.c +0
-0@@ -223,9 +227,29 @@ void stream_start(const char *job_id,BlockDriverState *bs, BlockDriverState *iter;block/stream.c void stream_start(const char *job_id, · line 237
Quoted 10 lines
block/stream.c +5
-1bool bs_read_only; int basic_flags = BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE_UNCHANGED ; - BlockDriverState *bottom = bdrv_find_overlay(bs, base); + BlockDriverState *base_overlay = bdrv_find_overlay(bs, base); + BlockDriverState *above_base; - if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) { + if (!base_overlay) { + error_setg(errp, "'%s' is not in the backing chain of '%s'", + base->node_name, bs->node_name);Sorry, I am not clear with the error message.
In this case, there is no an intermediate COW node but the base, if not NULL, is
in the backing chain of bs, isn't it?
Quoted 11 lines
block/stream.c +5
-0+ return; + } + + /* +* Find the node directly above @base. @base_overlay is a COW overlay, so
+ * it must have abdrv_cow_child(), but it is the immediate overlay of
+ * @base, so between the two there can only be filters. + */
+above_base=base_overlay;Quoted 8 lines
+ if (
bdrv_cow_bs(above_base)!= base) {
+above_base=bdrv_cow_bs(above_base);
+ while (bdrv_filter_bs(above_base)!= base) { +above_base=bdrv_filter_bs(above_base); + }
+ }
+
+ if (bdrv_freeze_backing_chain(bs, above_base, errp)< 0) {When a concurrent stream job tries to freeze or remove the
above_basenode,Quoted 4 lines
we will encounter the frozen node error. The
above_basenode is a part of theconcurrent job frozen chain.
Correct.
Quoted 32 lines
Quoted 10 lines
return;
}
@@ -255,14 +279,19 @@ void stream_start(const char*job_id,
BlockDriverState *bs,
* and resizes. Reassign the base node pointer because the
backing BS of the
* bottom node might change after the call tobdrv_reopen_set_read_only()
* due to parallel block jobs running.
+ *above_basenode might change after the call toYes, if not frozen.
Quoted 7 lines
+ *
bdrv_reopen_set_read_only()due to parallel block jobs running.
*/
- base =backing_bs(bottom);
- for (iter =backing_bs(bs); iter && iter != base; iter =backing_bs(iter)) {
+ base =bdrv_filter_or_cow_bs(above_base);
+ for (iter =bdrv_filter_or_cow_bs(bs); iter != base;Quoted 8 lines
+ iter =
bdrv_filter_or_cow_bs(iter))
+ {
block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
}
-s->bottom= bottom;
+s->base_overlay=base_overlay;
+s->above_base=above_base;Generally, being the filter for a concurrent job, the
above_basenode may be deleted any timeand we will keep the dangling pointer. It may happen even earlier if
above_baseis not frozen.
Quoted 6 lines
Quoted 1 line
If it is, as it here, we may get the frozen link error then.
I’m not sure what you mean here. Freezing it was absolutely intentional. A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.
Max
The nodes we freeze should be in one context of the relevant job:
filter->top_node->intermediate_node(s)
We would not include the base or any filter above it to the frozen chain
because they are of a different job context.
Once 'this' job is completed, we set the current backing child of the base_overlay
and may not care of its character. If that is another job filter, it will be replaced
with the proper node afterwards.
Andrey
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 54 lines · Andrey Shinkevich
Quoted 46 lines · Max Reitz
Quoted 37 lines · Andrey Shinkevich
Quoted 36 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63
++++++++++++++++++++++++++++++++------------
blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +## @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +0
-0Quoted 8 lines
Quoted 7 lines
block/stream.c +1
-1@@ -31,7 +31,8 @@ enum { typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream fromthis) */
+ BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.Quoted 6 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen. So the
above_basecannot disappear.
Quoted 8 lines
Once we insert a filter above the top bs of the stream job, the parallel jobs in
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
I’m not sure whether that’s a problem with this series specifically.
Quoted 11 lines
Quoted 5 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we keep
above_baseand use its immediate child.We can do the same with the
base_overlay.If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
And what if the user manually added some filter above the base (i.e. below base_overlay) that they want to keep after the job?
Quoted 30 lines
Quoted 23 lines
If we don’t keep
above_base, then we’re basically left guessing as to what should be the backing file after the stream job.Quoted 20 lines
Quoted 12 lines
BlockdevOnError
on_error;
char*backing_file_str;
boolbs_read_only;block/stream.c +1
-1@@ -53,7 +54,7 @@ static void stream_abort(Job *job) if (s->chain_frozen) { BlockJob *bjob = &s->common; - bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom); + bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base); } } @@ -62,14 +63,15 @@ static int stream_prepare(Job *job) StreamBlockJob *s = container_of(job, StreamBlockJob,Quoted 6 lines
common.job);
BlockJob *bjob =&s->common;
BlockDriverState *bs =blk_bs(bjob->blk);
- BlockDriverState *base =backing_bs(s->bottom);
+ BlockDriverState*unfiltered_bs=bdrv_skip_filters(bs); + BlockDriverState *base =bdrv_filter_or_cow_bs(s->above_base);The initial base node may be a top node for a concurrent commit job and
Quoted 4 lines
Quoted 1 line
may disappear.
Then it would just be replaced by another node, though, so
above_basekeeps a child. The @base here is not necessarily the initial @base, and that’s intentional.Not really. In my example,
above_basebecomes a dangling
Quoted 7 lines
pointer because after the commit job finishes, its filter that should belong to the
commit job frozen chain will be deleted. If we freeze the link to the
above_basefor this job, the iotests #30 will not pass.
So it doesn’t become a dangling pointer, because it’s frozen.
030 passes after this series, so I’m not sure whether I can consider that problem part of this series.
I think if adding a filter node becomes a problem, we have to consider relaxing the restrictions when we do that, not now.
Quoted 153 lines
Quoted 146 lines
Quoted 1 line
base =
bdrv_filter_or_cow_bs(s->base_overlay)is more reliable.But also wrong. The point of keeping
above_basearound is to get its child here to use that child as the new backing child of the top node.Quoted 104 lines
Quoted 67 lines
Error
*local_err= NULL;
int ret = 0;
-bdrv_unfreeze_backing_chain(bs, s->bottom); +bdrv_unfreeze_backing_chain(bs, s->above_base);s->chain_frozen= false;
- if (bs->backing) {
+ if (bdrv_cow_child(unfiltered_bs)) {
const char*base_id= NULL,*base_fmt= NULL; if (base) {
base_id=s->backing_file_str;block/stream.c +2
-2@@ -77,8 +79,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } - bdrv_set_backing_hd(bs, base, &local_err); - ret = bdrv_change_backing_file(bs, base_id, base_fmt); + bdrv_set_backing_hd(unfiltered_bs, base, &local_err); + ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt); if (local_err) { error_report_err(local_err);return
-EPERM;block/stream.c +2
-1@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job, Error **errp) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockBackend *blk = s->common.blk; BlockDriverState *bs = blk_bs(blk); - bool enable_cor = !backing_bs(s->bottom); + BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); + bool enable_cor = !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset = 0; uint64_t delay_ns = 0; int error = 0;
int64_tn = 0; /* bytes */
- if (bs ==s->bottom) {
+ if (unfiltered_bs==s->base_overlay) {
/* Nothing to stream */
return 0;
}block/stream.c +1
-0@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n); + ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK, &n); if (ret == 1) { /* Allocated in the top, no need to copy. */ } else if (ret >= 0) { /* Copy if allocated in the intermediate images. Limitto the
* known-unallocated area [offset, offset+n*BDRV_SECTOR_SIZE). */
- ret = bdrv_is_allocated_above(backing_bs(bs),s->bottom, true,
+ ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +s->base_overlay, true, offset, n, &n); /* Finish early if end of backing file has been reached */
if (ret == 0 && n == 0) {block/stream.c +0
-0@@ -223,9 +227,29 @@ void stream_start(const char *job_id,BlockDriverState *bs, BlockDriverState *iter; bool bs_read_only;int
basic_flags=BLK_PERM_CONSISTENT_READ|Quoted 8 lines
BLK_PERM_WRITE_UNCHANGED;
- BlockDriverState *bottom =bdrv_find_overlay(bs, base); + BlockDriverState*base_overlay=bdrv_find_overlay(bs, base); + BlockDriverState*above_base;
- if (bdrv_freeze_backing_chain(bs, bottom, errp)< 0) { + if (!base_overlay) {
+ error_setg(errp, "'%s' is not in the backing chain of '%s'", +base->node_name,bs->node_name);Sorry, I am not clear with the error message.
In this case, there is no an intermediate COW node but the base, if not NULL, is
in the backing chain of bs, isn't it?
Quoted 11 lines
+ return;
+ }
+
+ /*
+ * Find the node directly above @base. @base_overlay is a COW overlay, so
+ * it must have abdrv_cow_child(), but it is the immediate overlay of
+ * @base, so between the two there can only be filters. + */
+above_base=base_overlay;Quoted 8 lines
+ if (
bdrv_cow_bs(above_base)!= base) {
+above_base=bdrv_cow_bs(above_base);
+ while (bdrv_filter_bs(above_base)!= base) { +above_base=bdrv_filter_bs(above_base); + }
+ }
+
+ if (bdrv_freeze_backing_chain(bs, above_base, errp)< 0) {When a concurrent stream job tries to freeze or remove the
above_basenode,Quoted 4 lines
we will encounter the frozen node error. The
above_basenode is a part of theconcurrent job frozen chain.
Correct.
Quoted 32 lines
Quoted 10 lines
return;
}
@@ -255,14 +279,19 @@ void stream_start(const char*job_id, BlockDriverState *bs,
* and resizes. Reassign the base node pointer because the backing BS of the
* bottom node might change after the call tobdrv_reopen_set_read_only()
* due to parallel block jobs running.
+ *above_basenode might change after the call toYes, if not frozen.
Quoted 7 lines
+ *
bdrv_reopen_set_read_only()due to parallel block jobs running. */
- base =backing_bs(bottom);
- for (iter =backing_bs(bs); iter && iter != base; iter =backing_bs(iter)) {
+ base =bdrv_filter_or_cow_bs(above_base);
+ for (iter =bdrv_filter_or_cow_bs(bs); iter != base;Quoted 8 lines
+ iter =
bdrv_filter_or_cow_bs(iter))
+ {
block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
}
-s->bottom= bottom;
+s->base_overlay=base_overlay;
+s->above_base=above_base;Generally, being the filter for a concurrent job, the
above_basenode may be deleted any timeand we will keep the dangling pointer. It may happen even earlier if
above_baseis not frozen.Quoted 6 lines
Quoted 1 line
If it is, as it here, we may get the frozen link error then.
I’m not sure what you mean here. Freezing it was absolutely intentional. A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.
Max
Quoted 7 lines
The nodes we freeze should be in one context of the relevant job:
filter->top_node->intermediate_node(s)We would not include the base or any filter above it to the frozen chain
because they are of a different job context.
They aren’t really, because we need to know the backing node of @device after the job.
Quoted 7 lines
Once 'this' job is completed, we set the current backing child of the
base_overlayand may not care of its character. If that is another job filter, it will be replaced
with the proper node afterwards.
But what if there are filters above the base that the user wants to keep after the job?
Max
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 72 lines · Max Reitz
Quoted 54 lines · Andrey Shinkevich
Quoted 46 lines · Max Reitz
Quoted 37 lines · Andrey Shinkevich
Quoted 36 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #block/stream.c +0
-0Quoted 8 lines
Quoted 7 lines
block/stream.c +1
-1@@ -31,7 +31,8 @@ enum { typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream fromthis) */
+ BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.Quoted 6 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen.
So theabove_basecannot disappear.Quoted 8 lines
Once we insert a filter above the top bs of the stream job, the parallel jobs in
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
I’m not sure whether that’s a problem with this series specifically.
Quoted 6 lines
Quoted 5 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we keep
above_baseand use its immediate child.
Quoted 8 lines
Quoted 5 lines
We can do the same with the
base_overlay.If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
And what if the user manually added some filter above the base (i.e. below
base_overlay) that they want to keep after the job?
It's automatically kept, if we use base_overlay->backing->bs as final backing node.
You mean, that they want it to be dropped?
so, assuming the following:
top -(backing)-> manually-inserted-filter -(file)-> base
and user do stream with base=base, and expects filter to be removed by stream job?
Hmm, yes, such use-case is broken with our proposed way...
====
Let me now clarify the problem we'll have with your way.
When stream don't have any filter, we can easily imagine two parallel stream jobs:
top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base
stream1: top=top, base=mid2
stream2: top=mid2, base=NULL
final picture is obvious:
top (merged with mid1) -(backing)-> mid2 (merged with base)
But we want stream job has own filter, like mirror. So the picture becomes more complex.
Assume stream2 starts first.
top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base
Now, when we run stream1, with your solution, stream1 will freeze stream2-filter (wrong thing, stream2 will fail to remove it if it finished first), and stream1 will remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to removing of its filter)..
But, with our proposed way (freeze only chain up to base_overlay inclusively, and use backing(base_overlay) as final backing), all will work as expected, and two parallel jobs will work..
====
So, these are two mutually exclusive cases.. I vote for freezing up to base_overlay, and use backing(base_overlay) as final backing, because:
1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)
2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:
- Stream job is not intended to remove filters, but to stream data. Filters between base_overlay and base don't contain any data and unrelated to stream process
- I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable. On the contrary, removing all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.
Next, with your way, what about filters, inserted above base during stream job? They will be between above_base and base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.
Quoted 224 lines
Quoted 31 lines
Quoted 24 lines
If we don’t keep
above_base, then we’re basically left guessing as to what should be the backing file after the stream job.Quoted 20 lines
Quoted 12 lines
BlockdevOnError
on_error;
char*backing_file_str;
boolbs_read_only;block/stream.c +1
-1@@ -53,7 +54,7 @@ static void stream_abort(Job *job) if (s->chain_frozen) { BlockJob *bjob = &s->common; - bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->bottom); + bdrv_unfreeze_backing_chain(blk_bs(bjob->blk), s->above_base); } } @@ -62,14 +63,15 @@ static int stream_prepare(Job *job) StreamBlockJob *s = container_of(job, StreamBlockJob,Quoted 6 lines
common.job);
BlockJob *bjob =&s->common;
BlockDriverState *bs =blk_bs(bjob->blk);
- BlockDriverState *base =backing_bs(s->bottom);
+ BlockDriverState*unfiltered_bs=bdrv_skip_filters(bs); + BlockDriverState *base =bdrv_filter_or_cow_bs(s->above_base);The initial base node may be a top node for a concurrent commit job and
Quoted 4 lines
Quoted 1 line
may disappear.
Then it would just be replaced by another node, though, so
above_basekeeps a child. The @base here is not necessarily the initial @base, and that’s intentional.Not really. In my example,
above_basebecomes a danglingQuoted 7 lines
pointer because after the commit job finishes, its filter that should belong to the
commit job frozen chain will be deleted. If we freeze the link to the
above_basefor this job, the iotests #30 will not pass.
So it doesn’t become a dangling pointer, because it’s frozen.
030 passes after this series, so I’m not sure whether I can consider that problem part of this series.
I think if adding a filter node becomes a problem, we have to consider relaxing the restrictions when we do that, not now.
Quoted 153 lines
Quoted 146 lines
Quoted 1 line
base =
bdrv_filter_or_cow_bs(s->base_overlay)is more reliable.But also wrong. The point of keeping
above_basearound is to get its child here to use that child as the new backing child of the top node.Quoted 104 lines
Quoted 65 lines
Error
*local_err= NULL;
int ret = 0;
-bdrv_unfreeze_backing_chain(bs, s->bottom);
+bdrv_unfreeze_backing_chain(bs, s->above_base);
s->chain_frozen= false;
- if (bs->backing) {
+ if (bdrv_cow_child(unfiltered_bs)) {
const char*base_id= NULL,*base_fmt= NULL; if (base) {
base_id=s->backing_file_str;block/stream.c +2
-2@@ -77,8 +79,8 @@ static int stream_prepare(Job *job) base_fmt = base->drv->format_name; } } - bdrv_set_backing_hd(bs, base, &local_err); - ret = bdrv_change_backing_file(bs, base_id, base_fmt); + bdrv_set_backing_hd(unfiltered_bs, base, &local_err); + ret = bdrv_change_backing_file(unfiltered_bs, base_id, base_fmt); if (local_err) { error_report_err(local_err);return
-EPERM;block/stream.c +4
-1@@ -109,14 +111,15 @@ static int coroutine_fn stream_run(Job *job, Error **errp) StreamBlockJob *s = container_of(job, StreamBlockJob, common.job); BlockBackend *blk = s->common.blk; BlockDriverState *bs = blk_bs(blk); - bool enable_cor = !backing_bs(s->bottom); + BlockDriverState *unfiltered_bs = bdrv_skip_filters(bs); + bool enable_cor = !bdrv_cow_child(s->base_overlay); int64_t len; int64_t offset = 0; uint64_t delay_ns = 0; int error = 0; int64_t n = 0; /* bytes */ - if (bs == s->bottom) { + if (unfiltered_bs == s->base_overlay) { /* Nothing to stream */ return 0; }@@ -150,13 +153,14 @@ static int coroutine_fn stream_run(Job *job, Error **errp) copy = false; - ret = bdrv_is_allocated(bs, offset, STREAM_CHUNK, &n); + ret = bdrv_is_allocated(unfiltered_bs, offset, STREAM_CHUNK, &n); if (ret == 1) { /* Allocated in the top, no need to copy. */ } else if (ret >= 0) { /* Copy if allocated in the intermediate images. Limitto the
* known-unallocated area [offset,
offset+n*BDRV_SECTOR_SIZE). */
- ret = bdrv_is_allocated_above(backing_bs(bs),s->bottom, true,
+ ret = bdrv_is_allocated_above(bdrv_cow_bs(unfiltered_bs), +s->base_overlay, true,
offset, n, &n); /* Finish early if end of backing file has been
reached */
if (ret == 0 && n == 0) {block/stream.c +0
-0@@ -223,9 +227,29 @@ void stream_start(const char *job_id,BlockDriverState *bs, BlockDriverState *iter;block/stream.c void stream_start(const char *job_id, · line 237
Quoted 10 lines
block/stream.c +5
-1bool bs_read_only; int basic_flags = BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE_UNCHANGED ; - BlockDriverState *bottom = bdrv_find_overlay(bs, base); + BlockDriverState *base_overlay = bdrv_find_overlay(bs, base); + BlockDriverState *above_base; - if (bdrv_freeze_backing_chain(bs, bottom, errp) < 0) { + if (!base_overlay) { + error_setg(errp, "'%s' is not in the backing chain of '%s'", + base->node_name, bs->node_name);Sorry, I am not clear with the error message.
In this case, there is no an intermediate COW node but the base, if not NULL, is
in the backing chain of bs, isn't it?
Quoted 11 lines
block/stream.c +5
-0+ return; + } + + /* +* Find the node directly above @base. @base_overlay is a COW overlay, so
+ * it must have abdrv_cow_child(), but it is the immediate overlay of
+ * @base, so between the two there can only be filters. + */
+above_base=base_overlay;Quoted 8 lines
+ if (
bdrv_cow_bs(above_base)!= base) {
+above_base=bdrv_cow_bs(above_base);
+ while (bdrv_filter_bs(above_base)!= base) { +above_base=bdrv_filter_bs(above_base); + }
+ }
+
+ if (bdrv_freeze_backing_chain(bs, above_base, errp)< 0) {When a concurrent stream job tries to freeze or remove the
above_basenode,Quoted 4 lines
we will encounter the frozen node error. The
above_basenode is a part of theconcurrent job frozen chain.
Correct.
Quoted 32 lines
Quoted 10 lines
return;
}
@@ -255,14 +279,19 @@ void stream_start(const char*job_id,
BlockDriverState *bs,
* and resizes. Reassign the base node pointer because the
backing BS of the
* bottom node might change after the call tobdrv_reopen_set_read_only()
* due to parallel block jobs running.
+ *above_basenode might change after the call toYes, if not frozen.
Quoted 7 lines
+ *
bdrv_reopen_set_read_only()due to parallel block jobs running.
*/
- base =backing_bs(bottom);
- for (iter =backing_bs(bs); iter && iter != base; iter =backing_bs(iter)) {
+ base =bdrv_filter_or_cow_bs(above_base);
+ for (iter =bdrv_filter_or_cow_bs(bs); iter != base;Quoted 8 lines
+ iter =
bdrv_filter_or_cow_bs(iter))
+ {
block_job_add_bdrv(&s->common, "intermediate node", iter, 0, basic_flags, &error_abort);
}
-s->bottom= bottom;
+s->base_overlay=base_overlay;
+s->above_base=above_base;Generally, being the filter for a concurrent job, the
above_basenode may be deleted any timeand we will keep the dangling pointer. It may happen even earlier if
above_baseis not frozen.Quoted 6 lines
Quoted 1 line
If it is, as it here, we may get the frozen link error then.
I’m not sure what you mean here. Freezing it was absolutely intentional. A dangling pointer would be a problem, but that’s why it’s frozen, so it stays around and can’t be deleted any time.
Max
Quoted 7 lines
The nodes we freeze should be in one context of the relevant job:
filter->top_node->intermediate_node(s)We would not include the base or any filter above it to the frozen chain
because they are of a different job context.
They aren’t really, because we need to know the backing node of @device after the job.
Quoted 7 lines
Once 'this' job is completed, we set the current backing child of the
base_overlayand may not care of its character. If that is another job filter, it will be replaced
with the proper node afterwards.
But what if there are filters above the base that the user wants to keep after the job?
Max
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 60 lines · Andrey Shinkevich
Quoted 48 lines · Max Reitz
Quoted 39 lines · Andrey Shinkevich
Quoted 38 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63
++++++++++++++++++++++++++++++++------------
blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +## @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +0
-0@@ -31,7 +31,8 @@ enum {Quoted 8 lines
Quoted 7 lines
block/stream.c +1
-1typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream fromthis) */
+ BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.Quoted 7 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen.
So theabove_basecannot disappear.Once we insert a filter above the top bs of the stream job, the parallel jobs in
Quoted 7 lines
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
I’m not sure whether that’s a problem with this series specifically.
Quoted 14 lines
Quoted 7 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we
keepabove_baseand use its immediate child.We can do the same with the
base_overlay.If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
Quoted 8 lines
Quoted 2 lines
And what if the user manually added some filter above the base (i.e. below
base_overlay) that they want to keep after the job?It's automatically kept, if we use
base_overlay->backing->bsas final backing node.You mean, that they want it to be dropped?
Er, yes. Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.
Quoted 16 lines
so, assuming the following:
top -(backing)-> manually-inserted-filter -(file)-> base
and user do stream with base=base, and expects filter to be removed by stream job?
Hmm, yes, such use-case is broken with our proposed way...
====
Let me now clarify the problem we'll have with your way.
When stream don't have any filter, we can easily imagine two parallel stream jobs:
Quoted 8 lines
top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base
stream1: top=top, base=mid2
stream2: top=mid2, base=NULLfinal picture is obvious:
top (merged with mid1) -(backing)-> mid2 (merged with base)
Yes, and I don’t think this currently working case is broken by this series.
Quoted 1 line
But we want stream job has own filter, like mirror.
Which it does not have yet, right? Which is why I was saying that I don’t think this is a problem with this series. We could try to address it later.
Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?
Quoted 6 lines
So the picture becomes more complex.
Assume stream2 starts first.
top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base
stream2-filter would be on top of mid2, right?
Quoted 7 lines
Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..
Note that the user first needs to pass “mid2” as the base to the stream job stream1. Why don’t they just pass “stream2-filter”? In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.
I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.
If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.
Quoted 3 lines
But, with our proposed way (freeze only chain up to
base_overlayinclusively, and usebacking(base_overlay)as final backing), all will work as expected, and two parallel jobs will work..
I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.
I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?
Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.
Quoted 9 lines
====
So, these are two mutually exclusive cases.. I vote for freezing up to
base_overlay, and usebacking(base_overlay)as final backing, because:1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)
Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.
You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.
(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)
Quoted 6 lines
2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:
- Stream job is not intended to remove filters, but to stream data. Filters between
base_overlayand base don't contain any data and unrelated to stream process
Well, it is intended to remove nodes. You can only remove data-bearing nodes by moving data around. I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.
Quoted 4 lines
- I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.
Yes.
Although it could be argued that it is a handy way to remove filters, in a backing chain at least. (Thanks to bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)
*shrug*
Quoted 3 lines
On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.
I don’t think so, actually. Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain? Or a COR node, shouldn’t that go above the chain after streaming?
I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here. The only reason to drop them is because that’s what I expect users to expect of the stream job.
Quoted 7 lines
Next, with your way, what about filters, inserted above base during stream job? They will be between
above_baseand base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.
Hm. The documentation seems the same to me. Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.
The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.
OK, so after all this text, maybe some more original problem searching.
I think it the root of the problem is that the stream job takes a @base
parameter, but as of c624b015bf14fe01, it doesn’t really matter anymore.
Maybe c624b015bf14fe01 should have introduced a new parameter for users
to specify the bottom node instead of @base.
Well, that would have made everything a parameter mess, but it would have saved us the trouble now.
In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it. So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.
I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming. (If you do graph modifications during the job, that’s on you, because since c624b015bf14fe01 we can’t keep the base frozen.)
Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change. The problem I have with it is that you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.
I think it would be nice if we could have something that remains truthful to the current documentation. If just ignoring implicit filters above @base would work, then I’d find that nice. If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.
Max
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 129 lines · Max Reitz
Quoted 87 lines · Vladimir Sementsov-Ogievskiy
Quoted 86 lines · Max Reitz
Quoted 60 lines · Andrey Shinkevich
Quoted 48 lines · Max Reitz
Quoted 39 lines · Andrey Shinkevich
Quoted 38 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #block/stream.c +0
-0@@ -31,7 +31,8 @@ enum {Quoted 8 lines
Quoted 7 lines
block/stream.c +1
-1typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream fromthis) */
+ BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.Quoted 7 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen.
So theabove_basecannot disappear.Once we insert a filter above the top bs of the stream job, the parallel jobs in
Quoted 7 lines
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
I’m not sure whether that’s a problem with this series specifically.
Quoted 14 lines
Quoted 7 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we
keepabove_baseand use its immediate child.We can do the same with the
base_overlay.If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
Quoted 8 lines
Quoted 2 lines
And what if the user manually added some filter above the base (i.e. below
base_overlay) that they want to keep after the job?It's automatically kept, if we use
base_overlay->backing->bsas final backing node.You mean, that they want it to be dropped?
Er, yes. Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.
Quoted 16 lines
so, assuming the following:
top -(backing)-> manually-inserted-filter -(file)-> base
and user do stream with base=base, and expects filter to be removed by stream job?
Hmm, yes, such use-case is broken with our proposed way...
====
Let me now clarify the problem we'll have with your way.
When stream don't have any filter, we can easily imagine two parallel stream jobs:
Quoted 8 lines
top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base
stream1: top=top, base=mid2
stream2: top=mid2, base=NULLfinal picture is obvious:
top (merged with mid1) -(backing)-> mid2 (merged with base)
Yes, and I don’t think this currently working case is broken by this series.
Quoted 1 line
But we want stream job has own filter, like mirror.
Quoted 7 lines
Which it does not have yet, right? Which is why I was saying that I don’t think this is a problem with this series. We could try to address it later.
Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?
I think, we can address it later, but it would be good to fit into one release cycle with these series, to not make incompatible behavior changes later.
Quoted 9 lines
Quoted 7 lines
So the picture becomes more complex.
Assume stream2 starts first.
top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base
stream2-filter would be on top of mid2, right?
Right. In my picture, "-(backing)->" means backing link. Hmm, most probably stream-filter is COR, which actually have file child. It doesn't matter here.
Quoted 9 lines
Quoted 8 lines
Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..
Quoted 4 lines
Note that the user first needs to pass “mid2” as the base to the stream job stream1. Why don’t they just pass “stream2-filter”? In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.
Hmm. I'm sure we already tried/discussed this.. But I don't remember the results. Seems, such logic should work as well. Andrey don't you remember? I think, the only difficulty is to update iotests 30 (in our stream-filter series) to use node-names, not filename.. Can we go this way?
Quoted 4 lines
I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.
Agree. I think stream-filter should be explicit, like recently introduced backup-top.
Quoted 6 lines
If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.
Quoted 8 lines
Quoted 3 lines
But, with our proposed way (freeze only chain up to
base_overlayinclusively, and usebacking(base_overlay)as final backing), all will work as expected, and two parallel jobs will work..I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.
But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.
Quoted 3 lines
I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?
I don't, but I like the idea :)
Quoted 4 lines
Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.
And again, this is impossible even with your way. I have an idea:
What about making the whole thing explicit?
We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.
Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)). Benefit: user have guarantee of what would be final backing node.
Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.
But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..
Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying exact node-names in it?..
Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.
What do you think of it? Could we break this scenario in one release without deprecation and don't care? Than I think my idea about base vs bottom-node arguments for stream job may be applied. Or what to do?
If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.
Quoted 16 lines
Quoted 10 lines
====
So, these are two mutually exclusive cases.. I vote for freezing up to
base_overlay, and usebacking(base_overlay)as final backing, because:1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)
Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.
Quoted 5 lines
You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.
(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)
We can break existing user scenarios.. Seems unlikely. I don't know.
Quoted 61 lines
Quoted 7 lines
2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:
- Stream job is not intended to remove filters, but to stream data.
Filters betweenbase_overlayand base don't contain any data and unrelated to stream processWell, it is intended to remove nodes. You can only remove data-bearing nodes by moving data around. I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.
Quoted 4 lines
- I think, that filters are "more related" to their children than to
their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.Yes.
Although it could be argued that it is a handy way to remove filters, in a backing chain at least. (Thanks to
bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)*shrug*
Quoted 3 lines
On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.I don’t think so, actually. Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain? Or a COR node, shouldn’t that go above the chain after streaming?
I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here. The only reason to drop them is because that’s what I expect users to expect of the stream job.
Quoted 7 lines
Next, with your way, what about filters, inserted above base during stream job? They will be between
above_baseand base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.Hm. The documentation seems the same to me. Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.
The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.
Quoted 5 lines
OK, so after all this text, maybe some more original problem searching.
I think it the root of the problem is that the stream job takes a @base
parameter, but as ofc624b015bf14fe01, it doesn’t really matter anymore.
Maybec624b015bf14fe01should have introduced a new parameter for users
to specify the bottom node instead of @base.
Yes, absolutely agree. If we do it now, would it be incompatible change or not?
Quoted 16 lines
Well, that would have made everything a parameter mess, but it would have saved us the trouble now.
In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it. So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.
I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming. (If you do graph modifications during the job, that’s on you, because since
c624b015bf14fe01we can’t keep the base frozen.)
Quoted 5 lines
Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change.
I also try to think about future introduction of stream filter. It shouldn't be a breaking change as well. But now I think it can be done the hard-way if needed: just work without a filter if filter-node-name is not given. but probably I care to much. Finally we can just drop the test-case from 030 or insert additional intermediate node into it..
Quoted 8 lines
The problem I have with it is that
you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.I think it would be nice if we could have something that remains truthful to the current documentation.
Then, we should freeze base node again, so just revert c624b015bf14fe01
I go and see it cover-letter of the series introduced c624b015bf14fe01:
This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job and can change before the stream job is completed. We avoid that dependency by introducing the bottom node.
Hmm. Bad that we didn't add an iotest, but the series solved existing problem:
parallel stream and commit, as commit already has filter. Commit has filter with "backing" child, so, I assume that the case worked prior to introducing frozen chains, broken then by frozen chains and then fixed by c624b015bf14fe01. Still I don't know are there some real users of such parallel jobs..
Quoted 5 lines
If just ignoring implicit
filters above @base would work, then I’d find that nice. If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.
OK, at this point, I think, I'm OK with your patch in context of these series.
But I fill that something more should be done. Could we just revert c624b015bf14fe01 as "non corresponding to specification"?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 211 lines · Vladimir Sementsov-Ogievskiy
Quoted 131 lines · Max Reitz
Quoted 89 lines · Vladimir Sementsov-Ogievskiy
Quoted 88 lines · Max Reitz
Quoted 62 lines · Andrey Shinkevich
Quoted 50 lines · Max Reitz
Quoted 41 lines · Andrey Shinkevich
Quoted 40 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63
++++++++++++++++++++++++++++++++------------
blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +## @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +0
-0@@ -31,7 +31,8 @@ enum {Quoted 8 lines
Quoted 7 lines
block/stream.c +1
-1typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream fromthis) */
+ BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.Quoted 7 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen.
So theabove_basecannot disappear.Once we insert a filter above the top bs of the stream job, the parallel jobs in
Quoted 7 lines
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
I’m not sure whether that’s a problem with this series specifically.
Quoted 14 lines
Quoted 7 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we
keepabove_baseand use its immediate child.We can do the same with the
base_overlay.If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
Quoted 8 lines
Quoted 2 lines
And what if the user manually added some filter above the base (i.e. below
base_overlay) that they want to keep after the job?It's automatically kept, if we use
base_overlay->backing->bsas final backing node.You mean, that they want it to be dropped?
Er, yes. Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.
Quoted 16 lines
so, assuming the following:
top -(backing)-> manually-inserted-filter -(file)-> base
and user do stream with base=base, and expects filter to be removed by stream job?
Hmm, yes, such use-case is broken with our proposed way...
====
Let me now clarify the problem we'll have with your way.
When stream don't have any filter, we can easily imagine two parallel stream jobs:
Quoted 8 lines
top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base
stream1: top=top, base=mid2
stream2: top=mid2, base=NULLfinal picture is obvious:
top (merged with mid1) -(backing)-> mid2 (merged with base)
Yes, and I don’t think this currently working case is broken by this series.
Quoted 1 line
But we want stream job has own filter, like mirror.
Quoted 7 lines
Which it does not have yet, right? Which is why I was saying that I don’t think this is a problem with this series. We could try to address it later.
Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?
I think, we can address it later, but it would be good to fit into one release cycle with these series, to not make incompatible behavior changes later.
Quoted 9 lines
Quoted 7 lines
So the picture becomes more complex.
Assume stream2 starts first.
top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base
stream2-filter would be on top of mid2, right?
Right. In my picture, "-(backing)->" means backing link. Hmm, most probably stream-filter is COR, which actually have file child. It doesn't matter here.
Quoted 9 lines
Quoted 8 lines
Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..Quoted 4 lines
Note that the user first needs to pass “mid2” as the base to the stream job stream1. Why don’t they just pass “stream2-filter”? In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.
Hmm. I'm sure we already tried/discussed this.. But I don't remember the results. Seems, such logic should work as well. Andrey don't you remember? I think, the only difficulty is to update iotests 30 (in our stream-filter series) to use node-names, not filename.. Can we go this way?
Quoted 4 lines
I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.
Agree. I think stream-filter should be explicit, like recently introduced backup-top.
Quoted 6 lines
If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.
Quoted 8 lines
Quoted 3 lines
But, with our proposed way (freeze only chain up to
base_overlayinclusively, and usebacking(base_overlay)as final backing), all will work as expected, and two parallel jobs will work..I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.
But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.
Quoted 3 lines
I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?
I don't, but I like the idea :)
Quoted 4 lines
Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.
And again, this is impossible even with your way. I have an idea:
What about making the whole thing explicit?
We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.
Quoted 8 lines
Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)). Benefit: user have guarantee of what would be final backing node.
Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.
But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..
Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying exact node-names in it?..
That is what I am doing now but have not completed yet. Actually, when we insert the COR-filter for a stream job, the cases with concurrent jobs fail in the iotests:030. I am trying to cope with the issues and still in progress. The problem I am forseeing is that if a user specifies the filter node as a base for a job, the name of the filter will be written down to the QCOW2 header on the disk. It can happen if bdrv_refresh_filename() returns the json-name of the filter. Max separated the backing file of QCOW2 header from the one, let's say, in the running program with this series. Imagine, If the user doesn't specify the backing file in a QAPI command, the program will use the name of the filter read from the QCOW2 header. Let me please finish the experiment and I will back to you with results or issues...
Andrey
Quoted 162 lines
Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.
What do you think of it? Could we break this scenario in one release without deprecation and don't care? Than I think my idea about base vs bottom-node arguments for stream job may be applied. Or what to do?
If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.
Quoted 16 lines
Quoted 10 lines
====
So, these are two mutually exclusive cases.. I vote for freezing up to
base_overlay, and usebacking(base_overlay)as final backing, because:1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)
Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.
Quoted 5 lines
You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.
(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)
We can break existing user scenarios.. Seems unlikely. I don't know.
Quoted 61 lines
Quoted 7 lines
2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:
- Stream job is not intended to remove filters, but to stream data. Filters between
base_overlayand base don't contain any data and unrelated to stream processWell, it is intended to remove nodes. You can only remove data-bearing nodes by moving data around. I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.
Quoted 4 lines
- I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.
Yes.
Although it could be argued that it is a handy way to remove filters, in a backing chain at least. (Thanks to
bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)*shrug*
Quoted 3 lines
On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.I don’t think so, actually. Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain? Or a COR node, shouldn’t that go above the chain after streaming?
I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here. The only reason to drop them is because that’s what I expect users to expect of the stream job.
Quoted 7 lines
Next, with your way, what about filters, inserted above base during stream job? They will be between
above_baseand base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.Hm. The documentation seems the same to me. Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.
The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.
Quoted 6 lines
OK, so after all this text, maybe some more original problem searching. I think it the root of the problem is that the stream job takes a @base
parameter, but as ofc624b015bf14fe01, it doesn’t really matter anymore. Maybec624b015bf14fe01should have introduced a new parameter for users to specify the bottom node instead of @base.Yes, absolutely agree. If we do it now, would it be incompatible change or not?
Quoted 16 lines
Well, that would have made everything a parameter mess, but it would have saved us the trouble now.
In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it. So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.
I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming. (If you do graph modifications during the job, that’s on you, because since
c624b015bf14fe01we can’t keep the base frozen.)Quoted 5 lines
Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change.
I also try to think about future introduction of stream filter. It shouldn't be a breaking change as well. But now I think it can be done the hard-way if needed: just work without a filter if filter-node-name is not given. but probably I care to much. Finally we can just drop the test-case from 030 or insert additional intermediate node into it..
Quoted 8 lines
The problem I have with it is that
you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.I think it would be nice if we could have something that remains truthful to the current documentation.
Then, we should freeze base node again, so just revert
c624b015bf14fe01I go and see it cover-letter of the series introduced
c624b015bf14fe01:This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job and can change before the stream job is completed. We avoid that dependency by introducing the bottom node.
Hmm. Bad that we didn't add an iotest, but the series solved existing problem:
parallel stream and commit, as commit already has filter. Commit has filter with "backing" child, so, I assume that the case worked prior to introducing frozen chains, broken then by frozen chains and then fixed byc624b015bf14fe01. Still I don't know are there some real users of such parallel jobs..Quoted 5 lines
If just ignoring implicit
filters above @base would work, then I’d find that nice. If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.OK, at this point, I think, I'm OK with your patch in context of these series.
But I fill that something more should be done. Could we just revert
c624b015bf14fe01as "non corresponding to specification"?
Andrey Shinkevich
Reviewed-byRe: [PATCH v7 14/47] stream: Deal with filters
In reply to Vladimir Sementsov-OgievskiyQuoted 381 lines · Vladimir Sementsov-Ogievskiy
Quoted 131 lines · Max Reitz
Quoted 89 lines · Vladimir Sementsov-Ogievskiy
Quoted 88 lines · Max Reitz
Quoted 62 lines · Andrey Shinkevich
Quoted 50 lines · Max Reitz
Quoted 41 lines · Andrey Shinkevich
Quoted 40 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job
independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63
++++++++++++++++++++++++++++++++------------
blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasqapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +## @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
block/stream.c +0
-0@@ -31,7 +31,8 @@ enum {Quoted 8 lines
Quoted 7 lines
block/stream.c +1
-1typedef struct StreamBlockJob { BlockJob common; - BlockDriverState *bottom; + BlockDriverState *base_overlay; /* COW overlay (stream fromthis) */
+ BlockDriverState *above_base; /* Node directly above the base */Keeping the
base_overlayis enough to complete the stream job.Quoted 7 lines
Depends on the definition. If we decide it isn’t enough, then it isn’t enough.
Quoted 1 line
The
above_basemay disappear during the job and we can't rely on it.In this version of this series, it may not, because the chain is frozen.
So theabove_basecannot disappear.Once we insert a filter above the top bs of the stream job, the parallel jobs in
Quoted 7 lines
the iotests #030 will fail with 'frozen link error'. It is because of the
independent parallel stream or commit jobs that insert/remove their filters
asynchroniously.
I’m not sure whether that’s a problem with this series specifically.
Quoted 14 lines
Quoted 7 lines
We can discuss whether we should allow it to disappear, but I think not.
The problem is, we need something to set as the backing file after streaming. How do we figure out what that should be? My proposal is we
keepabove_baseand use its immediate child.We can do the same with the
base_overlay.If the backing node turns out to be a filter, the proper backing child will
be set after the filter is removed. So, we shouldn't care.
Quoted 8 lines
Quoted 2 lines
And what if the user manually added some filter above the base (i.e. below
base_overlay) that they want to keep after the job?It's automatically kept, if we use
base_overlay->backing->bsas final backing node.You mean, that they want it to be dropped?
Er, yes. Point is, the graph structure below with @base at the root may be different than the one right below @base_overlay.
Quoted 16 lines
so, assuming the following:
top -(backing)-> manually-inserted-filter -(file)-> base
and user do stream with base=base, and expects filter to be removed by stream job?
Hmm, yes, such use-case is broken with our proposed way...
====
Let me now clarify the problem we'll have with your way.
When stream don't have any filter, we can easily imagine two parallel stream jobs:
Quoted 8 lines
top -(backing)-> mid1 -(backing)-> mid2 -(backing)-> base
stream1: top=top, base=mid2
stream2: top=mid2, base=NULLfinal picture is obvious:
top (merged with mid1) -(backing)-> mid2 (merged with base)
Yes, and I don’t think this currently working case is broken by this series.
Quoted 1 line
But we want stream job has own filter, like mirror.
Quoted 7 lines
Which it does not have yet, right? Which is why I was saying that I don’t think this is a problem with this series. We could try to address it later.
Or do you think we can’t address it later because right now all filter cases are broken anyway so now would be the time to make a breaking change (which the suggestion to not use @base as the final backing node is)?
I think, we can address it later, but it would be good to fit into one release cycle with these series, to not make incompatible behavior changes later.
Quoted 9 lines
Quoted 7 lines
So the picture becomes more complex.
Assume stream2 starts first.
top -(backing)-> mid1 -(backing)-> stream2-filter -(backing)-> mid2 -(backing)-> base
stream2-filter would be on top of mid2, right?
Right. In my picture, "-(backing)->" means backing link. Hmm, most probably stream-filter is COR, which actually have file child. It doesn't matter here.
Quoted 9 lines
Quoted 8 lines
Now, when we run stream1, with your solution, stream1 will freeze stream2-filter
(wrong thing, stream2 will fail to remove it if it finished first), and stream1 will
remove stream2-filter on finish (which is wrong as well, stream2 is not prepared to
removing of its filter)..Quoted 4 lines
Note that the user first needs to pass “mid2” as the base to the stream job stream1. Why don’t they just pass “stream2-filter”? In my model, the user should specify exactly which node they want not to be touched by this stream job, and so that would be stream2-filter, not mid1.
Hmm. I'm sure we already tried/discussed this.. But I don't remember the results. Seems, such logic should work as well. Andrey don't you remember? I think, the only difficulty is to update iotests 30 (in our stream-filter series) to use node-names, not filename.. Can we go this way?
Quoted 4 lines
I feel like the answer to this question has to do with implicit nodes. AFAIU you wanted to remove them, so I don’t think we’d want to special-case them here.
Agree. I think stream-filter should be explicit, like recently introduced backup-top.
Quoted 6 lines
If you think that we can’t expect users to pass “stream2-filter” because currently it should work with “mid2”, then that’s a case of implicit nodes and it means we should ascend from @base up to the first non-implicit node to get the @above_base we want.
Quoted 8 lines
Quoted 3 lines
But, with our proposed way (freeze only chain up to
base_overlayinclusively, and usebacking(base_overlay)as final backing), all will work as expected, and two parallel jobs will work..I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.
But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.
Quoted 3 lines
I suppose you’re arguing that streaming through filters basically just doesn’t work at all right now, so we’re free to do whatever?
I don't, but I like the idea :)
Quoted 4 lines
Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.
And again, this is impossible even with your way. I have an idea:
What about making the whole thing explicit?
We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.
Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)). Benefit: user have guarantee of what would be final backing node.
Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.
But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..
Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying exact node-names in it?..
Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.
What do you think of it? Could we break this scenario in one release without deprecation and don't care? Than I think my idea about base vs bottom-node arguments for stream job may be applied. Or what to do?
If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.
Quoted 16 lines
Quoted 10 lines
====
So, these are two mutually exclusive cases.. I vote for freezing up to
base_overlay, and usebacking(base_overlay)as final backing, because:1. I can't imaging other way to fix the case with parallel streams with filters (it's not a problem of current master, but we have pending series which will introduce stream job filter, and the problem will appear and even break iotest 30)
Besides the question of whether the top job could just use the bottom job’s filter node as the base, there’s also the alternative of admitting defeat and declaring that you just cannot use a single node in two streams, because we didn’t find a way to make it work after all.
Quoted 5 lines
You could still create a temporary overlay in between that’s never used and then drop it with a trivial stream afterwards.
(But that just in case specifying the bottom job’s stream node somehow wouldn’t work.)
We can break existing user scenarios.. Seems unlikely. I don't know.
Quoted 61 lines
Quoted 7 lines
2. I don't think that removing filters above base node by stream job is so important case to break parallel stream jobs in future:
- Stream job is not intended to remove filters, but to stream data. Filters between
base_overlayand base don't contain any data and unrelated to stream processWell, it is intended to remove nodes. You can only remove data-bearing nodes by moving data around. I suspect if there was a way to get the to-be-removed nodes removed without having to move their data around, that would be popular.
Quoted 4 lines
- I think, that filters are "more related" to their children than to their parents. So, removing filters related to base node, when we just remove all data-containing nodes between top and base (and are not going to remove base node) is at least questionable.
Yes.
Although it could be argued that it is a handy way to remove filters, in a backing chain at least. (Thanks to
bdrv_find_overlay(), @base and @top still need to refer to different levels of the backing chain, but if we lifted that restriction, I suppose it could work for any filter chain.)*shrug*
Quoted 3 lines
On the contrary, removing
all intermediate data containing nodes _together_ with their filters is absolutely correct thing to do.I don’t think so, actually. Like, you have a throttle node somewhere in the chain, shouldn’t you maybe want to move it down below the chain? Or a COR node, shouldn’t that go above the chain after streaming?
I’m not making an argument here, I just don’t quite understand why you’d bring up what happens with intermediate filters here. The only reason to drop them is because that’s what I expect users to expect of the stream job.
Quoted 7 lines
Next, with your way, what about filters, inserted above base during stream job? They will be between
above_baseand base, and will not be removed. So with your way, filters above base, existing before job start will be frozen during the job and removed after it, but filters appended above base during the job will be untouched. With our way, just all base node related filters are untouched by the job. It seems simpler definition for me and simpler to document.Hm. The documentation seems the same to me. Either it’s “The backing node (at the end of the job) of @base’s parent node (when starting the job)” or “The backing node (at the end of the job) of the next non-filter node above @base (when starting the job)”.
The problem you describe (that @above_base at the end of the job isn’t necessarily above @base anymore) also exists with your suggestion, namely that you can add overlays above @base after the job has started, so @base_overlay at the end of the job isn’t necessarily the first non-filter node above @base anymore.
Quoted 6 lines
OK, so after all this text, maybe some more original problem searching. I think it the root of the problem is that the stream job takes a @base
parameter, but as ofc624b015bf14fe01, it doesn’t really matter anymore. Maybec624b015bf14fe01should have introduced a new parameter for users to specify the bottom node instead of @base.Yes, absolutely agree. If we do it now, would it be incompatible change or not?
Quoted 16 lines
Well, that would have made everything a parameter mess, but it would have saved us the trouble now.
In any case, the problem we have now is that we want a way to automagically find out which node the bottom node should be, because the user can’t specify it. So the documentation is always going to be written as “The backing node (at the end of the job) of $bottom”, where “$bottom” is what we’re interested in figuring out.
I thought it would be best if we stick as close as possible to the spirit of the current documentation, which basically requires @base-node to be the backing node of the top after streaming. (If you do graph modifications during the job, that’s on you, because since
c624b015bf14fe01we can’t keep the base frozen.)Quoted 5 lines
Your suggestion to do basically what you consider to be right, which comes at the caveat of being untruthful to the current documentation even if there are no graph modifications during the job. Luckily, the stream job right now doesn’t work in the cases we’re looking at, so it wouldn’t be a breaking change.
I also try to think about future introduction of stream filter. It shouldn't be a breaking change as well. But now I think it can be done the hard-way if needed: just work without a filter if filter-node-name is not given. but probably I care to much. Finally we can just drop the test-case from 030 or insert additional intermediate node into it..
Quoted 8 lines
The problem I have with it is that
you’re assuming what is right and what isn’t (i.e. “Who would want to remove filter nodes directly above @base”) without giving the user a chance to specify.I think it would be nice if we could have something that remains truthful to the current documentation.
Then, we should freeze base node again, so just revert
c624b015bf14fe01I go and see it cover-letter of the series introduced
c624b015bf14fe01:This series introduces a bottom intermediate node that eliminates the dependency on the base that may change while stream job is running. It happens when stream/commit parallel jobs are running on the same backing chain. The base node of the stream job may be a top node of the parallel commit job and can change before the stream job is completed. We avoid that dependency by introducing the bottom node.
Hmm. Bad that we didn't add an iotest, but the series solved existing problem:
parallel stream and commit, as commit already has filter. Commit has filter with "backing" child, so, I assume that the case worked prior to introducing frozen chains, broken then by frozen chains and then fixed byc624b015bf14fe01. Still I don't know are there some real users of such parallel jobs..Quoted 5 lines
If just ignoring implicit
filters above @base would work, then I’d find that nice. If it doesn’t, I suppose there’s indeed little we can do but to indeed forego @above_base and just use @base_overlay for all cases.OK, at this point, I think, I'm OK with your patch in context of these series.
But I fill that something more should be done. Could we just revert
c624b015bf14fe01as "non corresponding to specification"?
Re: [PATCH v7 14/47] stream: Deal with filters
In reply to Vladimir Sementsov-OgievskiyQuoted 2 lines · Vladimir Sementsov-Ogievskiy
Quoted 1 line · Max Reitz
On 07.08.20 12:29, Vladimir Sementsov-Ogievskiy wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
But, with our proposed way (freeze only chain up to
base_overlayinclusively, and usebacking(base_overlay)as final backing), all will work as expected, and two parallel jobs will work..
Quoted 8 lines
Quoted 4 lines
I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.
But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.
Yeah, but after c624b015bf, that’s just how it is. I thought the best would be an approach where if there are no graph changes during the job, you would indeed end up with @base as the backing file afterwards.
[...]
Quoted 8 lines
Quoted 3 lines
Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.
And again, this is impossible even with your way. I have an idea:
What about making the whole thing explicit?
Quoted 8 lines
We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.
Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)).
Benefit: user have guarantee of what would be final backing node.
Sounds very nice to me, but...
Quoted 7 lines
Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.
But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..
...yeah. Hm. What a pain, but right, we can just deprecate it.
Unfortunately, I don’t think there’s any way we could issue a warning (we’d want to deprecate using the @base node in something outside of the stream job, and we can’t really detect this case to issue a warning). So it would be a deprecation found only in the deprecation notes and the QAPI spec...
Quoted 6 lines
Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying exact node-names in it?..
Quoted 8 lines
Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.
What do you think of it? Could we break this scenario in one release without deprecation and don't care?
I don’t know, but I’m afraid I don’t think we can.
Quoted 11 lines
Than I think my idea about base vs
bottom-node arguments for stream job may be applied. Or what to do?If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.
If that works...
OK. So what I think we can do is first just take this patch as part of this series. Then, we add @bottom-node separately and deprecate @base not being frozen.
If it’s feasible to not add a stream filter node until the deprecation period is over, then that should work.
Max
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 52 lines · Max Reitz
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Quoted 1 line · Max Reitz
On 07.08.20 12:29, Vladimir Sementsov-Ogievskiy wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
But, with our proposed way (freeze only chain up to
base_overlayinclusively, and usebacking(base_overlay)as final backing), all will work as expected, and two parallel jobs will work..Quoted 8 lines
Quoted 4 lines
I don’t think it will work as expected because users can no longer specify which node should be the base node after streaming. And the QAPI schema says that base-node is to become the backing file of the top node after streaming.
But this will never work with either way: base node may disappear during stream. Even with you way, they only stable thing is "above-base", which backing child may be completely another node at stream finish.
Yeah, but after
c624b015bf, that’s just how it is. I thought the best would be an approach where if there are no graph changes during the job, you would indeed end up with @base as the backing file afterwards.[...]
Quoted 8 lines
Quoted 3 lines
Well, that still leaves the problem that users should be able to specify which node is to become the base after streaming, and that that node maybe shouldn’t be restricted to immediate children of COW images.
And again, this is impossible even with your way. I have an idea:
What about making the whole thing explicit?
Quoted 8 lines
We add an optional parameter to stream-job: bottom-node, which is mutally exclusive with specifying base.
Then, if user specified base node, we freeze base as well, so it can't disappear. User will not be able to start parallel stream with this base node as top (because new stream can not insert a filter into frozen chain), but for sure it's rare case, used only in iotest 30 :)).
Benefit: user have guarantee of what would be final backing node.Sounds very nice to me, but...
Quoted 7 lines
Otherwise, if user specified bottom-node, we use the way of this patch. So user can run parallel streams (iotest 30 will have to use bottom-node argument). No guarantee of final base-node, it would be backing of bottom-node at job finish.
But, this is incompatible change, and we probably should wait for 3 releases for deprecation of old behavior..
Quoted 7 lines
...yeah. Hm. What a pain, but right, we can just deprecate it.
Unfortunately, I don’t think there’s any way we could issue a warning (we’d want to deprecate using the @base node in something outside of the stream job, and we can’t really detect this case to issue a warning). So it would be a deprecation found only in the deprecation notes and the QAPI spec...
Hmm.. add "bool frozen_only_warn" field to BdrvChild and print warning instead of fail where c->frozen cause failure?
I can make a patch, if you don't think that its too much.
Quoted 13 lines
Quoted 13 lines
Anyway, I feel now, that you convinced me. I'm not sure that we will not have to change it make filter work, but not reason to change something now. Andrey, could you try to rebase your series on top of this and fix iotest 30 by just specifying exact node-names in it?..
Hmmm. My thought goes further. Seems, that in this way, introducing explicit filter would be incompatible change anyway: it will break scenario with parallel stream jobs, when user specifies filenames, not node names (user will have to specify filter-node name as base for another stream job, as you said). So, it's incompatible anyway.
Quoted 4 lines
Quoted 2 lines
What do you think of it? Could we break this scenario in one release without deprecation and don't care?
I don’t know, but I’m afraid I don’t think we can.
Actually, we already done so once:
Freezing of base was introduced in 4.0, when c624b015bf comes in 4.1 (there were no real user bugs or feature requests, it was just my [bad] idea, related to the problem around our stream-filter and parallel jobs in iotest 30).
So, at least for 4.0 we had frozen base.. Were there any bugs? But now we live with not-frozen base for several releases..
Quoted 13 lines
Quoted 4 lines
Than I think my idea about base vs
bottom-node arguments for stream job may be applied. Or what to do?Quoted 8 lines
If we can't break this scenario without a deprecation, we'll have to implement "implicit" filter, like for mirror, when filter-node-name is not specified. And for this implicit filter we'll need additional logic (closer to what I've proposed in a previous mail). Or, try to keep stream without a filter (not insert it at all and behave the old way), when filter-node-name is not specified. Than new features based on filter will be available only when filter-node-name is specified, but this is OK. The latter seems better for me.
Quoted 5 lines
If that works...
OK. So what I think we can do is first just take this patch as part of this series.
Yes, let's start with it.
Quoted 2 lines
Then, we add @bottom-node separately and deprecate @base
not being frozen.
I think we can just deprecate, and not add the bottom-node. If someone will come during deprecation period and say that he have such use-case, we'll add @bottom-node. Otherwise we'll just start to freeze base node again.
Quoted 6 lines
If it’s feasible to not add a stream filter node until the deprecation period is over, then that should work.
Max
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 14/47] stream: Deal with filters
In reply to Max ReitzQuoted 18 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0
qapi/block-core.json line 2490
Quoted 5 lines
qapi/block-core.json +2
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was
Forgot one thing. To me, it would be more understandable to read
"...to point to the base as backing node..." because it may be thought as a backing
node of the base.
Andrey
Quoted 5 lines
qapi/block-core.json +2
-0+# not specified) instead of modifying @device itself. +# # @job-id: identifier for the newly-created block job. If # omitted, the device name will be used. (Since 2.7) #
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 30 lines · Andrey Shinkevich
Quoted 20 lines · Max Reitz
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 4 +++
block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++-3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop thebacking file
Quoted 6 lines
# and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base wasForgot one thing. To me, it would be more understandable to read
Quoted 4 lines
"...to point to the base as backing node..." because it may be thought as a backing
node of the base.
This doesn’t sound like it’s about understandability; “point to the base as backing node” and “point to base’s backing node [as backing node]” are semantically different.
Was my phrasing just wrong? @base should be the backing node, so yours seems correct.
Max
Re: [PATCH v7 14/47] stream: Deal with filters
In reply to Max ReitzAm 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 18 lines
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)qapi/block-core.json +0
-0
qapi/block-core.json line 2491
Quoted 6 lines
qapi/block-core.json +3
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself.
Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).
Should we also document what using a filter node for @base means?
The code changes look good.
Kevin
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 14 lines · Kevin Wolf
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 13 lines
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)
Quoted 15 lines
qapi/block-core.json line 2491
Quoted 11 lines
qapi/block-core.json +3
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself.Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).
Oh, yes.
(I thought I had noticed that already at some point and fixed it locally... But apparently not.)
Quoted 1 line
Should we also document what using a filter node for @base means?
Hm. What does it mean? I think the more interesting case is what it means if above_base is a filter, right?
Maybe we can put in somewhere in the “If a base file is specified then sectors are not copied from that base file and its backing chain.” But the more I think about it, the less I know what we could add to it. What happens if there are filters above @base is that their data isn’t copied, because that’s exactly the data in @base.
Max
Re: [PATCH v7 14/47] stream: Deal with filters
Am 19.08.2020 um 16:47 hat Max Reitz geschrieben:
Quoted 38 lines · Kevin Wolf
Quoted 14 lines
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 13 lines
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)Quoted 15 lines
qapi/block-core.json line 2491
Quoted 11 lines
qapi/block-core.json +3
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself.Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).
Oh, yes.
(I thought I had noticed that already at some point and fixed it locally... But apparently not.)
Quoted 1 line
Should we also document what using a filter node for @base means?
Quoted 8 lines
Hm. What does it mean? I think the more interesting case is what it means if
above_baseis a filter, right?Maybe we can put in somewhere in the “If a base file is specified then sectors are not copied from that base file and its backing chain.” But the more I think about it, the less I know what we could add to it. What happens if there are filters above @base is that their data isn’t copied, because that’s exactly the data in @base.
The interesting part is probably the graph reconfiguration at the end of the job. Which is actually already documented:
# When streaming completes the image file will have the base
# file as its backing file.
Of course, this is not entirely correct any more (because the base may have changed).
If @base is a filter, what backing file path do we write into the top layer? A json: filename including the filter? Is this worth mentioning or do you consider it obvious?
Kevin
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 51 lines · Kevin Wolf
Am 19.08.2020 um 16:47 hat Max Reitz geschrieben:
Quoted 38 lines · Kevin Wolf
Quoted 14 lines
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 13 lines
Because of the (not so recent anymore) changes that make the stream job independent of the base node and instead track the node above it, we have to split that "bottom" node into two cases: The bottom COW node, and the node directly above the base node (which may be an R/W filter or the bottom COW node).
Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 4 +++ block/stream.c | 63 ++++++++++++++++++++++++++++++++------------ blockdev.c | 4 ++- 3 files changed, 53 insertions(+), 18 deletions(-)Quoted 15 lines
qapi/block-core.json line 2491
Quoted 11 lines
qapi/block-core.json +3
-0@@ -2486,6 +2486,10 @@ # On successful completion the image file is updated to drop the backing file # and the BLOCK_JOB_COMPLETED event is emitted. # +# In case @device is a filter node, block-stream modifies the first non-filter +# overlay node below it to point to base's backing node (or NULL if @base was +# not specified) instead of modifying @device itself.Not to @base's backing node, but to @base itself (or actually, to above_base's backing node, which is initially @base, but may have changed when the job is completed).
Oh, yes.
(I thought I had noticed that already at some point and fixed it locally... But apparently not.)
Quoted 1 line
Should we also document what using a filter node for @base means?
Quoted 8 lines
Hm. What does it mean? I think the more interesting case is what it means if
above_baseis a filter, right?Maybe we can put in somewhere in the “If a base file is specified then sectors are not copied from that base file and its backing chain.” But the more I think about it, the less I know what we could add to it. What happens if there are filters above @base is that their data isn’t copied, because that’s exactly the data in @base.
The interesting part is probably the graph reconfiguration at the end of the job. Which is actually already documented:
Quoted 8 lines
# When streaming completes the image file will have the base
# file as its backing file.Of course, this is not entirely correct any more (because the base may have changed).
If @base is a filter, what backing file path do we write into the top layer? A json: filename including the filter?
Yes.
Or, actually. Now that I read the code... It takes @base’s filename before the stream job and then uses that. So if @base has changed during the job, then it still uses the old filename.
But that’s not really due to this series.
Quoted 2 lines
Is this worth mentioning
or do you consider it obvious?
Hm. I consider it obvious, yes. @base becomes @top’s backing file (at least without any graph changes while the job is running), so naturally what’s written into the image header is @base’s filename – which is a json:{} filename.
On second thought, @backing-file’s description mysteriously states that “QEMU will automatically determine the backing file string to use”. Which makes sense because it would clearly not make sense to describe what’s actually happening, which is to use @base’s filename at job start regardless of whether it’s still there at the end of the job.
So I suppose I have the choice of either documenting exactly what’s happening, even though it doesn’t make much sense, or just not, keeping it mysterious.
So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)
The more I think about it, the more I think there are problems beyond the scope of this series here. :/
Max
Re: [PATCH v7 14/47] stream: Deal with filters
On 20.08.20 10:31, Max Reitz wrote:
[...]
Quoted 4 lines
So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)
(I tried triggering that, but, oh, it’s strdup’ed() in stream_start(). I’m a bit daft.)
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 4 lines · Max Reitz
On 20.08.20 10:31, Max Reitz wrote:
[...]
Quoted 8 lines
Quoted 4 lines
So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)
(I tried triggering that, but, oh, it’s strdup’ed() in
stream_start(). I’m a bit daft.)
If it's broken anyway, probably we can just revert c624b015bf and start to freeze base again?
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v7 14/47] stream: Deal with filters
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Max Reitz
On 20.08.20 10:31, Max Reitz wrote:
[...]
Quoted 4 lines
So all in all, I believe the biggest surprise about what’s written into the top layer isn’t that it may be a json:{} filename, but the filename of a node that maybe doesn’t even exist anymore? (Oh, no, please don’t tell me you can delete it and get an invalid pointer read...)
Quoted 7 lines
Quoted 3 lines
(I tried triggering that, but, oh, it’s strdup’ed() in
stream_start(). I’m a bit daft.)If it's broken anyway, probably we can just revert
c624b015bfand start to freeze base again?
Well, it’s only broken if you care about the backing filename string that’s written to @top. So it isn’t broken altogether.
Though, well. If we all agree to just revert it and maybe add a @bottom parameter instead, then I suppose we could do it.
(Maybe in a follow-up, though.)
Max
PATCH v7 15/47 block: Use CAFs when working with backing chains
In reply to Max ReitzUse child access functions when iterating through backing chains so filters do not break the chain.
In addition, bdrv_find_overlay() will now always return the actual overlay; that is, it will never return a filter node but only one with a COW backing file (there may be filter nodes between that node and @bs).
--- block.c | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-)
block.c +29 -12
@@ -4724,7 +4724,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file. * * active is the current topmost image. *
@@ -4736,11 +4737,18 @@ int bdrv_change_backing_file(BlockDriverState *bs,BlockDriverState *bdrv_find_overlay (BlockDriverState *active,BlockDriverState *bs) { -while (active && bs !=backing_bs (active)) { - active =backing_bs (active); + bs =bdrv_skip_filters (bs); + active =bdrv_skip_filters (active); + +while (active) { +BlockDriverState *next =bdrv_backing_chain_next (active); +if (bs == next) { +return active; + } + active = next; } -return active; +return NULL ; }/* Given a BDS, searches for the base layer. */
@@ -4892,9 +4900,7 @@ int bdrv_drop_intermediate(BlockDriverState *top, BlockDriverState *base,* other intermediate nodes have been dropped. * If 'top' is an implicit node (e.g. "commit_top") we should skip * it because no one inherits from it. We use explicit_top for that. */ -while (explicit_top && explicit_top->implicit) { - explicit_top =backing_bs (explicit_top); - } + explicit_top =bdrv_skip_implicit_filters (explicit_top); update_inherits_from =bdrv_inherits_from_recursive (base, explicit_top);/* success - we can delete the intermediate states, and link top->base */
@@ -5351,7 +5357,7 @@ BlockDriverState *bdrv_lookup_bs(const char *device,bool bdrv_chain_contains (BlockDriverState *top,BlockDriverState *base) {while (top && top != base) { - top =backing_bs (top); + top =bdrv_filter_or_cow_bs (top); }return top !=NULL ;
@@ -5607,6 +5613,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,int is_protocol =0 ;BlockDriverState *curr_bs =NULL ;BlockDriverState *retval =NULL ; +BlockDriverState *bs_below;if (!bs || !bs->drv || !backing_file) {return NULL ;
@@ -5617,7 +5624,17 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs, is_protocol =path_has_protocol (backing_file); -for (curr_bs = bs; curr_bs->backing; curr_bs = curr_bs->backing->bs) { +/* +* Being largely a legacy function, skip any filters here +* (because filters do not have normal filenames, so they cannot +* match anyway; and allowing json:{} filenames is a bit out of +* scope). +*/ +for (curr_bs =bdrv_skip_filters (bs); +bdrv_cow_child (curr_bs) !=NULL ; + curr_bs = bs_below) + { + bs_below =bdrv_backing_chain_next (curr_bs);/* If either of the filename paths is actually a protocol, then * compare unmodified paths; otherwise make paths relative */
@@ -5625,7 +5642,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,char *backing_file_full_ret;if (strcmp (backing_file, curr_bs->backing_file) ==0 ) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; }/* Also check against the full backing filename for the image */
@@ -5635,7 +5652,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,bool equal =strcmp (backing_file, backing_file_full_ret) ==0 ;g_free (backing_file_full_ret);if (equal) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; } }
@@ -5661,7 +5678,7 @@ BlockDriverState *bdrv_find_backing_image(BlockDriverState *bs,g_free (filename_tmp);if (strcmp (backing_file_full, filename_full) ==0 ) { - retval = curr_bs->backing->bs; + retval = bs_below;break ; } }
Signature · 1 line
2.26.2
Re: [PATCH v7 15/47] block: Use CAFs when working with backing chains
Quoted 17 lines · Max Reitz
Use child access functions when iterating through backing chains so filters do not break the chain.
In addition,
bdrv_find_overlay()will now always return the actual overlay; that is, it will never return a filter node but only one with a COW backing file (there may be filter nodes between that node and @bs).Signed-off-by Max Reitz <[email protected]>--- block.c | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-)block.c +0
-0
block.c int bdrv_change_backing_file(BlockDriverState *bs, · line 4728
Quoted 6 lines
block.c +2
-1@@ -4724,7 +4724,8 @@ int bdrv_change_backing_file(BlockDriverState *bs, }/* -* Finds the image layer in the chain that has 'bs' as its backing file. +* Finds the image layer in the chain that has 'bs' (or a filter on +* top of it) as its backing file.
One can optionally say "Finds the first non-filter parent of bs in the chain".
Quoted 1 line
...
PATCH v7 16/47 block: Use bdrv_cow_child() in bdrv_co_truncate()
In reply to Max ReitzThe condition modified here is not about potentially filtered children, but only about COW sources (i.e. traditional backing files).
--- block/io.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
block/io.c +4 -3
@@ -3308,7 +3308,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, Error **errp) {BlockDriverState *bs = child->bs; -BdrvChild *filtered; +BdrvChild *filtered, *backing;BlockDriver *drv = bs->drv;BdrvTrackedRequest req;int64_t old_size, new_bytes;
@@ -3361,6 +3361,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, } filtered =bdrv_filter_child (bs); + backing =bdrv_cow_child (bs);/* * If the image has a backing file that is large enough that it would
@@ -3372,10 +3373,10 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,* backing file, taking care of keeping things consistent with that backing * file is the user's responsibility. */ -if (new_bytes && bs->backing) { +if (new_bytes && backing) {int64_t backing_len; - backing_len =bdrv_getlength (backing_bs (bs)); + backing_len =bdrv_getlength (backing->bs);if (backing_len <0 ) { ret = backing_len;error_setg_errno (errp, -ret,"Could not get backing file size" );
Signature · 1 line
2.26.2
Re: [PATCH v7 16/47] block: Use bdrv_cow_child() in bdrv_co_truncate()
Quoted 32 lines · Max Reitz
The condition modified here is not about potentially filtered children, but only about COW sources (i.e. traditional backing files).
Signed-off-by Max Reitz <[email protected]>--- block/io.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)block/io.c +2
-1@@ -3308,7 +3308,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, Error **errp) {BlockDriverState *bs = child->bs; -BdrvChild *filtered; +BdrvChild *filtered, *backing;BlockDriver *drv = bs->drv;BdrvTrackedRequest req;int64_t old_size, new_bytes;@@ -3361,6 +3361,7 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, } filtered =bdrv_filter_child (bs); + backing =bdrv_cow_child (bs);/* * If the image has a backing file that is large enough that it would @@ -3372,10 +3373,10 @@ int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact,* backing file, taking care of keeping things consistent with that backing * file is the user's responsibility.
block/io.c int coroutine_fn bdrv_co_truncate(BdrvChild *child, int64_t offset, bool exact, · line 3382
Quoted 10 lines
block/io.c +2
-2*/ -if (new_bytes && bs->backing) { +if (new_bytes && backing) {int64_t backing_len; - backing_len =bdrv_getlength (backing_bs (bs)); + backing_len =bdrv_getlength (backing->bs);if (backing_len <0 ) { ret = backing_len;error_setg_errno (errp, -ret,"Could not get backing file size" );
PATCH v7 17/47 block: Re-evaluate backing file handling in reopen
In reply to Max ReitzReopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use bs->backing because these functions are about the "backing" child (which may or may not be the COW backing file).
--- block.c | 46 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 8 deletions(-)
block.c +38 -8
@@ -4026,26 +4026,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, } } +/* +* Ensure that @bs can really handle backing files, because we are +* about to give it one (or swap the existing one) +*/ +if (bs->drv->is_filter) { +/* Filters always have a file or a backing child */ +if (!bs->backing) { +error_setg (errp,"'%s' is a %s filter node that does not support a " +"backing child" , bs->node_name, bs->drv->format_name); +return -EINVAL ; + } + }else if (!bs->drv->supports_backing) { +error_setg (errp,"Driver '%s' of node '%s' does not support backing " +"files" , bs->drv->format_name, bs->node_name); +return -EINVAL ; + } +/* * Find the "actual" backing file by skipping all links that point * to an implicit node, if any (e.g. a commit filter node). +* We cannot use any of the bdrv_skip_*() functions here because +* those return the first explicit node, while we are looking for +* its overlay here. */ overlay_bs = bs; -while (backing_bs (overlay_bs) &&backing_bs (overlay_bs)->implicit) { - overlay_bs =backing_bs (overlay_bs); +while (bdrv_filter_or_cow_bs (overlay_bs) && +bdrv_filter_or_cow_bs (overlay_bs)->implicit) + { + overlay_bs =bdrv_filter_or_cow_bs (overlay_bs); }/* If we want to replace the backing file we need some extra checks */ -if (new_backing_bs !=backing_bs (overlay_bs)) { +if (new_backing_bs !=bdrv_filter_or_cow_bs (overlay_bs)) {/* Check for implicit nodes between bs and its backing file */ if (bs != overlay_bs) {error_setg (errp,"Cannot change backing link if '%s' has " "an implicit backing file" , bs->node_name);return -EPERM ; } -/* Check if the backing link that we want to replace is frozen */ -if (bdrv_is_backing_chain_frozen (overlay_bs,backing_bs (overlay_bs), - errp)) { +/* +* Check if the backing link that we want to replace is frozen. +* Note that +* bdrv_filter_or_cow_child(overlay_bs) == overlay_bs->backing, +* because we know that overlay_bs == bs, and that @bs +* either is a filter that uses ->backing or a COW format BDS +* with bs->drv->supports_backing == true. +*/ +if (bdrv_is_backing_chain_frozen (overlay_bs, +child_bs (overlay_bs->backing), errp)) + {return -EPERM ; } reopen_state->replace_backing_bs = true;
@@ -4196,7 +4226,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,* its metadata. Otherwise the 'backing' option can be omitted. */ if (drv->supports_backing && reopen_state->backing_missing && - (backing_bs (reopen_state->bs) || reopen_state->bs->backing_file[0 ])) { + (reopen_state->bs->backing || reopen_state->bs->backing_file[0 ])) {error_setg (errp,"backing is missing for '%s'" , reopen_state->bs->node_name); ret = -EINVAL ;
@@ -4337,7 +4367,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)* from bdrv_set_backing_hd()) has the new values. */ if (reopen_state->replace_backing_bs) { -BlockDriverState *old_backing_bs =backing_bs (bs); +BlockDriverState *old_backing_bs =child_bs (bs->backing);assert (!old_backing_bs || !old_backing_bs->implicit);/* Abort the permission update on the backing bs we're detaching */ if (old_backing_bs) {
Signature · 1 line
2.26.2
Re: [PATCH v7 17/47] block: Re-evaluate backing file handling in reopen
Quoted 42 lines · Max Reitz
Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use
bs->backingbecause these functions are about the "backing" child (which may or may not be the COW backing file).Signed-off-by Max Reitz <[email protected]>--- block.c | 46 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 8 deletions(-)block.c +19
-0@@ -4026,26 +4026,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, } } +/* +* Ensure that @bs can really handle backing files, because we are +* about to give it one (or swap the existing one) +*/ +if (bs->drv->is_filter) { +/* Filters always have a file or a backing child */ +if (!bs->backing) { +error_setg (errp,"'%s' is a %s filter node that does not support a " +"backing child" , bs->node_name, bs->drv->format_name); +return -EINVAL ; + } + }else if (!bs->drv->supports_backing) { +error_setg (errp,"Driver '%s' of node '%s' does not support backing " +"files" , bs->drv->format_name, bs->node_name); +return -EINVAL ; + } +/* * Find the "actual" backing file by skipping all links that point * to an implicit node, if any (e.g. a commit filter node). +* We cannot use any of the bdrv_skip_*() functions here because +* those return the first explicit node, while we are looking for
block.c static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, · line 4058
Quoted 10 lines
block.c +5
-2+* its overlay here. */ overlay_bs = bs; -while (backing_bs (overlay_bs) &&backing_bs (overlay_bs)->implicit) { - overlay_bs =backing_bs (overlay_bs); +while (bdrv_filter_or_cow_bs (overlay_bs) && +bdrv_filter_or_cow_bs (overlay_bs)->implicit) + { + overlay_bs =bdrv_filter_or_cow_bs (overlay_bs); }
I believe that little optimization would work properly:
for (BlockDriverState *below_bs = bdrv_filter_or_cow_bs(overlay_bs); below_bs && below_bs->implicit;
below_bs = bdrv_filter_or_cow_bs(overlay_bs)) { overlay_bs = below_bs;
}
Quoted 13 lines
block.c +1
-4/* If we want to replace the backing file we need some extra checks */ -if (new_backing_bs !=backing_bs (overlay_bs)) { +if (new_backing_bs !=bdrv_filter_or_cow_bs (overlay_bs)) {/* Check for implicit nodes between bs and its backing file */ if (bs != overlay_bs) {error_setg (errp,"Cannot change backing link if '%s' has " "an implicit backing file" , bs->node_name);return -EPERM ; } -/* Check if the backing link that we want to replace is frozen */ -if (bdrv_is_backing_chain_frozen (overlay_bs,backing_bs (overlay_bs), - errp)) {
block.c static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, · line 4077
Quoted 10 lines
block.c +10
-0+/* +* Check if the backing link that we want to replace is frozen. +* Note that +* bdrv_filter_or_cow_child(overlay_bs) == overlay_bs->backing, +* because we know that overlay_bs == bs, and that @bs +* either is a filter that uses ->backing or a COW format BDS +* with bs->drv->supports_backing == true. +*/ +if (bdrv_is_backing_chain_frozen (overlay_bs, +child_bs (overlay_bs->backing), errp))
What would be wrong with bdrv_filter_or_cow_bs(overlay_bs) here?
block.c int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, · line 4199
Quoted 9 lines
block.c +1
-1+ {return -EPERM ; } reopen_state->replace_backing_bs = true;@@ -4196,7 +4226,7 @@ int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue,* its metadata. Otherwise the 'backing' option can be omitted. */ if (drv->supports_backing && reopen_state->backing_missing && - (backing_bs (reopen_state->bs) || reopen_state->bs->backing_file[0 ])) {
= BlockDriverState*
block.c int bdrv_reopen_prepare(BDRVReopenState *reopen_state, BlockReopenQueue *queue, · line 4229
Quoted 1 line
block.c +1
-0+ (reopen_state->bs->backing || reopen_state->bs->backing_file[0])) {
= BdrvChild*
Are we OK with that?
Quoted 12 lines
block.c +1
-1error_setg (errp,"backing is missing for '%s'" , reopen_state->bs->node_name); ret = -EINVAL ;@@ -4337,7 +4367,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state)* from bdrv_set_backing_hd()) has the new values. */ if (reopen_state->replace_backing_bs) { -BlockDriverState *old_backing_bs =backing_bs (bs); +BlockDriverState *old_backing_bs =child_bs (bs->backing);assert (!old_backing_bs || !old_backing_bs->implicit);/* Abort the permission update on the backing bs we're detaching */ if (old_backing_bs) {
Re: [PATCH v7 17/47] block: Re-evaluate backing file handling in reopen
Quoted 60 lines · Andrey Shinkevich
Quoted 47 lines · Max Reitz
Reopening a node's backing child needs a bit of special handling because the "backing" child has different defaults than all other children (among other things). Adding filter support here is a bit more difficult than just using the child access functions. In fact, we often have to directly use
bs->backingbecause these functions are about the "backing" child (which may or may not be the COW backing file).Signed-off-by Max Reitz <[email protected]>---
block.c| 46 ++++++++++++++++++++++++++++++++++++++--------1 file changed, 38 insertions(+), 8 deletions(-)block.c +0
-0@@ -4026,26 +4026,56 @@ static int bdrv_reopen_parse_backing(BDRVReopenState *reopen_state, } }+ /*
+ * Ensure that @bs can really handle backing files, because we are + * about to give it one (or swap the existing one)
+ */
+ if (bs->drv->is_filter) {
+ /* Filters always have a file or a backing child */ + if (!bs->backing) {
+ error_setg(errp, "'%s' is a %s filter node that does not support a "
+ "backing child",bs->node_name,bs->drv->format_name);
+ return-EINVAL;
+ }
+ } else if (!bs->drv->supports_backing) {
+ error_setg(errp, "Driver '%s' of node '%s' does not support backing "
+ "files",bs->drv->format_name,bs->node_name); + return-EINVAL;
+ }
+
/*
* Find the "actual" backing file by skipping all links that point
* to an implicit node, if any (e.g. a commit filter node). + * We cannot use any of the bdrv_skip_*() functions here because + * those return the first explicit node, while we are looking for + * its overlay here.
*/Quoted 8 lines
overlay_bs= bs;
- while (backing_bs(overlay_bs)&& backing_bs(overlay_bs)->implicit) { -overlay_bs=backing_bs(overlay_bs);
+ while (bdrv_filter_or_cow_bs(overlay_bs)&&
+ bdrv_filter_or_cow_bs(overlay_bs)->implicit)
+ {
+overlay_bs=bdrv_filter_or_cow_bs(overlay_bs);
}I believe that little optimization would work properly:
Quoted 5 lines
for (BlockDriverState
*below_bs=bdrv_filter_or_cow_bs(overlay_bs);below_bs&&below_bs->implicit;
below_bs=bdrv_filter_or_cow_bs(overlay_bs)) {overlay_bs=below_bs;
}
Looks good, thanks.
Quoted 26 lines
Quoted 18 lines
/* If we want to replace the backing file we need some extra checks */
- if (new_backing_bs!=backing_bs(overlay_bs)) {
+ if (new_backing_bs!=bdrv_filter_or_cow_bs(overlay_bs)) { /* Check for implicit nodes between bs and its backing file */ if (bs !=overlay_bs) {
error_setg(errp, "Cannot change backing link if '%s' has " "an implicit backing file", bs->node_name); return-EPERM;
}
- /* Check if the backing link that we want to replace is frozen */
- if (bdrv_is_backing_chain_frozen(overlay_bs,backing_bs(overlay_bs),
- errp)) {
+ /*
+ * Check if the backing link that we want to replace is frozen. + * Note thatQuoted 8 lines
+ *
bdrv_filter_or_cow_child(overlay_bs)==overlay_bs->backing, + * because we know thatoverlay_bs== bs, and that @bs + * either is a filter that uses ->backing or a COW format BDS + * withbs->drv->supports_backing== true.
+ */
+ if (bdrv_is_backing_chain_frozen(overlay_bs,
+child_bs(overlay_bs->backing), errp))
Quoted 1 line
What would be wrong with
bdrv_filter_or_cow_bs(overlay_bs)here?
As the comment says, it’s the same thing.
I prefer ->backing here, because this function is about reopening the ->backing child.
Quoted 12 lines
Quoted 11 lines
+ {
return-EPERM;
}reopen_state->replace_backing_bs= true;block.c +0
-0@@ -4196,7 +4226,7 @@ int bdrv_reopen_prepare(BDRVReopenState*reopen_state, BlockReopenQueue *queue, * its metadata. Otherwise the 'backing' option can be omitted. */ if (
drv->supports_backing&&reopen_state->backing_missing&& - (backing_bs(reopen_state->bs)|| reopen_state->bs->backing_file[0])) {= BlockDriverState*
Quoted 6 lines
Quoted 2 lines
+ (
reopen_state->bs->backing||
reopen_state->bs->backing_file[0])) {= BdrvChild*
Are we OK with that?
Sure, the question is whether it’s non-NULL, and BdrvChild.bs is always non-NULL.
Max
PATCH v7 18/47 block: Flush all children in generic code
In reply to Max ReitzIf the driver does not support .bdrv_co_flush() so bdrv_co_flush() itself has to flush the children of the given node, it should not flush just bs->file->bs, but in fact all children that might have been written to (judging from the permissions taken on them).
This is a bug fix for qcow2 images with an external data file, as they so far did not flush that data_file node.
In any case, the BLKDBG_EVENT() should be emitted on the primary child, because that is where a blkdebug node would be if there is any.
--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
block/io.c +17 -6
@@ -2769,6 +2769,8 @@ static int coroutine_fn bdrv_flush_co_entry(void *opaque) int coroutine_fn bdrv_co_flush(BlockDriverState *bs) { +BdrvChild *primary_child =bdrv_primary_child (bs); +BdrvChild *child;int current_gen;int ret =0 ;
@@ -2798,7 +2800,7 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs) }/* Write back cached data to the OS even with cache=unsafe */ -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_OS ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_OS );if (bs->drv->bdrv_co_flush_to_os) { ret = bs->drv->bdrv_co_flush_to_os (bs);if (ret <0 ) {
@@ -2808,15 +2810,15 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* But don't actually force it to the disk with cache=unsafe */ if (bs->open_flags &BDRV_O_NO_FLUSH ) { - goto flush_parent; + goto flush_children; }/* Check if we really need to flush anything */ if (bs->flushed_gen == current_gen) { - goto flush_parent; + goto flush_children; } -BLKDBG_EVENT (bs->file,BLKDBG_FLUSH_TO_DISK ); +BLKDBG_EVENT (primary_child,BLKDBG_FLUSH_TO_DISK );if (!bs->drv) {/* bs->drv->bdrv_co_flush() might have ejected the BDS * (even in case of apparent success) */
@@ -2860,8 +2862,17 @@ int coroutine_fn bdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */ -flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret =0 ; + QLIST_FOREACH(child, &bs->children, next) { +if (child->perm & (BLK_PERM_WRITE |BLK_PERM_WRITE_UNCHANGED )) { +int this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + } + } + out:/* Notify any pending flushes that we have completed */ if (ret ==0 ) {
Signature · 1 line
2.26.2
Re: [PATCH v7 18/47] block: Flush all children in generic code
Quoted 16 lines · Max Reitz
If the driver does not support .bdrv_co_flush() so
bdrv_co_flush()itself has to flush the children of the given node, it should not flush justbs->file->bs, but in fact all children that might have been written to (judging from the permissions taken on them).This is a bug fix for qcow2 images with an external data file, as they so far did not flush that
data_filenode.In any case, the
BLKDBG_EVENT()should be emitted on the primary child, because that is where a blkdebug node would be if there is any.Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/io.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-)
...
Quoted 10 lines
@@ int
coroutine_fnbdrv_co_flush(BlockDriverState *bs)/* Now flush the underlying protocol. It will also have BDRV_O_NO_FLUSH * in the case of cache=unsafe, so there are no useless flushes. */block/io.c +4
-2-flush_parent: - ret = bs->file ?bdrv_co_flush (bs->file->bs) :0 ; +flush_children: + ret =0 ; + QLIST_FOREACH(child, &bs->children, next) { +if (child->perm & (BLK_PERM_WRITE |BLK_PERM_WRITE_UNCHANGED )) {
block/io.c int coroutine_fn bdrv_co_flush(BlockDriverState *bs) · line 2878
Quoted 10 lines
block/io.c +7
-0+int this_child_ret =bdrv_co_flush (child->bs); +if (!ret) { + ret = this_child_ret; + } + } + } + out:/* Notify any pending flushes that we have completed */ if (ret ==0 ) {
I have not tested if the running application do reaches the flush_parent point to flush the data_file but with the code it looks OK.
PATCH v7 19/47 vmdk: Drop vmdk_co_flush()
In reply to Max ReitzBefore HEAD^, we needed this because bdrv_co_flush() by itself would only flush bs->file. With HEAD^, bdrv_co_flush() will flush all children on which a WRITE or WRITE_UNCHANGED permission has been taken. Thus, vmdk no longer needs to do it itself.
--- block/vmdk.c | 16 ---------------- 1 file changed, 16 deletions(-)
block/vmdk.c +0 -16
@@ -2802,21 +2802,6 @@ static void vmdk_close(BlockDriverState *bs)error_free (s->migration_blocker); } -static coroutine_fn int vmdk_co_flush(BlockDriverState *bs) -{ -BDRVVmdkState *s = bs->opaque; -int i, err; -int ret =0 ; - -for (i =0 ; i < s->num_extents; i++) { - err =bdrv_co_flush (s->extents[i].file->bs); -if (err <0 ) { - ret = err; - } - } -return ret; -} -static int64_t vmdk_get_allocated_file_size (BlockDriverState *bs) {int i;
@@ -3075,7 +3060,6 @@ static BlockDriver bdrv_vmdk = { .bdrv_close = vmdk_close, .bdrv_co_create_opts = vmdk_co_create_opts, .bdrv_co_create = vmdk_co_create, - .bdrv_co_flush_to_disk = vmdk_co_flush, .bdrv_co_block_status = vmdk_co_block_status, .bdrv_get_allocated_file_size = vmdk_get_allocated_file_size, .bdrv_has_zero_init = vmdk_has_zero_init,
Signature · 1 line
2.26.2
Re: [PATCH v7 19/47] vmdk: Drop vmdk_co_flush()
Quoted 37 lines · Max Reitz
Before HEAD^, we needed this because
bdrv_co_flush()by itself would only flushbs->file. With HEAD^,bdrv_co_flush()will flush all children on which a WRITE orWRITE_UNCHANGEDpermission has been taken. Thus, vmdk no longer needs to do it itself.Signed-off-by Max Reitz <[email protected]>--- block/vmdk.c | 16 ---------------- 1 file changed, 16 deletions(-)block/vmdk.c +0
-15@@ -2802,21 +2802,6 @@ static void vmdk_close(BlockDriverState *bs)error_free (s->migration_blocker); } -static coroutine_fn int vmdk_co_flush(BlockDriverState *bs) -{ -BDRVVmdkState *s = bs->opaque; -int i, err; -int ret =0 ; - -for (i =0 ; i < s->num_extents; i++) { - err =bdrv_co_flush (s->extents[i].file->bs); -if (err <0 ) { - ret = err; - } - } -return ret; -} -static int64_t vmdk_get_allocated_file_size (BlockDriverState *bs) {int i;
block/vmdk.c static BlockDriver bdrv_vmdk = { · line 3078
Quoted 4 lines
block/vmdk.c +0
-1@@ -3075,7 +3060,6 @@ static BlockDriver bdrv_vmdk = { .bdrv_close = vmdk_close, .bdrv_co_create_opts = vmdk_co_create_opts, .bdrv_co_create = vmdk_co_create, - .bdrv_co_flush_to_disk = vmdk_co_flush,
After HEAD^ applied, wouldn't we get an endless recursion in bdrv_co_flush() if the HEAD (this patch) had not been merged into HEAD^?
Andrey
Quoted 3 lines
block/vmdk.c +0
-0.bdrv_co_block_status = vmdk_co_block_status, .bdrv_get_allocated_file_size = vmdk_get_allocated_file_size, .bdrv_has_zero_init = vmdk_has_zero_init,
Re: [PATCH v7 19/47] vmdk: Drop vmdk_co_flush()
Quoted 37 lines · Andrey Shinkevich
Quoted 36 lines · Max Reitz
Before HEAD^, we needed this because
bdrv_co_flush()by itself would only flush bs->file. With HEAD^,bdrv_co_flush()will flush all children on which a WRITE orWRITE_UNCHANGEDpermission has been taken. Thus, vmdk no longer needs to do it itself.Signed-off-by Max Reitz <[email protected]>---
block/vmdk.c| 16 ----------------1 file changed, 16 deletions(-)block/vmdk.c +0
-14@@ -2802,21 +2802,6 @@ static void vmdk_close(BlockDriverState *bs) error_free(s->migration_blocker); } -static coroutine_fn int vmdk_co_flush(BlockDriverState *bs) -{ - BDRVVmdkState *s = bs->opaque; - int i, err; - int ret = 0; - - for (i = 0; i < s->num_extents; i++) { - err = bdrv_co_flush(s->extents[i].file->bs); - if (err < 0) { - ret = err; - } - } - return ret; -} - static int64_t vmdk_get_allocated_file_size(BlockDriverState *bs) { int i;@@ -3075,7 +3060,6 @@ static BlockDriver bdrv_vmdk = {
Quoted 8 lines
block/vmdk.c static BlockDriver bdrv_vmdk = { · line 3078
Quoted 4 lines
block/vmdk.c +0
-1.bdrv_close = vmdk_close, .bdrv_co_create_opts = vmdk_co_create_opts, .bdrv_co_create = vmdk_co_create, - .bdrv_co_flush_to_disk = vmdk_co_flush,After HEAD^ applied, wouldn't we get an endless recursion in
bdrv_co_flush()if the HEAD (this patch) had not been merged into HEAD^?
Hm, how so? HEAD^ just flushes all children, just like vmdk_co_flush() does. So it seems to me just like double the work. (Which is unfortunate but shouldn’t be a problem.)
Max
PATCH v7 20/47 block: Iterate over children in refresh_limits
In reply to Max ReitzInstead of looking at just bs->file and bs->backing, we should look at all children that could end up receiving forwarded requests.
--- block/io.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)
block/io.c +16 -16
@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)void bdrv_refresh_limits (BlockDriverState *bs,Error **errp) {BlockDriver *drv = bs->drv; +BdrvChild *c; +bool have_limits;Error *local_err =NULL ;memset (&bs->bl,0 ,sizeof (bs->bl));
@@ -149,14 +151,21 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) drv->bdrv_co_preadv_part) ? 1 : 512;/* Take some limits from the children as a default */ -if (bs->file) { -bdrv_refresh_limits (bs->file->bs, &local_err); -if (local_err) { -error_propagate (errp, local_err); -return ; + have_limits = false; + QLIST_FOREACH(c, &bs->children, next) { +if (c->role & (BDRV_CHILD_DATA |BDRV_CHILD_FILTERED |BDRV_CHILD_COW )) + { +bdrv_refresh_limits (c->bs, &local_err); +if (local_err) { +error_propagate (errp, local_err); +return ; + } +bdrv_merge_limits (&bs->bl, &c->bs->bl); + have_limits = true; } -bdrv_merge_limits (&bs->bl, &bs->file->bs->bl); - }else { + } + +if (!have_limits) { bs->bl.min_mem_alignment =512 ; bs->bl.opt_mem_alignment = qemu_real_host_page_size;
@@ -164,15 +173,6 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) bs->bl.max_iov =IOV_MAX ; } -if (bs->backing) { -bdrv_refresh_limits (bs->backing->bs, &local_err); -if (local_err) { -error_propagate (errp, local_err); -return ; - } -bdrv_merge_limits (&bs->bl, &bs->backing->bs->bl); - } -/* Then let the driver override it */ if (drv->bdrv_refresh_limits) { drv->bdrv_refresh_limits (bs, errp);
Signature · 1 line
2.26.2
Re: [PATCH v7 20/47] block: Iterate over children in refresh_limits
Quoted 37 lines · Max Reitz
Instead of looking at just
bs->fileandbs->backing, we should look at all children that could end up receiving forwarded requests.Signed-off-by Max Reitz <[email protected]>--- block/io.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-)block/io.c +9
-5@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src)void bdrv_refresh_limits (BlockDriverState *bs,Error **errp) {BlockDriver *drv = bs->drv; +BdrvChild *c; +bool have_limits;Error *local_err =NULL ;memset (&bs->bl,0 ,sizeof (bs->bl));@@ -149,14 +151,21 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) drv->bdrv_co_preadv_part) ? 1 : 512;/* Take some limits from the children as a default */ -if (bs->file) { -bdrv_refresh_limits (bs->file->bs, &local_err); -if (local_err) { -error_propagate (errp, local_err); -return ; + have_limits = false; + QLIST_FOREACH(c, &bs->children, next) { +if (c->role & (BDRV_CHILD_DATA |BDRV_CHILD_FILTERED |BDRV_CHILD_COW )) + { +bdrv_refresh_limits (c->bs, &local_err); +if (local_err) { +error_propagate (errp, local_err);
block/io.c void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) · line 168
Quoted 10 lines
block/io.c +7
-2+return ; + } +bdrv_merge_limits (&bs->bl, &c->bs->bl); + have_limits = true; } -bdrv_merge_limits (&bs->bl, &bs->file->bs->bl); - }else { + } + +if (!have_limits) {
This conditioned piece of code worked with (bs->file == NULL) only.
Now, it works only if there are neither bs->file, nor bs->backing, nor else filtered children.
Is it OK and doesn't break the logic for all cases?
Andrey
Quoted 19 lines
block/io.c +0
-9bs->bl.min_mem_alignment =512 ; bs->bl.opt_mem_alignment = qemu_real_host_page_size;@@ -164,15 +173,6 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) bs->bl.max_iov =IOV_MAX ; } -if (bs->backing) { -bdrv_refresh_limits (bs->backing->bs, &local_err); -if (local_err) { -error_propagate (errp, local_err); -return ; - } -bdrv_merge_limits (&bs->bl, &bs->backing->bs->bl); - } -/* Then let the driver override it */ if (drv->bdrv_refresh_limits) { drv->bdrv_refresh_limits (bs, errp);
Re: [PATCH v7 20/47] block: Iterate over children in refresh_limits
Quoted 53 lines · Andrey Shinkevich
Quoted 38 lines · Max Reitz
Instead of looking at just
bs->fileandbs->backing, we should look at all children that could end up receiving forwarded requests.Signed-off-by Max Reitz <[email protected]>---
block/io.c| 32 ++++++++++++++++----------------1 file changed, 16 insertions(+), 16 deletions(-)block/io.c +9
-5@@ -135,6 +135,8 @@ static void bdrv_merge_limits(BlockLimits *dst, const BlockLimits *src) void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) { BlockDriver *drv = bs->drv; + BdrvChild *c; + bool have_limits; Error *local_err = NULL; memset(&bs->bl, 0, sizeof(bs->bl));@@ -149,14 +151,21 @@ void bdrv_refresh_limits(BlockDriverState *bs, Error **errp) drv->bdrv_co_preadv_part) ? 1 : 512; /* Take some limits from the children as a default */ - if (bs->file) { - bdrv_refresh_limits(bs->file->bs, &local_err); - if (local_err) { - error_propagate(errp, local_err); - return; + have_limits = false; + QLIST_FOREACH(c, &bs->children, next) { + if (c->role & (BDRV_CHILD_DATA | BDRV_CHILD_FILTERED |BDRV_CHILD_COW )) + { + bdrv_refresh_limits(c->bs, &local_err); + if (local_err) { + error_propagate(errp, local_err);block/io.c void bdrv_refresh_limits(BlockDriverState *bs, · line 169
Quoted 10 lines
block/io.c +7
-2+ return; + } + bdrv_merge_limits(&bs->bl, &c->bs->bl); + have_limits = true; } - bdrv_merge_limits(&bs->bl, &bs->file->bs->bl); - } else { + } + + if (!have_limits) {This conditioned piece of code worked with (
bs->file== NULL) only.
Quoted 4 lines
Now, it works only if there are neither
bs->file, norbs->backing, nor else filtered children.Is it OK and doesn't break the logic for all cases?
Hm. Good question.
I think the answer is it’s OK.
For DATA and FILTERED, it makes absolute sense to just use their alignments. For COW, maybe not so much? But if there’s a COW child, there has to be a DATA child as well (in practice). So we’ll always consider its alignment, too.
(And hypothetically speaking, if there was a COW child but no DATA child, then the only alignment we need to observe is in fact the one of the COW child.)
Max
PATCH v7 21/47 block: Use CAFs in bdrv_refresh_filename()
In reply to Max Reitzbdrv_refresh_filename() and the kind of related bdrv_dirname() should look to the primary child when they wish to copy the underlying file's filename.
--- block.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)
block.c +21 -8
@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs) {BlockDriver *drv = bs->drv;BdrvChild *child; +BlockDriverState *primary_child_bs;QDict *opts;bool backing_overridden;bool generate_json_filename;/* Whether our default implementation should
@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)qobject_unref (bs->full_open_options); bs->full_open_options = opts; + primary_child_bs =bdrv_primary_bs (bs); +if (drv->bdrv_refresh_filename) {/* Obsolete information is of no use here, so drop the old file name * information before refreshing it */ bs->exact_filename[0 ] ='\0' ; drv->bdrv_refresh_filename (bs); - }else if (bs->file) { -/* Try to reconstruct valid information from the underlying file */ + }else if (primary_child_bs) { +/* +* Try to reconstruct valid information from the underlying +* file -- this only works for format nodes (filter nodes +* cannot be probed and as such must be selected by the user +* either through an options dict, or through a special +* filename which the filter driver must construct in its +* .bdrv_refresh_filename() implementation). +*/ bs->exact_filename[0 ] ='\0' ;/* * We can use the underlying file's filename if: * - it has a filename, +* - the current BDS is not a filter, * - the file is a protocol BDS, and * - opening that file (as this BDS's format) will automatically create * the BDS tree we have right now, that is:
@@ -6888,11 +6899,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)* - no non-file child of this BDS has been overridden by the user * Both of these conditions are represented by generate_json_filename. */ -if (bs->file->bs->exact_filename[0 ] && - bs->file->bs->drv->bdrv_file_open && - !generate_json_filename) +if (primary_child_bs->exact_filename[0 ] && + primary_child_bs->drv->bdrv_file_open && + !drv->is_filter && !generate_json_filename) { -strcpy (bs->exact_filename, bs->file->bs->exact_filename); +strcpy (bs->exact_filename, primary_child_bs->exact_filename); } }
@@ -6912,6 +6923,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)char *bdrv_dirname (BlockDriverState *bs,Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *child_bs;if (!drv) {error_setg (errp,"Node '%s' is ejected" , bs->node_name);
@@ -6922,8 +6934,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp)return drv->bdrv_dirname (bs, errp); } -if (bs->file) { -return bdrv_dirname (bs->file->bs, errp); + child_bs =bdrv_primary_bs (bs); +if (child_bs) { +return bdrv_dirname (child_bs, errp); }bdrv_refresh_filename (bs);
Signature · 1 line
2.26.2
Re: [PATCH v7 21/47] block: Use CAFs in bdrv_refresh_filename()
Quoted 34 lines · Max Reitz
bdrv_refresh_filename()and the kind of relatedbdrv_dirname()should look to the primary child when they wish to copy the underlying file's filename.Signed-off-by Max Reitz <[email protected]>--- block.c | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-)block.c +3
-1@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs) {BlockDriver *drv = bs->drv;BdrvChild *child; +BlockDriverState *primary_child_bs;QDict *opts;bool backing_overridden;bool generate_json_filename;/* Whether our default implementation should @@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs)qobject_unref (bs->full_open_options); bs->full_open_options = opts; + primary_child_bs =bdrv_primary_bs (bs); +if (drv->bdrv_refresh_filename) {/* Obsolete information is of no use here, so drop the old file name * information before refreshing it */ bs->exact_filename[0 ] ='\0' ; drv->bdrv_refresh_filename (bs); - }else if (bs->file) {
block.c void bdrv_refresh_filename(BlockDriverState *bs) · line 6886
Quoted 10 lines
block.c +9
-1-/* Try to reconstruct valid information from the underlying file */ + }else if (primary_child_bs) { +/* +* Try to reconstruct valid information from the underlying +* file -- this only works for format nodes (filter nodes +* cannot be probed and as such must be selected by the user +* either through an options dict, or through a special +* filename which the filter driver must construct in its +* .bdrv_refresh_filename() implementation). +*/
The caller may not be aware of a filter node and intend to refresh the name of underlying format node.
In that case, the filter driver should redirect the call to the format node.
What are situations the name of the filter itself should be refreshed in?
If there are any, should we do both actions or choose either?
Andrey
block.c void bdrv_refresh_filename(BlockDriverState *bs) · line 6893
Quoted 7 lines
block.c +1
-0bs->exact_filename[0 ] ='\0' ;/* * We can use the underlying file's filename if: * - it has a filename, +* - the current BDS is not a filter,
Should we check the function input parameter for being a filter's BS here, in this function, and handle the case here or let the filter driver function do that or else the caller should check it?
Andrey
Quoted 39 lines
block.c +8
-6* - the file is a protocol BDS, and * - opening that file (as this BDS's format) will automatically create * the BDS tree we have right now, that is: @@ -6888,11 +6899,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)* - no non-file child of this BDS has been overridden by the user * Both of these conditions are represented by generate_json_filename. */ -if (bs->file->bs->exact_filename[0 ] && - bs->file->bs->drv->bdrv_file_open && - !generate_json_filename) +if (primary_child_bs->exact_filename[0 ] && + primary_child_bs->drv->bdrv_file_open && + !drv->is_filter && !generate_json_filename) { -strcpy (bs->exact_filename, bs->file->bs->exact_filename); +strcpy (bs->exact_filename, primary_child_bs->exact_filename); } }@@ -6912,6 +6923,7 @@ void bdrv_refresh_filename(BlockDriverState *bs)char *bdrv_dirname (BlockDriverState *bs,Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *child_bs;if (!drv) {error_setg (errp,"Node '%s' is ejected" , bs->node_name);@@ -6922,8 +6934,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp)return drv->bdrv_dirname (bs, errp); } -if (bs->file) { -return bdrv_dirname (bs->file->bs, errp); + child_bs =bdrv_primary_bs (bs); +if (child_bs) { +return bdrv_dirname (child_bs, errp); }bdrv_refresh_filename (bs);
Re: [PATCH v7 21/47] block: Use CAFs in bdrv_refresh_filename()
Quoted 48 lines · Andrey Shinkevich
Quoted 35 lines · Max Reitz
bdrv_refresh_filename()and the kind of relatedbdrv_dirname()should look to the primary child when they wish to copy the underlying file's filename.Signed-off-by Max Reitz <[email protected]>---
block.c| 29 +++++++++++++++++++++--------1 file changed, 21 insertions(+), 8 deletions(-)block.c +1
-0@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs) { BlockDriver *drv = bs->drv; BdrvChild *child; + BlockDriverState *primary_child_bs; QDict *opts; bool backing_overridden;bool
generate_json_filename; /* Whether our default implementation shouldblock.c +1
-2@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs) qobject_unref(bs->full_open_options); bs->full_open_options = opts; + primary_child_bs = bdrv_primary_bs(bs); + if (drv->bdrv_refresh_filename) { /* Obsolete information is of no use here, so drop the old file name * information before refreshing it */ bs->exact_filename[0] = '\0'; drv->bdrv_refresh_filename(bs); - } else if (bs->file) { - /* Try to reconstruct valid information from the underlyingfile */
+ } else if (primary_child_bs) {Quoted 8 lines
+ /*
+ * Try to reconstruct valid information from the underlying + * file -- this only works for format nodes (filter nodes + * cannot be probed and as such must be selected by the user + * either through an options dict, or through a special + * filename which the filter driver must construct in its + * .bdrv_refresh_filename() implementation).
+ */The caller may not be aware of a filter node and intend to refresh the name of underlying format node.
Quoted 8 lines
In that case, the filter driver should redirect the call to the format node.
What are situations the name of the filter itself should be refreshed in?
If there are any, should we do both actions or choose either?
Andrey
I ment the node FILE name.
Andrey
Quoted 51 lines
Quoted 6 lines
bs->exact_filename[0] = '\0';
/*
* We can use the underlying file's filename if:
* - it has a filename,
+ * - the current BDS is not a filter,Should we check the function input parameter for being a filter's BS here, in this function, and handle the case here or let the filter driver function do that or else the caller should check it?
Andrey
Quoted 37 lines
* - the file is a protocol BDS, and
* - opening that file (as this BDS's format) will automatically create * the BDS tree we have right now, that is:block.c +4
-4@@ -6888,11 +6899,11 @@ void bdrv_refresh_filename(BlockDriverState *bs)* - no non-file child of this BDS has been overridden by the user * Both of these conditions are represented by generate_json_filename. */ - if (bs->file->bs->exact_filename[0] && - bs->file->bs->drv->bdrv_file_open && - !generate_json_filename) + if (primary_child_bs->exact_filename[0] && + primary_child_bs->drv->bdrv_file_open && + !drv->is_filter && !generate_json_filename) { - strcpy(bs->exact_filename, bs->file->bs->exact_filename); + strcpy(bs->exact_filename, primary_child_bs->exact_filename); } }@@ -6912,6 +6923,7 @@ void
bdrv_refresh_filename(BlockDriverState *bs)char *bdrv_dirname(BlockDriverState *bs, Error **errp)
{
BlockDriver *drv =bs->drv;
+ BlockDriverState*child_bs;
if (!drv) {
error_setg(errp, "Node '%s' is ejected", bs->node_name);block.c +3
-1@@ -6922,8 +6934,9 @@ char *bdrv_dirname(BlockDriverState *bs, Error **errp) return drv->bdrv_dirname(bs, errp); } - if (bs->file) { - return bdrv_dirname(bs->file->bs, errp); + child_bs = bdrv_primary_bs(bs); + if (child_bs) { + return bdrv_dirname(child_bs, errp); } bdrv_refresh_filename(bs);
Re: [PATCH v7 21/47] block: Use CAFs in bdrv_refresh_filename()
In reply to Andrey ShinkevichQuoted 51 lines · Andrey Shinkevich
Quoted 37 lines · Max Reitz
bdrv_refresh_filename()and the kind of relatedbdrv_dirname()should look to the primary child when they wish to copy the underlying file's filename.Signed-off-by Max Reitz <[email protected]>---
block.c| 29 +++++++++++++++++++++--------1 file changed, 21 insertions(+), 8 deletions(-)block.c +1
-0@@ -6797,6 +6797,7 @@ void bdrv_refresh_filename(BlockDriverState *bs) { BlockDriver *drv = bs->drv; BdrvChild *child; + BlockDriverState *primary_child_bs; QDict *opts; bool backing_overridden;bool
generate_json_filename; /* Whether our default implementation shouldblock.c +1
-0@@ -6866,20 +6867,30 @@ void bdrv_refresh_filename(BlockDriverState *bs) qobject_unref(bs->full_open_options); bs->full_open_options = opts; + primary_child_bs = bdrv_primary_bs(bs); + if (drv->bdrv_refresh_filename) { /* Obsolete information is of no use here, so drop the oldfile name
* information before refreshing it */ bs->exact_filename[0] = '\0';drv->bdrv_refresh_filename(bs);
- } else if (bs->file) {
- /* Try to reconstruct valid information from the underlying file */
+ } else if (primary_child_bs) {Quoted 8 lines
+ /*
+ * Try to reconstruct valid information from the underlying + * file -- this only works for format nodes (filter nodes + * cannot be probed and as such must be selected by the user + * either through an options dict, or through a special + * filename which the filter driver must construct in its + * .bdrv_refresh_filename() implementation).
+ */The caller may not be aware of a filter node and intend to refresh the name of underlying format node.
Quoted 2 lines
In that case, the filter driver should redirect the call to the format node.
It shouldn’t. We can only return a plain filename if passing that filename to qemu (e.g. to -drive) will result in the same block graph configuration.
This is what the comment means by “filter nodes cannot be probed”: If there is a filter node, we must generate a json:{} filename, because otherwise reopening the block device with -drive by passing the filename generated here would result in a configuration where the filter is missing.
Quoted 1 line
What are situations the name of the filter itself should be refreshed in?
Hypothetically, if a filename could specify a filter. For example, say the filename “filter[copy-on-read]:foo.qcow2” would result in qemu creating a COR filter on top of a qcow2 node, then we could generate such a filename.
In practice, filters cannot be configured through plain filenames (apart from blkverify, but that’s a debugging feature, so it doesn’t really matter), so there is no such situation. All filter nodes should have an empty exact_filename and thus get a json:{} pseudo-filename.
Quoted 12 lines
If there are any, should we do both actions or choose either?
Andrey
Quoted 5 lines
bs->exact_filename[0] = '\0';
/*
* We can use the underlying file's filename if:
* - it has a filename,
+ * - the current BDS is not a filter,
Quoted 3 lines
Should we check the function input parameter for being a filter's BS here, in this function, and handle the case here or let the filter driver function do that or else the caller should check it?
bdrv_refresh_filename() is called whenever some node in the block graph has changed, just to refresh its filename (after that change). The caller generally doesn’t really care about the result, so it doesn’t matter whether the node is a filter or not (i.e., whether it gets a plain filename or not).
I don’t think the caller should check it, and in this implementation we simply have to handle filter nodes correctly: That is, not give them a plain filename.
Max
PATCH v7 22/47 block: Use CAF in bdrv_co_rw_vmstate()
In reply to Max ReitzIf a node whose driver does not provide VM state functions has a metadata child, the VM state should probably go there; if it is a filter, the VM state should probably go there. It follows that we should generally go down to the primary child.
--- block/io.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
block/io.c +3 -2
@@ -2646,6 +2646,7 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, bool is_read) {BlockDriver *drv = bs->drv; +BlockDriverState *child_bs =bdrv_primary_bs (bs);int ret = -ENOTSUP ;bdrv_inc_in_flight (bs);
@@ -2658,8 +2659,8 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, }else { ret = drv->bdrv_save_vmstate (bs, qiov, pos); } - }else if (bs->file) { - ret =bdrv_co_rw_vmstate (bs->file->bs, qiov, pos, is_read); + }else if (child_bs) { + ret =bdrv_co_rw_vmstate (child_bs, qiov, pos, is_read); }bdrv_dec_in_flight (bs);
Signature · 1 line
2.26.2
Re: [PATCH v7 22/47] block: Use CAF in bdrv_co_rw_vmstate()
Quoted 24 lines · Max Reitz
If a node whose driver does not provide VM state functions has a metadata child, the VM state should probably go there; if it is a filter, the VM state should probably go there. It follows that we should generally go down to the primary child.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/io.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)block/io.c +1
-0@@ -2646,6 +2646,7 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, bool is_read) {BlockDriver *drv = bs->drv; +BlockDriverState *child_bs =bdrv_primary_bs (bs);int ret = -ENOTSUP ;bdrv_inc_in_flight (bs);
block/io.c bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, · line 2666
Quoted 10 lines
block/io.c +2
-2@@ -2658,8 +2659,8 @@ bdrv_co_rw_vmstate(BlockDriverState *bs, QEMUIOVector *qiov, int64_t pos, }else { ret = drv->bdrv_save_vmstate (bs, qiov, pos); } - }else if (bs->file) { - ret =bdrv_co_rw_vmstate (bs->file->bs, qiov, pos, is_read); + }else if (child_bs) { + ret =bdrv_co_rw_vmstate (child_bs, qiov, pos, is_read); }bdrv_dec_in_flight (bs);
PATCH v7 23/47 block/snapshot: Fix fallback
In reply to Max ReitzIf the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there is more than one such child. Furthermore, we really only can fall back to bs->file and bs->backing, because bdrv_snapshot_goto() has to modify the child link (notably, set it to NULL).
--- block/snapshot.c | 104 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 21 deletions(-)
block/snapshot.c +83 -21
@@ -147,6 +147,56 @@ bool bdrv_snapshot_find_by_id_and_name(BlockDriverState *bs,return ret; } +/** +* Return a pointer to the child BDS pointer to which we can fall +* back if the given BDS does not support snapshots. +* Return NULL if there is no BDS to (safely) fall back to. +* +* We need to return an indirect pointer because bdrv_snapshot_goto() +* has to modify the BdrvChild pointer. +*/ +static BdrvChild **bdrv_snapshot_fallback_ptr (BlockDriverState *bs) +{ +BdrvChild **fallback; +BdrvChild *child; + +/* +* The only BdrvChild pointers that are safe to modify (and which +* we can thus return a reference to) are bs->file and +* bs->backing. +*/ + fallback = &bs->file; +if (!*fallback && bs->drv && bs->drv->is_filter) { + fallback = &bs->backing; + } + +if (!*fallback) { +return NULL ; + } + +/* +* Check that there are no other children that would need to be +* snapshotted. If there are, it is not safe to fall back to +* *fallback. +*/ + QLIST_FOREACH(child, &bs->children, next) { +if (child->role & (BDRV_CHILD_DATA |BDRV_CHILD_METADATA | +BDRV_CHILD_FILTERED ) && + child != *fallback) + { +return NULL ; + } + } + +return fallback; +} + +static BlockDriverState *bdrv_snapshot_fallback (BlockDriverState *bs) +{ +BdrvChild **child_ptr =bdrv_snapshot_fallback_ptr (bs); +return child_ptr ? (*child_ptr)->bs :NULL ; +} +int bdrv_can_snapshot (BlockDriverState *bs) {BlockDriver *drv = bs->drv;
@@ -155,8 +205,9 @@ int bdrv_can_snapshot(BlockDriverState *bs) }if (!drv->bdrv_snapshot_create) { -if (bs->file !=NULL ) { -return bdrv_can_snapshot (bs->file->bs); +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs); +if (fallback_bs) { +return bdrv_can_snapshot (fallback_bs); }return 0 ; }
@@ -168,14 +219,15 @@ int bdrv_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_create) {return drv->bdrv_snapshot_create (bs, sn_info); } -if (bs->file) { -return bdrv_snapshot_create (bs->file->bs, sn_info); +if (fallback_bs) { +return bdrv_snapshot_create (fallback_bs, sn_info); }return -ENOTSUP ; }
@@ -185,6 +237,7 @@ int bdrv_snapshot_goto(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BdrvChild **fallback_ptr;int ret, open_ret;if (!drv) {
@@ -205,39 +258,46 @@ int bdrv_snapshot_goto(BlockDriverState *bs,return ret; } -if (bs->file) { -BlockDriverState *file; -QDict *options =qdict_clone_shallow (bs->options); + fallback_ptr =bdrv_snapshot_fallback_ptr (bs); +if (fallback_ptr) { +QDict *options;QDict *file_options;Error *local_err =NULL ; +BlockDriverState *fallback_bs = (*fallback_ptr)->bs; +char *subqdict_prefix =g_strdup_printf ("%s." , (*fallback_ptr)->name); + + options =qdict_clone_shallow (bs->options); - file = bs->file->bs;/* Prevent it from getting deleted when detached from bs */ -bdrv_ref (file); +bdrv_ref (fallback_bs); -qdict_extract_subqdict (options, &file_options,"file." ); +qdict_extract_subqdict (options, &file_options, subqdict_prefix);qobject_unref (file_options); -qdict_put_str (options,"file" ,bdrv_get_node_name (file)); +g_free (subqdict_prefix); + +qdict_put_str (options, (*fallback_ptr)->name, +bdrv_get_node_name (fallback_bs));if (drv->bdrv_close) { drv->bdrv_close (bs); } -bdrv_unref_child (bs, bs->file); - bs->file =NULL ; - ret =bdrv_snapshot_goto (file, snapshot_id, errp); +bdrv_unref_child (bs, *fallback_ptr); + *fallback_ptr =NULL ; + + ret =bdrv_snapshot_goto (fallback_bs, snapshot_id, errp); open_ret = drv->bdrv_open (bs, options, bs->open_flags, &local_err);qobject_unref (options);if (open_ret <0 ) { -bdrv_unref (file); +bdrv_unref (fallback_bs); bs->drv =NULL ;/* A bdrv_snapshot_goto() error takes precedence */ error_propagate (errp, local_err);return ret <0 ? ret : open_ret; } -assert (bs->file->bs == file); -bdrv_unref (file); +assert (fallback_bs == (*fallback_ptr)->bs); +bdrv_unref (fallback_bs);return ret; }
@@ -273,6 +333,7 @@ int bdrv_snapshot_delete(BlockDriverState *bs, Error **errp) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);int ret;if (!drv) {
@@ -289,8 +350,8 @@ int bdrv_snapshot_delete(BlockDriverState *bs,if (drv->bdrv_snapshot_delete) { ret = drv->bdrv_snapshot_delete (bs, snapshot_id, name, errp); - }else if (bs->file) { - ret =bdrv_snapshot_delete (bs->file->bs, snapshot_id, name, errp); + }else if (fallback_bs) { + ret =bdrv_snapshot_delete (fallback_bs, snapshot_id, name, errp); }else { error_setg(errp, "Block format '%s' used by device '%s' " "does not support internal snapshot deletion",
@@ -306,14 +367,15 @@ int bdrv_snapshot_list(BlockDriverState *bs, QEMUSnapshotInfo **psn_info) {BlockDriver *drv = bs->drv; +BlockDriverState *fallback_bs =bdrv_snapshot_fallback (bs);if (!drv) {return -ENOMEDIUM ; }if (drv->bdrv_snapshot_list) {return drv->bdrv_snapshot_list (bs, psn_info); } -if (bs->file) { -return bdrv_snapshot_list (bs->file->bs, psn_info); +if (fallback_bs) { +return bdrv_snapshot_list (fallback_bs, psn_info); }return -ENOTSUP ; }
Signature · 1 line
2.26.2
Re: [PATCH v7 23/47] block/snapshot: Fix fallback
Quoted 3 lines · Max Reitz
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there
I guess it comes to COW children like BDRV_CHILD_DATA | BDRV_CHILD_METADATA rather than non-COW ones, does it?
Andrey
Quoted 11 lines
is more than one such child. Furthermore, we really only can fall back to
bs->fileandbs->backing, becausebdrv_snapshot_goto()has to modify the child link (notably, set it to NULL).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>--- block/snapshot.c | 104 +++++++++++++++++++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 21 deletions(-)block/snapshot.c +0
-0
...
Quoted 16 lines
block/snapshot.c +16
-0+/* +* Check that there are no other children that would need to be +* snapshotted. If there are, it is not safe to fall back to +* *fallback. +*/ + QLIST_FOREACH(child, &bs->children, next) { +if (child->role & (BDRV_CHILD_DATA |BDRV_CHILD_METADATA | +BDRV_CHILD_FILTERED ) && + child != *fallback) + { +return NULL ; + } + } + +return fallback; +}
...
Re: [PATCH v7 23/47] block/snapshot: Fix fallback
Quoted 7 lines · Andrey Shinkevich
Quoted 3 lines · Max Reitz
If the top node's driver does not provide snapshot functionality and we want to fall back to a node down the chain, we need to snapshot all non-COW children. For simplicity's sake, just do not fall back if there
I guess it comes to COW children like
BDRV_CHILD_DATA| BDRV_CHILD_METADATA rather than non-COW ones, does it?
The BDRV_CHILD_COW is mutually exclusive with DATA, METADATA and FILTERED per definition.
Sorry about the question.
Quoted 37 lines
Andrey
Quoted 11 lines
is more than one such child. Furthermore, we really only can fall back to
bs->fileandbs->backing, becausebdrv_snapshot_goto()has to modify the child link (notably, set it to NULL).Suggested-by Vladimir Sementsov-Ogievskiy <[email protected]>Signed-off-by Max Reitz <[email protected]>---
block/snapshot.c| 104 +++++++++++++++++++++++++++++++++++++----------1 file changed, 83 insertions(+), 21 deletions(-)block/snapshot.c +0
-0...
Quoted 16 lines
+ /*
+ * Check that there are no other children that would need to be + * snapshotted. If there are, it is not safe to fall back to + * *fallback.
+ */
+QLIST_FOREACH(child, &bs->children, next){
+ if (child->role& (BDRV_CHILD_DATA|BDRV_CHILD_METADATA| +BDRV_CHILD_FILTERED) &&
+ child != *fallback)
+ {
+ return NULL;
+ }
+ }block/snapshot.c +3
-0+ + return fallback; +}...
Reviewed-by Andrey Shinkevich <[email protected]>
PATCH v7 24/47 block: Use CAFs for debug breakpoints
In reply to Max ReitzWhen looking for a blkdebug node (which implements debug breakpoints), use bdrv_primary_bs() to iterate through the graph, because that is where a blkdebug node would be.
--- block.c | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-)
block.c +3 -13
@@ -5562,17 +5562,7 @@ void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event)static BlockDriverState *bdrv_find_debug_node (BlockDriverState *bs) {while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) { -if (bs->file) { - bs = bs->file->bs; -continue ; - } - -if (bs->drv->is_filter && bs->backing) { - bs = bs->backing->bs; -continue ; - } - -break ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {
@@ -5607,7 +5597,7 @@ int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)int bdrv_debug_resume (BlockDriverState *bs,const char *tag) {while (bs && (!bs->drv || !bs->drv->bdrv_debug_resume)) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_resume) {
@@ -5620,7 +5610,7 @@ int bdrv_debug_resume(BlockDriverState *bs, const char *tag)bool bdrv_debug_is_suspended (BlockDriverState *bs,const char *tag) {while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
Signature · 1 line
2.26.2
Re: [PATCH v7 24/47] block: Use CAFs for debug breakpoints
Quoted 42 lines · Max Reitz
When looking for a blkdebug node (which implements debug breakpoints), use
bdrv_primary_bs()to iterate through the graph, because that is where a blkdebug node would be.Signed-off-by Max Reitz <[email protected]>--- block.c | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-)block.c +2
-12@@ -5562,17 +5562,7 @@ void bdrv_debug_event(BlockDriverState *bs, BlkdebugEvent event)static BlockDriverState *bdrv_find_debug_node (BlockDriverState *bs) {while (bs && bs->drv && !bs->drv->bdrv_debug_breakpoint) { -if (bs->file) { - bs = bs->file->bs; -continue ; - } - -if (bs->drv->is_filter && bs->backing) { - bs = bs->backing->bs; -continue ; - } - -break ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_breakpoint) {@@ -5607,7 +5597,7 @@ int bdrv_debug_remove_breakpoint(BlockDriverState *bs, const char *tag)int bdrv_debug_resume (BlockDriverState *bs,const char *tag) {while (bs && (!bs->drv || !bs->drv->bdrv_debug_resume)) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_resume) {
block.c int bdrv_debug_resume(BlockDriverState *bs, const char *tag) · line 5616
Quoted 8 lines
block.c +1
-1@@ -5620,7 +5610,7 @@ int bdrv_debug_resume(BlockDriverState *bs, const char *tag)bool bdrv_debug_is_suspended (BlockDriverState *bs,const char *tag) {while (bs && bs->drv && !bs->drv->bdrv_debug_is_suspended) { - bs = bs->file ? bs->file->bs :NULL ; + bs =bdrv_primary_bs (bs); }if (bs && bs->drv && bs->drv->bdrv_debug_is_suspended) {
PATCH v7 25/47 block: Def. impl.s for get_allocated_file_size
In reply to Max ReitzIf every BlockDriver were to implement bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return -ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizes
For the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child
(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.
--- include/block/block_int.h | 5 ++++ block.c | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)
include/block/block_int.h +5 -0
@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,int64_t *pnum, int64_t *map, BlockDriverState **file); + +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs); +const char *bdrv_get_parent_name (const BlockDriverState *bs);void blk_dev_change_media_cb (BlockBackend *blk,bool load,Error **errp);bool blk_dev_has_removable_media (BlockBackend *blk);
block.c +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.26.2
Re: [PATCH v7 25/47] block: Def. impl.s for get_allocated_file_size
Quoted 90 lines · Max Reitz
If every BlockDriver were to implement
bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return-ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizesFor the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 5 ++++ block.c | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)include/block/block_int.h +5
-0@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,int64_t *pnum, int64_t *map, BlockDriverState **file); + +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs); +const char *bdrv_get_parent_name (const BlockDriverState *bs);void blk_dev_change_media_cb (BlockBackend *blk,bool load,Error **errp);bool blk_dev_has_removable_media (BlockBackend *blk);block.c +44
-0@@ -5003,6 +5003,57 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)return -ENOTSUP ; } +/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* block drivers that want it to sum all children they store data on. +* (This excludes backing children.) +*/ +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs) +{ +BdrvChild *child; +int64_t child_size, sum =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child->role & (BDRV_CHILD_DATA |BDRV_CHILD_METADATA | +BDRV_CHILD_FILTERED )) + { + child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size; + } + sum += child_size; + } + } + +return sum; +} + +/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* block drivers that want it to return only the size of a node's +* primary child. +*/ +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs) +{ +BlockDriverState *primary_bs; + + primary_bs =bdrv_primary_bs (bs); +if (!primary_bs) { +return -ENOTSUP ; + } + +return bdrv_get_allocated_file_size (primary_bs); +} + +/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5059
Quoted 10 lines
block.c +7
-0+* protocol block drivers that just do not support it. +*/ +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs) +{ +return -ENOTSUP ; +} +/* * bdrv_measure: * @drv: Format driver
Re: [PATCH v7 25/47] block: Def. impl.s for get_allocated_file_size
In reply to Max ReitzAm 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 60 lines
If every BlockDriver were to implement
bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return-ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizesFor the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 5 ++++ block.c | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)include/block/block_int.h +5
-0@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,int64_t *pnum, int64_t *map, BlockDriverState **file); + +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs); +const char *bdrv_get_parent_name (const BlockDriverState *bs);void blk_dev_change_media_cb (BlockBackend *blk,bool load,Error **errp);bool blk_dev_has_removable_media (BlockBackend *blk);block.c +14
-0@@ -5003,6 +5003,57 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)return -ENOTSUP ; } +/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* block drivers that want it to sum all children they store data on. +* (This excludes backing children.) +*/ +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs) +{ +BdrvChild *child; +int64_t child_size, sum =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child->role & (BDRV_CHILD_DATA |BDRV_CHILD_METADATA | +BDRV_CHILD_FILTERED )) + {
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5029
Quoted 10 lines
block.c +10
-0+ child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size; + } + sum += child_size; + } + } + +return sum; +}
The only user apart from bdrv_get_allocated_file_size() is blkverify. As I argued that blkverify shouldn't use it, this can become static.
Quoted 6 lines
block.c +6
-0+/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* block drivers that want it to return only the size of a node's +* primary child. +*/ +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs)
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5046
Quoted 10 lines
block.c +10
-0+{ +BlockDriverState *primary_bs; + + primary_bs =bdrv_primary_bs (bs); +if (!primary_bs) { +return -ENOTSUP ; + } + +return bdrv_get_allocated_file_size (primary_bs); +}
This can become static, too (never used as a callback), and possibly even be inlined.
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5055
Quoted 8 lines
block.c +8
-0+/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* protocol block drivers that just do not support it. +*/ +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs) +{ +return -ENOTSUP ; +}
Also never used as a callback. I think inlining it would almost certainly make more sense.
Kevin
Re: [PATCH v7 25/47] block: Def. impl.s for get_allocated_file_size
Quoted 95 lines · Kevin Wolf
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 60 lines
If every BlockDriver were to implement
bdrv_get_allocated_file_size(), there are basically three ways it would be handled:
(1) For protocol drivers: Figure out the actual allocated file size in some protocol-specific way
(2) For protocol drivers: If that is not possible (or we just have not bothered to implement it yet), return-ENOTSUP
(3) For drivers with children: Return the sum of some or all their children's sizesFor the drivers we have, case (3) boils down to either:
(a) The sum of all children's sizes
(b) The size of the primary child(2), (3a) and (3b) can be implemented generically, so this patch adds such generic implementations for drivers to use.
Signed-off-by Max Reitz <[email protected]>--- include/block/block_int.h | 5 ++++ block.c | 51 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+)include/block/block_int.h +5
-0@@ -1318,6 +1318,11 @@ int coroutine_fn bdrv_co_block_status_from_backing(BlockDriverState *bs,int64_t *pnum, int64_t *map, BlockDriverState **file); + +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs); +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs); +const char *bdrv_get_parent_name (const BlockDriverState *bs);void blk_dev_change_media_cb (BlockBackend *blk,bool load,Error **errp);bool blk_dev_has_removable_media (BlockBackend *blk);block.c +14
-0@@ -5003,6 +5003,57 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)return -ENOTSUP ; } +/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* block drivers that want it to sum all children they store data on. +* (This excludes backing children.) +*/ +int64_t bdrv_sum_allocated_file_size (BlockDriverState *bs) +{ +BdrvChild *child; +int64_t child_size, sum =0 ; + + QLIST_FOREACH(child, &bs->children, next) { +if (child->role & (BDRV_CHILD_DATA |BDRV_CHILD_METADATA | +BDRV_CHILD_FILTERED )) + {block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5029
Quoted 10 lines
block.c +10
-0+ child_size =bdrv_get_allocated_file_size (child->bs); +if (child_size <0 ) { +return child_size; + } + sum += child_size; + } + } + +return sum; +}The only user apart from
bdrv_get_allocated_file_size()is blkverify. As I argued that blkverify shouldn't use it, this can become static.Quoted 6 lines
block.c +6
-0+/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* block drivers that want it to return only the size of a node's +* primary child. +*/ +int64_t bdrv_primary_allocated_file_size (BlockDriverState *bs)block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5046
Quoted 10 lines
block.c +10
-0+{ +BlockDriverState *primary_bs; + + primary_bs =bdrv_primary_bs (bs); +if (!primary_bs) { +return -ENOTSUP ; + } + +return bdrv_get_allocated_file_size (primary_bs); +}This can become static, too (never used as a callback), and possibly even be inlined.
Quoted 11 lines
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5055
Quoted 8 lines
block.c +8
-0+/** +* Implementation of BlockDriver.bdrv_get_allocated_file_size() for +* protocol block drivers that just do not support it. +*/ +int64_t bdrv_notsup_allocated_file_size (BlockDriverState *bs) +{ +return -ENOTSUP ; +}Also never used as a callback. I think inlining it would almost certainly make more sense.
I think they’re all artifacts from the development process, yeah.
Originally, I wanted to make .bdrv_get_allocated_file_size() mandatory, but then I saw that led nowhere and could be done well generically.
Max
PATCH v7 26/47 block: Improve get_allocated_file_size's default
In reply to Max ReitzThere are two practical problems with bdrv_get_allocated_file_size()'s default right now:
(1) For drivers with children, we should generally sum all their sizes instead of just passing the request through to bs->file. The latter is good for filters, but not so much for format drivers.
(2) Filters need not have bs->file, so we should actually go to the
filtered child instead of hard-coding bs->file.
And we can make the whole default implementation more idiomatic by using the three generic functions added by the previous patch.
--- block.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
block.c +14 -3
@@ -4997,10 +4997,21 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (drv->bdrv_file_open) { +/* +* Protocol drivers default to -ENOTSUP (most of their data is +* not stored in any of their children (if they even have any), +* so there is no generic way to figure it out). +*/ +return bdrv_notsup_allocated_file_size (bs); + }else if (drv->is_filter) { +/* Filter drivers default to the size of their primary child */ +return bdrv_primary_allocated_file_size (bs); + }else { +/* Other drivers default to summing their children's sizes */ +return bdrv_sum_allocated_file_size (bs); } -return -ENOTSUP ; }/**
Signature · 1 line
2.26.2
Re: [PATCH v7 26/47] block: Improve get_allocated_file_size's default
Quoted 36 lines · Max Reitz
There are two practical problems with bdrv_get_allocated_file_size()'s default right now:
(1) For drivers with children, we should generally sum all their sizes instead of just passing the request through tobs->file. The latter is good for filters, but not so much for format drivers.(2) Filters need not have
bs->file, so we should actually go to thefiltered child instead of hard-coding bs->file.And we can make the whole default implementation more idiomatic by using the three generic functions added by the previous patch.
Signed-off-by Max Reitz <[email protected]>--- block.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)block.c +9
-2@@ -4997,10 +4997,21 @@ int64_t bdrv_get_allocated_file_size(BlockDriverState *bs)if (drv->bdrv_get_allocated_file_size) {return drv->bdrv_get_allocated_file_size (bs); } -if (bs->file) { -return bdrv_get_allocated_file_size (bs->file->bs); + +if (drv->bdrv_file_open) { +/* +* Protocol drivers default to -ENOTSUP (most of their data is +* not stored in any of their children (if they even have any), +* so there is no generic way to figure it out). +*/ +return bdrv_notsup_allocated_file_size (bs); + }else if (drv->is_filter) {
block.c int64_t bdrv_get_allocated_file_size(BlockDriverState *bs) · line 5017
Quoted 10 lines
block.c +5
-1+/* Filter drivers default to the size of their primary child */ +return bdrv_primary_allocated_file_size (bs); + }else { +/* Other drivers default to summing their children's sizes */ +return bdrv_sum_allocated_file_size (bs); } -return -ENOTSUP ; }/**
PATCH v7 27/47 blkverify: Use bdrv_sum_allocated_file_size()
In reply to Max Reitzblkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.
--- block/blkverify.c | 1 + 1 file changed, 1 insertion(+)
block/blkverify.c +1 -0
@@ -323,6 +323,7 @@ static BlockDriver bdrv_blkverify = { .bdrv_getlength = blkverify_getlength, .bdrv_refresh_filename = blkverify_refresh_filename, .bdrv_dirname = blkverify_dirname, + .bdrv_get_allocated_file_size = bdrv_sum_allocated_file_size, .bdrv_co_preadv = blkverify_co_preadv, .bdrv_co_pwritev = blkverify_co_pwritev,
Signature · 1 line
2.26.2
Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()
Quoted 15 lines · Max Reitz
blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.
Signed-off-by Max Reitz <[email protected]>--- block/blkverify.c | 1 + 1 file changed, 1 insertion(+)block/blkverify.c +0
-0
block/blkverify.c static BlockDriver bdrv_blkverify = { · line 329
Quoted 7 lines
block/blkverify.c +1
-0@@ -323,6 +323,7 @@ static BlockDriver bdrv_blkverify = { .bdrv_getlength = blkverify_getlength, .bdrv_refresh_filename = blkverify_refresh_filename, .bdrv_dirname = blkverify_dirname, + .bdrv_get_allocated_file_size = bdrv_sum_allocated_file_size, .bdrv_co_preadv = blkverify_co_preadv, .bdrv_co_pwritev = blkverify_co_pwritev,
Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()
In reply to Max ReitzAm 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 4 lines
blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.
Hm, but so are the children of, say, backup-top. I don't think you're suggesting that backup-top should add the sizes of both images, even though the backup job is actively increasing the allocated size of the non-primary node, much like blkverify.
So I believe returning only the allocated size of the primary child in blkverify would be more consistent with what we do elsewhere.
Kevin
Re: [PATCH v7 27/47] blkverify: Use bdrv_sum_allocated_file_size()
Quoted 8 lines · Kevin Wolf
Am 25.06.2020 um 17:21 hat Max Reitz geschrieben:
Quoted 4 lines
blkverify is a filter, so bdrv_get_allocated_file_size()'s default implementation will return only the size of its filtered child. However, because both of its children are disk images, it makes more sense to sum both of their allocated sizes.
Hm, but so are the children of, say, backup-top. I don't think you're suggesting that backup-top should add the sizes of both images,
Can be argued either way.
I lean on the side of that it should not, because: The backup is external. The job is copying data off. So it isn’t really directly data that serves qemu, which can be seen from the fact that it’s never read.
Which is not the case for quorum and blkverify. Though one can argue that blkverify is different from quorum here in that it doesn’t read data from the non-filtered child to serve a guest device, but just to verify it node-internally.
Quoted 6 lines
even
though the backup job is actively increasing the allocated size of the non-primary node, much like blkverify.So I believe returning only the allocated size of the primary child in blkverify would be more consistent with what we do elsewhere.
For me, blkverify is basically an archaic mode of quorum, and for quorum it’s clear that we should sum the sizes. Which is why I thought summing the sizes would be more consistent.
But honestly, I just don’t care about blkverify whatsoever. I don’t believe anyone actually cares about whether what blkverify returns for .bdrv_get_allocated_file_size() is consistent. I believe we could return 42 and nobody would bat an eyelash.
(But that’s the curse of this series. I have to touch stuff that nobody cares about, and then we have discussions on stuff nobody cares about.)
So from that POV I’m happy to drop this patch if it means there’s just one less opportunity to have a discussion on blkverify.
Max
PATCH v7 28/47 block/null: Implement bdrv_get_allocated_file_size
In reply to Max ReitzIt is trivial, so we might as well do it.
--- block/null.c | 7 +++++++ tests/qemu-iotests/153.out | 2 +- tests/qemu-iotests/184.out | 6 ++++-- 3 files changed, 12 insertions(+), 3 deletions(-)
block/null.c +7 -0
@@ -262,6 +262,11 @@ static void null_refresh_filename(BlockDriverState *bs) bs->drv->format_name); } +static int64_t null_allocated_file_size (BlockDriverState *bs) +{ +return 0 ; +} +static const char *const null_strong_runtime_opts[] = {BLOCK_OPT_SIZE ,NULL_OPT_ZEROES ,
@@ -277,6 +282,7 @@ static BlockDriver bdrv_null_co = { .bdrv_file_open = null_file_open, .bdrv_parse_filename = null_co_parse_filename, .bdrv_getlength = null_getlength, + .bdrv_get_allocated_file_size = null_allocated_file_size, .bdrv_co_preadv = null_co_preadv, .bdrv_co_pwritev = null_co_pwritev,
@@ -297,6 +303,7 @@ static BlockDriver bdrv_null_aio = { .bdrv_file_open = null_file_open, .bdrv_parse_filename = null_aio_parse_filename, .bdrv_getlength = null_getlength, + .bdrv_get_allocated_file_size = null_allocated_file_size, .bdrv_aio_preadv = null_aio_preadv, .bdrv_aio_pwritev = null_aio_pwritev,
tests/qemu-iotests/153.out +1 -1
@@ -461,7 +461,7 @@ No conflict: image: null-co:// file format: null-co virtual size: 1 GiB (1073741824 bytes) -disk size: unavailable +disk size: 0 B Conflict: qemu-img: --force-share/-U conflicts with image options
tests/qemu-iotests/184.out +4 -2
@@ -29,7 +29,8 @@ Testing: "image": { "virtual-size": 1073741824, "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}", - "format": "throttle" + "format": "throttle", + "actual-size": SIZE }, "iops_wr": 0, "ro": false,
@@ -56,7 +57,8 @@ Testing: "image": { "virtual-size": 1073741824, "filename": "null-co://", - "format": "null-co" + "format": "null-co", + "actual-size": SIZE }, "iops_wr": 0, "ro": false,
Signature · 1 line
2.26.2
Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size
Quoted 59 lines · Max Reitz
It is trivial, so we might as well do it.
Signed-off-by Max Reitz <[email protected]>--- block/null.c | 7 +++++++ tests/qemu-iotests/153.out | 2 +- tests/qemu-iotests/184.out | 6 ++++-- 3 files changed, 12 insertions(+), 3 deletions(-)block/null.c +7
-0@@ -262,6 +262,11 @@ static void null_refresh_filename(BlockDriverState *bs) bs->drv->format_name); } +static int64_t null_allocated_file_size (BlockDriverState *bs) +{ +return 0 ; +} +static const char *const null_strong_runtime_opts[] = {BLOCK_OPT_SIZE ,NULL_OPT_ZEROES ,@@ -277,6 +282,7 @@ static BlockDriver bdrv_null_co = { .bdrv_file_open = null_file_open, .bdrv_parse_filename = null_co_parse_filename, .bdrv_getlength = null_getlength, + .bdrv_get_allocated_file_size = null_allocated_file_size, .bdrv_co_preadv = null_co_preadv, .bdrv_co_pwritev = null_co_pwritev,@@ -297,6 +303,7 @@ static BlockDriver bdrv_null_aio = { .bdrv_file_open = null_file_open, .bdrv_parse_filename = null_aio_parse_filename, .bdrv_getlength = null_getlength, + .bdrv_get_allocated_file_size = null_allocated_file_size, .bdrv_aio_preadv = null_aio_preadv, .bdrv_aio_pwritev = null_aio_pwritev,tests/qemu-iotests/153.out +1
-1@@ -461,7 +461,7 @@ No conflict: image: null-co:// file format: null-co virtual size: 1 GiB (1073741824 bytes) -disk size: unavailable +disk size: 0 B Conflict: qemu-img: --force-share/-U conflicts with image optionstests/qemu-iotests/184.out +0
-0
tests/qemu-iotests/184.out Testing: · line 33
Quoted 6 lines
tests/qemu-iotests/184.out +2
-1@@ -29,7 +29,8 @@ Testing: "image": { "virtual-size": 1073741824, "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}", - "format": "throttle" + "format": "throttle", + "actual-size": SIZE
If we remove the _filter_generated_node_ids() in the current implementation of the test #184, we will get '"actual-size": 0'. It might be more informative when analyzing the output in 184.out.
Andrey
tests/qemu-iotests/184.out Testing: · line 64
Quoted 13 lines
tests/qemu-iotests/184.out +2
-1}, "iops_wr": 0, "ro": false,@@ -56,7 +57,8 @@ Testing: "image": { "virtual-size": 1073741824, "filename": "null-co://", - "format": "null-co" + "format": "null-co", + "actual-size": SIZE }, "iops_wr": 0, "ro": false,
Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size
Quoted 61 lines · Andrey Shinkevich
Quoted 60 lines · Max Reitz
It is trivial, so we might as well do it.
Signed-off-by Max Reitz <[email protected]>---
block/null.c | 7 +++++++
tests/qemu-iotests/153.out | 2 +-
tests/qemu-iotests/184.out | 6 ++++--3 files changed, 12 insertions(+), 3 deletions(-)block/null.c +4
-0@@ -262,6 +262,11 @@ static voidnull_refresh_filename (BlockDriverState *bs) bs->drv->format_name); } +static int64_t null_allocated_file_size(BlockDriverState *bs) +{ + return 0; +} + static const char *const null_strong_runtime_opts[] = { BLOCK_OPT_SIZE,
NULL_OPT_ZEROES,block/null.c +2
-0@@ -277,6 +282,7 @@ static BlockDriver bdrv_null_co = { .bdrv_file_open = null_file_open, .bdrv_parse_filename = null_co_parse_filename, .bdrv_getlength = null_getlength, + .bdrv_get_allocated_file_size = null_allocated_file_size, .bdrv_co_preadv = null_co_preadv, .bdrv_co_pwritev = null_co_pwritev,@@ -297,6 +303,7 @@ static BlockDriver bdrv_null_aio = { .bdrv_file_open = null_file_open, .bdrv_parse_filename = null_aio_parse_filename, .bdrv_getlength = null_getlength, + .bdrv_get_allocated_file_size = null_allocated_file_size, .bdrv_aio_preadv = null_aio_preadv, .bdrv_aio_pwritev = null_aio_pwritev,tests/qemu-iotests/153.out +0
-0@@ -461,7 +461,7 @@ No conflict:image: null-co://
file format: null-co
virtual size: 1 GiB (1073741824 bytes)tests/qemu-iotests/153.out +1
-1-disk size: unavailable +disk size: 0 BConflict:
qemu-img: --force-share/-U conflicts with image optionstests/qemu-iotests/184.out +0
-0@@ -29,7 +29,8 @@ Testing: "image": { "virtual-size": 1073741824, "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}",
Quoted 8 lines
tests/qemu-iotests/184.out Testing: · line 33
Quoted 3 lines
tests/qemu-iotests/184.out +2
-1- "format": "throttle" + "format": "throttle", + "actual-size": SIZEIf we remove the
_filter_generated_node_ids()in the current implementation of the test #184, we will get '"actual-size": 0'. It might be more informative when analyzing the output in 184.out.
You mean _filter_actual_image_size? Yeah, why not, it doesn’t seem necessary here.
Max
Re: [PATCH v7 28/47] block/null: Implement bdrv_get_allocated_file_size
Quoted 10 lines · Max Reitz
Quoted 9 lines · Andrey Shinkevich
Quoted 8 lines · Max Reitz
It is trivial, so we might as well do it.
Signed-off-by Max Reitz <[email protected]>--- block/null.c | 7 +++++++ tests/qemu-iotests/153.out | 2 +- tests/qemu-iotests/184.out | 6 ++++-- 3 files changed, 12 insertions(+), 3 deletions(-)
...
Quoted 16 lines
Quoted 8 lines
Quoted 8 lines
tests/qemu-iotests/184.out +0
-0@@ -29,7 +29,8 @@ Testing: "image": { "virtual-size": 1073741824, "filename": "json:{\"throttle-group\": \"group0\",Quoted 8 lines
tests/qemu-iotests/184.out Testing: · line 33
Quoted 4 lines
tests/qemu-iotests/184.out +2
-1\"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}", - "format": "throttle" + "format": "throttle", + "actual-size": SIZEIf we remove the
_filter_generated_node_ids()in the current implementation of the test #184, we will get '"actual-size": 0'. It might be more informative when analyzing the output in 184.out.
Quoted 5 lines
You mean
_filter_actual_image_size? Yeah, why not, it doesn’t seem necessary here.Max
Yes Max, you are right, I ment the _filter_actual_image_size. It was my copy/paste mistake.
Andrey
PATCH v7 29/47 blockdev: Use CAF in external_snapshot_prepare()
In reply to Max ReitzThis allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
--- blockdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
blockdev.c +6 -1
@@ -1549,7 +1549,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } -if (state->new_bs->backing !=NULL ) { +if (state->new_bs->drv->is_filter) { +error_setg (errp,"Filters cannot be used as overlays" ); + goto out; + } + +if (bdrv_cow_child (state->new_bs)) {error_setg (errp,"The overlay already has a backing image" ); goto out; }
Signature · 1 line
2.26.2
Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()
Quoted 14 lines · Max Reitz
This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
Signed-off-by Max Reitz <[email protected]>--- blockdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)blockdev.c +0
-0
blockdev.c static void external_snapshot_prepare(BlkActionState *common, · line 1552
Quoted 5 lines
blockdev.c +1
-1@@ -1549,7 +1549,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } -if (state->new_bs->backing !=NULL ) { +if (state->new_bs->drv->is_filter) {
Is there a chance to get a filter here? If so, is that when a user specifies the file name of such a kind “filter[filter-name]:foo.qcow2” or somehow else?
Andrey
blockdev.c static void external_snapshot_prepare(BlkActionState *common, · line 1560
Quoted 8 lines
blockdev.c +5
-0+error_setg (errp,"Filters cannot be used as overlays" ); + goto out; + } + +if (bdrv_cow_child (state->new_bs)) {error_setg (errp,"The overlay already has a backing image" ); goto out; }
Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()
Quoted 10 lines · Andrey Shinkevich
Quoted 9 lines · Max Reitz
This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
Signed-off-by Max Reitz <[email protected]>---
blockdev.c| 7 ++++++-1 file changed, 6 insertions(+), 1 deletion(-)
Quoted 15 lines
blockdev.c static void · line 1553
Quoted 10 lines
blockdev.c +1
-0@@ -1549,7 +1549,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } - if (state->new_bs->backing != NULL) { + if (state->new_bs->drv->is_filter) {Is there a chance to get a filter here? If so, is that when a user specifies the file name of such a kind “filter[filter-name]:foo.qcow2” or somehow else?
It would be with blockdev-snapshot and by specifying a filter for @overlay. Technically that’s already caught by the check whether the overlay supports backing images (whether drv->supports_backing is true), but we might as well give a nicer error message.
Example:
{"execute":"qmp_capabilities"}
{"execute":"blockdev-add","arguments":
{"node-name":"overlay","driver":"copy-on-read",
"file":{"driver":"null-co"}}}
{"execute":"blockdev-add","arguments":
{"node-name":"base","driver":"null-co"}}
{"execute":"blockdev-snapshot","arguments":
{"node":"base","overlay":"overlay"}}
Max
Re: [PATCH v7 29/47] blockdev: Use CAF in external_snapshot_prepare()
Quoted 38 lines · Max Reitz
Quoted 10 lines · Andrey Shinkevich
Quoted 9 lines · Max Reitz
This allows us to differentiate between filters and nodes with COW backing files: Filters cannot be used as overlays at all (for this function).
Signed-off-by Max Reitz <[email protected]>--- blockdev.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)Quoted 14 lines
blockdev.c static void · line 1553
Quoted 10 lines
blockdev.c +1
-0@@ -1549,7 +1549,12 @@ static void external_snapshot_prepare(BlkActionState *common, goto out; } - if (state->new_bs->backing != NULL) { + if (state->new_bs->drv->is_filter) {Is there a chance to get a filter here? If so, is that when a user specifies the file name of such a kind “filter[filter-name]:foo.qcow2” or somehow else?
It would be with blockdev-snapshot and by specifying a filter for @overlay. Technically that’s already caught by the check whether the overlay supports backing images (whether
drv->supports_backingis true), but we might as well give a nicer error message.Example:
{"execute":"qmp_capabilities"}
{"execute":"blockdev-add","arguments":
{"node-name":"overlay","driver":"copy-on-read",
"file":{"driver":"null-co"}}}
Quoted 8 lines
{"execute":"blockdev-add","arguments":
{"node-name":"base","driver":"null-co"}}{"execute":"blockdev-snapshot","arguments":
{"node":"base","overlay":"overlay"}}Max
Thank you for the example.
Andrey
PATCH v7 30/47 block: Report data child for query-blockstats
In reply to Max ReitzIt makes no sense to report the block stats of a purely metadata-storing child in query-blockstats. So if the primary child does not have any data, try to find a unique data-storing child.
--- block/qapi.c | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)
block/qapi.c +29 -2
@@ -526,6 +526,7 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk)static BlockStats *bdrv_query_bds_stats (BlockDriverState *bs,bool blk_level) { +BdrvChild *parent_child;BlockStats *s =NULL ; s =g_malloc0 (sizeof (*s));
@@ -555,9 +556,35 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, s->has_driver_specific = true; } -if (bs->file) { + parent_child =bdrv_primary_child (bs); +if (!parent_child || + !(parent_child->role & (BDRV_CHILD_DATA |BDRV_CHILD_FILTERED ))) + { +BdrvChild *c; + +/* +* Look for a unique data-storing child. We do not need to look for +* filtered children, as there would be only one and it would have been +* the primary child. +*/ + parent_child =NULL ; + QLIST_FOREACH(c, &bs->children, next) { +if (c->role &BDRV_CHILD_DATA ) { +if (parent_child) { +/* +* There are multiple data-storing children and we cannot +* choose between them. +*/ + parent_child =NULL ; +break ; + } + parent_child = c; + } + } + } +if (parent_child) { s->has_parent = true; - s->parent =bdrv_query_bds_stats (bs->file->bs, blk_level); + s->parent =bdrv_query_bds_stats (parent_child->bs, blk_level); }if (blk_level && bs->backing) {
Signature · 1 line
2.26.2
Re: [PATCH v7 30/47] block: Report data child for query-blockstats
Quoted 49 lines · Max Reitz
It makes no sense to report the block stats of a purely metadata-storing child in query-blockstats. So if the primary child does not have any data, try to find a unique data-storing child.
Signed-off-by Max Reitz <[email protected]>--- block/qapi.c | 31 +++++++++++++++++++++++++++++-- 1 file changed, 29 insertions(+), 2 deletions(-)block/qapi.c +24
-1@@ -526,6 +526,7 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk)static BlockStats *bdrv_query_bds_stats (BlockDriverState *bs,bool blk_level) { +BdrvChild *parent_child;BlockStats *s =NULL ; s =g_malloc0 (sizeof (*s));@@ -555,9 +556,35 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, s->has_driver_specific = true; } -if (bs->file) { + parent_child =bdrv_primary_child (bs); +if (!parent_child || + !(parent_child->role & (BDRV_CHILD_DATA |BDRV_CHILD_FILTERED ))) + { +BdrvChild *c; + +/* +* Look for a unique data-storing child. We do not need to look for +* filtered children, as there would be only one and it would have been +* the primary child. +*/ + parent_child =NULL ; + QLIST_FOREACH(c, &bs->children, next) { +if (c->role &BDRV_CHILD_DATA ) { +if (parent_child) { +/* +* There are multiple data-storing children and we cannot +* choose between them. +*/ + parent_child =NULL ; +break ; + } + parent_child = c;
block/qapi.c static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, · line 590
Quoted 10 lines
block/qapi.c +5
-1+ } + } + } +if (parent_child) { s->has_parent = true; - s->parent =bdrv_query_bds_stats (bs->file->bs, blk_level); + s->parent =bdrv_query_bds_stats (parent_child->bs, blk_level); }if (blk_level && bs->backing) {
PATCH v7 31/47 block: Use child access functions for QAPI queries
In reply to Max Reitzquery-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just bs->backing or COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.
--- block/qapi.c | 33 ++++++++++++++++++++------------- tests/qemu-iotests/184.out | 8 +++++++- 2 files changed, 27 insertions(+), 14 deletions(-)
block/qapi.c +20 -13
@@ -163,9 +163,13 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,break ; } -if (bs0->drv && bs0->backing) { +if (bs0->drv &&bdrv_filter_or_cow_child (bs0)) { +/* +* Put any filtered child here (for backwards compatibility to when +* we put bs0->backing here, which might be any filtered child). +*/ info->backing_file_depth++; - bs0 = bs0->backing->bs; + bs0 =bdrv_filter_or_cow_bs (bs0); (*p_image_info)->has_backing_image = true; p_image_info = &((*p_image_info)->backing_image); }else {
@@ -174,9 +178,8 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,/* Skip automatically inserted nodes that the user isn't aware of for * query-block (blk != NULL), but not for query-named-block-nodes */ -while (blk && bs0->drv && bs0->implicit) { - bs0 =backing_bs (bs0); -assert (bs0); +if (blk) { + bs0 =bdrv_skip_implicit_filters (bs0); } }
@@ -362,9 +365,7 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,char *qdev;/* Skip automatically inserted nodes that the user isn't aware of */ -while (bs && bs->drv && bs->implicit) { - bs =backing_bs (bs); - } + bs =bdrv_skip_implicit_filters (bs); info->device =g_strdup (blk_name (blk)); info->type =g_strdup ("unknown" );
@@ -527,6 +528,7 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, bool blk_level) {BdrvChild *parent_child; +BlockDriverState *filter_or_cow_bs;BlockStats *s =NULL ; s =g_malloc0 (sizeof (*s));
@@ -539,9 +541,8 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs,/* Skip automatically inserted nodes that the user isn't aware of in * a BlockBackend-level command. Stay at the exact node for a node-level * command. */ -while (blk_level && bs->drv && bs->implicit) { - bs =backing_bs (bs); -assert (bs); +if (blk_level) { + bs =bdrv_skip_implicit_filters (bs); }if (bdrv_get_node_name (bs)[0 ]) {
@@ -587,9 +588,15 @@ static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, s->parent =bdrv_query_bds_stats (parent_child->bs, blk_level); } -if (blk_level && bs->backing) { + filter_or_cow_bs =bdrv_filter_or_cow_bs (bs); +if (blk_level && filter_or_cow_bs) { +/* +* Put any filtered or COW child here (for backwards +* compatibility to when we put bs0->backing here, which might +* be either) +*/ s->has_backing = true; - s->backing =bdrv_query_bds_stats (bs->backing->bs, blk_level); + s->backing =bdrv_query_bds_stats (filter_or_cow_bs, blk_level); }return s;
tests/qemu-iotests/184.out +7 -1
@@ -27,6 +27,12 @@ Testing: "iops_rd": 0, "detect_zeroes": "off", "image": { + "backing-image": { + "virtual-size": 1073741824, + "filename": "null-co://", + "format": "null-co", + "actual-size": SIZE + }, "virtual-size": 1073741824, "filename": "json:{\"throttle-group\": \"group0\", \"driver\": \"throttle\", \"file\": {\"driver\": \"null-co\"}}", "format": "throttle",
@@ -35,7 +41,7 @@ Testing: "iops_wr": 0, "ro": false, "node-name": "throttle0", - "backing_file_depth": 0, + "backing_file_depth": 1, "drv": "throttle", "iops": 0, "bps_wr": 0,
Signature · 1 line
2.26.2
Re: [PATCH v7 31/47] block: Use child access functions for QAPI queries
Quoted 11 lines · Max Reitz
query-block, query-named-block-nodes, and query-blockstats now return any filtered child under "backing", not just
bs->backingor COW children. This is so that filters do not interrupt the reported backing chain. This changes the output for iotest 184, as the throttled node now appears as a backing child.Signed-off-by Max Reitz <[email protected]>--- block/qapi.c | 33 ++++++++++++++++++++------------- tests/qemu-iotests/184.out | 8 +++++++- 2 files changed, 27 insertions(+), 14 deletions(-)
...
PATCH v7 32/47 block-copy: Use CAF to find sync=top base
In reply to Max Reitz--- block/block-copy.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/block-copy.c +2 -2
@@ -437,8 +437,8 @@ static int block_copy_block_status(BlockCopyState *s, int64_t offset,BlockDriverState *base;int ret; -if (s->skip_unallocated && s->source->bs->backing) { - base = s->source->bs->backing->bs; +if (s->skip_unallocated) { + base =bdrv_backing_chain_next (s->source->bs); }else { base =NULL ; }
Signature · 1 line
2.26.2
Re: [PATCH v7 32/47] block-copy: Use CAF to find sync=top base
Quoted 10 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/block-copy.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)block/block-copy.c +0
-0
block/block-copy.c static int block_copy_block_status(BlockCopyState *s, int64_t offset, · line 444
Quoted 10 lines
block/block-copy.c +2
-2@@ -437,8 +437,8 @@ static int block_copy_block_status(BlockCopyState *s, int64_t offset,BlockDriverState *base;int ret; -if (s->skip_unallocated && s->source->bs->backing) { - base = s->source->bs->backing->bs; +if (s->skip_unallocated) { + base =bdrv_backing_chain_next (s->source->bs); }else { base =NULL ; }
PATCH v7 33/47 mirror: Deal with filters
In reply to Max ReitzThis includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "target_backing_bs", because that is what it really refers to.
--- qapi/block-core.json | 6 ++- block/mirror.c | 118 +++++++++++++++++++++++++++++++++---------- blockdev.c | 36 +++++++++---- 3 files changed, 121 insertions(+), 39 deletions(-)
qapi/block-core.json +4 -2
@@ -1943,7 +1943,8 @@ # # @replaces: with sync=full graph node name to be replaced by the new # image when a whole image copy is done. This can be used to repair -# broken Quorum files. (Since 2.1) +# broken Quorum files. By default, @device is replaced, although +# implicitly created filters on it are kept. (Since 2.1) # # @mode: whether and how QEMU should create a new image, default is # 'absolute-paths'.
@@ -2254,7 +2255,8 @@ # # @replaces: with sync=full graph node name to be replaced by the new # image when a whole image copy is done. This can be used to repair -# broken Quorum files. +# broken Quorum files. By default, @device is replaced, although +# implicitly created filters on it are kept. # # @speed: the maximum speed, in bytes per second #
block/mirror.c +91 -27
206 lines not on this page · raw message
blockdev.c +26 -10
@@ -2886,6 +2886,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs, bool has_auto_dismiss, bool auto_dismiss, Error **errp) { + BlockDriverState *unfiltered_bs;int job_flags =JOB_DEFAULT ;if (!has_speed) {
@@ -2937,10 +2938,19 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,return ; } -if (!bs->backing && sync ==MIRROR_SYNC_MODE_TOP ) { +if (!bdrv_backing_chain_next (bs) && sync ==MIRROR_SYNC_MODE_TOP ) { sync =MIRROR_SYNC_MODE_FULL ; } +if (!has_replaces) { +/* We want to mirror from @bs, but keep implicit filters on top */ + unfiltered_bs =bdrv_skip_implicit_filters (bs); +if (unfiltered_bs != bs) { + replaces = unfiltered_bs->node_name; + has_replaces = true; + } + } +if (has_replaces) {BlockDriverState *to_replace_bs;AioContext *replace_aio_context;
@@ -2987,7 +2997,7 @@ static void blockdev_mirror_common(const char *job_id, BlockDriverState *bs,void qmp_drive_mirror (DriveMirror *arg,Error **errp) {BlockDriverState *bs; -BlockDriverState *source, *target_bs; +BlockDriverState *target_backing_bs, *target_bs;AioContext *aio_context;AioContext *old_context;BlockMirrorBackingMode backing_mode;
@@ -2996,6 +3006,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)int flags;int64_t size;const char *format = arg->format; +const char *replaces_node_name =NULL ;bool zero_target;int ret;
@@ -3022,12 +3033,12 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) } flags = bs->open_flags |BDRV_O_RDWR ; - source =backing_bs (bs); -if (!source && arg->sync ==MIRROR_SYNC_MODE_TOP ) { + target_backing_bs =bdrv_cow_bs (bdrv_skip_filters (bs)); +if (!target_backing_bs && arg->sync ==MIRROR_SYNC_MODE_TOP ) { arg->sync =MIRROR_SYNC_MODE_FULL ; }if (arg->sync ==MIRROR_SYNC_MODE_NONE ) { - source = bs; + target_backing_bs = bs; } size =bdrv_getlength (bs);
@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces; }if (arg->mode ==NEW_IMAGE_MODE_ABSOLUTE_PATHS ) {
@@ -3053,7 +3065,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)/* Don't open backing image in create() */ flags |=BDRV_O_NO_BACKING ; -if ((arg->sync ==MIRROR_SYNC_MODE_FULL || !source) +if ((arg->sync ==MIRROR_SYNC_MODE_FULL || !target_backing_bs) && arg->mode !=NEW_IMAGE_MODE_EXISTING ) {/* create new image w/o backing file */
@@ -3061,15 +3073,19 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)bdrv_img_create (arg->target, format,NULL ,NULL ,NULL , size, flags, false, &local_err); }else { +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing = +bdrv_skip_implicit_filters (target_backing_bs); +switch (arg->mode) {case NEW_IMAGE_MODE_EXISTING :break ;case NEW_IMAGE_MODE_ABSOLUTE_PATHS :/* create new image with backing file */ -bdrv_refresh_filename (source); +bdrv_refresh_filename (explicit_backing);bdrv_img_create (arg->target, format, - source->filename, - source->drv->format_name, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err);break ;default :
@@ -3119,7 +3135,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp)aio_context_acquire (aio_context); blockdev_mirror_common(arg->has_job_id ? arg->job_id : NULL, bs, target_bs, - arg->has_replaces, arg->replaces, arg->sync, + !!replaces_node_name, replaces_node_name, arg->sync, backing_mode, zero_target, arg->has_speed, arg->speed, arg->has_granularity, arg->granularity,
Signature · 1 line
2.26.2
Re: [PATCH v7 33/47] mirror: Deal with filters
Quoted 14 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "
target_backing_bs", because that is what it really refers to.Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 6 ++- block/mirror.c | 118 +++++++++++++++++++++++++++++++++---------- blockdev.c | 36 +++++++++---- 3 files changed, 121 insertions(+), 39 deletions(-)
...
Quoted 13 lines
block/mirror.c +1
-0@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob {BlockBackend *target;BlockDriverState *mirror_top_bs;BlockDriverState *base; +BlockDriverState *base_overlay;/* The name of the graph node to replace */ char *replaces;
block/mirror.c static int mirror_exit_common(Job *job) · line 683
Quoted 8 lines
block/mirror.c +3
-2@@ -677,8 +678,10 @@ static int mirror_exit_common(Job *job) &error_abort);if (!abort && s->backing_mode ==MIRROR_SOURCE_BACKING_CHAIN ) {BlockDriverState *backing = s->is_none_mode ? src : s->base; -if (backing_bs (target_bs) != backing) { -bdrv_set_backing_hd (target_bs, backing, &local_err); +BlockDriverState *unfiltered_target =bdrv_skip_filters (target_bs); + +if (bdrv_cow_bs (unfiltered_target) != backing) {
I just worry about a filter node of the concurrent job right below the unfiltered_target. The filter has unfiltered_target in its parent list. Will that filter node be replaced correctly then?
Andrey
...
Quoted 11 lines
block/mirror.c +11
-0+/* +* The topmost node with +* bdrv_skip_filters(filtered_target) == bdrv_skip_filters(target) +*/ + filtered_target =bdrv_cow_bs (bdrv_find_overlay (bs, target)); + +assert (bdrv_skip_filters (filtered_target) == +bdrv_skip_filters (target)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block
block/mirror.c static BlockJob *mirror_start_job( · line 1778
Quoted 10 lines
block/mirror.c +10
-0+* ourselves at s->base (if writes are blocked for a node, they are +* also blocked for its backing file). The other options would be a +* second filter driver above s->base (== target). +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter =bdrv_filter_or_cow_bs (bs); iter != target; + iter =bdrv_filter_or_cow_bs (iter)) + { +if (iter == filtered_target) {
For one filter node only?
Quoted 14 lines
block/mirror.c +8
-2+/* +* From here on, all nodes are filters on the base. +* This allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }
...
block/mirror.c void qmp_drive_mirror(DriveMirror *arg, Error **errp) · line 3056
Quoted 5 lines
block/mirror.c +1
-0@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces;
What is the idea behind the variables substitution?
Probably, the patch might be split out.
Andrey
Re: [PATCH v7 33/47] mirror: Deal with filters
Quoted 33 lines · Andrey Shinkevich
Quoted 14 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "
target_backing_bs", because that is what it really refers to.Signed-off-by Max Reitz <[email protected]>---
qapi/block-core.json| 6 ++-
block/mirror.c | 118 +++++++++++++++++++++++++++++++++---------- blockdev.c | 36 +++++++++----3 files changed, 121 insertions(+), 39 deletions(-)...
Quoted 17 lines
block/mirror.c +1
-2@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob { BlockBackend *target; BlockDriverState *mirror_top_bs; BlockDriverState *base; + BlockDriverState *base_overlay; /* The name of the graph node to replace */ char *replaces;@@ -677,8 +678,10 @@ static int mirror_exit_common(Job *job) &error_abort); if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) { BlockDriverState *backing = s->is_none_mode ? src : s->base; - if (backing_bs(target_bs) != backing) { - bdrv_set_backing_hd(target_bs, backing, &local_err);
Quoted 8 lines
block/mirror.c static int mirror_exit_common(Job *job) · line 683
Quoted 4 lines
block/mirror.c +3
-0+ BlockDriverState *unfiltered_target =bdrv_skip_filters (target_bs); + + if (bdrv_cow_bs(unfiltered_target) != backing) {I just worry about a filter node of the concurrent job right below the
unfiltered_target.
Having a concurrent job on the target sounds extremely problematic in itself (because at least for most of the mirror job, the target isn’t in a consistent state). Is that a real use case?
Quoted 2 lines
The filter has
unfiltered_targetin its parent list. Will that filter node be replaced correctly then?
I’m also not quite sure what you mean. We need to attach the source’s backing chain to the target here, so we go down to the first node that might accept COW backing files (by invoking bdrv_skip_filters()). That should be correct no matter what kind of filters are on it.
Quoted 19 lines
Quoted 19 lines
block/mirror.c +3
-0+ /* +* The topmost node with +* bdrv_skip_filters(filtered_target) ==
bdrv_skip_filters(target)
+ */
+filtered_target=bdrv_cow_bs(bdrv_find_overlay(bs, target));block/mirror.c +7
-0+ + assert(bdrv_skip_filters(filtered_target) == + bdrv_skip_filters(target)); + + /* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are
+ * also blocked for its backing file). The other options would be a
+ * second filter driver aboves->base(== target). + */
+iter_shared_perms=BLK_PERM_WRITE_UNCHANGED|BLK_PERM_WRITE;
Quoted 8 lines
Quoted 5 lines
+
+ for (iter =bdrv_filter_or_cow_bs(bs); iter != target; + iter =bdrv_filter_or_cow_bs(iter))
+ {
+ if (iter ==filtered_target) {For one filter node only?
No, iter_shared_perms is never reset, so it retains the BLK_PERM_CONSISTENT_READ flag until the end of the loop.
Quoted 17 lines
Quoted 16 lines
+ /*
+ * From here on, all nodes are filters on the base. + * This allows us to shareBLK_PERM_CONSISTENT_READ. + */
+iter_shared_perms|=BLK_PERM_CONSISTENT_READ; + }
+
ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0,
-BLK_PERM_WRITE_UNCHANGED|BLK_PERM_WRITE,
- errp);
+iter_shared_perms, errp); if (ret < 0) {
goto fail;
}...
Quoted 9 lines
block/mirror.c void qmp_drive_mirror(DriveMirror *arg, Error · line 3057
Quoted 6 lines
block/mirror.c +1
-0@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces;What is the idea behind the variables substitution?
Looks like a remnant from v6, where there was an
if (arg->has_replaces) {
...
replaces_node_name = arg->replaces;
} else if (unfiltered_bs != bs) {
replaces_node_name = unfiltered_bs->node_name;
}But I moved that logic to blockdev_mirror_common() in this version.
So it’s just useless now and replaces_node_name shouldn’t exist.
Max
Re: [PATCH v7 33/47] mirror: Deal with filters
Quoted 41 lines · Max Reitz
Quoted 32 lines · Andrey Shinkevich
Quoted 14 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "
target_backing_bs", because that is what it really refers to.Signed-off-by Max Reitz <[email protected]>--- qapi/block-core.json | 6 ++- block/mirror.c | 118 +++++++++++++++++++++++++++++++++---------- blockdev.c | 36 +++++++++---- 3 files changed, 121 insertions(+), 39 deletions(-)...
Quoted 16 lines
block/mirror.c +1
-1@@ -42,6 +42,7 @@ typedef struct MirrorBlockJob { BlockBackend *target; BlockDriverState *mirror_top_bs; BlockDriverState *base; + BlockDriverState *base_overlay; /* The name of the graph node to replace */ char *replaces;@@ -677,8 +678,10 @@ static int mirror_exit_common(Job *job) &error_abort); if (!abort && s->backing_mode == MIRROR_SOURCE_BACKING_CHAIN) { BlockDriverState *backing = s->is_none_mode ? src : s->base; - if (backing_bs(target_bs) != backing) {Quoted 8 lines
block/mirror.c static int mirror_exit_common(Job *job) · line 683
Quoted 5 lines
block/mirror.c +3
-1- bdrv_set_backing_hd(target_bs, backing, &local_err); + BlockDriverState *unfiltered_target =bdrv_skip_filters (target_bs); + + if (bdrv_cow_bs(unfiltered_target) != backing) {I just worry about a filter node of the concurrent job right below the
unfiltered_target.
Quoted 3 lines
Having a concurrent job on the target sounds extremely problematic in itself (because at least for most of the mirror job, the target isn’t in a consistent state). Is that a real use case?
It might be at the TestParallelOps of iotests #30 but I am not sure now. I am going to apply my series with copy-on-read filter for the stream job above this one and will see then.
Andrey
Quoted 7 lines
Quoted 3 lines
The filter has
unfiltered_targetin its parent list. Will that filter node be replaced correctly then?I’m also not quite sure what you mean. We need to attach the source’s backing chain to the target here, so we go down to the first node that might accept COW backing files (by invoking
bdrv_skip_filters()). That should be correct no matter what kind of filters are on it.
I ment when a filter is removed with the bdrv_replace_node() afterwards. As I mentioned above, I am going to test the case later.
Andrey
Quoted 26 lines
Quoted 18 lines
Quoted 18 lines
block/mirror.c +3
-0+ /* +* The topmost node with +* bdrv_skip_filters(filtered_target) ==
bdrv_skip_filters(target)
+ */
+filtered_target=bdrv_cow_bs(bdrv_find_overlay(bs, target));block/mirror.c +7
-0+ + assert(bdrv_skip_filters(filtered_target) == + bdrv_skip_filters(target)); + + /* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block +* ourselves at s->base (if writes are blocked for a node, they are
+ * also blocked for its backing file). The other options would be a
+ * second filter driver aboves->base(== target). + */Quoted 8 lines
Quoted 6 lines
+
iter_shared_perms=BLK_PERM_WRITE_UNCHANGED|BLK_PERM_WRITE; +
+ for (iter =bdrv_filter_or_cow_bs(bs); iter != target; + iter =bdrv_filter_or_cow_bs(iter))
+ {
+ if (iter ==filtered_target) {For one filter node only?
Quoted 2 lines
No,
iter_shared_permsis never reset, so it retains theBLK_PERM_CONSISTENT_READflag until the end of the loop.
Yes, that's right. Clear.
Andrey
Quoted 41 lines
Quoted 18 lines
Quoted 17 lines
+ /*
+ * From here on, all nodes are filters on the base. + * This allows us to shareBLK_PERM_CONSISTENT_READ. + */
+iter_shared_perms|=BLK_PERM_CONSISTENT_READ; + }
+
ret = block_job_add_bdrv(&s->common, "intermediate
node", iter, 0,
-BLK_PERM_WRITE_UNCHANGED|BLK_PERM_WRITE,
- errp);
+iter_shared_perms, errp);
if (ret < 0) {
goto fail;
}...
Quoted 8 lines
block/mirror.c void qmp_drive_mirror(DriveMirror *arg, Error · line 3057
Quoted 6 lines
block/mirror.c +1
-0@@ -3042,6 +3053,7 @@ void qmp_drive_mirror(DriveMirror *arg, Error **errp) " named node of the graph"); goto out; } + replaces_node_name = arg->replaces;What is the idea behind the variables substitution?
Looks like a remnant from v6, where there was an
if (arg->has_replaces) { ... replaces_node_name = arg->replaces; } else if (unfiltered_bs != bs) { replaces_node_name = unfiltered_bs->node_name; }But I moved that logic to
blockdev_mirror_common()in this version.So it’s just useless now and
replaces_node_nameshouldn’t exist.Max
Re: [PATCH v7 33/47] mirror: Deal with filters
In reply to Max ReitzAm 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 10 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "
target_backing_bs", because that is what it really refers to.Signed-off-by Max Reitz <[email protected]>
block/mirror.c static BlockJob *mirror_start_job( · line 1727
Quoted 7 lines
block/mirror.c +1
-0@@ -1682,6 +1721,7 @@ static BlockJob *mirror_start_job( s->zero_target = zero_target; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;
Is this valid without freezing the links between base_overlay and base?
Actually, I guess we should freeze everything between bs and base (for base != NULL) and it's a preexisting problem that just happens to affect this code, too.
Or maybe freezing everything is too much. We only want to make sure that no non-filter is inserted between base and base_overlay and that base (and now base_overlay) always stay in the backing chain of bs. But what options apart from freezing do we have to achieve this?
Why is using base_overlay even better than using base? Assuming there is a good reason, maybe the commit message could spell it out.
Kevin
Re: [PATCH v7 33/47] mirror: Deal with filters
Quoted 10 lines · Kevin Wolf
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 8 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission for active commits where the base is smaller than the top).
Use this opportunity to rename qmp_drive_mirror()'s "source" BDS to "
target_backing_bs", because that is what it really refers to.Signed-off-by Max Reitz <[email protected]>
Quoted 10 lines
block/mirror.c static BlockJob *mirror_start_job( · line 1727
Quoted 8 lines
block/mirror.c +1
-0@@ -1682,6 +1721,7 @@ static BlockJob *mirror_start_job( s->zero_target = zero_target; s->copy_mode = copy_mode; s->base = base; + s->base_overlay =bdrv_find_overlay (bs, base); s->granularity = granularity; s->buf_size =ROUND_UP (buf_size, granularity); s->unmap = unmap;Is this valid without freezing the links between
base_overlayand base?
Er...
Quoted 3 lines
Actually, I guess we should freeze everything between bs and base (for base != NULL) and it's a preexisting problem that just happens to affect this code, too.
Yes, that’s how it looks to me, too. I don’t think that has anything to do with this patch.
Quoted 4 lines
Or maybe freezing everything is too much. We only want to make sure that no non-filter is inserted between base and
base_overlayand that base (and nowbase_overlay) always stay in the backing chain of bs. But what options apart from freezing do we have to achieve this?
I don’t know of any, and I don’t know whether anyone would actually care if we were to just freeze everything.
Quoted 2 lines
Why is using
base_overlayeven better than using base? Assuming there is a good reason, maybe the commit message could spell it out.
The problem is that querying the block status for a filter node falls through to the underlying data-carrying node. So if there’s a filter on top of @base, and we query for is_allocated_above above @base, then we’ll include @base, which we do not want.
Max
PATCH v7 34/47 backup: Deal with filters
In reply to Max Reitz--- block/backup-top.c | 2 +- block/backup.c | 9 +++++---- blockdev.c | 19 +++++++++++++++---- 3 files changed, 21 insertions(+), 9 deletions(-)
block/backup-top.c +1 -1
@@ -291,7 +291,7 @@ void bdrv_backup_top_drop(BlockDriverState *bs) s->active = false;bdrv_child_refresh_perms (bs, bs->backing, &error_abort); -bdrv_replace_node (bs,backing_bs (bs), &error_abort); +bdrv_replace_node (bs, bs->backing->bs, &error_abort);bdrv_set_backing_hd (bs,NULL , &error_abort);bdrv_drained_end (bs);
block/backup.c +5 -4
@@ -297,6 +297,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target, {int ret;BlockDriverInfo bdi; +bool target_does_cow =bdrv_backing_chain_next (target);/* * If there is no backing file on the target, we cannot rely on COW if our
@@ -304,7 +305,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,* targets with a backing file, try to avoid COW if possible. */ ret =bdrv_get_info (target, &bdi); -if (ret == -ENOTSUP && !target->backing) { +if (ret == -ENOTSUP && !target_does_cow) {/* Cluster size is not defined */ warn_report ("The target block device doesn't provide " "information about the block size and it doesn't have a "
@@ -313,14 +314,14 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,"this default, the backup may be unusable" , BACKUP_CLUSTER_SIZE_DEFAULT);return BACKUP_CLUSTER_SIZE_DEFAULT ; - }else if (ret <0 && !target->backing) { + }else if (ret <0 && !target_does_cow) {error_setg_errno (errp, -ret,"Couldn't determine the cluster size of the target image, " "which has no backing file" );error_append_hint (errp,"Aborting, since this may create an unusable destination image\n" );return ret; - }else if (ret <0 && target->backing) { + }else if (ret <0 && target_does_cow) {/* Not fatal; just trudge on ahead. */ return BACKUP_CLUSTER_SIZE_DEFAULT ; }
@@ -371,7 +372,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,return NULL ; } -if (compress && !block_driver_can_compress (target->drv)) { +if (compress && !bdrv_supports_compressed_writes (target)) {error_setg (errp,"Compression is not supported for this drive %s" ,bdrv_get_device_name (target));return NULL ;
blockdev.c +15 -4
@@ -1728,7 +1728,13 @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)* on top of. */ if (backup->sync ==MIRROR_SYNC_MODE_TOP ) { - source =backing_bs (bs); +/* +* Backup will not replace the source by the target, so none +* of the filters skipped here will be removed (in contrast to +* mirror). Therefore, we can skip all of them when looking +* for the first COW relationship. +*/ + source =bdrv_cow_bs (bdrv_skip_filters (bs));if (!source) { backup->sync =MIRROR_SYNC_MODE_FULL ; }
@@ -1748,9 +1754,14 @@ static void drive_backup_prepare(BlkActionState *common, Error **errp)if (backup->mode !=NEW_IMAGE_MODE_EXISTING ) {assert (backup->format);if (source) { -bdrv_refresh_filename (source); -bdrv_img_create (backup->target, backup->format, source->filename, - source->drv->format_name,NULL , +/* Implicit filters should not appear in the filename */ +BlockDriverState *explicit_backing = +bdrv_skip_implicit_filters (source); + +bdrv_refresh_filename (explicit_backing); +bdrv_img_create (backup->target, backup->format, + explicit_backing->filename, + explicit_backing->drv->format_name,NULL , size, flags, false, &local_err); }else { bdrv_img_create(backup->target, backup->format, NULL, NULL, NULL,
Signature · 1 line
2.26.2
Re: [PATCH v7 34/47] backup: Deal with filters
Quoted 13 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>--- block/backup-top.c | 2 +- block/backup.c | 9 +++++---- blockdev.c | 19 +++++++++++++++---- 3 files changed, 21 insertions(+), 9 deletions(-)block/backup.c +0
-0
block/backup.c static int64_t backup_calculate_cluster_size(BlockDriverState *target, · line 301
Quoted 5 lines
block/backup.c +1
-0@@ -297,6 +297,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target, {int ret;BlockDriverInfo bdi; +bool target_does_cow =bdrv_backing_chain_next (target);
Wouldn't it better to make the explicit type conversion or use "!!" approach?
Andrey
Quoted 9 lines
block/backup.c +1
-1/* * If there is no backing file on the target, we cannot rely on COW if our @@ -304,7 +305,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target,* targets with a backing file, try to avoid COW if possible. */ ret =bdrv_get_info (target, &bdi); -if (ret == -ENOTSUP && !target->backing) { +if (ret == -ENOTSUP && !target_does_cow) {/* Cluster size is not defined */
...
Re: [PATCH v7 34/47] backup: Deal with filters
Quoted 9 lines · Andrey Shinkevich
Quoted 8 lines · Max Reitz
Signed-off-by Max Reitz <[email protected]>---
block/backup-top.c| 2 +-
block/backup.c | 9 +++++----
blockdev.c | 19 +++++++++++++++----3 files changed, 21 insertions(+), 9 deletions(-)
Quoted 15 lines
Quoted 11 lines
block/backup.c +1
-0@@ -297,6 +297,7 @@ static int64_t backup_calculate_cluster_size(BlockDriverState *target, { int ret; BlockDriverInfo bdi; + bool target_does_cow = bdrv_backing_chain_next(target);Wouldn't it better to make the explicit type conversion or use "!!" approach?
I don’t know. O:)
I personally don’t like too may exclamation marks because I feel like the code is screaming at me. So I tend to use them only where necessary.
As for doing an explicit cast... I think I’ll keep that in mind to reduce my future use of !!. But in this case, the type name is in the same line, so I feel like it’s sufficiently clear.
Max
PATCH v7 35/47 commit: Deal with filters
In reply to Max ReitzThis includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
--- block/block-backend.c | 9 +++- block/commit.c | 96 +++++++++++++++++++++++++--------- block/monitor/block-hmp-cmds.c | 2 +- blockdev.c | 4 +- 4 files changed, 81 insertions(+), 30 deletions(-)
block/block-backend.c +7 -2
@@ -2271,8 +2271,13 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs); +if (blk_is_inserted (blk)) { +BlockDriverState *non_filter; +int ret; + +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); + ret =bdrv_commit (non_filter);if (ret <0 ) {aio_context_release (aio_context);return ret;
block/commit.c +70 -26
173 lines not on this page · raw message
block/monitor/block-hmp-cmds.c +1 -1
@@ -217,7 +217,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)return ; } - bs =blk_bs (blk); + bs =bdrv_skip_implicit_filters (blk_bs (blk)); aio_context =bdrv_get_aio_context (bs);aio_context_acquire (aio_context);
blockdev.c +3 -1
@@ -2690,7 +2690,9 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,assert (bdrv_get_aio_context (base_bs) == aio_context); -for (iter = top_bs; iter !=backing_bs (base_bs); iter =backing_bs (iter)) { +for (iter = top_bs; iter !=bdrv_filter_or_cow_bs (base_bs); + iter =bdrv_filter_or_cow_bs (iter)) + {if (bdrv_op_is_blocked (iter,BLOCK_OP_TYPE_COMMIT_TARGET , errp)) { goto out; }
Signature · 1 line
2.26.2
Re: [PATCH v7 35/47] commit: Deal with filters
Quoted 11 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/block-backend.c | 9 +++- block/commit.c | 96 +++++++++++++++++++++++++--------- block/monitor/block-hmp-cmds.c | 2 +- blockdev.c | 4 +- 4 files changed, 81 insertions(+), 30 deletions(-)
...
Quoted 18 lines
block/commit.c +18
-0+/* +* Block all nodes between top and base, because they will +* disappear from the chain after this operation. +* Note that this assumes that the user is fine with removing all +* nodes (including R/W filters) between top and base. Assuring +* this is the responsibility of the interface (i.e. whoever calls +* commit_start()). +*/ + s->base_overlay =bdrv_find_overlay (top, base); +assert (s->base_overlay); + +/* +* The topmost node with +* bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base) +*/ + filtered_base =bdrv_cow_bs (s->base_overlay); +assert (bdrv_skip_filters (filtered_base) ==bdrv_skip_filters (base)); +
block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 352
Quoted 10 lines
block/commit.c +10
-0+/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves +* at s->base (if writes are blocked for a node, they are also blocked +* for its backing file). The other options would be a second filter +* driver above s->base. +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter = top; iter != base; iter =bdrv_filter_or_cow_bs (iter)) { +if (iter == filtered_base) {
The question same to mirroring:
in case of multiple filters, one above another, the permission is extended for the filtered_base only.
Andrey
Quoted 14 lines
block/commit.c +8
-2+/* +* From here on, all nodes are filters on the base. This +* allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }
...
Re: [PATCH v7 35/47] commit: Deal with filters
Quoted 50 lines · Andrey Shinkevich
Quoted 11 lines · Max Reitz
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>---
block/block-backend.c | 9 +++-
block/commit.c | 96 +++++++++++++++++++++++++---------block/monitor/block-hmp-cmds.c| 2 +-
blockdev.c | 4 +-4 files changed, 81 insertions(+), 30 deletions(-)...
Quoted 25 lines
+ /*
+ * Block all nodes between top and base, because they will + * disappear from the chain after this operation.
+ * Note that this assumes that the user is fine with removing all + * nodes (including R/W filters) between top and base. Assuring + * this is the responsibility of the interface (i.e. whoever calls + *commit_start()).
+ */
+s->base_overlay=bdrv_find_overlay(top, base);
+assert(s->base_overlay);
+
+ /*
+ * The topmost node with
+ *bdrv_skip_filters(filtered_base)==bdrv_skip_filters(base)+ */
+filtered_base=bdrv_cow_bs(s->base_overlay);
+assert(bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base));
+
+ /*
+ * XXXBLK_PERM_WRITEneeds to be allowed so we don't block ourselves
+ * ats->base(if writes are blocked for a node, they are also blocked
+ * for its backing file). The other options would be a secondQuoted 8 lines
filter
+ * driver aboves->base.
+ */
+iter_shared_perms=BLK_PERM_WRITE_UNCHANGED|BLK_PERM_WRITE; +
+ for (iter = top; iter != base; iter =bdrv_filter_or_cow_bs(iter)) {
+ if (iter ==filtered_base) {The question same to mirroring:
Quoted 4 lines
in case of multiple filters, one above another, the permission is extended for the
filtered_baseonly.Andrey
The question has been answered already.
Andrey
Quoted 23 lines
Quoted 16 lines
+ /*
+ * From here on, all nodes are filters on the base. This + * allows us to shareBLK_PERM_CONSISTENT_READ.
+ */
+iter_shared_perms|=BLK_PERM_CONSISTENT_READ;
+ }
+
ret = block_job_add_bdrv(&s->common, "intermediate node", iter, 0, -BLK_PERM_WRITE_UNCHANGED|BLK_PERM_WRITE,
- errp);
+iter_shared_perms, errp);
if (ret < 0) {
goto fail;
}...
Reviewed-by Andrey Shinkevich <[email protected]>
Re: [PATCH v7 35/47] commit: Deal with filters
In reply to Max ReitzAm 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 16 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/block-backend.c | 9 +++- block/commit.c | 96 +++++++++++++++++++++++++--------- block/monitor/block-hmp-cmds.c | 2 +- blockdev.c | 4 +- 4 files changed, 81 insertions(+), 30 deletions(-)block/block-backend.c +0
-0
block/block-backend.c int blk_commit_all(void) · line 2275
Quoted 5 lines
block/block-backend.c +0
-2@@ -2271,8 +2271,13 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs);
The old code didn't try to commit nodes that don't have a backing file.
block/block-backend.c int blk_commit_all(void) · line 2280
Quoted 7 lines
block/block-backend.c +7
-0+if (blk_is_inserted (blk)) { +BlockDriverState *non_filter; +int ret; + +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); + ret =bdrv_commit (non_filter);
The new one tries unconditionally. For nodes without a backing file, bdrv_commit() will return -ENOTSUP, so the whole function fails.
(First real bug at patch 35. I almost thought I wouldn't find any!)
Quoted 8 lines
block/block-backend.c +0
-0if (ret <0 ) {aio_context_release (aio_context);return ret;block/commit.c +0
-0
block/commit.c typedef struct CommitBlockJob { · line 43
Quoted 7 lines
block/commit.c +1
-0@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {BlockBackend *top;BlockBackend *base;BlockDriverState *base_bs; +BlockDriverState *base_overlay;BlockdevOnError on_error;bool base_read_only;bool chain_frozen;
Hm, again this mysterious base_overlay. I know that stream introduced it to avoid freezing the link to base, but commit doesn't seem to do that.
Is it to avoid using the block status of filter drivers between base_overlay and base? If so, I guess that goes back to the question I raised earlier in this series: What is the block status supposed to tell for filter nodes?
But anyway, in contrast to mirror, commit actually freezes the chain between commit_top_bs and base, so it should be safe at least.
Quoted 41 lines
block/commit.c +12
-3@@ -89,7 +90,7 @@ static void commit_abort(Job *job)* XXX Can (or should) we somehow keep 'consistent read' blocked even * after the failed/cancelled commit job is gone? If we already wrote * something to base, the intermediate images aren't valid any more. */ -bdrv_replace_node (s->commit_top_bs,backing_bs (s->commit_top_bs), +bdrv_replace_node (s->commit_top_bs, s->commit_top_bs->backing->bs, &error_abort);bdrv_unref (s->commit_top_bs);@@ -153,7 +154,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)break ; }/* Copy if allocated above the base */ - ret =bdrv_is_allocated_above (blk_bs (s->top),blk_bs (s->base), false, + ret =bdrv_is_allocated_above (blk_bs (s->top), s->base_overlay, true, offset,COMMIT_BUFFER_SIZE , &n); copy = (ret ==1 );trace_commit_one_iteration (s, offset, n, ret);@@ -253,15 +254,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,CommitBlockJob *s;BlockDriverState *iter;BlockDriverState *commit_top_bs =NULL ; +BlockDriverState *filtered_base;Error *local_err =NULL ; +int64_t base_size, top_size; +uint64_t perms, iter_shared_perms;int ret;assert (top != bs); -if (top == base) { +if (bdrv_skip_filters (top) ==bdrv_skip_filters (base)) {error_setg (errp,"Invalid files for merge: top and base are the same" );return ; } + base_size =bdrv_getlength (base); +if (base_size <0 ) { +error_setg_errno (errp, -base_size,"Could not inquire base image size" ); +return ; + } +
block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 284
Quoted 10 lines
block/commit.c +10
-0+ top_size =bdrv_getlength (top); +if (top_size <0 ) { +error_setg_errno (errp, -top_size,"Could not inquire top image size" ); +return ; + } + + perms =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE ; +if (base_size < top_size) { + perms |=BLK_PERM_RESIZE ; + }
base_perms would indicate which permissions these are (particularly because it's not the next thing that requires permissions, but only used further down the function).
Quoted 105 lines
block/commit.c +44
-19s =block_job_create (job_id, &commit_job_driver,NULL , bs,0 ,BLK_PERM_ALL , speed, creation_flags,NULL ,NULL , errp);if (!s) {@@ -301,17 +322,43 @@ void commit_start(const char *job_id, BlockDriverState *bs, s->commit_top_bs = commit_top_bs; -/* Block all nodes between top and base, because they will -* disappear from the chain after this operation. */ -assert (bdrv_chain_contains (top, base)); -for (iter = top; iter != base; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves -* at s->base (if writes are blocked for a node, they are also blocked -* for its backing file). The other options would be a second filter -* driver above s->base. */ +/* +* Block all nodes between top and base, because they will +* disappear from the chain after this operation. +* Note that this assumes that the user is fine with removing all +* nodes (including R/W filters) between top and base. Assuring +* this is the responsibility of the interface (i.e. whoever calls +* commit_start()). +*/ + s->base_overlay =bdrv_find_overlay (top, base); +assert (s->base_overlay); + +/* +* The topmost node with +* bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base) +*/ + filtered_base =bdrv_cow_bs (s->base_overlay); +assert (bdrv_skip_filters (filtered_base) ==bdrv_skip_filters (base)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves +* at s->base (if writes are blocked for a node, they are also blocked +* for its backing file). The other options would be a second filter +* driver above s->base. +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter = top; iter != base; iter =bdrv_filter_or_cow_bs (iter)) { +if (iter == filtered_base) { +/* +* From here on, all nodes are filters on the base. This +* allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -328,9 +375,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, } s->base =blk_new (s->common.job.aio_context, -BLK_PERM_CONSISTENT_READ - |BLK_PERM_WRITE - |BLK_PERM_RESIZE , + perms,BLK_PERM_CONSISTENT_READ |BLK_PERM_GRAPH_MOD |BLK_PERM_WRITE_UNCHANGED );@@ -398,19 +443,22 @@ int bdrv_commit(BlockDriverState *bs)if (!drv)return -ENOMEDIUM ; -if (!bs->backing) { + backing_file_bs =bdrv_cow_bs (bs); + +if (!backing_file_bs) {return -ENOTSUP ; }if (bdrv_op_is_blocked (bs,BLOCK_OP_TYPE_COMMIT_SOURCE ,NULL ) || -bdrv_op_is_blocked (bs->backing->bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) { +bdrv_op_is_blocked (backing_file_bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) + {return -EBUSY ; } - ro = bs->backing->bs->read_only; + ro = backing_file_bs->read_only;if (ro) { -if (bdrv_reopen_set_read_only (bs->backing->bs, false,NULL )) { +if (bdrv_reopen_set_read_only (backing_file_bs, false,NULL )) {return -EACCES ; } }@@ -428,8 +476,6 @@ int bdrv_commit(BlockDriverState *bs) }/* Insert commit_top block node above backing, so we can write to it */ - backing_file_bs =backing_bs (bs); - commit_top_bs =bdrv_new_open_driver (&bdrv_commit_top,NULL ,BDRV_O_RDWR , &local_err);if (commit_top_bs ==NULL ) {
block/commit.c ro_cleanup: · line 564
Quoted 7 lines
block/commit.c +1
-3@@ -515,15 +561,13 @@ ro_cleanup:qemu_vfree (buf);blk_unref (backing); -if (backing_file_bs) { -bdrv_set_backing_hd (bs, backing_file_bs, &error_abort); - } +bdrv_set_backing_hd (bs, backing_file_bs, &error_abort);
This means that bdrv_set_backing_hd() is now called to undo a change that hasn't even been made yet. This fails (with &error_abort) if the backing chain is frozen.
On the other hand, the other bdrv_set_backing_hd() calls in the same function would fail the same way.
block/commit.c ro_cleanup: · line 573
Quoted 10 lines
block/commit.c +1
-1bdrv_unref (commit_top_bs);blk_unref (src);if (ro) {/* ignoring error return here */ -bdrv_reopen_set_read_only (bs->backing->bs, true,NULL ); +bdrv_reopen_set_read_only (backing_file_bs, true,NULL ); }return ret;
Kevin
Re: [PATCH v7 35/47] commit: Deal with filters
Quoted 33 lines · Kevin Wolf
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 16 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/block-backend.c | 9 +++- block/commit.c | 96 +++++++++++++++++++++++++--------- block/monitor/block-hmp-cmds.c | 2 +- blockdev.c | 4 +- 4 files changed, 81 insertions(+), 30 deletions(-)block/block-backend.c +0
-0block/block-backend.c int blk_commit_all(void) · line 2275
Quoted 5 lines
block/block-backend.c +0
-2@@ -2271,8 +2271,13 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs);The old code didn't try to commit nodes that don't have a backing file.
block/block-backend.c int blk_commit_all(void) · line 2280
Quoted 7 lines
block/block-backend.c +7
-0+if (blk_is_inserted (blk)) { +BlockDriverState *non_filter; +int ret; + +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); + ret =bdrv_commit (non_filter);
Quoted 2 lines
The new one tries unconditionally. For nodes without a backing file,
bdrv_commit()will return-ENOTSUP, so the whole function fails.
:(
Hm. Should I fix it by checking for bdrv_cow_bs(bdrv_skip_implicit_filters())? Or bdrv_backing_chain_next() and change the bdrv_skip_implicit_filters() to a bdrv_skip_filters()? I feel like that would make even more sense.
Quoted 1 line
(First real bug at patch 35. I almost thought I wouldn't find any!)
:)
Quoted 19 lines
Quoted 8 lines
block/block-backend.c +0
-0if (ret <0 ) {aio_context_release (aio_context);return ret;block/commit.c +0
-0block/commit.c typedef struct CommitBlockJob { · line 43
Quoted 7 lines
block/commit.c +1
-0@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {BlockBackend *top;BlockBackend *base;BlockDriverState *base_bs; +BlockDriverState *base_overlay;BlockdevOnError on_error;bool base_read_only;bool chain_frozen;Hm, again this mysterious
base_overlay. I know that stream introduced it to avoid freezing the link to base, but commit doesn't seem to do that.
Quoted 2 lines
Is it to avoid using the block status of filter drivers between
base_overlayand base?
Yes.
Quoted 3 lines
If so, I guess that goes back to the question I
raised earlier in this series: What is the block status supposed to tell for filter nodes?
Honestly, I would really like to get away without having to answer that question in this series. Intuitively, I feel like falling through to the next data-bearing layer is not something most callers want. But I’d rather investigate that question separately from this series (even though that likely means we’ll never do it), and just treat it as it is in this series.
Quoted 50 lines
But anyway, in contrast to mirror, commit actually freezes the chain between
commit_top_bsand base, so it should be safe at least.Quoted 47 lines
block/commit.c +18
-3@@ -89,7 +90,7 @@ static void commit_abort(Job *job)* XXX Can (or should) we somehow keep 'consistent read' blocked even * after the failed/cancelled commit job is gone? If we already wrote * something to base, the intermediate images aren't valid any more. */ -bdrv_replace_node (s->commit_top_bs,backing_bs (s->commit_top_bs), +bdrv_replace_node (s->commit_top_bs, s->commit_top_bs->backing->bs, &error_abort);bdrv_unref (s->commit_top_bs);@@ -153,7 +154,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)break ; }/* Copy if allocated above the base */ - ret =bdrv_is_allocated_above (blk_bs (s->top),blk_bs (s->base), false, + ret =bdrv_is_allocated_above (blk_bs (s->top), s->base_overlay, true, offset,COMMIT_BUFFER_SIZE , &n); copy = (ret ==1 );trace_commit_one_iteration (s, offset, n, ret);@@ -253,15 +254,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,CommitBlockJob *s;BlockDriverState *iter;BlockDriverState *commit_top_bs =NULL ; +BlockDriverState *filtered_base;Error *local_err =NULL ; +int64_t base_size, top_size; +uint64_t perms, iter_shared_perms;int ret;assert (top != bs); -if (top == base) { +if (bdrv_skip_filters (top) ==bdrv_skip_filters (base)) {error_setg (errp,"Invalid files for merge: top and base are the same" );return ; } + base_size =bdrv_getlength (base); +if (base_size <0 ) { +error_setg_errno (errp, -base_size,"Could not inquire base image size" ); +return ; + } + + top_size =bdrv_getlength (top); +if (top_size <0 ) { +error_setg_errno (errp, -top_size,"Could not inquire top image size" ); +return ; + } +
Quoted 8 lines
block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 284
Quoted 4 lines
block/commit.c +4
-0+ perms =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE ; +if (base_size < top_size) { + perms |=BLK_PERM_RESIZE ; + }
base_permswould indicate which permissions these are (particularly because it's not the next thing that requires permissions, but only used further down the function).
%s/\<perms\>/base_perms/? Sure.
Quoted 117 lines
Quoted 105 lines
block/commit.c +44
-19s =block_job_create (job_id, &commit_job_driver,NULL , bs,0 ,BLK_PERM_ALL , speed, creation_flags,NULL ,NULL , errp);if (!s) {@@ -301,17 +322,43 @@ void commit_start(const char *job_id, BlockDriverState *bs, s->commit_top_bs = commit_top_bs; -/* Block all nodes between top and base, because they will -* disappear from the chain after this operation. */ -assert (bdrv_chain_contains (top, base)); -for (iter = top; iter != base; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves -* at s->base (if writes are blocked for a node, they are also blocked -* for its backing file). The other options would be a second filter -* driver above s->base. */ +/* +* Block all nodes between top and base, because they will +* disappear from the chain after this operation. +* Note that this assumes that the user is fine with removing all +* nodes (including R/W filters) between top and base. Assuring +* this is the responsibility of the interface (i.e. whoever calls +* commit_start()). +*/ + s->base_overlay =bdrv_find_overlay (top, base); +assert (s->base_overlay); + +/* +* The topmost node with +* bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base) +*/ + filtered_base =bdrv_cow_bs (s->base_overlay); +assert (bdrv_skip_filters (filtered_base) ==bdrv_skip_filters (base)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves +* at s->base (if writes are blocked for a node, they are also blocked +* for its backing file). The other options would be a second filter +* driver above s->base. +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter = top; iter != base; iter =bdrv_filter_or_cow_bs (iter)) { +if (iter == filtered_base) { +/* +* From here on, all nodes are filters on the base. This +* allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -328,9 +375,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, } s->base =blk_new (s->common.job.aio_context, -BLK_PERM_CONSISTENT_READ - |BLK_PERM_WRITE - |BLK_PERM_RESIZE , + perms,BLK_PERM_CONSISTENT_READ |BLK_PERM_GRAPH_MOD |BLK_PERM_WRITE_UNCHANGED );@@ -398,19 +443,22 @@ int bdrv_commit(BlockDriverState *bs)if (!drv)return -ENOMEDIUM ; -if (!bs->backing) { + backing_file_bs =bdrv_cow_bs (bs); + +if (!backing_file_bs) {return -ENOTSUP ; }if (bdrv_op_is_blocked (bs,BLOCK_OP_TYPE_COMMIT_SOURCE ,NULL ) || -bdrv_op_is_blocked (bs->backing->bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) { +bdrv_op_is_blocked (backing_file_bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) + {return -EBUSY ; } - ro = bs->backing->bs->read_only; + ro = backing_file_bs->read_only;if (ro) { -if (bdrv_reopen_set_read_only (bs->backing->bs, false,NULL )) { +if (bdrv_reopen_set_read_only (backing_file_bs, false,NULL )) {return -EACCES ; } }@@ -428,8 +476,6 @@ int bdrv_commit(BlockDriverState *bs) }/* Insert commit_top block node above backing, so we can write to it */ - backing_file_bs =backing_bs (bs); - commit_top_bs =bdrv_new_open_driver (&bdrv_commit_top,NULL ,BDRV_O_RDWR , &local_err);if (commit_top_bs ==NULL ) {block/commit.c ro_cleanup: · line 564
Quoted 7 lines
block/commit.c +1
-3@@ -515,15 +561,13 @@ ro_cleanup:qemu_vfree (buf);blk_unref (backing); -if (backing_file_bs) { -bdrv_set_backing_hd (bs, backing_file_bs, &error_abort); - } +bdrv_set_backing_hd (bs, backing_file_bs, &error_abort);This means that
bdrv_set_backing_hd()is now called to undo a change that hasn't even been made yet. This fails (with&error_abort) if the backing chain is frozen.
Quoted 2 lines
On the other hand, the other
bdrv_set_backing_hd()calls in the same function would fail the same way.
True. :)
Still, maybe there’s an op blocker from a concurrent job, so we go to the failure path and then we’d abort here. So better to guard it by checking whether bdrv_cow_bs(bs) != backing_file_bs.
Max
Re: [PATCH v7 35/47] commit: Deal with filters
Am 20.08.2020 um 13:27 hat Max Reitz geschrieben:
Quoted 37 lines · Kevin Wolf
Quoted 33 lines
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 16 lines
This includes some permission limiting (for example, we only need to take the RESIZE permission if the base is smaller than the top).
Signed-off-by Max Reitz <[email protected]>--- block/block-backend.c | 9 +++- block/commit.c | 96 +++++++++++++++++++++++++--------- block/monitor/block-hmp-cmds.c | 2 +- blockdev.c | 4 +- 4 files changed, 81 insertions(+), 30 deletions(-)block/block-backend.c +0
-0block/block-backend.c int blk_commit_all(void) · line 2275
Quoted 5 lines
block/block-backend.c +0
-2@@ -2271,8 +2271,13 @@ int blk_commit_all(void)AioContext *aio_context =blk_get_aio_context (blk);aio_context_acquire (aio_context); -if (blk_is_inserted (blk) && blk->root->bs->backing) { -int ret =bdrv_commit (blk->root->bs);The old code didn't try to commit nodes that don't have a backing file.
block/block-backend.c int blk_commit_all(void) · line 2280
Quoted 7 lines
block/block-backend.c +7
-0+if (blk_is_inserted (blk)) { +BlockDriverState *non_filter; +int ret; + +/* Legacy function, so skip implicit filters */ + non_filter =bdrv_skip_implicit_filters (blk->root->bs); + ret =bdrv_commit (non_filter);Quoted 2 lines
The new one tries unconditionally. For nodes without a backing file,
bdrv_commit()will return-ENOTSUP, so the whole function fails.
Quoted 6 lines
:(
Hm. Should I fix it by checking for bdrv_cow_bs(bdrv_skip_implicit_filters())? Or
bdrv_backing_chain_next()and change thebdrv_skip_implicit_filters()to abdrv_skip_filters()? I feel like that would make even more sense.
I agree that bdrv_skip_filters() makes more sense. If I have a qcow2 image and an explicit throttle filter on top, there is no reason to skip this image.
bdrv_backing_chain_next() or bdrv_cow_bs() should be the same in a boolean context, so I'd vote for bdrv_cow_bs() because it has less work to do to get the same result.
Quoted 32 lines
Quoted 1 line
(First real bug at patch 35. I almost thought I wouldn't find any!)
:)
Quoted 19 lines
Quoted 8 lines
block/block-backend.c +0
-0if (ret <0 ) {aio_context_release (aio_context);return ret;block/commit.c +0
-0block/commit.c typedef struct CommitBlockJob { · line 43
Quoted 7 lines
block/commit.c +1
-0@@ -37,6 +37,7 @@ typedef struct CommitBlockJob {BlockBackend *top;BlockBackend *base;BlockDriverState *base_bs; +BlockDriverState *base_overlay;BlockdevOnError on_error;bool base_read_only;bool chain_frozen;Hm, again this mysterious
base_overlay. I know that stream introduced it to avoid freezing the link to base, but commit doesn't seem to do that.Quoted 2 lines
Is it to avoid using the block status of filter drivers between
base_overlayand base?Yes.
Quoted 3 lines
If so, I guess that goes back to the question I
raised earlier in this series: What is the block status supposed to tell for filter nodes?
Quoted 6 lines
Honestly, I would really like to get away without having to answer that question in this series. Intuitively, I feel like falling through to the next data-bearing layer is not something most callers want. But I’d rather investigate that question separately from this series (even though that likely means we’ll never do it), and just treat it as it is in this series.
Well, I'm asking the question because not having the answer makes us jump through hoops in this series to accomodate a behaviour it probably shouldn't even have. (Because I agree that filters should probably keep DATA clear, i.e. they are never the layer that defines the content.)
Additional node references (i.e. references that are not edges in the graph) always make the design more complicated and require us to consider more things like what happens on graph changes. So it's a question of maintainability.
Quoted 59 lines
Quoted 50 lines
But anyway, in contrast to mirror, commit actually freezes the chain between
commit_top_bsand base, so it should be safe at least.Quoted 47 lines
block/commit.c +18
-3@@ -89,7 +90,7 @@ static void commit_abort(Job *job)* XXX Can (or should) we somehow keep 'consistent read' blocked even * after the failed/cancelled commit job is gone? If we already wrote * something to base, the intermediate images aren't valid any more. */ -bdrv_replace_node (s->commit_top_bs,backing_bs (s->commit_top_bs), +bdrv_replace_node (s->commit_top_bs, s->commit_top_bs->backing->bs, &error_abort);bdrv_unref (s->commit_top_bs);@@ -153,7 +154,7 @@ static int coroutine_fn commit_run(Job *job, Error **errp)break ; }/* Copy if allocated above the base */ - ret =bdrv_is_allocated_above (blk_bs (s->top),blk_bs (s->base), false, + ret =bdrv_is_allocated_above (blk_bs (s->top), s->base_overlay, true, offset,COMMIT_BUFFER_SIZE , &n); copy = (ret ==1 );trace_commit_one_iteration (s, offset, n, ret);@@ -253,15 +254,35 @@ void commit_start(const char *job_id, BlockDriverState *bs,CommitBlockJob *s;BlockDriverState *iter;BlockDriverState *commit_top_bs =NULL ; +BlockDriverState *filtered_base;Error *local_err =NULL ; +int64_t base_size, top_size; +uint64_t perms, iter_shared_perms;int ret;assert (top != bs); -if (top == base) { +if (bdrv_skip_filters (top) ==bdrv_skip_filters (base)) {error_setg (errp,"Invalid files for merge: top and base are the same" );return ; } + base_size =bdrv_getlength (base); +if (base_size <0 ) { +error_setg_errno (errp, -base_size,"Could not inquire base image size" ); +return ; + } + + top_size =bdrv_getlength (top); +if (top_size <0 ) { +error_setg_errno (errp, -top_size,"Could not inquire top image size" ); +return ; + } +Quoted 8 lines
block/commit.c void commit_start(const char *job_id, BlockDriverState *bs, · line 284
Quoted 4 lines
block/commit.c +4
-0+ perms =BLK_PERM_CONSISTENT_READ |BLK_PERM_WRITE ; +if (base_size < top_size) { + perms |=BLK_PERM_RESIZE ; + }
base_permswould indicate which permissions these are (particularly because it's not the next thing that requires permissions, but only used further down the function).
Quoted 1 line
%s/\<perms\>/base_perms/? Sure.
Sorry, I admit this wasn't phrased very clearly. But yes, renaming the variable this way is what I meant.
Quoted 117 lines
Quoted 117 lines
Quoted 105 lines
block/commit.c +44
-19s =block_job_create (job_id, &commit_job_driver,NULL , bs,0 ,BLK_PERM_ALL , speed, creation_flags,NULL ,NULL , errp);if (!s) {@@ -301,17 +322,43 @@ void commit_start(const char *job_id, BlockDriverState *bs, s->commit_top_bs = commit_top_bs; -/* Block all nodes between top and base, because they will -* disappear from the chain after this operation. */ -assert (bdrv_chain_contains (top, base)); -for (iter = top; iter != base; iter =backing_bs (iter)) { -/* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves -* at s->base (if writes are blocked for a node, they are also blocked -* for its backing file). The other options would be a second filter -* driver above s->base. */ +/* +* Block all nodes between top and base, because they will +* disappear from the chain after this operation. +* Note that this assumes that the user is fine with removing all +* nodes (including R/W filters) between top and base. Assuring +* this is the responsibility of the interface (i.e. whoever calls +* commit_start()). +*/ + s->base_overlay =bdrv_find_overlay (top, base); +assert (s->base_overlay); + +/* +* The topmost node with +* bdrv_skip_filters(filtered_base) == bdrv_skip_filters(base) +*/ + filtered_base =bdrv_cow_bs (s->base_overlay); +assert (bdrv_skip_filters (filtered_base) ==bdrv_skip_filters (base)); + +/* +* XXX BLK_PERM_WRITE needs to be allowed so we don't block ourselves +* at s->base (if writes are blocked for a node, they are also blocked +* for its backing file). The other options would be a second filter +* driver above s->base. +*/ + iter_shared_perms =BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE ; + +for (iter = top; iter != base; iter =bdrv_filter_or_cow_bs (iter)) { +if (iter == filtered_base) { +/* +* From here on, all nodes are filters on the base. This +* allows us to share BLK_PERM_CONSISTENT_READ. +*/ + iter_shared_perms |=BLK_PERM_CONSISTENT_READ ; + } + ret =block_job_add_bdrv (&s->common,"intermediate node" , iter,0 , -BLK_PERM_WRITE_UNCHANGED |BLK_PERM_WRITE , - errp); + iter_shared_perms, errp);if (ret <0 ) { goto fail; }@@ -328,9 +375,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, } s->base =blk_new (s->common.job.aio_context, -BLK_PERM_CONSISTENT_READ - |BLK_PERM_WRITE - |BLK_PERM_RESIZE , + perms,BLK_PERM_CONSISTENT_READ |BLK_PERM_GRAPH_MOD |BLK_PERM_WRITE_UNCHANGED );@@ -398,19 +443,22 @@ int bdrv_commit(BlockDriverState *bs)if (!drv)return -ENOMEDIUM ; -if (!bs->backing) { + backing_file_bs =bdrv_cow_bs (bs); + +if (!backing_file_bs) {return -ENOTSUP ; }if (bdrv_op_is_blocked (bs,BLOCK_OP_TYPE_COMMIT_SOURCE ,NULL ) || -bdrv_op_is_blocked (bs->backing->bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) { +bdrv_op_is_blocked (backing_file_bs,BLOCK_OP_TYPE_COMMIT_TARGET ,NULL )) + {return -EBUSY ; } - ro = bs->backing->bs->read_only; + ro = backing_file_bs->read_only;if (ro) { -if (bdrv_reopen_set_read_only (bs->backing->bs, false,NULL )) { +if (bdrv_reopen_set_read_only (backing_file_bs, false,NULL )) {return -EACCES ; } }@@ -428,8 +476,6 @@ int bdrv_commit(BlockDriverState *bs) }/* Insert commit_top block node above backing, so we can write to it */ - backing_file_bs =backing_bs (bs); - commit_top_bs =bdrv_new_open_driver (&bdrv_commit_top,NULL ,BDRV_O_RDWR , &local_err);if (commit_top_bs ==NULL ) {block/commit.c ro_cleanup: · line 564
Quoted 7 lines
block/commit.c +1
-3@@ -515,15 +561,13 @@ ro_cleanup:qemu_vfree (buf);blk_unref (backing); -if (backing_file_bs) { -bdrv_set_backing_hd (bs, backing_file_bs, &error_abort); - } +bdrv_set_backing_hd (bs, backing_file_bs, &error_abort);This means that
bdrv_set_backing_hd()is now called to undo a change that hasn't even been made yet. This fails (with&error_abort) if the backing chain is frozen.
Quoted 8 lines
Quoted 2 lines
On the other hand, the other
bdrv_set_backing_hd()calls in the same function would fail the same way.True. :)
Still, maybe there’s an op blocker from a concurrent job, so we go to the failure path and then we’d abort here. So better to guard it by checking whether
bdrv_cow_bs(bs)!=backing_file_bs.
Certainly can't hurt.
Kevin
PATCH v7 36/47 nbd: Use CAF when looking for dirty bitmap
In reply to Max ReitzWhen looking for a dirty bitmap to share, we should handle filters by just including them in the search (so they do not break backing chains).
--- nbd/server.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
nbd/server.c +3 -3
@@ -1561,13 +1561,13 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,if (bitmap) {BdrvDirtyBitmap *bm =NULL ; -while (true) { +while (bs) { bm =bdrv_find_dirty_bitmap (bs, bitmap); -if (bm !=NULL || bs->backing ==NULL ) { +if (bm !=NULL ) {break ; } - bs = bs->backing->bs; + bs =bdrv_filter_or_cow_bs (bs); }if (bm ==NULL ) {
Signature · 1 line
2.26.2
Re: [PATCH v7 36/47] nbd: Use CAF when looking for dirty bitmap
Quoted 19 lines · Max Reitz
When looking for a dirty bitmap to share, we should handle filters by just including them in the search (so they do not break backing chains).
Signed-off-by Max Reitz <[email protected]>--- nbd/server.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)nbd/server.c +1
-1@@ -1561,13 +1561,13 @@ NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset,if (bitmap) {BdrvDirtyBitmap *bm =NULL ; -while (true) { +while (bs) { bm =bdrv_find_dirty_bitmap (bs, bitmap);
nbd/server.c NBDExport *nbd_export_new(BlockDriverState *bs, uint64_t dev_offset, · line 1573
Quoted 10 lines
nbd/server.c +2
-2-if (bm !=NULL || bs->backing ==NULL ) { +if (bm !=NULL ) {break ; } - bs = bs->backing->bs; + bs =bdrv_filter_or_cow_bs (bs); }if (bm ==NULL ) {
PATCH v7 37/47 qemu-img: Use child access functions
In reply to Max ReitzThis changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
--- qemu-img.c | 36 ++++++++++++++++++++++-------------- tests/qemu-iotests/204.out | 1 + 2 files changed, 23 insertions(+), 14 deletions(-)
qemu-img.c +22 -14
@@ -1089,7 +1089,7 @@ static int img_commit(int argc, char **argv)/* This is different from QMP, which by default uses the deepest file in * the backing chain (i.e., the very base); however, the traditional * behavior of qemu-img commit is using the immediate backing file. */ - base_bs =backing_bs (bs); + base_bs =bdrv_backing_chain_next (bs);if (!base_bs) {error_setg (&local_err,"Image does not have a backing file" ); goto done;
@@ -1737,18 +1737,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)if (s->sector_next_status <= sector_num) {uint64_t offset = (sector_num - src_cur_offset) *BDRV_SECTOR_SIZE ;int64_t count; +BlockDriverState *src_bs =blk_bs (s->src[src_cur]); +BlockDriverState *base; + +if (s->target_has_backing) { + base =bdrv_cow_bs (bdrv_skip_filters (src_bs)); + }else { + base =NULL ; + } do { count = n * BDRV_SECTOR_SIZE; - if (s->target_has_backing) { - ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset, - count, &count, NULL, NULL); - } else { - ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL, - offset, count, &count, NULL, - NULL); - } + ret = bdrv_block_status_above(src_bs, base, offset, count, &count, + NULL, NULL); if (ret < 0) { if (s->salvage) {
@@ -2673,7 +2675,8 @@ static int img_convert(int argc, char **argv)* s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only * for optimizations, so such a case is not fatal. */ - s.target_backing_sectors =bdrv_nb_sectors (out_bs->backing->bs); + s.target_backing_sectors = +bdrv_nb_sectors (bdrv_backing_chain_next (out_bs)); }else { s.target_backing_sectors =-1 ; }
@@ -3044,6 +3047,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, depth =0 ;for (;;) { + bs =bdrv_skip_filters (bs); ret =bdrv_block_status (bs, offset, bytes, &bytes, &map, &file);if (ret <0 ) {return ret;
@@ -3052,7 +3056,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,if (ret & (BDRV_BLOCK_ZERO |BDRV_BLOCK_DATA )) {break ; } - bs =backing_bs (bs); + bs =bdrv_cow_bs (bs);if (bs ==NULL ) { ret =0 ;break ;
@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;
@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);
@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),
@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need * to take action */ - ret =bdrv_is_allocated_above (backing_bs (bs), prefix_chain_bs, - false, offset, n, &n); + ret =bdrv_is_allocated_above (bdrv_cow_bs (unfiltered_bs), + prefix_chain_bs, false, + offset, n, &n);if (ret <0 ) {error_report ("error while reading image metadata: %s" ,strerror (-ret));
tests/qemu-iotests/204.out +1 -0
@@ -59,5 +59,6 @@ Offset Length File 0x900000 0x2400000 TEST_DIR/t.IMGFMT 0x3c00000 0x1100000 TEST_DIR/t.IMGFMT 0x6a00000 0x400000 TEST_DIR/t.IMGFMT +0x6e00000 0x1200000 TEST_DIR/t.IMGFMT.base No errors were found on the image. *** done
Signature · 1 line
2.26.2
Re: [PATCH v7 37/47] qemu-img: Use child access functions
Quoted 123 lines · Max Reitz
This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
Signed-off-by Max Reitz <[email protected]>--- qemu-img.c | 36 ++++++++++++++++++++++-------------- tests/qemu-iotests/204.out | 1 + 2 files changed, 23 insertions(+), 14 deletions(-)qemu-img.c +22
-14@@ -1089,7 +1089,7 @@ static int img_commit(int argc, char **argv)/* This is different from QMP, which by default uses the deepest file in * the backing chain (i.e., the very base); however, the traditional * behavior of qemu-img commit is using the immediate backing file. */ - base_bs =backing_bs (bs); + base_bs =bdrv_backing_chain_next (bs);if (!base_bs) {error_setg (&local_err,"Image does not have a backing file" ); goto done;@@ -1737,18 +1737,20 @@ static int convert_iteration_sectors(ImgConvertState *s, int64_t sector_num)if (s->sector_next_status <= sector_num) {uint64_t offset = (sector_num - src_cur_offset) *BDRV_SECTOR_SIZE ;int64_t count; +BlockDriverState *src_bs =blk_bs (s->src[src_cur]); +BlockDriverState *base; + +if (s->target_has_backing) { + base =bdrv_cow_bs (bdrv_skip_filters (src_bs)); + }else { + base =NULL ; + } do { count = n * BDRV_SECTOR_SIZE; - if (s->target_has_backing) { - ret = bdrv_block_status(blk_bs(s->src[src_cur]), offset, - count, &count, NULL, NULL); - } else { - ret = bdrv_block_status_above(blk_bs(s->src[src_cur]), NULL, - offset, count, &count, NULL, - NULL); - } + ret = bdrv_block_status_above(src_bs, base, offset, count, &count, + NULL, NULL); if (ret < 0) { if (s->salvage) {@@ -2673,7 +2675,8 @@ static int img_convert(int argc, char **argv)* s.target_backing_sectors has to be negative, which it will * be automatically). The backing file length is used only * for optimizations, so such a case is not fatal. */ - s.target_backing_sectors =bdrv_nb_sectors (out_bs->backing->bs); + s.target_backing_sectors = +bdrv_nb_sectors (bdrv_backing_chain_next (out_bs)); }else { s.target_backing_sectors =-1 ; }@@ -3044,6 +3047,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset, depth =0 ;for (;;) { + bs =bdrv_skip_filters (bs); ret =bdrv_block_status (bs, offset, bytes, &bytes, &map, &file);if (ret <0 ) {return ret;@@ -3052,7 +3056,7 @@ static int get_block_status(BlockDriverState *bs, int64_t offset,if (ret & (BDRV_BLOCK_ZERO |BDRV_BLOCK_DATA )) {break ; } - bs =backing_bs (bs); + bs =bdrv_cow_bs (bs);if (bs ==NULL ) { ret =0 ;break ;@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need * to take action */ - ret =bdrv_is_allocated_above (backing_bs (bs), prefix_chain_bs, - false, offset, n, &n); + ret =bdrv_is_allocated_above (bdrv_cow_bs (unfiltered_bs), + prefix_chain_bs, false, + offset, n, &n);if (ret <0 ) {error_report ("error while reading image metadata: %s" ,strerror (-ret));tests/qemu-iotests/204.out +0
-0
tests/qemu-iotests/204.out Offset Length File · line 64
Quoted 6 lines
tests/qemu-iotests/204.out +1
-0@@ -59,5 +59,6 @@ Offset Length File 0x900000 0x2400000 TEST_DIR/t.IMGFMT 0x3c00000 0x1100000 TEST_DIR/t.IMGFMT 0x6a00000 0x400000 TEST_DIR/t.IMGFMT +0x6e00000 0x1200000 TEST_DIR/t.IMGFMT.base No errors were found on the image. *** done
Re: [PATCH v7 37/47] qemu-img: Use child access functions
In reply to Max ReitzAm 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 35 lines
This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
Signed-off-by Max Reitz <[email protected]>qemu-img.c +4
-1@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need
qemu-img.c static int img_rebase(int argc, char **argv) · line 3753
Quoted 10 lines
qemu-img.c +3
-2* to take action */ - ret =bdrv_is_allocated_above (backing_bs (bs), prefix_chain_bs, - false, offset, n, &n); + ret =bdrv_is_allocated_above (bdrv_cow_bs (unfiltered_bs), + prefix_chain_bs, false, + offset, n, &n);if (ret <0 ) {error_report ("error while reading image metadata: %s" ,strerror (-ret));
img_rebase() has these additional calls:
/* If the cluster is allocated, we don't need to take action */
ret = bdrv_is_allocated(bs, offset, n, &n);And:
if (out_baseimg && *out_baseimg) {
ret = bdrv_change_backing_file(bs, out_baseimg, out_basefmt);
} else {
ret = bdrv_change_backing_file(bs, NULL, NULL);
}Shouldn't they use unfiltered_bs?
(Not that it's likely that anyone would use 'qemu-img rebase' with a filter, but while you're touching it...)
Kevin
Re: [PATCH v7 37/47] qemu-img: Use child access functions
Quoted 54 lines · Kevin Wolf
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 35 lines
This changes iotest 204's output, because blkdebug on top of a COW node used to make qemu-img map disregard the rest of the backing chain (the backing chain was broken by the filter). With this patch, the allocation in the base image is reported correctly.
Signed-off-by Max Reitz <[email protected]>qemu-img.c +4
-1@@ -3437,6 +3441,7 @@ static int img_rebase(int argc, char **argv)uint8_t *buf_old =NULL ;uint8_t *buf_new =NULL ;BlockDriverState *bs =NULL , *prefix_chain_bs =NULL ; +BlockDriverState *unfiltered_bs;char *filename;const char *fmt, *cache, *src_cache, *out_basefmt, *out_baseimg;int c, flags, src_flags, ret;@@ -3571,6 +3576,8 @@ static int img_rebase(int argc, char **argv) } bs =blk_bs (blk); + unfiltered_bs =bdrv_skip_filters (bs); +if (out_basefmt !=NULL ) {if (bdrv_find_format (out_basefmt) ==NULL ) {error_report ("Invalid format name: '%s'" , out_basefmt);@@ -3582,7 +3589,7 @@ static int img_rebase(int argc, char **argv)/* For safe rebasing we need to compare old and new backing file */ if (!unsafe) {QDict *options =NULL ; -BlockDriverState *base_bs =backing_bs (bs); +BlockDriverState *base_bs =bdrv_cow_bs (unfiltered_bs);if (base_bs) { blk_old_backing = blk_new(qemu_get_aio_context(),@@ -3738,8 +3745,9 @@ static int img_rebase(int argc, char **argv)* If cluster wasn't changed since prefix_chain, we don't need qemu-img.c static int img_rebase(int argc, char **argv) · line 3753
Quoted 10 lines
qemu-img.c +3
-2* to take action */ - ret =bdrv_is_allocated_above (backing_bs (bs), prefix_chain_bs, - false, offset, n, &n); + ret =bdrv_is_allocated_above (bdrv_cow_bs (unfiltered_bs), + prefix_chain_bs, false, + offset, n, &n);if (ret <0 ) {error_report ("error while reading image metadata: %s" ,strerror (-ret));
img_rebase()has these additional calls:/* If the cluster is allocated, we don't need to take action */ ret = bdrv_is_allocated(bs, offset, n, &n);And:
Quoted 7 lines
if (out_baseimg && *out_baseimg) { ret = bdrv_change_backing_file(bs, out_baseimg, out_basefmt); } else { ret = bdrv_change_backing_file(bs, NULL, NULL); }Shouldn't they use
unfiltered_bs?
Oh, yes, the second one definitely.
As for the first one, I don’t think there’s a difference. But why not, we really want to query unfiltered_bs, so it’s better to do so explicitly than through the implicit fall-through behavior of block_status.
Max
PATCH v7 38/47 block: Drop backing_bs()
In reply to Max ReitzWe want to make it explicit where bs->backing is used, and we have done so. The old role of backing_bs() is now effectively taken by bdrv_cow_bs().
--- include/block/block_int.h | 5 ----- 1 file changed, 5 deletions(-)
include/block/block_int.h +0 -5
@@ -991,11 +991,6 @@ typedef enum BlockMirrorBackingMode { MIRROR_LEAVE_BACKING_CHAIN, } BlockMirrorBackingMode; -static inline BlockDriverState *backing_bs (BlockDriverState *bs) -{ -return bs->backing ? bs->backing->bs :NULL ; -} -/* Essential block drivers which must always be statically linked into qemu, and * which therefore can be accessed without using bdrv_find_format() */
Signature · 1 line
2.26.2
Re: [PATCH v7 38/47] block: Drop backing_bs()
Quoted 16 lines · Max Reitz
We want to make it explicit where
bs->backingis used, and we have done so. The old role ofbacking_bs()is now effectively taken bybdrv_cow_bs().Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block_int.h | 5 ----- 1 file changed, 5 deletions(-)include/block/block_int.h +0
-0@@ -991,11 +991,6 @@ typedef enum BlockMirrorBackingMode { MIRROR_LEAVE_BACKING_CHAIN,
include/block/block_int.h typedef enum BlockMirrorBackingMode { · line 996
Quoted 10 lines
include/block/block_int.h +0
-5} BlockMirrorBackingMode; -static inline BlockDriverState *backing_bs (BlockDriverState *bs) -{ -return bs->backing ? bs->backing->bs :NULL ; -} -/* Essential block drivers which must always be statically linked into qemu, and * which therefore can be accessed without using bdrv_find_format() */
PATCH v7 39/47 blockdev: Fix active commit choice
In reply to Max ReitzWe have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.
--- blockdev.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)
blockdev.c +21 -3
@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,AioContext *aio_context;Error *local_err =NULL ;int job_flags =JOB_DEFAULT ; +uint64_t top_perm, top_shared;if (!has_speed) { speed =0 ;
@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, goto out; } -if (top_bs == bs) { +/* +* Active commit is required if and only if someone has taken a +* WRITE permission on the top node. Historically, we have always +* used active commit for top nodes, so continue that practice. +* (Active commit is never really wrong.) +*/ +bdrv_get_cumulative_perm (top_bs, &top_perm, &top_shared); +if (top_perm &BLK_PERM_WRITE || +bdrv_skip_filters (top_bs) ==bdrv_skip_filters (bs)) + {if (has_backing_file) {error_setg (errp,"'backing-file' specified," " but 'top' is the active layer" ); goto out; } -commit_active_start (has_job_id ? job_id :NULL , bs, base_bs, - job_flags, speed, on_error, +if (!has_job_id) { +/* +* Emulate here what block_job_create() does, because it +* is possible that @bs != @top_bs (the block job should +* be named after @bs, even if @top_bs is the actual +* source) +*/ + job_id =bdrv_get_device_name (bs); + } +commit_active_start (job_id, top_bs, base_bs, job_flags, speed, on_error, filter_node_name,NULL ,NULL , false, &local_err); }else {BlockDriverState *overlay_bs =bdrv_find_overlay (bs, top_bs);
Signature · 1 line
2.26.2
Re: [PATCH v7 39/47] blockdev: Fix active commit choice
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 22 lines
We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- blockdev.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)blockdev.c +1
-0@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,AioContext *aio_context;Error *local_err =NULL ;int job_flags =JOB_DEFAULT ; +uint64_t top_perm, top_shared;if (!has_speed) { speed =0 ;
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710
Quoted 7 lines
blockdev.c +3
-1@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, goto out; } -if (top_bs == bs) { +/* +* Active commit is required if and only if someone has taken a +* WRITE permission on the top node.
...or if someone wants to take a WRITE permission while the job is running.
Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713
Quoted 4 lines
blockdev.c +3
-0Historically, we have always +* used active commit for top nodes, so continue that practice. +* (Active commit is never really wrong.) +*/
Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".
This is even more problematic because the commit job doesn't unshare BLK_PERM_WRITE yet, so it would lead to silent corruption rather than an error.
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720
Quoted 7 lines
blockdev.c +4
-0+bdrv_get_cumulative_perm (top_bs, &top_perm, &top_shared); +if (top_perm &BLK_PERM_WRITE || +bdrv_skip_filters (top_bs) ==bdrv_skip_filters (bs)) + {if (has_backing_file) {error_setg (errp,"'backing-file' specified," " but 'top' is the active layer" );
Hm, this error message isn't accurate any more.
In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2729
Quoted 11 lines
blockdev.c +7
-2goto out; } - commit_active_start(has_job_id ? job_id : NULL, bs, base_bs, - job_flags, speed, on_error, +if (!has_job_id) { +/* +* Emulate here what block_job_create() does, because it +* is possible that @bs != @top_bs (the block job should +* be named after @bs, even if @top_bs is the actual +* source) +*/
Should it? Oh, yes, looks like it. block-commit is weird. :-)
Quoted 6 lines
blockdev.c +3
-0+ job_id =bdrv_get_device_name (bs); + } +commit_active_start (job_id, top_bs, base_bs, job_flags, speed, on_error, filter_node_name,NULL ,NULL , false, &local_err); }else {BlockDriverState *overlay_bs = bdrv_find_overlay(bs, top_bs);
Kevin
Re: [PATCH v7 39/47] blockdev: Fix active commit choice
Quoted 43 lines · Kevin Wolf
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 22 lines
We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- blockdev.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)blockdev.c +1
-0@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,AioContext *aio_context;Error *local_err =NULL ;int job_flags =JOB_DEFAULT ; +uint64_t top_perm, top_shared;if (!has_speed) { speed =0 ;blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710
Quoted 7 lines
blockdev.c +3
-1@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, goto out; } -if (top_bs == bs) { +/* +* Active commit is required if and only if someone has taken a +* WRITE permission on the top node. ...or if someone wants to take a WRITE permission while the job is running.
Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713
Quoted 4 lines
blockdev.c +3
-0Historically, we have always +* used active commit for top nodes, so continue that practice. +* (Active commit is never really wrong.) +*/
Quoted 4 lines
Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".
I think that’s what I meant by “historically”. Is “legacily” a word?
But sure, I can make it more explicit.
Quoted 12 lines
This is even more problematic because the commit job doesn't unshare
BLK_PERM_WRITEyet, so it would lead to silent corruption rather than an error.blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720
Quoted 7 lines
blockdev.c +4
-0+bdrv_get_cumulative_perm (top_bs, &top_perm, &top_shared); +if (top_perm &BLK_PERM_WRITE || +bdrv_skip_filters (top_bs) ==bdrv_skip_filters (bs)) + {if (has_backing_file) {error_setg (errp,"'backing-file' specified," " but 'top' is the active layer" );
Quoted 4 lines
Hm, this error message isn't accurate any more.
In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.
Hm. I wanted to agree, and then I wanted to come up with a QAPI documentation that fits the new behavior (because I think it makes more sense to change the QAPI documentation along with the behavior change, rather than to force us to allow backing-file for anything that isn’t on the top layer).
But in the process of coming up with a better description, I noticed that this doesn’t say “is a root node”, it says “is the active layer”. I would say a node in the active layer is a node that has some parent that has taken a WRITE permission on it. So actually I think that the documentation is right, and this code only now fits.
Though I do think this wants for some clarification. Perhaps “If 'top' is the active layer (i.e., is a node that may be written to), specifying a backing [...]”?
There’s more wrong with the specification, namely the whole part under @backing-file past the “(Since 2.1)”, starting with “If top == base”. I think all of that should go to the top level. (And “If top == active” should be changed to “If top is active (i.e., may be written to)”.)
Max
Re: [PATCH v7 39/47] blockdev: Fix active commit choice
Am 24.08.2020 um 15:18 hat Max Reitz geschrieben:
Quoted 76 lines · Kevin Wolf
Quoted 43 lines
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 22 lines
We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- blockdev.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)blockdev.c +1
-0@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,AioContext *aio_context;Error *local_err =NULL ;int job_flags =JOB_DEFAULT ; +uint64_t top_perm, top_shared;if (!has_speed) { speed =0 ;blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710
Quoted 7 lines
blockdev.c +3
-1@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, goto out; } -if (top_bs == bs) { +/* +* Active commit is required if and only if someone has taken a +* WRITE permission on the top node. ...or if someone wants to take a WRITE permission while the job is running.
Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713
Quoted 4 lines
blockdev.c +3
-0Historically, we have always +* used active commit for top nodes, so continue that practice. +* (Active commit is never really wrong.) +*/ Quoted 4 lines
Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".
I think that’s what I meant by “historically”. Is “legacily” a word?
But sure, I can make it more explicit.
Quoted 12 lines
This is even more problematic because the commit job doesn't unshare
BLK_PERM_WRITEyet, so it would lead to silent corruption rather than an error.blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720
Quoted 7 lines
blockdev.c +4
-0+bdrv_get_cumulative_perm (top_bs, &top_perm, &top_shared); +if (top_perm &BLK_PERM_WRITE || +bdrv_skip_filters (top_bs) ==bdrv_skip_filters (bs)) + {if (has_backing_file) {error_setg (errp,"'backing-file' specified," " but 'top' is the active layer" );Quoted 4 lines
Hm, this error message isn't accurate any more.
In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.
Hm. I wanted to agree, and then I wanted to come up with a QAPI documentation that fits the new behavior (because I think it makes more sense to change the QAPI documentation along with the behavior change, rather than to force us to allow backing-file for anything that isn’t on the top layer).
Quoted 5 lines
But in the process of coming up with a better description, I noticed that this doesn’t say “is a root node”, it says “is the active layer”. I would say a node in the active layer is a node that has some parent that has taken a WRITE permission on it. So actually I think that the documentation is right, and this code only now fits.
Then you may have not only "the" active layer, but multiple active layers. I find this a bit counterintuitive.
There is a simple reason why backing-file is an error for a root node:
It doesn't have overlays, so a value to write to the header of overlay images just doesn't make sense.
The same reasoning doesn't apply for writable images that do have overlays. Forbidding backing-file is a more arbitrary restriction there. I'm not saying that we can't make arbitrary restrictions where allowing an option is not worth the effort, but I feel they should be spelt out more explicitly instead of twisting words like "active layer" until they fit the code.
Quoted 3 lines
Though I do think this wants for some clarification. Perhaps “If 'top' is the active layer (i.e., is a node that may be written to), specifying a backing [...]”?
"If 'top' doesn't have an overlay image or is in use by a writer..."?
Quoted 4 lines
There’s more wrong with the specification, namely the whole part under @backing-file past the “(Since 2.1)”, starting with “If top == base”. I think all of that should go to the top level. (And “If top == active” should be changed to “If top is active (i.e., may be written to)”.)
At least the latter only becomes wrong with this patch, so I think it needs to be changed by this patch.
Kevin
Re: [PATCH v7 39/47] blockdev: Fix active commit choice
Quoted 77 lines · Kevin Wolf
Am 24.08.2020 um 15:18 hat Max Reitz geschrieben:
Quoted 76 lines · Kevin Wolf
Quoted 43 lines
Am 25.06.2020 um 17:22 hat Max Reitz geschrieben:
Quoted 22 lines
We have to perform an active commit whenever the top node has a parent that has taken the WRITE permission on it.
Signed-off-by Max Reitz <[email protected]>Reviewed-by Vladimir Sementsov-Ogievskiy <[email protected]>--- blockdev.c | 24 +++++++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-)blockdev.c +1
-0@@ -2589,6 +2589,7 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device,AioContext *aio_context;Error *local_err =NULL ;int job_flags =JOB_DEFAULT ; +uint64_t top_perm, top_shared;if (!has_speed) { speed =0 ;blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2710
Quoted 7 lines
blockdev.c +3
-1@@ -2704,14 +2705,31 @@ void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, goto out; } -if (top_bs == bs) { +/* +* Active commit is required if and only if someone has taken a +* WRITE permission on the top node. ...or if someone wants to take a WRITE permission while the job is running.
Future intentions of the user is something that we can't know, so maybe this should become an option in the future (not in this series, of course).
blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2713
Quoted 4 lines
blockdev.c +3
-0Historically, we have always +* used active commit for top nodes, so continue that practice. +* (Active commit is never really wrong.) +*/ Quoted 4 lines
Changing the practice would break compatibility with clients that start an active commit job and then attach it to a read-write device, so we must continue the practice. I think the comment should be clearer about this, it sounds more like "no reason, but why not".
I think that’s what I meant by “historically”. Is “legacily” a word?
But sure, I can make it more explicit.
Quoted 12 lines
This is even more problematic because the commit job doesn't unshare
BLK_PERM_WRITEyet, so it would lead to silent corruption rather than an error.blockdev.c void qmp_block_commit(bool has_job_id, const char *job_id, const char *device, · line 2720
Quoted 7 lines
blockdev.c +4
-0+bdrv_get_cumulative_perm (top_bs, &top_perm, &top_shared); +if (top_perm &BLK_PERM_WRITE || +bdrv_skip_filters (top_bs) ==bdrv_skip_filters (bs)) + {if (has_backing_file) {error_setg (errp,"'backing-file' specified," " but 'top' is the active layer" );Quoted 4 lines
Hm, this error message isn't accurate any more.
In fact, the implementation isn't consistent with the QAPI documentation any more, because backing-file is only an error for the top level.
Hm. I wanted to agree, and then I wanted to come up with a QAPI documentation that fits the new behavior (because I think it makes more sense to change the QAPI documentation along with the behavior change, rather than to force us to allow backing-file for anything that isn’t on the top layer).
Quoted 8 lines
Quoted 5 lines
But in the process of coming up with a better description, I noticed that this doesn’t say “is a root node”, it says “is the active layer”. I would say a node in the active layer is a node that has some parent that has taken a WRITE permission on it. So actually I think that the documentation is right, and this code only now fits.
Then you may have not only "the" active layer, but multiple active layers. I find this a bit counterintuitive.
Depends on what you count as a layer. I don’t think that’s a clearly defined term, is it? I only know of “active layer”, “format layer”, “protocol layer”, and you can at least have multiple format layers above each other. So I don’t find it counterintuitive.
But perhaps it’d be best to just get away from the term “active layer”, as you propose below.
Quoted 3 lines
There is a simple reason why backing-file is an error for a root node:
It doesn't have overlays, so a value to write to the header of overlay images just doesn't make sense.
Ah, yeah...
Quoted 6 lines
The same reasoning doesn't apply for writable images that do have overlays. Forbidding backing-file is a more arbitrary restriction there. I'm not saying that we can't make arbitrary restrictions where allowing an option is not worth the effort, but I feel they should be spelt out more explicitly instead of twisting words like "active layer" until they fit the code.
I’m all for spelling it out more explicitly. I just noticed that I couldn’t clearly distinguish “active layer” from “other” cases of nodes with writers on them, which is why I noted that “active” to me means the post-patch behavior already.
You’re right that there is no semantic reason for making it an error. So I just want it to be an error to be lazy. I hope you let me do that.
(I don’t think there’s much of a problem with it, considering that
commits on nodes that have the WRITE permission taken are basically just completely broken right now.)
Quoted 5 lines
Quoted 3 lines
Though I do think this wants for some clarification. Perhaps “If 'top' is the active layer (i.e., is a node that may be written to), specifying a backing [...]”?
"If 'top' doesn't have an overlay image or is in use by a writer..."?
I.e., avoiding the term “active layer” altogether? Sounds good. Only, I don’t know about “writer”... But it’s already used in BlockdevOptionsFile.dynamic-auto-read-only’s description, so I suppose we can use it here, too. (I just don’t know if as a non-block-layer-developer I’d know what it means.)
(Also, yes, you’re right, the current behavior of giving all root nodes an active commit of course remains, even when there are no writers.)
Quoted 7 lines
Quoted 4 lines
There’s more wrong with the specification, namely the whole part under @backing-file past the “(Since 2.1)”, starting with “If top == base”. I think all of that should go to the top level. (And “If top == active” should be changed to “If top is active (i.e., may be written to)”.)
At least the latter only becomes wrong with this patch, so I think it needs to be changed by this patch.
Sure. So I understand you agree with moving the whole chunk, right?
Max