backup: allow specifying minimum cluster size
v3 cover backup: allow specifying minimum cluster size
Discussion for v2:
https://lore.kernel.org/qemu-devel/[email protected]/
Changes in v3:
* Pass min_cluster_size option directly without checking has_min_cluster_size, because the default is 0 anyways.
* Calculate maximum of passed-in argument and default once at the beginning of block_copy_calculate_cluster_size()
* Update warning message to reflect actual value used
* Do not leak qdict in error case
* Use PRI{i,u}64 macros
Discussion for v1:
https://lore.kernel.org/qemu-devel/[email protected]/
-
Changes in v2:
* Use 'size' type in QAPI.
* Remove option in cbw_parse_options(), i.e. before parsing generic blockdev options.
* Reword commit messages hoping to describe the issue in a more straight-forward way.
In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because cbw_co_pdiscard_snapshot() will align down the discard requests and thus effectively ignore then.
To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source.
Fiona Ebner (2):
copy-before-write: allow specifying minimum cluster size
backup: add minimum cluster size to performance options
block/backup.c | 2 +- block/block-copy.c | 36 ++++++++++++++++++++++++++---------- block/copy-before-write.c | 14 +++++++++++++- block/copy-before-write.h | 1 + blockdev.c | 3 +++ include/block/block-copy.h | 1 + qapi/block-core.json | 17 ++++++++++++++--- 7 files changed, 59 insertions(+), 15 deletions(-)
Signature · 1 line
2.39.2
PATCH v3 1/2 copy-before-write: allow specifying minimum cluster size
In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because cbw_co_pdiscard_snapshot() will align down the discard requests and thus effectively ignore then.
To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source.
The type 'size' (corresponding to uint64_t in C) is used in QAPI to rule out negative inputs and for consistency with already existing @cluster-size parameters. Since block_copy_calculate_cluster_size() uses int64_t for its result, a check that the input is not too large is added in block_copy_state_new() before calling it. The calculation in block_copy_calculate_cluster_size() is done in the target int64_t type.
---
Changes in v3:
* Pass min_cluster_size option directly without checking has_min_cluster_size, because the default is 0 anyways.
* Calculate maximum of passed-in argument and default once at the beginning of block_copy_calculate_cluster_size()
* Update warning message to reflect actual value used
* Use PRI{i,u}64 macros
block/block-copy.c | 36 ++++++++++++++++++++++++++---------- block/copy-before-write.c | 5 ++++- include/block/block-copy.h | 1 + qapi/block-core.json | 8 +++++++- 4 files changed, 38 insertions(+), 12 deletions(-)
block/block-copy.c +26 -10
@@ -310,6 +310,7 @@ void block_copy_set_copy_opts(BlockCopyState *s, bool use_copy_range, }static int64_t block_copy_calculate_cluster_size (BlockDriverState *target, +int64_t min_cluster_size,Error **errp) {int ret;
@@ -319,6 +320,9 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target,GLOBAL_STATE_CODE ();GRAPH_RDLOCK_GUARD_MAINLOOP (); + min_cluster_size =MAX (min_cluster_size, + (int64_t )BLOCK_COPY_CLUSTER_SIZE_DEFAULT ); + target_does_cow =bdrv_backing_chain_next (target);/*
@@ -329,13 +333,13 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target, ret =bdrv_get_info (target, &bdi);if (ret == -ENOTSUP && !target_does_cow) {/* Cluster size is not defined */ -warn_report ("The target block device doesn't provide " -"information about the block size and it doesn't have a " -"backing file. The default block size of %u bytes is " -"used. If the actual block size of the target exceeds " -"this default, the backup may be unusable" , -BLOCK_COPY_CLUSTER_SIZE_DEFAULT ); -return BLOCK_COPY_CLUSTER_SIZE_DEFAULT ; +warn_report ("The target block device doesn't provide information about " +"the block size and it doesn't have a backing file. The " +"(default) block size of %" PRIi64" bytes is used. If the " +"actual block size of the target exceeds this value, the " +"backup may be unusable" , + min_cluster_size); +return min_cluster_size; }else if (ret <0 && !target_does_cow) {error_setg_errno (errp, -ret, "Couldn't determine the cluster size of the target image, "
@@ -345,16 +349,17 @@ static int64_t block_copy_calculate_cluster_size(BlockDriverState *target,return ret; }else if (ret <0 && target_does_cow) {/* Not fatal; just trudge on ahead. */ -return BLOCK_COPY_CLUSTER_SIZE_DEFAULT ; +return min_cluster_size; } -return MAX (BLOCK_COPY_CLUSTER_SIZE_DEFAULT , bdi.cluster_size); +return MAX (min_cluster_size, bdi.cluster_size); }BlockCopyState *block_copy_state_new (BdrvChild *source,BdrvChild *target,BlockDriverState *copy_bitmap_bs,const BdrvDirtyBitmap *bitmap,bool discard_source, +uint64_t min_cluster_size,Error **errp) {ERRP_GUARD ();
@@ -365,7 +370,18 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,GLOBAL_STATE_CODE (); - cluster_size =block_copy_calculate_cluster_size (target->bs, errp); +if (min_cluster_size >INT64_MAX ) { +error_setg (errp,"min-cluster-size too large: %" PRIu64" > %" PRIi64, + min_cluster_size,INT64_MAX ); +return NULL ; + }else if (min_cluster_size && !is_power_of_2 (min_cluster_size)) { +error_setg (errp,"min-cluster-size needs to be a power of 2" ); +return NULL ; + } + + cluster_size =block_copy_calculate_cluster_size (target->bs, + (int64_t )min_cluster_size, + errp);if (cluster_size <0 ) {return NULL ; }
block/copy-before-write.c +4 -1
@@ -417,6 +417,7 @@ static BlockdevOptions *cbw_parse_options(QDict *options, Error **errp)qdict_extract_subqdict (options,NULL ,"bitmap" );qdict_del (options,"on-cbw-error" );qdict_del (options,"cbw-timeout" ); +qdict_del (options,"min-cluster-size" ); out:visit_free (v);
@@ -476,8 +477,10 @@ static int cbw_open(BlockDriverState *bs, QDict *options, int flags, bs->file->bs->supported_zero_flags); s->discard_source = flags &BDRV_O_CBW_DISCARD_SOURCE ; + s->bcs =block_copy_state_new (bs->file, s->target, bs, bitmap, - flags &BDRV_O_CBW_DISCARD_SOURCE , errp); + flags &BDRV_O_CBW_DISCARD_SOURCE , + opts->min_cluster_size, errp);if (!s->bcs) {error_prepend (errp,"Cannot create block-copy-state: " );return -EINVAL ;
include/block/block-copy.h +1 -0
@@ -28,6 +28,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,BlockDriverState *copy_bitmap_bs, const BdrvDirtyBitmap *bitmap, bool discard_source, + uint64_t min_cluster_size, Error **errp);/* Function should be called prior any actual copy request */
qapi/block-core.json +7 -1
@@ -4638,12 +4638,18 @@ # @on-cbw-error parameter will decide how this failure is handled. # Default 0. (Since 7.1) # +# @min-cluster-size: Minimum size of blocks used by copy-before-write +# operations. Has to be a power of 2. No effect if smaller than +# the maximum of the target's cluster size and 64 KiB. Default 0. +# (Since 9.1) +# # Since: 6.2 ## { 'struct': 'BlockdevOptionsCbw', 'base': 'BlockdevOptionsGenericFormat', 'data': { 'target': 'BlockdevRef', '*bitmap': 'BlockDirtyBitmap', - '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32' } } + '*on-cbw-error': 'OnCbwError', '*cbw-timeout': 'uint32', + '*min-cluster-size': 'size' } } ## # @BlockdevOptions:
Signature · 1 line
2.39.2
Re: [PATCH v3 1/2] copy-before-write: allow specifying minimum cluster size
Quoted 18 lines · Fiona Ebner
In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because
cbw_co_pdiscard_snapshot()will align down the discard requests and thus effectively ignore then.To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source.
The type 'size' (corresponding to
uint64_tin C) is used in QAPI to rule out negative inputs and for consistency with already existing @cluster-size parameters. Sinceblock_copy_calculate_cluster_size()usesint64_tfor its result, a check that the input is not too large is added inblock_copy_state_new()before calling it. The calculation inblock_copy_calculate_cluster_size()is done in the targetint64_ttype.
Quoted 3 lines
Suggested-by Vladimir Sementsov-Ogievskiy<[email protected]>Signed-off-by Fiona Ebner<[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH v3 2/2 backup: add minimum cluster size to performance options
In reply to Fiona EbnerIn the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because cbw_co_pdiscard_snapshot() will align down the discard requests and thus effectively ignore then.
To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source.
---
Changes in v3:
* Use PRI{i,u}64 macros
* Do not leak qdict in error case
block/backup.c | 2 +- block/copy-before-write.c | 9 +++++++++ block/copy-before-write.h | 1 + blockdev.c | 3 +++ qapi/block-core.json | 9 +++++++-- 5 files changed, 21 insertions(+), 3 deletions(-)
block/backup.c +1 -1
@@ -458,7 +458,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, } cbw =bdrv_cbw_append (bs, target, filter_node_name, discard_source, - &bcs, errp); + perf->min_cluster_size, &bcs, errp);if (!cbw) { goto error; }
block/copy-before-write.c +9 -0
@@ -548,6 +548,7 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source,BlockDriverState *target, const char *filter_node_name, bool discard_source, + uint64_t min_cluster_size, BlockCopyState **bcs, Error **errp) {
@@ -567,6 +568,14 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source,qdict_put_str (opts,"file" ,bdrv_get_node_name (source));qdict_put_str (opts,"target" ,bdrv_get_node_name (target)); +if (min_cluster_size >INT64_MAX ) { +error_setg (errp,"min-cluster-size too large: %" PRIu64" > %" PRIi64, + min_cluster_size,INT64_MAX ); +qobject_unref (opts); +return NULL ; + } +qdict_put_int (opts,"min-cluster-size" , (int64_t )min_cluster_size); + top =bdrv_insert_node (source, opts, flags, errp);if (!top) {return NULL ;
block/copy-before-write.h +1 -0
@@ -40,6 +40,7 @@ BlockDriverState *bdrv_cbw_append(BlockDriverState *source,BlockDriverState *target, const char *filter_node_name, bool discard_source, + uint64_t min_cluster_size, BlockCopyState **bcs, Error **errp);void bdrv_cbw_drop (BlockDriverState *bs);
blockdev.c +3 -0
@@ -2655,6 +2655,9 @@ static BlockJob *do_backup_common(BackupCommon *backup,if (backup->x_perf->has_max_chunk) { perf.max_chunk = backup->x_perf->max_chunk; } +if (backup->x_perf->has_min_cluster_size) { + perf.min_cluster_size = backup->x_perf->min_cluster_size; + } } if ((backup->sync == MIRROR_SYNC_MODE_BITMAP) ||
qapi/block-core.json +7 -2
@@ -1551,11 +1551,16 @@ # it should not be less than job cluster size which is calculated # as maximum of target image cluster size and 64k. Default 0. # +# @min-cluster-size: Minimum size of blocks used by copy-before-write +# and background copy operations. Has to be a power of 2. No +# effect if smaller than the maximum of the target's cluster size +# and 64 KiB. Default 0. (Since 9.1) +# # Since: 6.0 ## { 'struct': 'BackupPerf', - 'data': { '*use-copy-range': 'bool', - '*max-workers': 'int', '*max-chunk': 'int64' } } + 'data': { '*use-copy-range': 'bool', '*max-workers': 'int', + '*max-chunk': 'int64', '*min-cluster-size': 'size' } } ## # @BackupCommon:
Signature · 1 line
2.39.2
Re: [PATCH v3 2/2] backup: add minimum cluster size to performance options
Quoted 6 lines · Fiona Ebner
In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because
cbw_co_pdiscard_snapshot()will align down the discard requests and thus effectively ignore then.
Quoted 7 lines
To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source.
Suggested-by Vladimir Sementsov-Ogievskiy<[email protected]>Signed-off-by Fiona Ebner<[email protected]>
Signature · 2 lines
Best regards, Vladimir
Vladimir Sementsov-Ogievskiy
AppliedQuoted 36 lines · Fiona Ebner
Discussion for v2:
https://lore.kernel.org/qemu-devel/[email protected]/Changes in v3:
* Passmin_cluster_sizeoption directly without checkinghas_min_cluster_size, because the default is 0 anyways.
* Calculate maximum of passed-in argument and default once at the beginning ofblock_copy_calculate_cluster_size()
* Update warning message to reflect actual value used
* Do not leak qdict in error case
* Use PRI{i,u}64 macrosDiscussion for v1:
https://lore.kernel.org/qemu-devel/[email protected]/
-
Changes in v2:
* Use 'size' type in QAPI.
* Remove option incbw_parse_options(), i.e. before parsing generic blockdev options.
* Reword commit messages hoping to describe the issue in a more straight-forward way.In the context of backup fleecing, discarding the source will not work when the fleecing image has a larger granularity than the one used for block-copy operations (can happen if the backup target has smaller cluster size), because
cbw_co_pdiscard_snapshot()will align down the discard requests and thus effectively ignore then.To make @discard-source work in such a scenario, allow specifying the minimum cluster size used for block-copy operations and thus in particular also the granularity for discard requests to the source.
Fiona Ebner (2):
copy-before-write: allow specifying minimum cluster size
backup: add minimum cluster size to performance options
Quoted 9 lines
block/backup.c | 2 +- block/block-copy.c | 36 ++++++++++++++++++++++++++---------- block/copy-before-write.c | 14 +++++++++++++- block/copy-before-write.h | 1 + blockdev.c | 3 +++ include/block/block-copy.h | 1 + qapi/block-core.json | 17 ++++++++++++++--- 7 files changed, 59 insertions(+), 15 deletions(-)
Thanks, applied to my block branch.
Signature · 2 lines
Best regards, Vladimir