backup performance: block_status + async
v2 cover backup performance: block_status + async
Hi all!
This a last part of original
"[RFC 00/24] backup performance: block_status + async", prepartions are already merged.
The series turn backup into series of block_copy_async calls, covering the whole disk, so we get block-status based paralallel async requests out of the box, which gives performance gain:
----------------- ---------------- ------------- -------------------------- -------------------------- ---------------- -------------------------------
mirror(upstream) backup(new) backup(new, no-copy-range) backup(new, copy-range-1w) backup(upstream) backup(upstream, no-copy-range)
hdd-ext4:hdd-ext4 18.86 +- 0.11 45.50 +- 2.35 19.22 +- 0.09 19.51 +- 0.09 22.85 +- 5.98 19.72 +- 0.35
hdd-ext4:ssd-ext4 8.99 +- 0.02 9.30 +- 0.01 8.97 +- 0.02 9.02 +- 0.02 9.68 +- 0.26 9.84 +- 0.12
ssd-ext4:hdd-ext4 9.09 +- 0.11 9.34 +- 0.10 9.34 +- 0.10 8.99 +- 0.01 11.37 +- 0.37 11.47 +- 0.30
ssd-ext4:ssd-ext4 4.07 +- 0.02 5.41 +- 0.05 4.05 +- 0.01 8.35 +- 0.58 9.83 +- 0.64 8.62 +- 0.35
hdd-xfs:hdd-xfs 18.90 +- 0.19 43.26 +- 2.47 19.62 +- 0.14 19.38 +- 0.16 19.55 +- 0.26 19.62 +- 0.12
hdd-xfs:ssd-xfs 8.93 +- 0.12 9.35 +- 0.03 8.93 +- 0.08 8.93 +- 0.05 9.79 +- 0.30 9.55 +- 0.15
ssd-xfs:hdd-xfs 9.15 +- 0.07 9.74 +- 0.28 9.29 +- 0.03 9.08 +- 0.05 10.85 +- 0.31 10.91 +- 0.30
ssd-xfs:ssd-xfs 4.06 +- 0.01 4.93 +- 0.02 4.04 +- 0.01 8.17 +- 0.42 9.52 +- 0.49 8.85 +- 0.46
ssd-ext4:nbd 9.96 +- 0.11 11.45 +- 0.15 11.45 +- 0.02 17.22 +- 0.06 34.45 +- 1.35 35.16 +- 0.37
nbd:ssd-ext4 9.84 +- 0.02 9.84 +- 0.04 9.80 +- 0.06 18.96 +- 0.06 30.89 +- 0.73 31.46 +- 0.21
----------------- ---------------- ------------- -------------------------- -------------------------- ---------------- -------------------------------The table shows, that copy_range is in bad relation with parallel async requests. copy_range brings real performance gain only on supporting fs, like btrfs. But even on such fs, I'm not sure that this is a good default behavior: if we do offload copy, so, that no real copy but just link block in backup the same blocks as in original, this means that further write from guest will lead to fragmentation of guest disk, when the aim of backup is to operate transparently for the guest.
So, in addition to these series I also suggest to disable copy_range by default.
===
How to test:
prepare images:
In a directories, where you want to place source and target images, prepare images by:
for img in test-source test-target; do
./qemu-img create -f raw $img 1000M;
./qemu-img bench -c 1000 -d 1 -f raw -s 1M -w --pattern=0xff $img
done
prepare similar image for nbd server, and start it somewhere by
qemu-nbd --persistent --nocache -f raw IMAGE
Then, run benchmark, like this:
./bench-backup.py --qemu new:../../x86_64-softmmu/qemu-system-x86_64 upstream:/work/src/qemu/up-backup-block-copy-master/x86_64-softmmu/qemu-system-x86_64 --dir hdd-ext4:/test-a hdd-xfs:/test-b ssd-ext4:/ssd ssd-xfs:/ssd-b --test $(for fs in ext4 xfs; do echo hdd-$fs:hdd-$fs hdd-$fs:ssd-$fs ssd-$fs:hdd-$fs ssd-$fs:ssd-$fs; done) --nbd 192.168.100.2 --test ssd-ext4:nbd nbd:ssd-ext4
(you may simply reduce number of directories/test-cases, use --help for
help)
===
Note, that I included here
"[PATCH] block/block-copy: block_copy_dirty_clusters: fix failure check" which was previously sent in separate, but still untouched in mailing list. It still may be applied separately.
Vladimir Sementsov-Ogievskiy (20):
block/block-copy: block_copy_dirty_clusters: fix failure check
iotests: 129 don't check backup "busy"
qapi: backup: add x-use-copy-range parameter block/block-copy: More explicit call_state block/block-copy: implement block_copy_async block/block-copy: add max_chunk and max_workers parameters block/block-copy: add ratelimit to block-copy block/block-copy: add block_copy_cancel
blockjob: add set_speed to BlockJobDriver
job: call job_enter from job_user_pause
qapi: backup: add x-max-chunk and x-max-workers parameters
iotests: 56: prepare for backup over block-copy
iotests: 129: prepare for backup over block-copy
iotests: 185: prepare for backup over block-copy
iotests: 219: prepare for backup over block-copy
iotests: 257: prepare for backup over block-copy
backup: move to block-copy
block/block-copy: drop unused argument of block_copy()
simplebench: bench_block_job: add cmd_options argument
simplebench: add bench-backup.py
qapi/block-core.json | 11 +- block/backup-top.h | 1 + include/block/block-copy.h | 45 +++- include/block/block_int.h | 8 + include/block/blockjob_int.h | 2 + block/backup-top.c | 6 +- block/backup.c | 170 ++++++++------ block/block-copy.c | 183 ++++++++++++--- block/replication.c | 1 + blockdev.c | 10 + blockjob.c | 6 + job.c | 1 + scripts/simplebench/bench-backup.py | 132 +++++++++++ scripts/simplebench/bench-example.py | 2 +- scripts/simplebench/bench_block_job.py | 13 +- tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/129 | 3 +- tests/qemu-iotests/185 | 3 +- tests/qemu-iotests/185.out | 2 +- tests/qemu-iotests/219 | 13 +- tests/qemu-iotests/257 | 1 + tests/qemu-iotests/257.out | 306 ++++++++++++------------- 22 files changed, 640 insertions(+), 287 deletions(-) create mode 100755 scripts/simplebench/bench-backup.py
Signature · 1 line
2.21.0
PATCH v2 4/20 block/block-copy: More explicit call_state
Refactor common path to use BlockCopyCallState pointer as parameter, to prepare it for use in asynchronous block-copy (at least, we'll need to run block-copy in a coroutine, passing the whole parameters as one pointer).
--- block/block-copy.c | 51 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 13 deletions(-)
block/block-copy.c +38 -13
@@ -30,7 +30,15 @@ static coroutine_fn int block_copy_task_entry(AioTask *task);typedef struct BlockCopyCallState { +/* IN parameters */ +BlockCopyState *s; +int64_t offset; +int64_t bytes; + +/* State */ bool failed; + +/* OUT parameters */ bool error_is_read; }BlockCopyCallState ;
@@ -541,15 +549,17 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s,* Returns 1 if dirty clusters found and successfully copied, 0 if no dirty * clusters found and -errno on failure. */ -static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s, -int64_t offset,int64_t bytes, -bool *error_is_read) +static int coroutine_fn +block_copy_dirty_clusters (BlockCopyCallState *call_state) { +BlockCopyState *s = call_state->s; +int64_t offset = call_state->offset; +int64_t bytes = call_state->bytes; +int ret =0 ;bool found_dirty = false;int64_t end = offset + bytes;AioTaskPool *aio =NULL ; -BlockCopyCallState call_state = {false, false};/* * block_copy() user is responsible for keeping source and target in same
@@ -565,7 +575,7 @@ static int coroutine_fn block_copy_dirty_clusters(BlockCopyState *s,BlockCopyTask *task;int64_t status_bytes; - task =block_copy_task_create (s, &call_state, offset, bytes); + task =block_copy_task_create (s, call_state, offset, bytes);if (!task) {/* No more dirty bits in the bitmap */ trace_block_copy_skip_range (s, offset, bytes);
@@ -630,15 +640,12 @@ out:aio_task_pool_free (aio); } -if (error_is_read && ret <0 ) { - *error_is_read = call_state.error_is_read; - }return ret <0 ? ret : found_dirty; }/* -* block_copy +* block_copy_common * * Copy requested region, accordingly to dirty bitmap. * Collaborate with parallel block_copy requests: if they succeed it will help
@@ -646,16 +653,16 @@ out:* it means that some I/O operation failed in context of _this_ block_copy call, * not some parallel operation. */ -int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, -bool *error_is_read) +static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) {int ret; do { - ret = block_copy_dirty_clusters(s, offset, bytes, error_is_read); + ret = block_copy_dirty_clusters(call_state); if (ret == 0) { - ret = block_copy_wait_one(s, offset, bytes); + ret = block_copy_wait_one(call_state->s, call_state->offset, + call_state->bytes); } /*
@@ -672,6 +679,24 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,return ret; } +int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, +bool *error_is_read) +{ +BlockCopyCallState call_state = { + .s = s, + .offset = start, + .bytes = bytes, + }; + +int ret =block_copy_common (&call_state); + +if (error_is_read && ret <0 ) { + *error_is_read = call_state.error_is_read; + } + +return ret; +} +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s) {return s->copy_bitmap;
Signature · 1 line
2.21.0
PATCH v2 1/20 block/block-copy: block_copy_dirty_clusters: fix failure check
In reply to Vladimir Sementsov-Ogievskiyret may be > 0 on success path at this point. Fix assertion, which may crash currently.
--- block/block-copy.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
block/block-copy.c +3 -1
@@ -622,8 +622,10 @@ out:* block_copy_task_run. If it fails, it means some task already failed * for real reason, let's return first failure. * Still, assert that we don't rewrite failure by success. +* +* Note: ret may be positive here because of block-status result. */ -assert (ret ==0 ||aio_task_pool_status (aio) <0 ); +assert (ret >=0 ||aio_task_pool_status (aio) <0 ); ret =aio_task_pool_status (aio);aio_task_pool_free (aio);
Signature · 1 line
2.21.0
PATCH v2 3/20 qapi: backup: add x-use-copy-range parameter
In reply to Vladimir Sementsov-OgievskiyAdd parameter to enable/disable copy_range. Keep current default for now (enabled).
--- qapi/block-core.json | 4 +++- block/backup-top.h | 1 + include/block/block-copy.h | 2 +- include/block/block_int.h | 1 + block/backup-top.c | 4 +++- block/backup.c | 4 +++- block/block-copy.c | 4 ++-- block/replication.c | 1 + blockdev.c | 5 +++++ 9 files changed, 20 insertions(+), 6 deletions(-)
qapi/block-core.json +3 -1
@@ -1405,6 +1405,8 @@ # above node specified by @drive. If this option is not given, # a node name is autogenerated. (Since: 4.2) # +# @x-use-copy-range: use copy offloading if possible. Default true. (Since 5.1) +# # Note: @on-source-error and @on-target-error only affect background # I/O. If an error occurs during a guest write request, the device's # rerror/werror actions will be used.
@@ -1419,7 +1421,7 @@ '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool', - '*filter-node-name': 'str' } } + '*filter-node-name': 'str', '*x-use-copy-range': 'bool' } } ## # @DriveBackup:
block/backup-top.h +1 -0
@@ -33,6 +33,7 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source,BlockDriverState *target, const char *filter_node_name, uint64_t cluster_size, + bool use_copy_range, BdrvRequestFlags write_flags, BlockCopyState **bcs, Error **errp);
include/block/block-copy.h +1 -1
@@ -22,7 +22,7 @@ typedef void (*ProgressBytesCallbackFunc)(int64_t bytes, void *opaque);typedef struct BlockCopyState BlockCopyState ;BlockCopyState *block_copy_state_new (BdrvChild *source,BdrvChild *target, -int64_t cluster_size, +int64_t cluster_size,bool use_copy_range,BdrvRequestFlags write_flags,Error **errp);
include/block/block_int.h +1 -0
@@ -1225,6 +1225,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,BitmapSyncMode bitmap_mode, bool compress, const char *filter_node_name, + bool use_copy_range, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,
block/backup-top.c +3 -1
@@ -188,6 +188,7 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source,BlockDriverState *target, const char *filter_node_name, uint64_t cluster_size, + bool use_copy_range, BdrvRequestFlags write_flags, BlockCopyState **bcs, Error **errp)
@@ -246,7 +247,8 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source, state->cluster_size = cluster_size; state->bcs =block_copy_state_new (top->backing, state->target, - cluster_size, write_flags, &local_err); + cluster_size, use_copy_range, + write_flags, &local_err);if (local_err) {error_prepend (&local_err,"Cannot create block-copy-state: " ); goto fail;
block/backup.c +3 -1
@@ -334,6 +334,7 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,BitmapSyncMode bitmap_mode, bool compress, const char *filter_node_name, + bool use_copy_range, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,
@@ -440,7 +441,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, (compress ?BDRV_REQ_WRITE_COMPRESSED :0 ), backup_top =bdrv_backup_top_append (bs, target, filter_node_name, - cluster_size, write_flags, &bcs, errp); + cluster_size, use_copy_range, + write_flags, &bcs, errp);if (!backup_top) { goto error; }
block/block-copy.c +2 -2
@@ -215,7 +215,7 @@ static uint32_t block_copy_max_transfer(BdrvChild *source, BdrvChild *target) }BlockCopyState *block_copy_state_new (BdrvChild *source,BdrvChild *target, -int64_t cluster_size, +int64_t cluster_size,bool use_copy_range,BdrvRequestFlags write_flags,Error **errp) {BlockCopyState *s;
@@ -257,7 +257,7 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,* We enable copy-range, but keep small copy_size, until first * successful copy_range (look at block_copy_do_copy). */ - s->use_copy_range = true; + s->use_copy_range = use_copy_range; s->copy_size =MAX (s->cluster_size,BLOCK_COPY_MAX_BUFFER ); }
block/replication.c +1 -0
@@ -563,6 +563,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job =backup_job_create (NULL , s->secondary_disk->bs, s->hidden_disk->bs,0 ,MIRROR_SYNC_MODE_NONE ,NULL ,0 , false,NULL , + true,BLOCKDEV_ON_ERROR_REPORT ,BLOCKDEV_ON_ERROR_REPORT ,JOB_INTERNAL , backup_job_completed, bs,NULL , &local_err);
blockdev.c +5 -0
@@ -2813,10 +2813,15 @@ static BlockJob *do_backup_common(BackupCommon *backup, job_flags |=JOB_MANUAL_DISMISS ; } +if (!backup->has_x_use_copy_range) { + backup->x_use_copy_range = true; + } + job =backup_job_create (backup->job_id, bs, target_bs, backup->speed, backup->sync, bmap, backup->bitmap_mode, backup->compress, backup->filter_node_name, + backup->x_use_copy_range, backup->on_source_error, backup->on_target_error, job_flags,NULL ,NULL , txn, errp);
Signature · 1 line
2.21.0
PATCH v2 5/20 block/block-copy: implement block_copy_async
In reply to Vladimir Sementsov-OgievskiyWe'll need async block-copy invocation to use in backup directly.
--- include/block/block-copy.h | 13 +++++++++++++ block/block-copy.c | 40 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+)
include/block/block-copy.h +13 -0
@@ -19,7 +19,10 @@#include "qemu/co-shared-resource.h" typedef void (*ProgressBytesCallbackFunc )(int64_t bytes,void *opaque); +typedef void (*BlockCopyAsyncCallbackFunc )(int ret,bool error_is_read, +void *opaque);typedef struct BlockCopyState BlockCopyState ; +typedef struct BlockCopyCallState BlockCopyCallState ; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, int64_t cluster_size, bool use_copy_range,
@@ -41,6 +44,16 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,bool *error_is_read); +/* +* Run block-copy in a coroutine, return state pointer. If finished early +* returns NULL (@cb is called anyway). +*/ +BlockCopyCallState *block_copy_async (BlockCopyState *s, +int64_t offset,int64_t bytes, +bool ratelimit,int max_workers, +int64_t max_chunk, +BlockCopyAsyncCallbackFunc cb); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);
block/block-copy.c +40 -0
@@ -34,9 +34,11 @@ typedef struct BlockCopyCallState {BlockCopyState *s;int64_t offset;int64_t bytes; +BlockCopyAsyncCallbackFunc cb;/* State */ bool failed; +bool finished;/* OUT parameters */ bool error_is_read;
@@ -676,6 +678,13 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)*/ }while (ret >0 ); +if (call_state->cb) { + call_state->cb (ret, call_state->error_is_read, + call_state->s->progress_opaque); + } + + call_state->finished = true; +return ret; }
@@ -697,6 +706,37 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes,return ret; } +static void coroutine_fn block_copy_async_co_entry(void *opaque) +{ +block_copy_common (opaque); +} + +BlockCopyCallState *block_copy_async (BlockCopyState *s, +int64_t offset,int64_t bytes, +bool ratelimit,int max_workers, +int64_t max_chunk, +BlockCopyAsyncCallbackFunc cb) +{ +BlockCopyCallState *call_state =g_new (BlockCopyCallState,1 ); +Coroutine *co =qemu_coroutine_create (block_copy_async_co_entry, + call_state); + + *call_state = (BlockCopyCallState ) { + .s = s, + .offset = offset, + .bytes = bytes, + .cb = cb, + }; + +qemu_coroutine_enter (co); + +if (call_state->finished) { +g_free (call_state); +return NULL ; + } + +return call_state; +}BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s) {return s->copy_bitmap;
Signature · 1 line
2.21.0
PATCH v2 9/20 blockjob: add set_speed to BlockJobDriver
In reply to Vladimir Sementsov-OgievskiyWe are going to use async block-copy call in backup, so we'll need to passthrough setting backup speed to block-copy call.
--- include/block/blockjob_int.h | 2 ++ blockjob.c | 6 ++++++ 2 files changed, 8 insertions(+)
include/block/blockjob_int.h +2 -0
@@ -52,6 +52,8 @@ struct BlockJobDriver {* besides job->blk to the new AioContext. */ void (*attached_aio_context)(BlockJob *job,AioContext *new_context); + +void (*set_speed)(BlockJob *job,int64_t speed); };/**
blockjob.c +6 -0
@@ -256,6 +256,7 @@ static bool job_timer_pending(Job *job)void block_job_set_speed (BlockJob *job,int64_t speed,Error **errp) { +const BlockJobDriver *drv =block_job_driver (job);int64_t old_speed = job->speed;if (job_apply_verb (&job->job,JOB_VERB_SET_SPEED , errp)) {
@@ -270,6 +271,11 @@ void block_job_set_speed(BlockJob *job, int64_t speed, Error **errp)ratelimit_set_speed (&job->limit, speed,BLOCK_JOB_SLICE_TIME ); job->speed = speed; + +if (drv->set_speed) { + drv->set_speed (job, speed); + } +if (speed && speed <= old_speed) {return ; }
Signature · 1 line
2.21.0
PATCH v2 12/20 iotests: 56: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyAfter introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, we'll have to wait for paused status, as we need to wait for several parallel request. So, let's gently wait instead of just asserting that job already paused.
--- tests/qemu-iotests/056 | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
tests/qemu-iotests/056 +6 -2
@@ -306,8 +306,12 @@ class BackupTest(iotests.QMPTestCase): event = self.vm.event_wait(name="BLOCK_JOB_ERROR", match={'data': {'device': 'drive0'}}) self.assertNotEqual(event, None) - # OK, job should be wedged - res = self.vm.qmp('query-block-jobs') + # OK, job should pause, but it can't do it immediately, as it can't + # cancel other parallel requests (which didn't fail) + while True: + res = self.vm.qmp('query-block-jobs') + if res['return'][0]['status'] == 'paused': + break self.assert_qmp(res, 'return[0]/status', 'paused') res = self.vm.qmp('block-job-dismiss', id='drive0') self.assert_qmp(res, 'error/desc',
Signature · 1 line
2.21.0
PATCH v2 2/20 iotests: 129 don't check backup "busy"
In reply to Vladimir Sementsov-OgievskiyBusy is racy, job has it's "pause-points" when it's not busy. Drop this check.
--- tests/qemu-iotests/129 | 1 - 1 file changed, 1 deletion(-)
tests/qemu-iotests/129 +0 -1
@@ -66,7 +66,6 @@ class TestStopWithBlockJob(iotests.QMPTestCase): result = self.vm.qmp("stop") self.assert_qmp(result, 'return', {}) result = self.vm.qmp("query-block-jobs") - self.assert_qmp(result, 'return[0]/busy', True) self.assert_qmp(result, 'return[0]/ready', False) def test_drive_mirror(self):
Signature · 1 line
2.21.0
PATCH v2 7/20 block/block-copy: add ratelimit to block-copy
In reply to Vladimir Sementsov-OgievskiyWe are going to directly use one async block-copy operation for backup job, so we need rate limitator.
We want to maintain current backup behavior: only background copying is limited and copy-before-write operations only participate in limit calculation. Therefore we need one rate limitator for block-copy state and boolean flag for block-copy call state for actual limitation.
Note, that we can't just calculate each chunk in limitator after successful copying: it will not save us from starting a lot of async sub-requests which will exceed limit too much. Instead let's use the following scheme on sub-request creation:
1. If at the moment limit is not exceeded, create the request and account it immediately.
2. If at the moment limit is already exceeded, drop create sub-request and handle limit instead (by sleep).
With this approach we'll never exceed the limit more than by one sub-request (which pretty much matches current backup behavior).
--- include/block/block-copy.h | 8 +++++++ block/block-copy.c | 44 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)
include/block/block-copy.h +8 -0
@@ -59,6 +59,14 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,int64_t max_chunk, BlockCopyAsyncCallbackFunc cb); +/* +* Set speed limit for block-copy instance. All block-copy operations related to +* this BlockCopyState will participate in speed calculation, but only +* block_copy_async calls with @ratelimit=true will be actually limited. +*/ +void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state, +uint64_t speed); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);
block/block-copy.c +44 -0
88 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v2 10/20 job: call job_enter from job_user_pause
In reply to Vladimir Sementsov-OgievskiyIf main job coroutine called job_yield (while some background process is in progress), we should give it a chance to call job_pause_point(). It will be used in backup, when moved on async block-copy.
--- job.c | 1 + 1 file changed, 1 insertion(+)
job.c +1 -0
@@ -578,6 +578,7 @@ void job_user_pause(Job *job, Error **errp) } job->user_paused = true;job_pause (job); +job_enter (job); }bool job_user_paused (Job *job)
Signature · 1 line
2.21.0
PATCH v2 13/20 iotests: 129: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyAfter introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, backup job may finish earlier than final assertion in do_test_stop. Let's require slow backup explicitly by specifying speed parameter.
--- tests/qemu-iotests/129 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
tests/qemu-iotests/129 +1 -1
@@ -76,7 +76,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase): def test_drive_backup(self): self.do_test_stop("drive-backup", device="drive0", target=self.target_img, - sync="full") + sync="full", speed=1024) def test_block_commit(self): self.do_test_stop("block-commit", device="drive0")
Signature · 1 line
2.21.0
PATCH v2 6/20 block/block-copy: add max_chunk and max_workers parameters
In reply to Vladimir Sementsov-OgievskiyThey will be used for backup.
--- include/block/block-copy.h | 5 +++++ block/block-copy.c | 10 ++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
include/block/block-copy.h +5 -0
@@ -47,6 +47,11 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,/* * Run block-copy in a coroutine, return state pointer. If finished early * returns NULL (@cb is called anyway). +* +* @max_workers means maximum of parallel coroutines to execute sub-requests, +* must be > 0. +* +* @max_chunk means maximum length for one IO operation. Zero means unlimited. */ BlockCopyCallState *block_copy_async(BlockCopyState *s, int64_t offset, int64_t bytes,
block/block-copy.c +8 -2
@@ -34,6 +34,8 @@ typedef struct BlockCopyCallState {BlockCopyState *s;int64_t offset;int64_t bytes; +int max_workers; +int64_t max_chunk;BlockCopyAsyncCallbackFunc cb;/* State */
@@ -144,10 +146,11 @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, int64_t offset, int64_t bytes) {BlockCopyTask *task; +int64_t max_chunk =MIN_NON_ZERO (s->copy_size, call_state->max_chunk);if (!bdrv_dirty_bitmap_next_dirty_area (s->copy_bitmap, offset, offset + bytes, - s->copy_size, &offset, &bytes)) + max_chunk, &offset, &bytes)) {return NULL ; }
@@ -616,7 +619,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) bytes = end - offset;if (!aio && bytes) { - aio =aio_task_pool_new (BLOCK_COPY_MAX_WORKERS ); + aio =aio_task_pool_new (call_state->max_workers); } ret =block_copy_task_run (aio, task);
@@ -695,6 +698,7 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, .s = s, .offset = start, .bytes = bytes, + .max_workers = BLOCK_COPY_MAX_WORKERS, };int ret =block_copy_common (&call_state);
@@ -726,6 +730,8 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, .offset = offset, .bytes = bytes, .cb = cb, + .max_workers = max_workers ?: BLOCK_COPY_MAX_WORKERS, + .max_chunk = max_chunk, };qemu_coroutine_enter (co);
Signature · 1 line
2.21.0
PATCH v2 8/20 block/block-copy: add block_copy_cancel
In reply to Vladimir Sementsov-OgievskiyAdd function to cancel running async block-copy call. It will be used in backup.
--- include/block/block-copy.h | 7 +++++++ block/block-copy.c | 22 +++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-)
include/block/block-copy.h +7 -0
@@ -67,6 +67,13 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state,uint64_t speed); +/* +* Cancel running block-copy call. +* Cancel leaves block-copy state valid: dirty bits are correct and you may use +* cancel + <run block_copy with same parameters> to emulate pause/resume. +*/ +void block_copy_cancel (BlockCopyCallState *call_state); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);
block/block-copy.c +19 -3
@@ -44,6 +44,8 @@ typedef struct BlockCopyCallState {bool failed;bool finished;QemuCoSleepState *sleep_state; +bool cancelled; +Coroutine *canceller;/* OUT parameters */ bool error_is_read;
@@ -582,7 +584,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)assert (QEMU_IS_ALIGNED (offset, s->cluster_size));assert (QEMU_IS_ALIGNED (bytes, s->cluster_size)); -while (bytes &&aio_task_pool_status (aio) ==0 ) { +while (bytes &&aio_task_pool_status (aio) ==0 && !call_state->cancelled) {BlockCopyTask *task;int64_t status_bytes;
@@ -693,7 +695,7 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) do { ret = block_copy_dirty_clusters(call_state); - if (ret == 0) { + if (ret == 0 && !call_state->cancelled) { ret = block_copy_wait_one(call_state->s, call_state->offset, call_state->bytes); }
@@ -707,13 +709,18 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)* 2. We have waited for some intersecting block-copy request * It may have failed and produced new dirty bits. */ - }while (ret >0 ); + }while (ret >0 && !call_state->cancelled);if (call_state->cb) { call_state->cb (ret, call_state->error_is_read, call_state->s->progress_opaque); } +if (call_state->canceller) { +aio_co_wake (call_state->canceller); + call_state->canceller =NULL ; + } + call_state->finished = true;return ret;
@@ -772,6 +779,15 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,return call_state; } + +void block_copy_cancel (BlockCopyCallState *call_state) +{ + call_state->cancelled = true; + call_state->canceller =qemu_coroutine_self (); +block_copy_kick (call_state); +qemu_coroutine_yield (); +} +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s) {return s->copy_bitmap;
Signature · 1 line
2.21.0
PATCH v2 11/20 qapi: backup: add x-max-chunk and x-max-workers parameters
In reply to Vladimir Sementsov-OgievskiyAdd new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
--- qapi/block-core.json | 9 ++++++++- include/block/block_int.h | 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-)
qapi/block-core.json +8 -1
@@ -1407,6 +1407,12 @@ # # @x-use-copy-range: use copy offloading if possible. Default true. (Since 5.1) # +# @x-max-workers: maximum of parallel requests for static data backup. This +# doesn't influence copy-before-write operations. (Since: 5.1) +# +# @x-max-chunk: maximum chunk length for static data backup. This doesn't +# influence copy-before-write operations. (Since: 5.1) +# # Note: @on-source-error and @on-target-error only affect background # I/O. If an error occurs during a guest write request, the device's # rerror/werror actions will be used.
@@ -1421,7 +1427,8 @@ '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool', - '*filter-node-name': 'str', '*x-use-copy-range': 'bool' } } + '*filter-node-name': 'str', '*x-use-copy-range': 'bool', + '*x-max-workers': 'int', '*x-max-chunk': 'int64' } } ## # @DriveBackup:
include/block/block_int.h +7 -0
@@ -1207,6 +1207,11 @@ void mirror_start(const char *job_id, BlockDriverState *bs,* @sync_mode: What parts of the disk image should be copied to the destination. * @sync_bitmap: The dirty bitmap if sync_mode is 'bitmap' or 'incremental' * @bitmap_mode: The bitmap synchronization policy to use. +* @max_workers: The limit for parallel requests for main backup loop. +* Must be >= 1. +* @max_chunk: The limit for one IO operation length in main backup loop. +* Must be not less than job cluster size or zero. Zero means no +* specific limit. * @on_source_error: The action to take upon error reading from the source. * @on_target_error: The action to take upon error writing to the target. * @creation_flags: Flags that control the behavior of the Job lifetime.
@@ -1226,6 +1231,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,bool compress, const char *filter_node_name, bool use_copy_range, + int max_workers, + int64_t max_chunk, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,
block/backup.c +21 -0
@@ -46,6 +46,8 @@ typedef struct BackupBlockJob {uint64_t len;uint64_t bytes_read;int64_t cluster_size; +int max_workers; +int64_t max_chunk;BlockCopyState *bcs; } BackupBlockJob;
@@ -335,6 +337,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,bool compress, const char *filter_node_name, bool use_copy_range, + int max_workers, + int64_t max_chunk, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,
@@ -355,6 +359,16 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL );assert (sync_bitmap || sync_mode !=MIRROR_SYNC_MODE_BITMAP ); +if (max_workers <1 ) { +error_setg (errp,"At least one worker needed" ); +return NULL ; + } + +if (max_chunk <0 ) { +error_setg (errp,"max-chunk is negative" ); +return NULL ; + } +if (bs == target) {error_setg (errp,"Source and target cannot be the same" );return NULL ;
@@ -422,6 +436,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,if (cluster_size <0 ) { goto error; } +if (max_chunk && max_chunk < cluster_size) { +error_setg (errp,"Required max-chunk (%" PRIi64") is less than backup " +"cluster size (%" PRIi64")" , max_chunk, cluster_size); +return NULL ; + }/* * If source is in backing chain of target assume that target is going to be
@@ -465,6 +484,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->bcs = bcs; job->cluster_size = cluster_size; job->len = len; + job->max_workers = max_workers; + job->max_chunk = max_chunk;block_copy_set_progress_callback (bcs, backup_progress_bytes_callback, job);block_copy_set_progress_meter (bcs, &job->common.job.progress);
block/replication.c +1 -1
@@ -563,7 +563,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job =backup_job_create (NULL , s->secondary_disk->bs, s->hidden_disk->bs,0 ,MIRROR_SYNC_MODE_NONE ,NULL ,0 , false,NULL , - true, + true,0 ,0 ,BLOCKDEV_ON_ERROR_REPORT ,BLOCKDEV_ON_ERROR_REPORT ,JOB_INTERNAL , backup_job_completed, bs,NULL , &local_err);
blockdev.c +5 -0
@@ -2743,6 +2743,9 @@ static BlockJob *do_backup_common(BackupCommon *backup,if (!backup->has_compress) { backup->compress = false; } +if (!backup->has_x_max_workers) { + backup->x_max_workers =64 ; + }if ((backup->sync ==MIRROR_SYNC_MODE_BITMAP ) || (backup->sync ==MIRROR_SYNC_MODE_INCREMENTAL )) {
@@ -2822,6 +2825,8 @@ static BlockJob *do_backup_common(BackupCommon *backup, backup->compress, backup->filter_node_name, backup->x_use_copy_range, + backup->x_max_workers, + backup->x_max_chunk, backup->on_source_error, backup->on_target_error, job_flags, NULL, NULL, txn, errp);
Signature · 1 line
2.21.0
PATCH v2 19/20 simplebench: bench_block_job: add cmd_options argument
In reply to Vladimir Sementsov-OgievskiyAdd argument to allow additional block-job options.
--- scripts/simplebench/bench-example.py | 2 +- scripts/simplebench/bench_block_job.py | 13 ++++++++----- 2 files changed, 9 insertions(+), 6 deletions(-)
scripts/simplebench/bench-example.py +1 -1
@@ -24,7 +24,7 @@ from bench_block_job import bench_block_copy, drv_file, drv_nbd def bench_func(env, case): """ Handle one "cell" of benchmarking table. """ - return bench_block_copy(env['qemu_binary'], env['cmd'], + return bench_block_copy(env['qemu_binary'], env['cmd'], {} case['source'], case['target'])
scripts/simplebench/bench_block_job.py +8 -5
@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3 # # Benchmark block jobs #
@@ -78,16 +78,19 @@ def bench_block_job(cmd, cmd_args, qemu_args): # Bench backup or mirror -def bench_block_copy(qemu_binary, cmd, source, target): +def bench_block_copy(qemu_binary, cmd, cmd_options, source, target): """Helper to run bench_block_job() for mirror or backup""" assert cmd in ('blockdev-backup', 'blockdev-mirror') source['node-name'] = 'source' target['node-name'] = 'target' - return bench_block_job(cmd, - {'job-id': 'job0', 'device': 'source', - 'target': 'target', 'sync': 'full'}, + cmd_options['job-id'] = 'job0' + cmd_options['device'] = 'source' + cmd_options['target'] = 'target' + cmd_options['sync'] = 'full' + + return bench_block_job(cmd, cmd_options, [qemu_binary, '-blockdev', json.dumps(source), '-blockdev', json.dumps(target)])
Signature · 1 line
2.21.0
PATCH v2 15/20 iotests: 219: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyThe further change of moving backup to be a on block-copy call will make copying chunk-size and cluster-size a separate things. So, even with 64k cluster sized qcow2 image, default chunk would be 1M. Test 219 depends on specified chunk-size. Update it for explicit chunk-size for backup as for mirror.
--- tests/qemu-iotests/219 | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
tests/qemu-iotests/219 +7 -6
@@ -203,13 +203,13 @@ with iotests.FilePath('disk.img') as disk_path, \ # but related to this also automatic state transitions like job # completion), but still get pause points often enough to avoid making this # test very slow, it's important to have the right ratio between speed and - # buf_size. + # copy-chunk-size. # - # For backup, buf_size is hard-coded to the source image cluster size (64k), - # so we'll pick the same for mirror. The slice time, i.e. the granularity - # of the rate limiting is 100ms. With a speed of 256k per second, we can - # get four pause points per second. This gives us 250ms per iteration, - # which should be enough to stay deterministic. + # Chose 64k copy-chunk-size both for mirror (by buf_size) and backup (by + # x-max-chunk). The slice time, i.e. the granularity of the rate limiting + # is 100ms. With a speed of 256k per second, we can get four pause points + # per second. This gives us 250ms per iteration, which should be enough to + # stay deterministic. test_job_lifecycle(vm, 'drive-mirror', has_ready=True, job_args={ 'device': 'drive0-node',
@@ -226,6 +226,7 @@ with iotests.FilePath('disk.img') as disk_path, \ 'target': copy_path, 'sync': 'full', 'speed': 262144, + 'x-max-chunk': 65536, 'auto-finalize': auto_finalize, 'auto-dismiss': auto_dismiss, })
Signature · 1 line
2.21.0
PATCH v2 14/20 iotests: 185: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyThe further change of moving backup to be a on block-copy call will make copying chunk-size and cluster-size a separate things. So, even with 64k cluster sized qcow2 image, default chunk would be 1M. 185 test however assumes, that with speed limited to 64K, one iteration would result in offset=64K. It will change, as first iteration would result in offset=1M independently of speed.
So, let's explicitly specify, what test wants: set max-chunk to 64K, so that one iteration is 64K. Note, that we don't need to limit max-workers, as block-copy rate limitator will handle the situation and wouldn't start new workers when speed limit is obviously reached.
--- tests/qemu-iotests/185 | 3 ++- tests/qemu-iotests/185.out | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
tests/qemu-iotests/185 +2 -1
@@ -182,7 +182,8 @@ _send_qemu_cmd $h \ 'target': '$TEST_IMG.copy', 'format': '$IMGFMT', 'sync': 'full', - 'speed': 65536 } }" \ + 'speed': 65536, + 'x-max-chunk': 65536 } }" \ "return" # If we don't sleep here 'quit' command races with disk I/O
tests/qemu-iotests/185.out +1 -1
@@ -61,7 +61,7 @@ Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 size=67108864 cluster_size=65536 l { 'execute': 'qmp_capabilities' } {"return": {}} -{ 'execute': 'drive-backup', 'arguments': { 'device': 'disk', 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', 'speed': 65536 } } +{ 'execute': 'drive-backup', 'arguments': { 'device': 'disk', 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', 'speed': 65536, 'x-max-chunk': 65536 } } Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "disk"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "disk"}}
Signature · 1 line
2.21.0
PATCH v2 16/20 iotests: 257: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyIotest 257 dumps a lot of in-progress information of backup job, such as offset and bitmap dirtiness. Further commit will move backup to be one block-copy call, which will introduce async parallel requests instead of plain cluster-by-cluster copying. To keep things deterministic, allow only one worker (only one copy request at a time) for this test.
--- tests/qemu-iotests/257 | 1 + tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 2 files changed, 154 insertions(+), 153 deletions(-)
tests/qemu-iotests/257 +1 -0
@@ -191,6 +191,7 @@ def blockdev_backup(vm, device, target, sync, **kwargs): target=target, sync=sync, filter_node_name='backup-top', + x_max_workers=1, **kwargs) return result
tests/qemu-iotests/257.out +153 -153
1,043 lines not on this page · raw message
Signature · 1 line
2.21.0
PATCH v2 20/20 simplebench: add bench-backup.py
In reply to Vladimir Sementsov-OgievskiyAdd script to benchmark new backup architecture.
--- scripts/simplebench/bench-backup.py | 132 ++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100755 scripts/simplebench/bench-backup.py
scripts/simplebench/bench-backup.py new file +132 -0
@@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +# +# Bench backup block-job +# +# Copyright (c) 2020 Virtuozzo International GmbH. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import argparse +import simplebench +from bench_block_job import bench_block_copy, drv_file, drv_nbd + + +def bench_func(env, case): + """ Handle one "cell" of benchmarking table. """ + cmd_options = env['cmd-options'] if 'cmd-options' in env else {} + return bench_block_copy(env['qemu-binary'], env['cmd'], + cmd_options, + case['source'], case['target']) + + +def bench(args): + test_cases = [] + + sources = {} + targets = {} + for d in args.dir: + label, path = d.split(':') + sources[label] = drv_file(path + '/test-source') + targets[label] = drv_file(path + '/test-target') + + if args.nbd: + nbd = args.nbd.split(':') + host = nbd[0] + port = '10809' if len(nbd) == 1 else nbd[1] + drv = drv_nbd(host, port) + sources['nbd'] = drv + targets['nbd'] = drv + + for t in args.test: + src, dst = t.split(':') + + test_cases.append({ + 'id': t, + 'source': sources[src], + 'target': targets[dst] + }) + + binaries = [] + upstream = None + for i, q in enumerate(args.qemu): + name_path = q.split(':') + if len(name_path) == 1: + binaries.append((f'q{i}', name_path[0])) + else: + binaries.append((name_path[0], name_path[1])) + if name_path[0] == 'upstream' or name_path[0] == 'master': + upstream = binaries[-1] + + test_envs = [] + if upstream: + label, path = upstream + test_envs.append({ + 'id': f'mirror({label})', + 'cmd': 'blockdev-mirror', + 'qemu-binary': path + }) + + for label, path in binaries: + test_envs.append({ + 'id': f'backup({label})', + 'cmd': 'blockdev-backup', + 'qemu-binary': path + }) + test_envs.append({ + 'id': f'backup({label}, no-copy-range)', + 'cmd': 'blockdev-backup', + 'cmd-options': {'x-use-copy-range': False}, + 'qemu-binary': path + }) + if label == 'new': + test_envs.append({ + 'id': f'backup({label}, copy-range-1w)', + 'cmd': 'blockdev-backup', + 'cmd-options': {'x-use-copy-range': True, + 'x-max-workers': 1}, + 'qemu-binary': path + }) + + result = simplebench.bench(bench_func, test_envs, test_cases, count=3) + print(simplebench.ascii(result)) + + +class ExtendAction(argparse.Action): + def __call__(self, parser, namespace, values, option_string=None): + items = getattr(namespace, self.dest) or [] + items.extend(values) + setattr(namespace, self.dest, items) + + +if __name__ == '__main__': + p = argparse.ArgumentParser('Backup benchmark') + p.add_argument('--qemu', nargs='+', help='Qemu binaries to compare, just ' + 'file path with label, like label:/path/to/qemu. Qemu ' + 'labeled "new" should support x-max-workers argument for ' + 'backup job, labeled "upstream" will be used also to run ' + 'mirror benchmark for comparison.', + action=ExtendAction) + p.add_argument('--dir', nargs='+', help='Directories, each containing ' + '"test-source" and/or "test-target" files, raw images to ' + 'used in benchmarking. File path with label, like ' + 'label:/path/to/directory', action=ExtendAction) + p.add_argument('--nbd', help='host:port for remote NBD ' + 'image, (or just host, for default port 10809). Use it in ' + 'tests, label is "nbd" (but you cannot create test ' + 'nbd:nbd).') + p.add_argument('--test', nargs='+', help='Tests, in form ' + 'source-dir-label:target-dir-label', action=ExtendAction) + + bench(p.parse_args())
Signature · 1 line
2.21.0
PATCH v2 17/20 backup: move to block-copy
In reply to Vladimir Sementsov-OgievskiyThis brings async request handling and block-status driven chunk sizes to backup out of the box, which improves backup performance.
--- include/block/block-copy.h | 9 +-- block/backup.c | 145 +++++++++++++++++++------------------ block/block-copy.c | 21 ++---- 3 files changed, 83 insertions(+), 92 deletions(-)
include/block/block-copy.h +2 -7
@@ -18,7 +18,6 @@#include "block/block.h" #include "qemu/co-shared-resource.h" -typedef void (*ProgressBytesCallbackFunc )(int64_t bytes,void *opaque);typedef void (*BlockCopyAsyncCallbackFunc )(int ret,bool error_is_read,void *opaque);typedef struct BlockCopyState BlockCopyState ;
@@ -29,11 +28,6 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,BdrvRequestFlags write_flags, Error **errp); -void block_copy_set_progress_callback ( -BlockCopyState *s, -ProgressBytesCallbackFunc progress_bytes_callback, -void *progress_opaque); -void block_copy_set_progress_meter (BlockCopyState *s,ProgressMeter *pm);void block_copy_state_free (BlockCopyState *s);
@@ -57,7 +51,8 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, int64_t offset, int64_t bytes, bool ratelimit, int max_workers, int64_t max_chunk, - BlockCopyAsyncCallbackFunc cb); + BlockCopyAsyncCallbackFunc cb, + void *cb_opaque);/* * Set speed limit for block-copy instance. All block-copy operations related to
block/backup.c +76 -69
202 lines not on this page · raw message
block/block-copy.c +5 -16
@@ -39,6 +39,7 @@ typedef struct BlockCopyCallState {int64_t max_chunk;bool ratelimit;BlockCopyAsyncCallbackFunc cb; +void *cb_opaque;/* State */ bool failed;
@@ -103,9 +104,6 @@ typedef struct BlockCopyState {bool skip_unallocated;ProgressMeter *progress; -/* progress_bytes_callback: called when some copying progress is done. */ -ProgressBytesCallbackFunc progress_bytes_callback; -void *progress_opaque;SharedResource *mem;
@@ -287,15 +285,6 @@ BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target,return s; } -void block_copy_set_progress_callback ( -BlockCopyState *s, -ProgressBytesCallbackFunc progress_bytes_callback, -void *progress_opaque) -{ - s->progress_bytes_callback = progress_bytes_callback; - s->progress_opaque = progress_opaque; -} -void block_copy_set_progress_meter (BlockCopyState *s,ProgressMeter *pm) { s->progress = pm;
@@ -443,7 +432,6 @@ static coroutine_fn int block_copy_task_entry(AioTask *task) t->call_state->error_is_read = error_is_read; }else {progress_work_done (t->s->progress, t->bytes); - t->s->progress_bytes_callback (t->bytes, t->s->progress_opaque); }co_put_to_shres (t->s->mem, t->bytes);block_copy_task_end (t, ret);
@@ -712,8 +700,7 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) }while (ret >0 && !call_state->cancelled);if (call_state->cb) { - call_state->cb (ret, call_state->error_is_read, - call_state->s->progress_opaque); + call_state->cb (ret, call_state->error_is_read, call_state->cb_opaque); }if (call_state->canceller) {
@@ -754,7 +741,8 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, int64_t offset, int64_t bytes, bool ratelimit, int max_workers, int64_t max_chunk, - BlockCopyAsyncCallbackFunc cb) + BlockCopyAsyncCallbackFunc cb, + void *cb_opaque) { BlockCopyCallState *call_state = g_new(BlockCopyCallState, 1); Coroutine *co = qemu_coroutine_create(block_copy_async_co_entry,
@@ -766,6 +754,7 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, .bytes = bytes, .ratelimit = ratelimit, .cb = cb, + .cb_opaque = cb_opaque, .max_workers = max_workers ?: BLOCK_COPY_MAX_WORKERS, .max_chunk = max_chunk, };
Signature · 1 line
2.21.0
Quoted 12 lines · Vladimir Sementsov-Ogievskiy
Hi all!
This a last part of original
"[RFC 00/24] backup performance:block_status+ async", prepartions are already merged.The series turn backup into series of
block_copy_asynccalls, covering the whole disk, so we get block-status based paralallel async requests out of the box, which gives performance gain:----------------- ---------------- ------------- -------------------------- -------------------------- ---------------- -------------------------------
mirror(upstream) backup(new)backup(new, no-copy-range)backup(new, copy-range-1w)backup(upstream)backup(upstream, no-copy-range)
Quoted 11 lines
hdd-ext4:hdd-ext4 18.86 +- 0.11 45.50 +- 2.35 19.22 +- 0.09 19.51 +- 0.09 22.85 +- 5.98 19.72 +- 0.35 hdd-ext4:ssd-ext4 8.99 +- 0.02 9.30 +- 0.01 8.97 +- 0.02 9.02 +- 0.02 9.68 +- 0.26 9.84 +- 0.12 ssd-ext4:hdd-ext4 9.09 +- 0.11 9.34 +- 0.10 9.34 +- 0.10 8.99 +- 0.01 11.37 +- 0.37 11.47 +- 0.30 ssd-ext4:ssd-ext4 4.07 +- 0.02 5.41 +- 0.05 4.05 +- 0.01 8.35 +- 0.58 9.83 +- 0.64 8.62 +- 0.35 hdd-xfs:hdd-xfs 18.90 +- 0.19 43.26 +- 2.47 19.62 +- 0.14 19.38 +- 0.16 19.55 +- 0.26 19.62 +- 0.12 hdd-xfs:ssd-xfs 8.93 +- 0.12 9.35 +- 0.03 8.93 +- 0.08 8.93 +- 0.05 9.79 +- 0.30 9.55 +- 0.15 ssd-xfs:hdd-xfs 9.15 +- 0.07 9.74 +- 0.28 9.29 +- 0.03 9.08 +- 0.05 10.85 +- 0.31 10.91 +- 0.30 ssd-xfs:ssd-xfs 4.06 +- 0.01 4.93 +- 0.02 4.04 +- 0.01 8.17 +- 0.42 9.52 +- 0.49 8.85 +- 0.46 ssd-ext4:nbd 9.96 +- 0.11 11.45 +- 0.15 11.45 +- 0.02 17.22 +- 0.06 34.45 +- 1.35 35.16 +- 0.37 nbd:ssd-ext4 9.84 +- 0.02 9.84 +- 0.04 9.80 +- 0.06 18.96 +- 0.06 30.89 +- 0.73 31.46 +- 0.21 ----------------- ---------------- ------------- -------------------------- -------------------------- ---------------- -------------------------------
I should add, that nbd results may be damaged by the fact that node with nbd server is my desktop, which was used for another tasks in parallel. Still I don't think it really hurt.
Quoted 90 lines
The table shows, that
copy_rangeis in bad relation with parallel async requests.copy_rangebrings real performance gain only on supporting fs, like btrfs. But even on such fs, I'm not sure that this is a good default behavior: if we do offload copy, so, that no real copy but just link block in backup the same blocks as in original, this means that further write from guest will lead to fragmentation of guest disk, when the aim of backup is to operate transparently for the guest.So, in addition to these series I also suggest to disable
copy_rangeby default.===
How to test:
prepare images:
In a directories, where you want to place source and target images, prepare images by:for img in test-source test-target; do
./qemu-img create -f raw $img 1000M;
./qemu-img bench -c 1000 -d 1 -f raw -s 1M -w --pattern=0xff $img
doneprepare similar image for nbd server, and start it somewhere by
qemu-nbd --persistent --nocache -f raw IMAGEThen, run benchmark, like this:
./bench-backup.py --qemu new:../../x86_64-softmmu/qemu-system-x86_64 upstream:/work/src/qemu/up-backup-block-copy-master/x86_64-softmmu/qemu-system-x86_64 --dir hdd-ext4:/test-a hdd-xfs:/test-b ssd-ext4:/ssd ssd-xfs:/ssd-b --test $(for fs in ext4 xfs; do echo hdd-$fs:hdd-$fs hdd-$fs:ssd-$fs ssd-$fs:hdd-$fs ssd-$fs:ssd-$fs; done) --nbd 192.168.100.2 --test ssd-ext4:nbd nbd:ssd-ext4(you may simply reduce number of directories/test-cases, use --help for
help)===
Note, that I included here
"[PATCH]block/block-copy:block_copy_dirty_clusters: fix failure check" which was previously sent in separate, but still untouched in mailing list. It still may be applied separately.Vladimir Sementsov-Ogievskiy (20):
block/block-copy:block_copy_dirty_clusters: fix failure check
iotests: 129 don't check backup "busy"
qapi: backup: add x-use-copy-range parameterblock/block-copy: More explicitcall_stateblock/block-copy: implementblock_copy_asyncblock/block-copy: addmax_chunkandmax_workersparametersblock/block-copy: add ratelimit to block-copyblock/block-copy: addblock_copy_cancel
blockjob: addset_speedto BlockJobDriver
job: calljob_enterfromjob_user_pause
qapi: backup: add x-max-chunk and x-max-workers parameters
iotests: 56: prepare for backup over block-copy
iotests: 129: prepare for backup over block-copy
iotests: 185: prepare for backup over block-copy
iotests: 219: prepare for backup over block-copy
iotests: 257: prepare for backup over block-copy
backup: move to block-copy
block/block-copy: drop unused argument ofblock_copy()
simplebench:bench_block_job: addcmd_optionsargument
simplebench: addbench-backup.pyqapi/block-core.json | 11 +- block/backup-top.h | 1 + include/block/block-copy.h | 45 +++- include/block/block_int.h | 8 + include/block/blockjob_int.h | 2 + block/backup-top.c | 6 +- block/backup.c | 170 ++++++++------ block/block-copy.c | 183 ++++++++++++--- block/replication.c | 1 + blockdev.c | 10 + blockjob.c | 6 + job.c | 1 + scripts/simplebench/bench-backup.py | 132 +++++++++++ scripts/simplebench/bench-example.py | 2 +- scripts/simplebench/bench_block_job.py | 13 +- tests/qemu-iotests/056 | 8 +- tests/qemu-iotests/129 | 3 +- tests/qemu-iotests/185 | 3 +- tests/qemu-iotests/185.out | 2 +- tests/qemu-iotests/219 | 13 +- tests/qemu-iotests/257 | 1 + tests/qemu-iotests/257.out | 306 ++++++++++++------------- 22 files changed, 640 insertions(+), 287 deletions(-)create mode 100755
scripts/simplebench/bench-backup.py
Signature · 2 lines
Best regards, Vladimir
PATCH v2 18/20 block/block-copy: drop unused argument of block_copy()
In reply to Vladimir Sementsov-Ogievskiy--- include/block/block-copy.h | 3 +-- block/backup-top.c | 2 +- block/block-copy.c | 11 ++--------- 3 files changed, 4 insertions(+), 12 deletions(-)
include/block/block-copy.h +1 -2
@@ -35,8 +35,7 @@ void block_copy_state_free(BlockCopyState *s);int64_t block_copy_reset_unallocated (BlockCopyState *s,int64_t offset,int64_t *count); -int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, -bool *error_is_read); +int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes);/* * Run block-copy in a coroutine, return state pointer. If finished early
block/backup-top.c +1 -1
@@ -61,7 +61,7 @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset, off =QEMU_ALIGN_DOWN (offset, s->cluster_size); end =QEMU_ALIGN_UP (offset + bytes, s->cluster_size); -return block_copy (s->bcs, off, end - off,NULL ); +return block_copy (s->bcs, off, end - off); } static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs,
block/block-copy.c +2 -9
@@ -713,8 +713,7 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)return ret; } -int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, -bool *error_is_read) +int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes) {BlockCopyCallState call_state = { .s = s,
@@ -723,13 +722,7 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, .max_workers = BLOCK_COPY_MAX_WORKERS, }; -int ret =block_copy_common (&call_state); - -if (error_is_read && ret <0 ) { - *error_is_read = call_state.error_is_read; - } - -return ret; +return block_copy_common (&call_state); } static void coroutine_fn block_copy_async_co_entry(void *opaque)
Signature · 1 line
2.21.0
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
TEST check-unit: tests/test-logging
TEST check-unit: tests/test-replication
**
ERROR:/tmp/qemu-test/src/tests/test-replication.c:428:test_secondary_start: assertion failed: (!local_err)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/test-replication.c:428:test_secondary_start: assertion failed: (!local_err)
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
TEST check-qtest-x86_64: tests/qtest/boot-order-test
TEST check-qtest-x86_64: tests/qtest/bios-tables-test
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=73ef29198bda41a1bce9aa3697266e4c', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-j0ydkht8/src/docker-src.2020-06-01-14.44.54.8388:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=73ef29198bda41a1bce9aa3697266e4c
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-j0ydkht8/src'
make: *** [docker-run-test-quick@centos7] Error 2
real 14m11.116s
user 0m9.306s
The full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Patchew URL: https://patchew.org/QEMU/[email protected]/
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the asan build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1 === TEST SCRIPT END ===
PASS 1 fdc-test /x86_64/fdc/cmos
PASS 2 fdc-test /x86_64/fdc/no_media_on_start
PASS 3 fdc-test /x86_64/fdc/read_without_media
==8167==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 fdc-test /x86_64/fdc/media_change
PASS 5 fdc-test /x86_64/fdc/sense_interrupt
PASS 6 fdc-test /x86_64/fdc/relative_seek
---
PASS 32 test-opts-visitor /visitor/opts/range/beyond
PASS 33 test-opts-visitor /visitor/opts/dict/unvisited
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-coroutine -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-coroutine"
==8242==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-coroutine /basic/no-dangling-access
PASS 2 test-coroutine /basic/lifecycle
PASS 3 test-coroutine /basic/yield
==8242==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc47dee000; bottom 0x7f11b97d2000; size: 0x00ea8e61c000 (1007411118080)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 4 test-coroutine /basic/nesting
---
PASS 12 test-aio /aio/event/flush
PASS 13 test-aio /aio/event/wait/no-flush-cb
PASS 14 test-aio /aio/timer/schedule
==8257==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 test-aio /aio/coroutine/queue-chaining
PASS 16 test-aio /aio-gsource/flush
PASS 17 test-aio /aio-gsource/bh/schedule
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/ide-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="ide-test" PASS 28 test-aio /aio-gsource/timer/schedule
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-aio-multithread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-aio-multithread"
==8268==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-aio-multithread /aio/multi/lifecycle
==8265==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 ide-test /x86_64/ide/identify
PASS 2 test-aio-multithread /aio/multi/schedule
==8285==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 ide-test /x86_64/ide/flush
PASS 3 test-aio-multithread /aio/multi/mutex/contended
==8296==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 ide-test /x86_64/ide/bmdma/simple_rw
==8307==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 ide-test /x86_64/ide/bmdma/trim
==8313==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 test-aio-multithread /aio/multi/mutex/handoff
PASS 5 test-aio-multithread /aio/multi/mutex/mcs
PASS 6 test-aio-multithread /aio/multi/mutex/pthread
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-throttle -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-throttle"
==8330==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-throttle /throttle/leak_bucket
PASS 2 test-throttle /throttle/compute_wait
PASS 3 test-throttle /throttle/init
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-thread-pool -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-thread-pool"
PASS 1 test-thread-pool /thread-pool/submit
PASS 2 test-thread-pool /thread-pool/submit-aio
==8334==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 test-thread-pool /thread-pool/submit-co
PASS 4 test-thread-pool /thread-pool/submit-many
PASS 5 test-thread-pool /thread-pool/cancel
==8401==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 test-thread-pool /thread-pool/cancel-async
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-hbitmap -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-hbitmap"
PASS 1 test-hbitmap /hbitmap/granularity
---
PASS 39 test-hbitmap /hbitmap/next_dirty_area/next_dirty_area_4
PASS 40 test-hbitmap /hbitmap/next_dirty_area/next_dirty_area_after_truncate
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-bdrv-drain -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bdrv-drain"
==8412==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bdrv-drain /bdrv-drain/nested
PASS 2 test-bdrv-drain /bdrv-drain/multiparent
PASS 3 test-bdrv-drain /bdrv-drain/set_aio_context
---
PASS 41 test-bdrv-drain /bdrv-drain/bdrv_drop_intermediate/poll
PASS 42 test-bdrv-drain /bdrv-drain/replace_child/mid-drain
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-bdrv-graph-mod -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-bdrv-graph-mod"
==8451==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-bdrv-graph-mod /bdrv-graph-mod/update-perm-tree
PASS 2 test-bdrv-graph-mod /bdrv-graph-mod/should-update-child
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-blockjob -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-blockjob"
==8455==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-blockjob /blockjob/ids
PASS 2 test-blockjob /blockjob/cancel/created
PASS 3 test-blockjob /blockjob/cancel/running
---
PASS 7 test-blockjob /blockjob/cancel/pending
PASS 8 test-blockjob /blockjob/cancel/concluded
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-blockjob-txn -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-blockjob-txn"
==8459==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-blockjob-txn /single/success
PASS 2 test-blockjob-txn /single/failure
PASS 3 test-blockjob-txn /single/cancel
---
PASS 6 test-blockjob-txn /pair/cancel
PASS 7 test-blockjob-txn /pair/fail-cancel-race
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-block-backend -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-block-backend"
==8465==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-block-backend /block-backend/drain_aio_error
PASS 2 test-block-backend /block-backend/drain_all_aio_error
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-block-iothread -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-block-iothread"
==8469==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-block-iothread /sync-op/pread
PASS 2 test-block-iothread /sync-op/pwrite
PASS 3 test-block-iothread /sync-op/load_vmstate
---
PASS 15 test-block-iothread /propagate/diamond
PASS 16 test-block-iothread /propagate/mirror
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-image-locking -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-image-locking"
==8461==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==8492==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-image-locking /image-locking/basic
PASS 2 test-image-locking /image-locking/set-perm-abort
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-x86-cpuid -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-x86-cpuid"
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-rcu-list -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-list"
PASS 1 test-rcu-list /rcu/qlist/single-threaded
PASS 2 test-rcu-list /rcu/qlist/short-few
==8578==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 test-rcu-list /rcu/qlist/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-rcu-simpleq -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-simpleq"
PASS 1 test-rcu-simpleq /rcu/qsimpleq/single-threaded
---
PASS 3 test-rcu-simpleq /rcu/qsimpleq/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-rcu-tailq -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-tailq"
PASS 1 test-rcu-tailq /rcu/qtailq/single-threaded
==8623==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-rcu-tailq /rcu/qtailq/short-few
PASS 3 test-rcu-tailq /rcu/qtailq/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-rcu-slist -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-rcu-slist"
PASS 1 test-rcu-slist /rcu/qslist/single-threaded
PASS 2 test-rcu-slist /rcu/qslist/short-few
==8689==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 test-rcu-slist /rcu/qslist/long-many
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-qdist -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qdist"
PASS 1 test-qdist /qdist/none
---
PASS 7 test-qdist /qdist/binning/expand
PASS 8 test-qdist /qdist/binning/shrink
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-qht -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qht"
==8702==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==8708==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-qht /qht/mode/default
PASS 2 test-qht /qht/mode/resize
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-qht-par -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qht-par"
---
PASS 15 test-crypto-secret /crypto/secret/crypt/missingiv
PASS 16 test-crypto-secret /crypto/secret/crypt/badiv
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-crypto-tlscredsx509 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-tlscredsx509"
==8781==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/perfectserver
PASS 2 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/perfectclient
PASS 3 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/goodca1
---
PASS 33 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/inactive2
PASS 34 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/inactive3
PASS 5 ide-test /x86_64/ide/bmdma/various_prdts
==8795==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==8795==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffe8f24000; bottom 0x7fc54fbfe000; size: 0x003a99326000 (251678318592)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 6 ide-test /x86_64/ide/bmdma/no_busmaster
---
PASS 39 test-crypto-tlscredsx509 /qcrypto/tlscredsx509/missingclient
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-crypto-tlssession -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-tlssession"
PASS 7 ide-test /x86_64/ide/flush/nodev
==8810==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 ide-test /x86_64/ide/flush/empty_drive
PASS 1 test-crypto-tlssession /qcrypto/tlssession/psk
==8815==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 test-crypto-tlssession /qcrypto/tlssession/basicca
PASS 3 test-crypto-tlssession /qcrypto/tlssession/differentca
PASS 9 ide-test /x86_64/ide/flush/retry_pci
PASS 4 test-crypto-tlssession /qcrypto/tlssession/altname1
==8821==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 ide-test /x86_64/ide/flush/retry_isa
PASS 5 test-crypto-tlssession /qcrypto/tlssession/altname2
==8827==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 test-crypto-tlssession /qcrypto/tlssession/altname3
PASS 11 ide-test /x86_64/ide/cdrom/pio
PASS 7 test-crypto-tlssession /qcrypto/tlssession/altname4
PASS 8 test-crypto-tlssession /qcrypto/tlssession/altname5
==8833==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 test-crypto-tlssession /qcrypto/tlssession/altname6
PASS 10 test-crypto-tlssession /qcrypto/tlssession/wildcard1
PASS 11 test-crypto-tlssession /qcrypto/tlssession/wildcard2
PASS 12 ide-test /x86_64/ide/cdrom/pio_large
PASS 12 test-crypto-tlssession /qcrypto/tlssession/wildcard3
==8839==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 ide-test /x86_64/ide/cdrom/dma
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/ahci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="ahci-test" PASS 13 test-crypto-tlssession /qcrypto/tlssession/wildcard4
PASS 14 test-crypto-tlssession /qcrypto/tlssession/wildcard5
==8853==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 ahci-test /x86_64/ahci/sanity
==8859==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 ahci-test /x86_64/ahci/pci_spec
==8865==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 ahci-test /x86_64/ahci/pci_enable
PASS 15 test-crypto-tlssession /qcrypto/tlssession/wildcard6
==8871==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 16 test-crypto-tlssession /qcrypto/tlssession/cachain
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-qga -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-qga"
PASS 4 ahci-test /x86_64/ahci/hba_spec
==8885==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-qga /qga/sync-delimited
PASS 2 test-qga /qga/sync
PASS 3 test-qga /qga/ping
---
PASS 16 test-qga /qga/invalid-args
PASS 17 test-qga /qga/fsfreeze-status
PASS 5 ahci-test /x86_64/ahci/hba_enable
==8894==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 ahci-test /x86_64/ahci/identify
PASS 18 test-qga /qga/blacklist
PASS 19 test-qga /qga/config
PASS 20 test-qga /qga/guest-exec
PASS 21 test-qga /qga/guest-exec-invalid
==8900==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 ahci-test /x86_64/ahci/max
==8918==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 22 test-qga /qga/guest-get-osinfo
PASS 23 test-qga /qga/guest-get-host-name
PASS 24 test-qga /qga/guest-get-timezone
---
PASS 1 test-util-sockets /util/socket/is-socket/bad
PASS 2 test-util-sockets /util/socket/is-socket/good
PASS 8 ahci-test /x86_64/ahci/reset
==8939==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==8939==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffcad268000; bottom 0x7fae285fe000; size: 0x004e84c6a000 (337235058688)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 9 ahci-test /x86_64/ahci/io/pio/lba28/simple/zero
==8947==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==8947==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd27324000; bottom 0x7f631d1fe000; size: 0x009a0a126000 (661593939968)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 3 test-util-sockets /util/socket/unix-abstract/good
---
PASS 10 ahci-test /x86_64/ahci/io/pio/lba28/simple/low
PASS 1 test-authz-simple /authz/simple
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-authz-list -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-authz-list"
==8958==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-authz-list /auth/list/complex
PASS 2 test-authz-list /auth/list/add-remove
PASS 3 test-authz-list /auth/list/default/deny
---
PASS 4 test-authz-listfile /auth/list/explicit/deny
PASS 5 test-authz-listfile /auth/list/explicit/allow
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-io-task -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-task"
==8958==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe1bf2e000; bottom 0x7f6d085fe000; size: 0x009113930000 (623098658816)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-io-task /crypto/task/complete
---
PASS 4 test-io-channel-file /io/channel/pipe/sync
PASS 5 test-io-channel-file /io/channel/pipe/async
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-io-channel-tls -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-channel-tls"
==9010==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9010==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc27a06000; bottom 0x7f1d9f5fe000; size: 0x00de88408000 (955768668160)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 12 ahci-test /x86_64/ahci/io/pio/lba28/double/zero
==9036==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9036==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff5fc1a000; bottom 0x7fe5227fe000; size: 0x001a3d41c000 (112696868864)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 13 ahci-test /x86_64/ahci/io/pio/lba28/double/low
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-io-channel-buffer -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-io-channel-buffer"
PASS 1 test-io-channel-buffer /io/channel/buf
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-base64 -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-base64"
==9042==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-base64 /util/base64/good
PASS 2 test-base64 /util/base64/embedded-nul
PASS 3 test-base64 /util/base64/not-nul-terminated
PASS 4 test-base64 /util/base64/invalid-chars
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-crypto-pbkdf -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-pbkdf"
==9042==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd21508000; bottom 0x7fbf6c7fe000; size: 0x003db4d0a000 (265026576384)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-crypto-pbkdf /crypto/pbkdf/rfc3962/sha1/iter1
---
PASS 3 test-crypto-afsplit /crypto/afsplit/sha256/big
PASS 4 test-crypto-afsplit /crypto/afsplit/sha1/1000
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-crypto-xts -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-xts"
==9067==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 test-crypto-xts /crypto/xts/t-1-key-32-ptx-32/basic
PASS 2 test-crypto-xts /crypto/xts/t-1-key-32-ptx-32/split
PASS 3 test-crypto-xts /crypto/xts/t-1-key-32-ptx-32/unaligned
---
PASS 17 test-crypto-xts /crypto/xts/t-21-key-32-ptx-31/basic
PASS 18 test-crypto-xts /crypto/xts/t-21-key-32-ptx-31/unaligned
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-crypto-block -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-crypto-block"
==9067==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffdbe6bb000; bottom 0x7f20b697c000; size: 0x00dd07d3f000 (949319102464)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 1 test-crypto-block /crypto/block/qcow
---
PASS 3 test-logging /logging/logfile_write_path
PASS 4 test-logging /logging/logfile_lock_path
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} tests/test-replication -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-replication"
==9094==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 ahci-test /x86_64/ahci/io/pio/lba28/long/zero
PASS 1 test-replication /replication/primary/read
PASS 2 test-replication /replication/primary/write
==9098==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9098==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd31d90000; bottom 0x7face31fe000; size: 0x00504eb92000 (344918138880)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 3 test-replication /replication/primary/start
---
PASS 5 test-replication /replication/primary/do_checkpoint
PASS 6 test-replication /replication/primary/get_error_all
PASS 16 ahci-test /x86_64/ahci/io/pio/lba28/long/low
==9104==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9104==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe8ecdd000; bottom 0x7f499e37c000; size: 0x00b4f0961000 (777130479616)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 7 test-replication /replication/secondary/read
PASS 17 ahci-test /x86_64/ahci/io/pio/lba28/long/high
==9110==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 test-replication /replication/secondary/write
PASS 18 ahci-test /x86_64/ahci/io/pio/lba28/short/zero
**
ERROR:/tmp/qemu-test/src/tests/test-replication.c:428:test_secondary_start: assertion failed: (!local_err)
ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/test-replication.c:428:test_secondary_start: assertion failed: (!local_err)
==9116==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
make: *** [/tmp/qemu-test/src/tests/Makefile.include:642: check-unit] Error 1
make: *** Waiting for unfinished jobs....
PASS 19 ahci-test /x86_64/ahci/io/pio/lba28/short/low
==9122==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 20 ahci-test /x86_64/ahci/io/pio/lba28/short/high
==9128==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9128==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc481e0000; bottom 0x7f7db47fe000; size: 0x007e939e2000 (543642492928)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 21 ahci-test /x86_64/ahci/io/pio/lba48/simple/zero
==9134==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9134==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc5fea2000; bottom 0x7fa30d3fe000; size: 0x005952aa4000 (383638978560)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 22 ahci-test /x86_64/ahci/io/pio/lba48/simple/low
==9140==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9140==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffcc8774000; bottom 0x7fd05effe000; size: 0x002c69776000 (190747992064)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 23 ahci-test /x86_64/ahci/io/pio/lba48/simple/high
==9146==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9146==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffeb4ff000; bottom 0x7fdeabdfe000; size: 0x00213f701000 (142798229504)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 24 ahci-test /x86_64/ahci/io/pio/lba48/double/zero
==9152==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9152==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffee8121000; bottom 0x7f783bbfe000; size: 0x0086ac523000 (578416685056)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 25 ahci-test /x86_64/ahci/io/pio/lba48/double/low
==9158==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9158==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd281e2000; bottom 0x7f9e0f1fe000; size: 0x005f18fe4000 (408441208832)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 26 ahci-test /x86_64/ahci/io/pio/lba48/double/high
==9164==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9164==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe211b8000; bottom 0x7f52125fe000; size: 0x00ac0ebba000 (738981552128)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 27 ahci-test /x86_64/ahci/io/pio/lba48/long/zero
==9170==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9170==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffd10c20000; bottom 0x7fd20c9fe000; size: 0x002b04222000 (184752939008)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 28 ahci-test /x86_64/ahci/io/pio/lba48/long/low
==9176==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9176==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc60c29000; bottom 0x7f1fbb3fe000; size: 0x00dca582b000 (947669610496)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 29 ahci-test /x86_64/ahci/io/pio/lba48/long/high
==9182==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 30 ahci-test /x86_64/ahci/io/pio/lba48/short/zero
==9188==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 31 ahci-test /x86_64/ahci/io/pio/lba48/short/low
==9194==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 32 ahci-test /x86_64/ahci/io/pio/lba48/short/high
==9200==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 33 ahci-test /x86_64/ahci/io/dma/lba28/fragmented
==9206==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 34 ahci-test /x86_64/ahci/io/dma/lba28/retry
==9212==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 35 ahci-test /x86_64/ahci/io/dma/lba28/simple/zero
==9218==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 36 ahci-test /x86_64/ahci/io/dma/lba28/simple/low
==9224==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 37 ahci-test /x86_64/ahci/io/dma/lba28/simple/high
==9230==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 38 ahci-test /x86_64/ahci/io/dma/lba28/double/zero
==9236==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 39 ahci-test /x86_64/ahci/io/dma/lba28/double/low
==9242==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 40 ahci-test /x86_64/ahci/io/dma/lba28/double/high
==9248==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9248==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fffc74a2000; bottom 0x7f3efd1fd000; size: 0x00c0ca2a5000 (828025491456)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 41 ahci-test /x86_64/ahci/io/dma/lba28/long/zero
==9255==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9255==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe05b4a000; bottom 0x7fa43bd23000; size: 0x0059c9e27000 (385639149568)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 42 ahci-test /x86_64/ahci/io/dma/lba28/long/low
==9262==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9262==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffe2266e000; bottom 0x7f81717fd000; size: 0x007cb0e71000 (535543877632)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 43 ahci-test /x86_64/ahci/io/dma/lba28/long/high
==9269==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 44 ahci-test /x86_64/ahci/io/dma/lba28/short/zero
==9275==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 45 ahci-test /x86_64/ahci/io/dma/lba28/short/low
==9281==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 46 ahci-test /x86_64/ahci/io/dma/lba28/short/high
==9287==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 47 ahci-test /x86_64/ahci/io/dma/lba48/simple/zero
==9293==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 48 ahci-test /x86_64/ahci/io/dma/lba48/simple/low
==9299==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 49 ahci-test /x86_64/ahci/io/dma/lba48/simple/high
==9305==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 50 ahci-test /x86_64/ahci/io/dma/lba48/double/zero
==9311==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 51 ahci-test /x86_64/ahci/io/dma/lba48/double/low
==9317==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 52 ahci-test /x86_64/ahci/io/dma/lba48/double/high
==9323==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9323==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffdf79c8000; bottom 0x7fa67edfd000; size: 0x005778bcb000 (375687786496)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 53 ahci-test /x86_64/ahci/io/dma/lba48/long/zero
==9330==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9330==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffdfe2ef000; bottom 0x7fa09e5fd000; size: 0x005d5fcf2000 (401039368192)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 54 ahci-test /x86_64/ahci/io/dma/lba48/long/low
==9337==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9337==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7ffc176af000; bottom 0x7f7dee5fd000; size: 0x007e290b2000 (541854474240)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 55 ahci-test /x86_64/ahci/io/dma/lba48/long/high
==9344==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 56 ahci-test /x86_64/ahci/io/dma/lba48/short/zero
==9350==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 57 ahci-test /x86_64/ahci/io/dma/lba48/short/low
==9356==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 58 ahci-test /x86_64/ahci/io/dma/lba48/short/high
==9362==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 59 ahci-test /x86_64/ahci/io/ncq/simple
==9368==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 60 ahci-test /x86_64/ahci/io/ncq/retry
==9374==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 61 ahci-test /x86_64/ahci/flush/simple
==9380==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 62 ahci-test /x86_64/ahci/flush/retry
==9386==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9392==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 63 ahci-test /x86_64/ahci/flush/migrate
==9400==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9406==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 64 ahci-test /x86_64/ahci/migrate/sanity
==9414==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9420==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 65 ahci-test /x86_64/ahci/migrate/dma/simple
==9428==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9434==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 66 ahci-test /x86_64/ahci/migrate/dma/halted
==9442==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9448==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 67 ahci-test /x86_64/ahci/migrate/ncq/simple
==9456==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9462==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 68 ahci-test /x86_64/ahci/migrate/ncq/halted
==9470==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 69 ahci-test /x86_64/ahci/cdrom/eject
==9475==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 70 ahci-test /x86_64/ahci/cdrom/dma/single
==9481==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 71 ahci-test /x86_64/ahci/cdrom/dma/multi
==9487==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 72 ahci-test /x86_64/ahci/cdrom/pio/single
==9493==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9493==WARNING: ASan is ignoring requested __asan_handle_no_return: stack top: 0x7fff53807000; bottom 0x7f4b91154000; size: 0x00b3c26b3000 (772060950528)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
PASS 73 ahci-test /x86_64/ahci/cdrom/pio/multi
==9499==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 74 ahci-test /x86_64/ahci/cdrom/pio/bcl
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/hd-geo-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="hd-geo-test" PASS 1 hd-geo-test /x86_64/hd-geo/ide/none
==9513==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 hd-geo-test /x86_64/hd-geo/ide/drive/cd_0
==9519==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/blank
==9525==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/lba
==9531==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 5 hd-geo-test /x86_64/hd-geo/ide/drive/mbr/chs
==9537==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 6 hd-geo-test /x86_64/hd-geo/ide/device/mbr/blank
==9543==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 7 hd-geo-test /x86_64/hd-geo/ide/device/mbr/lba
==9549==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 hd-geo-test /x86_64/hd-geo/ide/device/mbr/chs
==9555==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 hd-geo-test /x86_64/hd-geo/ide/device/user/chs
==9560==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 10 hd-geo-test /x86_64/hd-geo/ide/device/user/chst
==9566==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9570==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9574==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9578==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9582==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9586==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9590==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9594==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9597==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 hd-geo-test /x86_64/hd-geo/override/ide
==9604==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9608==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9612==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9616==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9620==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9624==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9628==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9632==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9635==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 12 hd-geo-test /x86_64/hd-geo/override/scsi
==9642==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9646==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9650==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9654==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9658==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9662==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9666==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9670==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9673==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 hd-geo-test /x86_64/hd-geo/override/scsi_2_controllers
==9680==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9684==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9688==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9692==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9695==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 hd-geo-test /x86_64/hd-geo/override/virtio_blk
==9702==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9706==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9709==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 hd-geo-test /x86_64/hd-geo/override/zero_chs
==9716==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9720==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9724==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9728==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9731==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 16 hd-geo-test /x86_64/hd-geo/override/scsi_hot_unplug
==9738==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9742==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9746==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9750==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
==9753==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 17 hd-geo-test /x86_64/hd-geo/override/virtio_hot_unplug
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/boot-order-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="boot-order-test" PASS 1 boot-order-test /x86_64/boot-order/pc
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9822==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP'
Using expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9828==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP'
Using expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9834==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.bridge'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9840==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.ipmikcs'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9846==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.cphp'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9853==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.memhp'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9859==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.numamem'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9865==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.dimmpxm'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9874==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/pc/FACP.acpihmat'
Looking for expected file 'tests/data/acpi/pc/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9881==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.bridge'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9887==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.mmio64'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9893==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.ipmibt'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9899==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.cphp'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9906==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.memhp'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9912==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.numamem'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9918==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.dimmpxm'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==9927==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Looking for expected file 'tests/data/acpi/q35/FACP.acpihmat'
Looking for expected file 'tests/data/acpi/q35/FACP'
---
PASS 1 i440fx-test /x86_64/i440fx/defaults
PASS 2 i440fx-test /x86_64/i440fx/pam
PASS 3 i440fx-test /x86_64/i440fx/firmware/bios
==10019==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 i440fx-test /x86_64/i440fx/firmware/pflash
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/fw_cfg-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="fw_cfg-test"
PASS 1 fw_cfg-test /x86_64/fw_cfg/signature
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/drive_del-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="drive_del-test"
PASS 1 drive_del-test /x86_64/drive_del/without-dev
PASS 2 drive_del-test /x86_64/drive_del/after_failed_device_add
==10112==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 drive_del-test /x86_64/blockdev/drive_del_device_del
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/wdt_ib700-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="wdt_ib700-test"
PASS 1 wdt_ib700-test /x86_64/wdt_ib700/pause
---
PASS 1 usb-hcd-uhci-test /x86_64/uhci/pci/init
PASS 2 usb-hcd-uhci-test /x86_64/uhci/pci/port1
PASS 3 usb-hcd-uhci-test /x86_64/uhci/pci/hotplug
==10307==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 usb-hcd-uhci-test /x86_64/uhci/pci/hotplug/usb-storage
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/usb-hcd-ehci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="usb-hcd-ehci-test"
PASS 1 usb-hcd-ehci-test /x86_64/ehci/pci/uhci-port-1
---
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/usb-hcd-xhci-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="usb-hcd-xhci-test"
PASS 1 usb-hcd-xhci-test /x86_64/xhci/pci/init
PASS 2 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug
==10325==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug/usb-uas
PASS 4 usb-hcd-xhci-test /x86_64/xhci/pci/hotplug/usb-ccid
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/cpu-plug-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="cpu-plug-test"
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10461==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 1 vmgenid-test /x86_64/vmgenid/vmgenid/set-guid
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10467==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 vmgenid-test /x86_64/vmgenid/vmgenid/set-guid-auto
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10473==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 vmgenid-test /x86_64/vmgenid/vmgenid/query-monitor
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/tpm-crb-swtpm-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="tpm-crb-swtpm-test"
SKIP 1 tpm-crb-swtpm-test /x86_64/tpm/crb-swtpm/test # SKIP swtpm not in PATH or missing --tpm2 support
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10572==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10578==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 3 migration-test /x86_64/migration/fd_proto
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10585==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10591==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 4 migration-test /x86_64/migration/validate_uuid
PASS 5 migration-test /x86_64/migration/validate_uuid_error
PASS 6 migration-test /x86_64/migration/validate_uuid_src_not_set
---
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10641==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10647==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 8 migration-test /x86_64/migration/auto_converge
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10655==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10661==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 9 migration-test /x86_64/migration/postcopy/unix
PASS 10 migration-test /x86_64/migration/postcopy/recovery
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10690==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10696==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 11 migration-test /x86_64/migration/precopy/unix
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10704==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10710==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 12 migration-test /x86_64/migration/precopy/tcp
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10718==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10724==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 13 migration-test /x86_64/migration/xbzrle/unix
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10732==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10738==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 14 migration-test /x86_64/migration/multifd/tcp/none
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10856==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 15 migration-test /x86_64/migration/multifd/tcp/cancel
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10912==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10918==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 16 migration-test /x86_64/migration/multifd/tcp/zlib
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10974==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Could not access KVM kernel module: No such file or directory
qemu-system-x86_64: -accel kvm: failed to initialize kvm: No such file or directory
qemu-system-x86_64: falling back to tcg
==10980==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 17 migration-test /x86_64/migration/multifd/tcp/zstd
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/test-x86-cpuid-compat -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="test-x86-cpuid-compat"
PASS 1 test-x86-cpuid-compat /x86/cpuid/parsing-plus-minus
---
PASS 1 machine-none-test /x86_64/machine/none/cpu_option
MALLOC_PERTURB_=${MALLOC_PERTURB_:-$(( ${RANDOM:-0} % 255 + 1))} QTEST_QEMU_BINARY=x86_64-softmmu/qemu-system-x86_64 QTEST_QEMU_IMG=qemu-img tests/qtest/qmp-test -m=quick -k --tap < /dev/null | ./scripts/tap-driver.pl --test-name="qmp-test"
PASS 1 qmp-test /x86_64/qmp/protocol
==11419==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 2 qmp-test /x86_64/qmp/oob
PASS 3 qmp-test /x86_64/qmp/preconfig
PASS 4 qmp-test /x86_64/qmp/missing-any-arg
---
PASS 16 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/i82562/pci-device/pci-device-tests/nop
PASS 17 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/i82801/pci-device/pci-device-tests/nop
PASS 18 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/ES1370/pci-device/pci-device-tests/nop
==11837==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 19 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/megasas/pci-device/pci-device-tests/nop
PASS 20 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/megasas/megasas-tests/dcmd/pd-get-info/fuzz
PASS 21 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/ne2k_pci/pci-device/pci-device-tests/nop
PASS 22 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/tulip/pci-device/pci-device-tests/nop
PASS 23 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/tulip/tulip-tests/tulip_large_tx
PASS 24 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/nvme/pci-device/pci-device-tests/nop
==11852==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 25 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/nvme/nvme-tests/oob-cmb-access
==11857==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 26 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/pcnet/pci-device/pci-device-tests/nop
PASS 27 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/pci-ohci/pci-device/pci-device-tests/nop
PASS 28 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/pci-ohci/pci-ohci-tests/ohci_pci-test-hotplug
---
PASS 35 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pci/sdhci/sdhci-tests/registers
PASS 36 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/tpci200/ipack/ipoctal232/ipoctal232-tests/nop
PASS 37 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/tpci200/pci-device/pci-device-tests/nop
==11917==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 38 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/pci-device/pci-device-tests/nop
PASS 39 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio/virtio-tests/nop
PASS 40 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p/virtio-9p-tests/config
---
PASS 50 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-9p-pci/virtio-9p/virtio-9p-tests/fs/readdir/basic
PASS 51 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-balloon-pci/pci-device/pci-device-tests/nop
PASS 52 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-balloon-pci/virtio/virtio-tests/nop
==11928==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 53 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/msix
==11934==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 54 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/idx
==11940==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 55 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/nxvirtq
==11946==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 56 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk-pci-tests/hotplug
==11952==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 57 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/indirect
==11958==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 58 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/config
==11964==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 59 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/basic
==11970==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 60 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-blk-pci/virtio-blk/virtio-blk-tests/resize
PASS 61 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/basic
PASS 62 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-net-pci/virtio-net/virtio-net-tests/rx_stop_cont
---
PASS 70 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-rng-pci/virtio-rng-pci-tests/hotplug
PASS 71 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-rng-pci/pci-device/pci-device-tests/nop
PASS 72 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-rng-pci/virtio/virtio-tests/nop
==12062==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 73 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/virtio-scsi-pci-tests/iothread-attach-node
==12073==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 74 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/pci-device/pci-device-tests/nop
==12078==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 75 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/virtio-scsi/virtio-scsi-tests/hotplug
==12083==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
PASS 76 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-scsi-pci/virtio-scsi/virtio-scsi-tests/unaligned-write-same
PASS 77 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-serial-pci/pci-device/pci-device-tests/nop
PASS 78 qos-test /x86_64/pc/i440FX-pcihost/pci-bus-pc/pci-bus/virtio-serial-pci/virtio/virtio-tests/nop
---
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=f7e7d249fa3d477f8f5f54bfb4705e3b', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-kde08xkh/src/docker-src.2020-06-01-15.00.26.19416:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=f7e7d249fa3d477f8f5f54bfb4705e3b
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-kde08xkh/src'
make: *** [docker-run-test-debug@fedora] Error 2
real 43m19.363s
user 0m9.290s
The full log is available at
http://patchew.org/logs/[email protected]/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Re: [PATCH v2 11/20] qapi: backup: add x-max-chunk and x-max-workers parameters
In reply to Vladimir Sementsov-OgievskiyQuoted 18 lines · Vladimir Sementsov-Ogievskiy
Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 9 ++++++++- include/block/block_int.h | 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-)
[..]
block/replication.c static void replication_start(ReplicationState *rs, ReplicationMode mode, · line 566
Quoted 8 lines
block/replication.c +1
-1@@ -563,7 +563,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job = backup_job_create( NULL, s->secondary_disk->bs, s->hidden_disk->bs, 0, MIRROR_SYNC_MODE_NONE, NULL, 0, false, NULL, - true, + true, 0, 0,
must be true, 1, 0, to setup 1 worker.
Quoted 5 lines
block/replication.c +0
-0BLOCKDEV_ON_ERROR_REPORT ,BLOCKDEV_ON_ERROR_REPORT ,JOB_INTERNAL , backup_job_completed, bs, NULL, &local_err);blockdev.c +0
-0
Signature · 2 lines
Best regards, Vladimir
Quoted 16 lines · 01.06.2020 21:59, [email protected] wrote:
Patchew URL: https://patchew.org/QEMU/[email protected]/
Hi,
This series failed the docker-quick@centos7 build test. Please find the testing commands and their output below. If you have Docker installed, you can probably reproduce it locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
make docker-image-centos7 V=1 NETWORK=1
time make docker-test-quick@centos7 SHOW_ENV=1 J=14 NETWORK=1 === TEST SCRIPT END ===
Quoted 5 lines
TEST check-unit: tests/test-logging
TEST check-unit: tests/test-replication
**
ERROR:/tmp/qemu-test/src/tests/test-replication.c:428:test_secondary_start: assertion failed: (!local_err) ERROR - Bail out! ERROR:/tmp/qemu-test/src/tests/test-replication.c:428:test_secondary_start: assertion failed: (!local_err)
bug in patch 11, we need to set up 1 backup worker in block/repliction, not zero.
Quoted 22 lines
make: *** [check-unit] Error 1
make: *** Waiting for unfinished jobs....
TEST check-qtest-x86_64: tests/qtest/boot-order-test
TEST check-qtest-x86_64: tests/qtest/bios-tables-test
---
raiseCalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=73ef29198bda41a1bce9aa3697266e4c', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=1', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-j0ydkht8/src/docker-src.2020-06-01-14.44.54.8388:/var/tmp/qemu:z,ro', 'qemu:centos7', '/var/tmp/qemu/run', 'test-quick']' returned non-zero exit status 2. filter=--filter=label=com.qemu.instance.uuid=73ef29198bda41a1bce9aa3697266e4c
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-j0ydkht8/src'
make: *** [docker-run-test-quick@centos7] Error 2real 14m11.116s
user 0m9.306sThe full log is available at
http://patchew.org/logs/[email protected]/testing.docker-quick@centos7/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to [email protected]
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 02/20] iotests: 129 don't check backup "busy"
In reply to Vladimir Sementsov-OgievskiyQuoted 2 lines · Vladimir Sementsov-Ogievskiy
Busy is racy, job has it's "pause-points" when it's not busy. Drop this check.
Possible, though I have to admit I don’t think I’ve ever seen 129 fail because of that, but rather because of:
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html
I.e. the fact that BB throttling doesn’t do anything and the job is basically just done before query-block-jobs.
But checking @busy is probably not the best thing to do, right.
Quoted 4 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/129 | 1 - 1 file changed, 1 deletion(-)
Re: [PATCH v2 03/20] qapi: backup: add x-use-copy-range parameter
In reply to Vladimir Sementsov-OgievskiyQuoted 2 lines · Vladimir Sementsov-Ogievskiy
Add parameter to enable/disable
copy_range. Keep current default for now (enabled).
Why x-, though? I can’t think of a reason why we would have to remove this.
Quoted 18 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 4 +++- block/backup-top.h | 1 + include/block/block-copy.h | 2 +- include/block/block_int.h | 1 + block/backup-top.c | 4 +++- block/backup.c | 4 +++- block/block-copy.c | 4 ++-- block/replication.c | 1 + blockdev.c | 5 +++++ 9 files changed, 20 insertions(+), 6 deletions(-)qapi/block-core.json +0
-0
qapi/block-core.json line 1408
Quoted 4 lines
qapi/block-core.json +1
-0@@ -1405,6 +1405,8 @@ # above node specified by @drive. If this option is not given, # a node name is autogenerated. (Since: 4.2) # +# @x-use-copy-range: use copy offloading if possible. Default true. (Since 5.1)
Would it make more sense to invert it to disable-copy-range? First, this would make for a cleaner meaning, because it would allow dropping the “if possible” part. Setting use-copy-range=true would intuitively imply to me that I get an error if copy-range cannot be used. Sure, there’s this little “if possible” in the documentation, but it goes against my intuition. disable-copy-range=true is intuitively clear. Second, this would give us a default of false, which is marginally nicer.
Max
Re: [PATCH v2 04/20] block/block-copy: More explicit call_state
In reply to Vladimir Sementsov-OgievskiyQuoted 14 lines · Vladimir Sementsov-Ogievskiy
Refactor common path to use BlockCopyCallState pointer as parameter, to prepare it for use in asynchronous block-copy (at least, we'll need to run block-copy in a coroutine, passing the whole parameters as one pointer).
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/block-copy.c | 51 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 13 deletions(-)block/block-copy.c +0
-0
[...]
block/block-copy.c out: · line 661
Quoted 13 lines
block/block-copy.c +2
-3@@ -646,16 +653,16 @@ out:* it means that some I/O operation failed in context of _this_ block_copy call, * not some parallel operation. */ -int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, -bool *error_is_read) +static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) {int ret; do { - ret = block_copy_dirty_clusters(s, offset, bytes, error_is_read); + ret = block_copy_dirty_clusters(call_state);
It’s possible that much of this code will change in a future patch of this series, but as it is, it might be nice to make block_copy_dirty_clusters’s argument a const pointer so it’s clear that the call to block_copy_wait_one() below will use the original @offset and @bytes values.
*shrug*
Quoted 8 lines
block/block-copy.c +2
-1if (ret ==0 ) { - ret =block_copy_wait_one (s, offset, bytes); + ret =block_copy_wait_one (call_state->s, call_state->offset, + call_state->bytes); }/*
Re: [PATCH v2 05/20] block/block-copy: implement block_copy_async
In reply to Vladimir Sementsov-OgievskiyQuoted 24 lines · Vladimir Sementsov-Ogievskiy
We'll need async block-copy invocation to use in backup directly.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 13 +++++++++++++ block/block-copy.c | 40 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+)include/block/block-copy.h +3
-0@@ -19,7 +19,10 @@#include "qemu/co-shared-resource.h" typedef void (*ProgressBytesCallbackFunc )(int64_t bytes,void *opaque); +typedef void (*BlockCopyAsyncCallbackFunc )(int ret,bool error_is_read, +void *opaque);typedef struct BlockCopyState BlockCopyState ; +typedef struct BlockCopyCallState BlockCopyCallState ; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, int64_t cluster_size, bool use_copy_range,
include/block/block-copy.h int64_t block_copy_reset_unallocated(BlockCopyState *s, · line 49
Quoted 6 lines
include/block/block-copy.h +3
-0@@ -41,6 +44,16 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,bool *error_is_read); +/* +* Run block-copy in a coroutine, return state pointer. If finished early +* returns NULL (@cb is called anyway).
Any special reason for doing so? Seems like the code would be a tad simpler if we just returned it either way. (And off the top of my head I’d guess it’d be easier for the caller if the returned value was always non-NULL, too.)
Quoted 27 lines
include/block/block-copy.h +7
-0+*/ +BlockCopyCallState *block_copy_async (BlockCopyState *s, +int64_t offset,int64_t bytes, +bool ratelimit,int max_workers, +int64_t max_chunk, +BlockCopyAsyncCallbackFunc cb); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);block/block-copy.c +2
-0@@ -34,9 +34,11 @@ typedef struct BlockCopyCallState {BlockCopyState *s;int64_t offset;int64_t bytes; +BlockCopyAsyncCallbackFunc cb;/* State */ bool failed; +bool finished;/* OUT parameters */ bool error_is_read;
block/block-copy.c static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) · line 683
Quoted 6 lines
block/block-copy.c +3
-0@@ -676,6 +678,13 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)*/ }while (ret >0 ); +if (call_state->cb) { + call_state->cb (ret, call_state->error_is_read, + call_state->s->progress_opaque);
I find it weird to pass progress_opaque here. Shouldn’t we just have a dedicated opaque object for this CB?
Quoted 24 lines
block/block-copy.c +17
-0+ } + + call_state->finished = true; +return ret; }@@ -697,6 +706,37 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes,return ret; } +static void coroutine_fn block_copy_async_co_entry(void *opaque) +{ +block_copy_common (opaque); +} + +BlockCopyCallState *block_copy_async (BlockCopyState *s, +int64_t offset,int64_t bytes, +bool ratelimit,int max_workers, +int64_t max_chunk, +BlockCopyAsyncCallbackFunc cb) +{ +BlockCopyCallState *call_state =g_new (BlockCopyCallState,1 ); + Coroutine *co = qemu_coroutine_create(block_copy_async_co_entry,
block/block-copy.c int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, · line 731
Quoted 10 lines
block/block-copy.c +10
-0+ call_state); + + *call_state = (BlockCopyCallState ) { + .s = s, + .offset = offset, + .bytes = bytes, + .cb = cb, + }; + +qemu_coroutine_enter (co);
Do we need/want any already-in-coroutine shenanigans here?
Max
Quoted 12 lines
block/block-copy.c +8
-0+ +if (call_state->finished) { +g_free (call_state); +return NULL ; + } + +return call_state; +}BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s) {return s->copy_bitmap;
Re: [PATCH v2 03/20] qapi: backup: add x-use-copy-range parameter
In reply to Max ReitzQuoted 5 lines · Max Reitz
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
Add parameter to enable/disable
copy_range. Keep current default for now (enabled).Why x-, though? I can’t think of a reason why we would have to remove this.
I add some x- arguments in these series:
1. I'm unsure about default values (still, it can be changed even without x- prefixes as I understand)
2. Probably all this should be wrapped into common "block-copy" options, to not add same arguments to different block-jobs, when they all (I believe in bright future :) move on block-copy.
So, this series is not about API, but for new backup architecture, and experimental APIs are needed only to experiment with performance and adjust some iotests.
Quoted 24 lines
Quoted 19 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 4 +++- block/backup-top.h | 1 + include/block/block-copy.h | 2 +- include/block/block_int.h | 1 + block/backup-top.c | 4 +++- block/backup.c | 4 +++- block/block-copy.c | 4 ++-- block/replication.c | 1 + blockdev.c | 5 +++++ 9 files changed, 20 insertions(+), 6 deletions(-)qapi/block-core.json +0
-0qapi/block-core.json line 1408
Quoted 4 lines
qapi/block-core.json +1
-0@@ -1405,6 +1405,8 @@ # above node specified by @drive. If this option is not given, # a node name is autogenerated. (Since: 4.2) # +# @x-use-copy-range: use copy offloading if possible. Default true. (Since 5.1)
Quoted 8 lines
Would it make more sense to invert it to disable-copy-range? First, this would make for a cleaner meaning, because it would allow dropping the “if possible” part. Setting use-copy-range=true would intuitively imply to me that I get an error if copy-range cannot be used. Sure, there’s this little “if possible” in the documentation, but it goes against my intuition. disable-copy-range=true is intuitively clear. Second, this would give us a default of false, which is marginally nicer.
Reasonable. But we also should consider disabling copy_range by default as I propose.. But yes, if we keep x- for now, and don't change default for copy_range in this series, I can invert it just for readability.
Thanks a lot for reviewing this!
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 05/20] block/block-copy: implement block_copy_async
In reply to Max ReitzQuoted 28 lines · Max Reitz
Quoted 27 lines · Vladimir Sementsov-Ogievskiy
We'll need async block-copy invocation to use in backup directly.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 13 +++++++++++++ block/block-copy.c | 40 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 53 insertions(+)include/block/block-copy.h +3
-0@@ -19,7 +19,10 @@#include "qemu/co-shared-resource.h" typedef void (*ProgressBytesCallbackFunc )(int64_t bytes,void *opaque); +typedef void (*BlockCopyAsyncCallbackFunc )(int ret,bool error_is_read, +void *opaque);typedef struct BlockCopyState BlockCopyState ; +typedef struct BlockCopyCallState BlockCopyCallState ; BlockCopyState *block_copy_state_new(BdrvChild *source, BdrvChild *target, int64_t cluster_size, bool use_copy_range,@@ -41,6 +44,16 @@ int64_t block_copy_reset_unallocated(BlockCopyState *s, int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,bool *error_is_read);
Quoted 8 lines
include/block/block-copy.h int64_t block_copy_reset_unallocated(BlockCopyState *s, · line 49
Quoted 3 lines
include/block/block-copy.h +3
-0+/* +* Run block-copy in a coroutine, return state pointer. If finished early +* returns NULL (@cb is called anyway). Any special reason for doing so? Seems like the code would be a tad simpler if we just returned it either way. (And off the top of my head I’d guess it’d be easier for the caller if the returned value was always non-NULL, too.)
Sounds reasonable, will check
Quoted 31 lines
Quoted 31 lines
include/block/block-copy.h +7
-0+*/ +BlockCopyCallState *block_copy_async (BlockCopyState *s, +int64_t offset,int64_t bytes, +bool ratelimit,int max_workers, +int64_t max_chunk, +BlockCopyAsyncCallbackFunc cb); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);block/block-copy.c +2
-0@@ -34,9 +34,11 @@ typedef struct BlockCopyCallState {BlockCopyState *s;int64_t offset;int64_t bytes; +BlockCopyAsyncCallbackFunc cb;/* State */ bool failed; +bool finished;/* OUT parameters */ bool error_is_read;@@ -676,6 +678,13 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)*/ }while (ret >0 );
Quoted 6 lines
block/block-copy.c static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) · line 683
Quoted 3 lines
block/block-copy.c +3
-0+if (call_state->cb) { + call_state->cb (ret, call_state->error_is_read, + call_state->s->progress_opaque);I find it weird to pass
progress_opaquehere. Shouldn’t we just have a dedicated opaque object for this CB?
I remember, it should be refactored later. But seems strange here, better to change.
Quoted 29 lines
Quoted 29 lines
block/block-copy.c +21
-0+ } + + call_state->finished = true; +return ret; }@@ -697,6 +706,37 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes,return ret; } +static void coroutine_fn block_copy_async_co_entry(void *opaque) +{ +block_copy_common (opaque); +} + +BlockCopyCallState *block_copy_async (BlockCopyState *s, +int64_t offset,int64_t bytes, +bool ratelimit,int max_workers, +int64_t max_chunk, +BlockCopyAsyncCallbackFunc cb) +{ +BlockCopyCallState *call_state =g_new (BlockCopyCallState,1 ); +Coroutine *co =qemu_coroutine_create (block_copy_async_co_entry, + call_state); + + *call_state = (BlockCopyCallState ) { + .s = s,
Quoted 9 lines
block/block-copy.c int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, · line 731
Quoted 6 lines
block/block-copy.c +6
-0+ .offset = offset, + .bytes = bytes, + .cb = cb, + }; + +qemu_coroutine_enter (co);Do we need/want any already-in-coroutine shenanigans here?
No: the aim of the function is to start a new coroutine in parallel, independently of are we already in some other coroutine or not.
Quoted 15 lines
Quoted 13 lines
block/block-copy.c +8
-0+ +if (call_state->finished) { +g_free (call_state); +return NULL ; + } + +return call_state; +}BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s) {return s->copy_bitmap;
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 05/20] block/block-copy: implement block_copy_async
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines · Max Reitz
Quoted 7 lines · Vladimir Sementsov-Ogievskiy
We'll need async block-copy invocation to use in backup directly.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
include/block/block-copy.h| 13 +++++++++++++ block/block-copy.c | 40 ++++++++++++++++++++++++++++++++++++++2 files changed, 53 insertions(+)
[...]
Quoted 22 lines
Quoted 12 lines
Quoted 12 lines
block/block-copy.c +12
-0+BlockCopyCallState *block_copy_async (BlockCopyState *s, + int64_t offset, int64_t bytes, + bool ratelimit, int max_workers, + int64_t max_chunk, + BlockCopyAsyncCallbackFunc cb) +{ + BlockCopyCallState *call_state = g_new(BlockCopyCallState, 1); + Coroutine *co = qemu_coroutine_create(block_copy_async_co_entry, + call_state); + + *call_state = (BlockCopyCallState) { + .s = s,Quoted 9 lines
Quoted 6 lines
+ .offset = offset,
+ .bytes = bytes,
+ .cb = cb,
+ };
+
+qemu_coroutine_enter(co);Do we need/want any already-in-coroutine shenanigans here?
Quoted 2 lines
No: the aim of the function is to start a new coroutine in parallel, independently of are we already in some other coroutine or not.
OK, that makes sense.
Re: [PATCH v2 06/20] block/block-copy: add max_chunk and max_workers parameters
In reply to Vladimir Sementsov-OgievskiyQuoted 14 lines · Vladimir Sementsov-Ogievskiy
They will be used for backup.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 5 +++++ block/block-copy.c | 10 ++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)include/block/block-copy.h +0
-0@@ -47,6 +47,11 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes,/*
include/block/block-copy.h int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, · line 57
Quoted 10 lines
include/block/block-copy.h +5
-0* Run block-copy in a coroutine, return state pointer. If finished early * returns NULL (@cb is called anyway). +* +* @max_workers means maximum of parallel coroutines to execute sub-requests, +* must be > 0. +* +* @max_chunk means maximum length for one IO operation. Zero means unlimited. */ BlockCopyCallState *block_copy_async(BlockCopyState *s, int64_t offset, int64_t bytes,
I only just now notice that @max_workers and @max_chunk were already added in the previous patch, even though they aren’t used there. Should we defer adding them until this patch?
Quoted 44 lines
block/block-copy.c +6
-2@@ -34,6 +34,8 @@ typedef struct BlockCopyCallState {BlockCopyState *s;int64_t offset;int64_t bytes; +int max_workers; +int64_t max_chunk;BlockCopyAsyncCallbackFunc cb;/* State */ @@ -144,10 +146,11 @@ static BlockCopyTask *block_copy_task_create(BlockCopyState *s, int64_t offset, int64_t bytes) {BlockCopyTask *task; +int64_t max_chunk =MIN_NON_ZERO (s->copy_size, call_state->max_chunk);if (!bdrv_dirty_bitmap_next_dirty_area (s->copy_bitmap, offset, offset + bytes, - s->copy_size, &offset, &bytes)) + max_chunk, &offset, &bytes)) {return NULL ; }@@ -616,7 +619,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) bytes = end - offset;if (!aio && bytes) { - aio =aio_task_pool_new (BLOCK_COPY_MAX_WORKERS ); + aio =aio_task_pool_new (call_state->max_workers); } ret =block_copy_task_run (aio, task);@@ -695,6 +698,7 @@ int coroutine_fn block_copy(BlockCopyState *s, int64_t start, int64_t bytes, .s = s, .offset = start, .bytes = bytes, + .max_workers = BLOCK_COPY_MAX_WORKERS, };int ret =block_copy_common (&call_state);
block/block-copy.c BlockCopyCallState *block_copy_async(BlockCopyState *s, · line 733
Quoted 4 lines
block/block-copy.c +1
-0@@ -726,6 +730,8 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, .offset = offset, .bytes = bytes, .cb = cb, + .max_workers = max_workers ?: BLOCK_COPY_MAX_WORKERS,
I thought this must be > 0?
block/block-copy.c BlockCopyCallState *block_copy_async(BlockCopyState *s, · line 737
Quoted 5 lines
block/block-copy.c +1
-0+ .max_chunk = max_chunk, };qemu_coroutine_enter (co);
And I now notice that there’s no newline after block_copy_async(). (Doesn’t concern this patch, of course, but the previous one.)
Max
Re: [PATCH v2 07/20] block/block-copy: add ratelimit to block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 2 lines · Vladimir Sementsov-Ogievskiy
We are going to directly use one async block-copy operation for backup job, so we need rate limitator.
%s/limitator/limiter/g, I think.
Quoted 5 lines
We want to maintain current backup behavior: only background copying is limited and copy-before-write operations only participate in limit calculation. Therefore we need one rate limitator for block-copy state and boolean flag for block-copy call state for actual limitation.
Quoted 10 lines
Note, that we can't just calculate each chunk in limitator after successful copying: it will not save us from starting a lot of async sub-requests which will exceed limit too much. Instead let's use the following scheme on sub-request creation:
1. If at the moment limit is not exceeded, create the request and account it immediately.
2. If at the moment limit is already exceeded, drop create sub-request and handle limit instead (by sleep).
With this approach we'll never exceed the limit more than by one sub-request (which pretty much matches current backup behavior).
Sounds reasonable.
Quoted 70 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 8 +++++++ block/block-copy.c | 44 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)include/block/block-copy.h +8
-0@@ -59,6 +59,14 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,int64_t max_chunk, BlockCopyAsyncCallbackFunc cb); +/* +* Set speed limit for block-copy instance. All block-copy operations related to +* this BlockCopyState will participate in speed calculation, but only +* block_copy_async calls with @ratelimit=true will be actually limited. +*/ +void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state, +uint64_t speed); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);block/block-copy.c +11
-0@@ -26,6 +26,7 @@#define BLOCK_COPY_MAX_BUFFER (1 * MiB)#define BLOCK_COPY_MAX_MEM (128 * MiB)#define BLOCK_COPY_MAX_WORKERS 64 +#define BLOCK_COPY_SLICE_TIME 100000000ULL/* ns */ static coroutine_fn int block_copy_task_entry(AioTask *task);@@ -36,11 +37,13 @@ typedef struct BlockCopyCallState {int64_t bytes;int max_workers;int64_t max_chunk; +bool ratelimit;BlockCopyAsyncCallbackFunc cb;/* State */ bool failed;bool finished; +QemuCoSleepState *sleep_state;/* OUT parameters */ bool error_is_read;@@ -103,6 +106,9 @@ typedef struct BlockCopyState {void *progress_opaque;SharedResource *mem; + +uint64_t speed; +RateLimit rate_limit; } BlockCopyState; static BlockCopyTask *find_conflicting_task(BlockCopyState *s,@@ -611,6 +617,21 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) } task->zeroes = ret &BDRV_BLOCK_ZERO ; +if (s->speed) { +if (call_state->ratelimit) { +uint64_t ns =ratelimit_calculate_delay (&s->rate_limit,0 ); +if (ns >0 ) { +block_copy_task_end (task, -EAGAIN );
block/block-copy.c block_copy_dirty_clusters(BlockCopyCallState *call_state) · line 634
Quoted 10 lines
block/block-copy.c +10
-0+g_free (task); +qemu_co_sleep_ns_wakeable (QEMU_CLOCK_REALTIME , ns, + &call_state->sleep_state); +continue ; + } + } + +ratelimit_calculate_delay (&s->rate_limit, task->bytes); + } +
Looks good.
Quoted 18 lines
block/block-copy.c +7
-0trace_block_copy_process (s, task->offset);co_get_from_shres (s->mem, task->bytes);@@ -649,6 +670,13 @@ out:return ret <0 ? ret : found_dirty; } +static void block_copy_kick (BlockCopyCallState *call_state) +{ +if (call_state->sleep_state) { +qemu_co_sleep_wake (call_state->sleep_state); + } +} +/* * block_copy_common *
block/block-copy.c BlockCopyCallState *block_copy_async(BlockCopyState *s, · line 760
Quoted 4 lines
block/block-copy.c +1
-0@@ -729,6 +757,7 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, .s = s, .offset = offset, .bytes = bytes, + .ratelimit = ratelimit,
Hm, same problem/question as in patch 6: Should the @ratelimit parameter really be added in patch 5 if it’s used only now?
Quoted 12 lines
block/block-copy.c +5
-0.cb = cb, .max_workers = max_workers ?: BLOCK_COPY_MAX_WORKERS, .max_chunk = max_chunk,@@ -752,3 +781,18 @@ void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip) { s->skip_unallocated = skip; } + +void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state, +uint64_t speed) +{ +uint64_t old_speed = s->speed;
block/block-copy.c void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip) · line 798
Quoted 10 lines
block/block-copy.c +10
-0+ + s->speed = speed; +if (speed >0 ) { +ratelimit_set_speed (&s->rate_limit, speed,BLOCK_COPY_SLICE_TIME ); + } + +if (call_state && old_speed && (speed > old_speed || speed ==0 )) { +block_copy_kick (call_state); + } +}
Hm. I’m interested in seeing how this is going to be used, i.e. what callers will pass for @call_state. I suppose it’s going to be the background operation for the whole device, but I wonder whether it actually makes sense to pass it. I mean, the caller could just call block_copy_kick() itself (unconditionally, because it’ll never hurt, I think).
Re: [PATCH v2 08/20] block/block-copy: add block_copy_cancel
In reply to Vladimir Sementsov-OgievskiyQuoted 59 lines · Vladimir Sementsov-Ogievskiy
Add function to cancel running async block-copy call. It will be used in backup.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 7 +++++++ block/block-copy.c | 22 +++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-)include/block/block-copy.h +7
-0@@ -67,6 +67,13 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state,uint64_t speed); +/* +* Cancel running block-copy call. +* Cancel leaves block-copy state valid: dirty bits are correct and you may use +* cancel + <run block_copy with same parameters> to emulate pause/resume. +*/ +void block_copy_cancel (BlockCopyCallState *call_state); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);block/block-copy.c +4
-2@@ -44,6 +44,8 @@ typedef struct BlockCopyCallState {bool failed;bool finished;QemuCoSleepState *sleep_state; +bool cancelled; +Coroutine *canceller;/* OUT parameters */ bool error_is_read;@@ -582,7 +584,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)assert (QEMU_IS_ALIGNED (offset, s->cluster_size));assert (QEMU_IS_ALIGNED (bytes, s->cluster_size)); -while (bytes &&aio_task_pool_status (aio) ==0 ) { +while (bytes &&aio_task_pool_status (aio) ==0 && !call_state->cancelled) {BlockCopyTask *task;int64_t status_bytes;@@ -693,7 +695,7 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) do { ret = block_copy_dirty_clusters(call_state); - if (ret == 0) { + if (ret == 0 && !call_state->cancelled) { ret = block_copy_wait_one(call_state->s, call_state->offset, call_state->bytes); }
block/block-copy.c static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) · line 712
Quoted 5 lines
block/block-copy.c +1
-1@@ -707,13 +709,18 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)* 2. We have waited for some intersecting block-copy request * It may have failed and produced new dirty bits. */ - }while (ret >0 ); + }while (ret >0 && !call_state->cancelled);
Would it be cleaner if block_copy_dirty_cluster() just returned -ECANCELED? Or would that pose a problem for its callers or the async callback?
Quoted 13 lines
block/block-copy.c +5
-0if (call_state->cb) { call_state->cb (ret, call_state->error_is_read, call_state->s->progress_opaque); } +if (call_state->canceller) { +aio_co_wake (call_state->canceller); + call_state->canceller =NULL ; + } + call_state->finished = true;return ret;
Re: [PATCH v2 09/20] blockjob: add set_speed to BlockJobDriver
In reply to Vladimir Sementsov-OgievskiyQuoted 8 lines · Vladimir Sementsov-Ogievskiy
We are going to use async block-copy call in backup, so we'll need to passthrough setting backup speed to block-copy call.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/blockjob_int.h | 2 ++ blockjob.c | 6 ++++++ 2 files changed, 8 insertions(+)
Re: [PATCH v2 10/20] job: call job_enter from job_user_pause
In reply to Vladimir Sementsov-OgievskiyQuoted 8 lines · Vladimir Sementsov-Ogievskiy
If main job coroutine called
job_yield(while some background process is in progress), we should give it a chance to calljob_pause_point(). It will be used in backup, when moved on async block-copy.Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- job.c | 1 + 1 file changed, 1 insertion(+)
Sounds reasonable to me, although I’d prefer an opinion from John.
So, a middle-weak:
Re: [PATCH v2 11/20] qapi: backup: add x-max-chunk and x-max-workers parameters
In reply to Vladimir Sementsov-OgievskiyQuoted 23 lines · Vladimir Sementsov-Ogievskiy
Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 9 ++++++++- include/block/block_int.h | 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-)qapi/block-core.json +0
-0
qapi/block-core.json line 1411
Quoted 5 lines
qapi/block-core.json +2
-0@@ -1407,6 +1407,12 @@ # # @x-use-copy-range: use copy offloading if possible. Default true. (Since 5.1) # +# @x-max-workers: maximum of parallel requests for static data backup. This +# doesn't influence copy-before-write operations. (Since: 5.1)
I think I’d prefer something with “background” rather than or in addition to “static”, like “maximum number of parallel requests for the sustained background backup operation”.
Quoted 65 lines
qapi/block-core.json +6
-1+# +# @x-max-chunk: maximum chunk length for static data backup. This doesn't +# influence copy-before-write operations. (Since: 5.1) +# # Note: @on-source-error and @on-target-error only affect background # I/O. If an error occurs during a guest write request, the device's # rerror/werror actions will be used.@@ -1421,7 +1427,8 @@ '*on-source-error': 'BlockdevOnError', '*on-target-error': 'BlockdevOnError', '*auto-finalize': 'bool', '*auto-dismiss': 'bool', - '*filter-node-name': 'str', '*x-use-copy-range': 'bool' } } + '*filter-node-name': 'str', '*x-use-copy-range': 'bool', + '*x-max-workers': 'int', '*x-max-chunk': 'int64' } } ## # @DriveBackup:include/block/block_int.h +7
-0@@ -1207,6 +1207,11 @@ void mirror_start(const char *job_id, BlockDriverState *bs,* @sync_mode: What parts of the disk image should be copied to the destination. * @sync_bitmap: The dirty bitmap if sync_mode is 'bitmap' or 'incremental' * @bitmap_mode: The bitmap synchronization policy to use. +* @max_workers: The limit for parallel requests for main backup loop. +* Must be >= 1. +* @max_chunk: The limit for one IO operation length in main backup loop. +* Must be not less than job cluster size or zero. Zero means no +* specific limit. * @on_source_error: The action to take upon error reading from the source. * @on_target_error: The action to take upon error writing to the target. * @creation_flags: Flags that control the behavior of the Job lifetime. @@ -1226,6 +1231,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,bool compress, const char *filter_node_name, bool use_copy_range, + int max_workers, + int64_t max_chunk, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,block/backup.c +4
-0@@ -46,6 +46,8 @@ typedef struct BackupBlockJob {uint64_t len;uint64_t bytes_read;int64_t cluster_size; +int max_workers; +int64_t max_chunk;BlockCopyState *bcs; } BackupBlockJob;@@ -335,6 +337,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,bool compress, const char *filter_node_name, bool use_copy_range, + int max_workers, + int64_t max_chunk, BlockdevOnError on_source_error, BlockdevOnError on_target_error, int creation_flags,
block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 368
Quoted 10 lines
block/backup.c +7
-0@@ -355,6 +359,16 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,assert (sync_mode !=MIRROR_SYNC_MODE_INCREMENTAL );assert (sync_bitmap || sync_mode !=MIRROR_SYNC_MODE_BITMAP ); +if (max_workers <1 ) { +error_setg (errp,"At least one worker needed" ); +return NULL ; + } + +if (max_chunk <0 ) { +error_setg (errp,"max-chunk is negative" );
Perhaps “must be positive or 0” instead? I think most error messages try to specify what is allowed instead of what isn’t.
block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 440
Quoted 12 lines
block/backup.c +5
-0+return NULL ; + } +if (bs == target) {error_setg (errp,"Source and target cannot be the same" );return NULL ;@@ -422,6 +436,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,if (cluster_size <0 ) { goto error; } +if (max_chunk && max_chunk < cluster_size) { + error_setg(errp, "Required max-chunk (%" PRIi64") is less than backup "
(missing a space after PRIi64)
block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 441
Quoted 1 line
block/backup.c +1
-0+ "cluster size (%" PRIi64 ")", max_chunk, cluster_size);
Should this be noted in the QAPI documentation? (And perhaps the fact that without copy offloading, we’ll never copy anything bigger than one cluster at a time anyway?)
Quoted 19 lines
block/backup.c +4
-0+return NULL ; + }/* * If source is in backing chain of target assume that target is going to be @@ -465,6 +484,8 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, job->bcs = bcs; job->cluster_size = cluster_size; job->len = len; + job->max_workers = max_workers; + job->max_chunk = max_chunk;block_copy_set_progress_callback (bcs, backup_progress_bytes_callback, job);block_copy_set_progress_meter (bcs, &job->common.job.progress);block/replication.c +0
-0
block/replication.c static void replication_start(ReplicationState *rs, ReplicationMode mode, · line 566
Quoted 5 lines
block/replication.c +1
-1@@ -563,7 +563,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job = backup_job_create( NULL, s->secondary_disk->bs, s->hidden_disk->bs, 0, MIRROR_SYNC_MODE_NONE, NULL, 0, false, NULL, - true, + true, 0, 0,
Passing 0 for max_workers will error out immediately, won’t it?
Quoted 3 lines
block/replication.c +0
-0BLOCKDEV_ON_ERROR_REPORT ,BLOCKDEV_ON_ERROR_REPORT ,JOB_INTERNAL , backup_job_completed, bs, NULL, &local_err);
Re: [PATCH v2 11/20] qapi: backup: add x-max-chunk and x-max-workers parameters
Quoted 18 lines · Max Reitz
Quoted 17 lines · Vladimir Sementsov-Ogievskiy
Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 9 ++++++++- include/block/block_int.h | 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-)
[...]
Quoted 4 lines
Quoted 4 lines
block/replication.c +0
-0
Quoted 8 lines
block/replication.c static void replication_start(ReplicationState *rs, ReplicationMode mode, · line 566
Quoted 6 lines
block/replication.c +1
-1@@ -563,7 +563,7 @@ static void replication_start(ReplicationState *rs, ReplicationMode mode, s->backup_job = backup_job_create( NULL, s->secondary_disk->bs, s->hidden_disk->bs, 0, MIRROR_SYNC_MODE_NONE, NULL, 0, false, NULL, - true, + true, 0, 0,Passing 0 for
max_workerswill error out immediately, won’t it?
Ah, oops. Saw your own reply only now. Yep, 1 worker would be nice. :)
Re: [PATCH v2 12/20] iotests: 56: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 16 lines · Vladimir Sementsov-Ogievskiy
After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, we'll have to wait for paused status, as we need to wait for several parallel request. So, let's gently wait instead of just asserting that job already paused.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/056 | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)tests/qemu-iotests/056 +0
-0@@ -306,8 +306,12 @@ class BackupTest(iotests.QMPTestCase): event = self.vm.event_wait(name="BLOCK_JOB_ERROR",
tests/qemu-iotests/056 class BackupTest(iotests.QMPTestCase): · line 314
Quoted 10 lines
tests/qemu-iotests/056 +6
-2match={'data': {'device': 'drive0'}}) self.assertNotEqual(event, None) - # OK, job should be wedged - res = self.vm.qmp('query-block-jobs') + # OK, job should pause, but it can't do it immediately, as it can't + # cancel other parallel requests (which didn't fail) + while True: + res = self.vm.qmp('query-block-jobs') + if res['return'][0]['status'] == 'paused': + break
A timeout around this would be nice, I think.
Quoted 4 lines
tests/qemu-iotests/056 +0
-0self.assert_qmp(res, 'return[0]/status', 'paused') res = self.vm.qmp('block-job-dismiss', id='drive0') self.assert_qmp(res, 'error/desc',
Re: [PATCH v2 13/20] iotests: 129: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 4 lines · Vladimir Sementsov-Ogievskiy
After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, backup job may finish earlier than final assertion in
do_test_stop. Let's require slow backup explicitly by specifying speed parameter.
Isn’t the problem really that block_set_io_throttle does absolutely nothing? (Which is a long-standing problem with 129. I personally just never run it, honestly.)
Quoted 19 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/129 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/qemu-iotests/129 +1
-1@@ -76,7 +76,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase): def test_drive_backup(self): self.do_test_stop("drive-backup", device="drive0", target=self.target_img, - sync="full") + sync="full", speed=1024) def test_block_commit(self): self.do_test_stop("block-commit", device="drive0")
Re: [PATCH v2 14/20] iotests: 185: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 1 line · Vladimir Sementsov-Ogievskiy
The further change of moving backup to be a on block-copy call will
-on?
Quoted 1 line
make copying chunk-size and cluster-size a separate things. So, even
s/a/two/
Quoted 8 lines
with 64k cluster sized qcow2 image, default chunk would be 1M. 185 test however assumes, that with speed limited to 64K, one iteration would result in offset=64K. It will change, as first iteration would result in offset=1M independently of speed.
So, let's explicitly specify, what test wants: set max-chunk to 64K, so that one iteration is 64K. Note, that we don't need to limit max-workers, as block-copy rate limitator will handle the situation and
*limitator
Quoted 13 lines
wouldn't start new workers when speed limit is obviously reached.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/185 | 3 ++- tests/qemu-iotests/185.out | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)tests/qemu-iotests/185 +0
-0
tests/qemu-iotests/185 _send_qemu_cmd $h \ · line 186
Quoted 6 lines
tests/qemu-iotests/185 +2
-1@@ -182,7 +182,8 @@ _send_qemu_cmd $h \ 'target': '$TEST_IMG.copy', 'format': '$IMGFMT', 'sync': 'full', - 'speed': 65536 } }" \ + 'speed': 65536, + 'x-max-chunk': 65536 } }" \
Out of curiosity, would it also suffice to disable copy offloading?
But anyway:
Quoted 17 lines
tests/qemu-iotests/185 +0
-0"return" # If we don't sleep here 'quit' command races with disk I/Otests/qemu-iotests/185.out +1
-1@@ -61,7 +61,7 @@ Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 size=67108864 cluster_size=65536 l { 'execute': 'qmp_capabilities' } {"return": {}} -{ 'execute': 'drive-backup', 'arguments': { 'device': 'disk', 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', 'speed': 65536 } } +{ 'execute': 'drive-backup', 'arguments': { 'device': 'disk', 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', 'speed': 65536, 'x-max-chunk': 65536 } } Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "disk"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "disk"}}
Re: [PATCH v2 15/20] iotests: 219: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 1 line · Vladimir Sementsov-Ogievskiy
The further change of moving backup to be a on block-copy call will
-on?
Quoted 1 line
make copying chunk-size and cluster-size a separate things. So, even
s/a/two/
Quoted 20 lines
with 64k cluster sized qcow2 image, default chunk would be 1M. Test 219 depends on specified chunk-size. Update it for explicit chunk-size for backup as for mirror.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/219 | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)tests/qemu-iotests/219 +1
-1@@ -203,13 +203,13 @@ with iotests.FilePath('disk.img') as disk_path, \ # but related to this also automatic state transitions like job # completion), but still get pause points often enough to avoid making this # test very slow, it's important to have the right ratio between speed and - # buf_size. + # copy-chunk-size. #
tests/qemu-iotests/219 with iotests.FilePath('disk.img') as disk_path, \ · line 212
Quoted 10 lines
tests/qemu-iotests/219 +5
-5- # For backup, buf_size is hard-coded to the source image cluster size (64k), - # so we'll pick the same for mirror. The slice time, i.e. the granularity - # of the rate limiting is 100ms. With a speed of 256k per second, we can - # get four pause points per second. This gives us 250ms per iteration, - # which should be enough to stay deterministic. + # Chose 64k copy-chunk-size both for mirror (by buf_size) and backup (by + # x-max-chunk). The slice time, i.e. the granularity of the rate limiting + # is 100ms. With a speed of 256k per second, we can get four pause points + # per second. This gives us 250ms per iteration, which should be enough to + # stay deterministic.
Don’t we also have to limit the number of workers to 1 so we actually keep 250 ms per iteration instead of just finishing four requests immediately, then pausing for a second?
Quoted 11 lines
tests/qemu-iotests/219 +1
-0test_job_lifecycle(vm, 'drive-mirror', has_ready=True, job_args={ 'device': 'drive0-node',@@ -226,6 +226,7 @@ with iotests.FilePath('disk.img') as disk_path, \ 'target': copy_path, 'sync': 'full', 'speed': 262144, + 'x-max-chunk': 65536, 'auto-finalize': auto_finalize, 'auto-dismiss': auto_dismiss, })
Re: [PATCH v2 16/20] iotests: 257: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 7 lines · Vladimir Sementsov-Ogievskiy
Iotest 257 dumps a lot of in-progress information of backup job, such as offset and bitmap dirtiness. Further commit will move backup to be one block-copy call, which will introduce async parallel requests instead of plain cluster-by-cluster copying. To keep things deterministic, allow only one worker (only one copy request at a time) for this test.
Quoted 5 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/257 | 1 + tests/qemu-iotests/257.out | 306 ++++++++++++++++++------------------- 2 files changed, 154 insertions(+), 153 deletions(-)
It’s a shame that we don’t run this test with the default configuration then, but I suppose that’s how it is. For now, at least.
Re: [PATCH v2 17/20] backup: move to block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 9 lines · Vladimir Sementsov-Ogievskiy
This brings async request handling and block-status driven chunk sizes to backup out of the box, which improves backup performance.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 9 +-- block/backup.c | 145 +++++++++++++++++++------------------ block/block-copy.c | 21 ++---- 3 files changed, 83 insertions(+), 92 deletions(-)
This patch feels like it should be multiple ones. I don’t see why a patch that lets backup use block-copy (more) should have to modify the block-copy code.
More specifically: I think that block_copy_set_progress_callback() should be removed in a separate patch. Also, moving @cb_opaque/@progress_opaque from BlockCopyState into BlockCopyCallState seems like a separate patch to me, too.
(I presume if the cb had had its own opaque object from patch 5 on, there wouldn’t be this problem now. We’d just stop using the progress callback in backup, and remove it in one separate patch.)
[...]
Quoted 6 lines
block/backup.c +0
-0@@ -44,42 +44,19 @@ typedef struct BackupBlockJob {BlockdevOnError on_source_error;
block/backup.c typedef struct BackupBlockJob { · line 53
Quoted 10 lines
block/backup.c +2
-1BlockdevOnError on_target_error;uint64_t len; -uint64_t bytes_read;int64_t cluster_size;int max_workers;int64_t max_chunk;BlockCopyState *bcs; + +BlockCopyCallState *bcs_call;
Can this be more descriptive? E.g. background_bcs? bg_bcs_call? bg_bcsc?
Quoted 6 lines
block/backup.c +2
-0+int ret; +bool error_is_read; } BackupBlockJob;static const BlockJobDriver backup_job_driver;
[...]
Quoted 11 lines
static int
coroutine_fnbackup_loop(BackupBlockJob *job){block/backup.c +5
-4-bool error_is_read; -int64_t offset; -BdrvDirtyBitmapIter *bdbi; -int ret =0 ; +while (true) {/* retry loop */ +assert (!job->bcs_call); + job->bcs_call = block_copy_async(job->bcs, 0, + QEMU_ALIGN_UP(job->len, + job->cluster_size),
block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 161
Quoted 10 lines
block/backup.c +4
-5+ true, job->max_workers, job->max_chunk, + backup_block_copy_callback, job); - bdbi =bdrv_dirty_iter_new (block_copy_dirty_bitmap (job->bcs)); -while ((offset =bdrv_dirty_iter_next (bdbi)) !=-1 ) { - do { - if (yield_and_check(job)) { - goto out; +while (job->bcs_call && !job->common.job.cancelled) { +/* wait and handle pauses */
Doesn’t someone need to reset BlockCopyCallState.cancelled when the job is unpaused? I can’t see anyone doing that.
Well, or even just reentering the block-copy operation after backup_pause() has cancelled it. Is there some magic I’m missing?
Does pausing (which leads to cancelling the block-copy operation) just yield to the CB being invoked, so the copy operation is considered done, and then we just enter the next iteration of the loop and try again? But cancelling the block-copy operation leads to it returning 0, AFAIR, so...
Quoted 6 lines
block/backup.c +5
-0+ +job_pause_point (&job->common.job); + +if (job->bcs_call && !job->common.job.cancelled) { +job_yield (&job->common.job); }
block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 172
Quoted 10 lines
block/backup.c +5
-5- ret =backup_do_cow (job, offset, job->cluster_size, &error_is_read); -if (ret <0 &&backup_error_action (job, error_is_read, -ret) == -BLOCK_ERROR_ACTION_REPORT ) - { - goto out; + } + +if (!job->bcs_call && job->ret ==0 ) { +/* Success */ +return 0 ;
...I would assume we return here when the job is paused.
block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 182
Quoted 11 lines
block/backup.c +9
-1+ } + +if (job->common.job.cancelled) { +if (job->bcs_call) { +block_copy_cancel (job->bcs_call); } - }while (ret <0 ); +return 0 ; + } + + if (!job->bcs_call && job->ret < 0 &&
Is it even possible for bcs_call to be non-NULL here?
block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 187
Quoted 5 lines
block/backup.c +5
-0+ (backup_error_action(job, job->error_is_read, -job->ret) == + BLOCK_ERROR_ACTION_REPORT)) + { +return job->ret; + }
So if we get an error, and the error action isn’t REPORT, we just try the whole operation again? That doesn’t sound very IGNORE-y to me.
Quoted 10 lines
}- out:
block/backup.c +1
-2-bdrv_dirty_iter_free (bdbi); -return ret; +g_assert_not_reached (); }static void backup_init_bcs_bitmap (BackupBlockJob *job)
block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 231
Quoted 8 lines
block/backup.c +2
-3@@ -246,9 +227,14 @@ static int coroutine_fn backup_run(Job *job, Error **errp)int64_t count;for (offset =0 ; offset < s->len; ) { -if (yield_and_check (s)) { - ret = -ECANCELED ; - goto out; +if (job_is_cancelled (job)) { +return -ECANCELED ;
I’d either drop the out label altogether, or use it here. It’s a bit weird to use it sometimes, but not all the time.
Quoted 18 lines
block/backup.c +11
-0+ } + +job_pause_point (job); + +if (job_is_cancelled (job)) { +return -ECANCELED ; } ret =block_copy_reset_unallocated (s->bcs, offset, &count);@@ -281,6 +267,25 @@ static int coroutine_fn backup_run(Job *job, Error **errp)return ret; } +static void coroutine_fn backup_pause(Job *job) +{ +BackupBlockJob *s =container_of (job, BackupBlockJob, common.job); + +if (s->bcs_call) {
block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 284
Quoted 10 lines
block/backup.c +10
-0+block_copy_cancel (s->bcs_call); + } +} + +static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) +{ +BackupBlockJob *s =container_of (job, BackupBlockJob, common); + +if (s->bcs) { +/* In block_job_create we yet don't have bcs */
Shouldn’t hurt to make it conditional, but how can we get here in block_job_create()?
Quoted 7 lines
block/backup.c +4
-0+block_copy_set_speed (s->bcs, s->bcs_call, speed); + } +} +static const BlockJobDriver backup_job_driver = { .job_driver = { .instance_size =sizeof (BackupBlockJob),
Re: [PATCH v2 18/20] block/block-copy: drop unused argument of block_copy()
In reply to Vladimir Sementsov-OgievskiyQuoted 6 lines · Vladimir Sementsov-Ogievskiy
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 3 +-- block/backup-top.c | 2 +- block/block-copy.c | 11 ++--------- 3 files changed, 4 insertions(+), 12 deletions(-)
Re: [PATCH v2 19/20] simplebench: bench_block_job: add cmd_options argument
In reply to Vladimir Sementsov-OgievskiyQuoted 7 lines · Vladimir Sementsov-Ogievskiy
Add argument to allow additional block-job options.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- scripts/simplebench/bench-example.py | 2 +- scripts/simplebench/bench_block_job.py | 13 ++++++++----- 2 files changed, 9 insertions(+), 6 deletions(-)
[...]
scripts/simplebench/bench_block_job.py line 1
Quoted 7 lines
scripts/simplebench/bench_block_job.py +1
-1@@ -1,4 +1,4 @@ -#!/usr/bin/env python +#!/usr/bin/env python3
Looks a bit unrelated. Apart from that:
Re: [PATCH v2 20/20] simplebench: add bench-backup.py
In reply to Vladimir Sementsov-OgievskiyQuoted 7 lines · Vladimir Sementsov-Ogievskiy
Add script to benchmark new backup architecture.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- scripts/simplebench/bench-backup.py | 132 ++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100755 scripts/simplebench/bench-backup.py
Looks reasonable. Not sure whether I can give an R-b, given my non-existing background on the simplebench scripts, and also that it’s probably not really necessary.
The only thing that sprung to my eye is that you give x-use-copy-range even when the “new” label isn’t given, which may be a problem. (Because AFAIU, the “new” label means that qemu is sufficiently new to support x-max-workers; and that’s basically equivalent to supporting x-use-copy-range.)
Max
Re: [PATCH v2 04/20] block/block-copy: More explicit call_state
In reply to Max ReitzQuoted 28 lines · Max Reitz
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
Refactor common path to use BlockCopyCallState pointer as parameter, to prepare it for use in asynchronous block-copy (at least, we'll need to run block-copy in a coroutine, passing the whole parameters as one pointer).
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- block/block-copy.c | 51 ++++++++++++++++++++++++++++++++++------------ 1 file changed, 38 insertions(+), 13 deletions(-)block/block-copy.c +0
-0[...]
Quoted 10 lines
block/block-copy.c +1
-2@@ -646,16 +653,16 @@ out:* it means that some I/O operation failed in context of _this_ block_copy call, * not some parallel operation. */ -int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, -bool *error_is_read) +static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) {int ret;
Quoted 9 lines
block/block-copy.c out: · line 661
Quoted 3 lines
block/block-copy.c +1
-1do { - ret = block_copy_dirty_clusters(s, offset, bytes, error_is_read); + ret = block_copy_dirty_clusters(call_state);It’s possible that much of this code will change in a future patch of this series, but as it is, it might be nice to make block_copy_dirty_clusters’s argument a const pointer so it’s clear that the call to
block_copy_wait_one()below will use the original @offset and @bytes values.
Hm. I'm trying this, and it doesn't work:
block_copy_task_entry() wants to change call_state:
t->call_state->failed = true;
Quoted 14 lines
*shrug*
Reviewed-by Max Reitz <[email protected]>Quoted 8 lines
block/block-copy.c +2
-1if (ret ==0 ) { - ret =block_copy_wait_one (s, offset, bytes); + ret =block_copy_wait_one (call_state->s, call_state->offset, + call_state->bytes); }/*
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 04/20] block/block-copy: More explicit call_state
Quoted 42 lines · Vladimir Sementsov-Ogievskiy
Quoted 32 lines · Max Reitz
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
Refactor common path to use BlockCopyCallState pointer as parameter, to prepare it for use in asynchronous block-copy (at least, we'll need to run block-copy in a coroutine, passing the whole parameters as one pointer).
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
block/block-copy.c| 51 ++++++++++++++++++++++++++++++++++------------1 file changed, 38 insertions(+), 13 deletions(-)block/block-copy.c +0
-0[...]
Quoted 14 lines
block/block-copy.c +1
-3@@ -646,16 +653,16 @@ out:* it means that some I/O operation failed in context of _this_ block_copy call, * not some parallel operation. */ -int coroutine_fn block_copy(BlockCopyState *s, int64_t offset, int64_t bytes, - bool *error_is_read) +static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) { int ret; do { - ret = block_copy_dirty_clusters(s, offset, bytes,Quoted 8 lines
block/block-copy.c out: · line 664
Quoted 2 lines
block/block-copy.c +1
-0error_is_read); + ret = block_copy_dirty_clusters(call_state);It’s possible that much of this code will change in a future patch of this series, but as it is, it might be nice to make block_copy_dirty_clusters’s argument a const pointer so it’s clear that the call to
block_copy_wait_one()below will use the original @offset and @bytes values.
Quoted 5 lines
Hm. I'm trying this, and it doesn't work:
block_copy_task_entry()wants to changecall_state:
t->call_state->failed= true;
Too bad :)
Quoted 14 lines
Quoted 12 lines
*shrug*
Reviewed-by Max Reitz <[email protected]>Quoted 7 lines
block/block-copy.c +2
-1if (ret == 0) { - ret = block_copy_wait_one(s, offset, bytes); + ret = block_copy_wait_one(call_state->s, call_state->offset, + call_state->bytes);}
/*
Re: [PATCH v2 17/20] backup: move to block-copy
In reply to Max ReitzQuoted 79 lines · Max Reitz
Quoted 9 lines · Vladimir Sementsov-Ogievskiy
This brings async request handling and block-status driven chunk sizes to backup out of the box, which improves backup performance.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 9 +-- block/backup.c | 145 +++++++++++++++++++------------------ block/block-copy.c | 21 ++---- 3 files changed, 83 insertions(+), 92 deletions(-)This patch feels like it should be multiple ones. I don’t see why a patch that lets backup use block-copy (more) should have to modify the block-copy code.
More specifically: I think that
block_copy_set_progress_callback()should be removed in a separate patch. Also, moving @cb_opaque/@progress_opaque from BlockCopyState into BlockCopyCallState seems like a separate patch to me, too.(I presume if the cb had had its own opaque object from patch 5 on, there wouldn’t be this problem now. We’d just stop using the progress callback in backup, and remove it in one separate patch.)
[...]
Quoted 6 lines
block/backup.c +0
-0@@ -44,42 +44,19 @@ typedef struct BackupBlockJob {BlockdevOnError on_source_error;block/backup.c typedef struct BackupBlockJob { · line 53
Quoted 10 lines
block/backup.c +2
-1BlockdevOnError on_target_error;uint64_t len; -uint64_t bytes_read;int64_t cluster_size;int max_workers;int64_t max_chunk;BlockCopyState *bcs; + +BlockCopyCallState *bcs_call;Can this be more descriptive? E.g.
background_bcs?bg_bcs_call?bg_bcsc?Quoted 6 lines
block/backup.c +2
-0+int ret; +bool error_is_read; } BackupBlockJob;static const BlockJobDriver backup_job_driver;[...]
Quoted 11 lines
static int coroutine_fn backup_loop(BackupBlockJob *job) {block/backup.c +5
-4-bool error_is_read; -int64_t offset; -BdrvDirtyBitmapIter *bdbi; -int ret =0 ; +while (true) {/* retry loop */ +assert (!job->bcs_call); + job->bcs_call = block_copy_async(job->bcs, 0, + QEMU_ALIGN_UP(job->len, + job->cluster_size),block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 161
Quoted 10 lines
block/backup.c +4
-5+ true, job->max_workers, job->max_chunk, + backup_block_copy_callback, job); - bdbi =bdrv_dirty_iter_new (block_copy_dirty_bitmap (job->bcs)); -while ((offset =bdrv_dirty_iter_next (bdbi)) !=-1 ) { - do { - if (yield_and_check(job)) { - goto out; +while (job->bcs_call && !job->common.job.cancelled) { +/* wait and handle pauses */ Doesn’t someone need to reset BlockCopyCallState.cancelled when the job is unpaused? I can’t see anyone doing that.
Quoted 6 lines
Well, or even just reentering the block-copy operation after
backup_pause()has cancelled it. Is there some magic I’m missing?Does pausing (which leads to cancelling the block-copy operation) just yield to the CB being invoked, so the copy operation is considered done, and then we just enter the next iteration of the loop and try again?
Yes, that's my idea: we cancel on pause and just run new block_copy operation on resume.
Quoted 2 lines
But cancelling the block-copy operation leads to it returning 0, AFAIR, so...
Looks like it should be fixed. By returning ECANCELED or by checking the bitmap.
Quoted 34 lines
Quoted 7 lines
block/backup.c +5
-0+ +job_pause_point (&job->common.job); + +if (job->bcs_call && !job->common.job.cancelled) { +job_yield (&job->common.job); }block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 172
Quoted 10 lines
block/backup.c +5
-5- ret =backup_do_cow (job, offset, job->cluster_size, &error_is_read); -if (ret <0 &&backup_error_action (job, error_is_read, -ret) == -BLOCK_ERROR_ACTION_REPORT ) - { - goto out; + } + +if (!job->bcs_call && job->ret ==0 ) { +/* Success */ +return 0 ;...I would assume we return here when the job is paused.
block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 182
Quoted 11 lines
block/backup.c +9
-1+ } + +if (job->common.job.cancelled) { +if (job->bcs_call) { +block_copy_cancel (job->bcs_call); } - }while (ret <0 ); +return 0 ; + } + + if (!job->bcs_call && job->ret < 0 &&Is it even possible for
bcs_callto be non-NULL here?
Quoted 8 lines
block/backup.c static BlockErrorAction backup_error_action(BackupBlockJob *job, · line 187
Quoted 5 lines
block/backup.c +5
-0+ (backup_error_action(job, job->error_is_read, -job->ret) == + BLOCK_ERROR_ACTION_REPORT)) + { +return job->ret; + }So if we get an error, and the error action isn’t REPORT, we just try the whole operation again? That doesn’t sound very IGNORE-y to me.
Not the whole. We have the dirty bitmap: clusters that was already copied are not touched more.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 07/20] block/block-copy: add ratelimit to block-copy
In reply to Max ReitzQuoted 154 lines · Max Reitz
Quoted 2 lines · Vladimir Sementsov-Ogievskiy
We are going to directly use one async block-copy operation for backup job, so we need rate limitator.
%s/limitator/limiter/g, I think.
Quoted 5 lines
We want to maintain current backup behavior: only background copying is limited and copy-before-write operations only participate in limit calculation. Therefore we need one rate limitator for block-copy state and boolean flag for block-copy call state for actual limitation.
Quoted 10 lines
Note, that we can't just calculate each chunk in limitator after successful copying: it will not save us from starting a lot of async sub-requests which will exceed limit too much. Instead let's use the following scheme on sub-request creation:
1. If at the moment limit is not exceeded, create the request and account it immediately.
2. If at the moment limit is already exceeded, drop create sub-request and handle limit instead (by sleep).
With this approach we'll never exceed the limit more than by one sub-request (which pretty much matches current backup behavior).Sounds reasonable.
Quoted 70 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 8 +++++++ block/block-copy.c | 44 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+)include/block/block-copy.h +8
-0@@ -59,6 +59,14 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,int64_t max_chunk, BlockCopyAsyncCallbackFunc cb); +/* +* Set speed limit for block-copy instance. All block-copy operations related to +* this BlockCopyState will participate in speed calculation, but only +* block_copy_async calls with @ratelimit=true will be actually limited. +*/ +void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state, +uint64_t speed); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);block/block-copy.c +11
-0@@ -26,6 +26,7 @@#define BLOCK_COPY_MAX_BUFFER (1 * MiB)#define BLOCK_COPY_MAX_MEM (128 * MiB)#define BLOCK_COPY_MAX_WORKERS 64 +#define BLOCK_COPY_SLICE_TIME 100000000ULL/* ns */ static coroutine_fn int block_copy_task_entry(AioTask *task);@@ -36,11 +37,13 @@ typedef struct BlockCopyCallState {int64_t bytes;int max_workers;int64_t max_chunk; +bool ratelimit;BlockCopyAsyncCallbackFunc cb;/* State */ bool failed;bool finished; +QemuCoSleepState *sleep_state;/* OUT parameters */ bool error_is_read;@@ -103,6 +106,9 @@ typedef struct BlockCopyState {void *progress_opaque;SharedResource *mem; + +uint64_t speed; +RateLimit rate_limit; } BlockCopyState; static BlockCopyTask *find_conflicting_task(BlockCopyState *s,@@ -611,6 +617,21 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) } task->zeroes = ret &BDRV_BLOCK_ZERO ; +if (s->speed) { +if (call_state->ratelimit) { +uint64_t ns =ratelimit_calculate_delay (&s->rate_limit,0 ); +if (ns >0 ) { +block_copy_task_end (task, -EAGAIN );block/block-copy.c block_copy_dirty_clusters(BlockCopyCallState *call_state) · line 634
Quoted 10 lines
block/block-copy.c +10
-0+g_free (task); +qemu_co_sleep_ns_wakeable (QEMU_CLOCK_REALTIME , ns, + &call_state->sleep_state); +continue ; + } + } + +ratelimit_calculate_delay (&s->rate_limit, task->bytes); + } +Looks good.
Quoted 18 lines
block/block-copy.c +7
-0trace_block_copy_process (s, task->offset);co_get_from_shres (s->mem, task->bytes);@@ -649,6 +670,13 @@ out:return ret <0 ? ret : found_dirty; } +static void block_copy_kick (BlockCopyCallState *call_state) +{ +if (call_state->sleep_state) { +qemu_co_sleep_wake (call_state->sleep_state); + } +} +/* * block_copy_common * block/block-copy.c BlockCopyCallState *block_copy_async(BlockCopyState *s, · line 760
Quoted 4 lines
block/block-copy.c +1
-0@@ -729,6 +757,7 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, .s = s, .offset = offset, .bytes = bytes, + .ratelimit = ratelimit,Hm, same problem/question as in patch 6: Should the @ratelimit parameter really be added in patch 5 if it’s used only now?
Quoted 21 lines
block/block-copy.c +14
-0.cb = cb, .max_workers = max_workers ?: BLOCK_COPY_MAX_WORKERS, .max_chunk = max_chunk,@@ -752,3 +781,18 @@ void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip) { s->skip_unallocated = skip; } + +void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state, +uint64_t speed) +{ +uint64_t old_speed = s->speed; + + s->speed = speed; +if (speed >0 ) { +ratelimit_set_speed (&s->rate_limit, speed,BLOCK_COPY_SLICE_TIME ); + } + +if (call_state && old_speed && (speed > old_speed || speed ==0 )) { +block_copy_kick (call_state); + }
Quoted 9 lines
block/block-copy.c void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip) · line 798
Quoted 1 line
block/block-copy.c +1
-0+}Hm. I’m interested in seeing how this is going to be used, i.e. what callers will pass for @call_state. I suppose it’s going to be the background operation for the whole device, but I wonder whether it actually makes sense to pass it. I mean, the caller could just call
block_copy_kick()itself (unconditionally, because it’ll never hurt, I think).
Yes, that's weird. I think the correct approach is to have list of call-states in block-copy-state, and kick them all on speed update.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 11/20] qapi: backup: add x-max-chunk and x-max-workers parameters
In reply to Max ReitzQuoted 19 lines · Max Reitz
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 9 ++++++++- include/block/block_int.h | 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-)
[..]
Quoted 9 lines
block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 440
Quoted 6 lines
block/backup.c +2
-0@@ -422,6 +436,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs,if (cluster_size <0 ) { goto error; } +if (max_chunk && max_chunk < cluster_size) { + error_setg(errp, "Required max-chunk (%" PRIi64") is less than backup "(missing a space after PRIi64)
Quoted 3 lines
block/backup.c BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, · line 441
Quoted 1 line
block/backup.c +1
-0+ "cluster size (%" PRIi64 ")", max_chunk, cluster_size);Should this be noted in the QAPI documentation?
Hmm.. It makes sense, but I don't know what to write: should be >= job cluster_size? But then I'll have to describe the logic of backup_calculate_cluster_size()...
Quoted 3 lines
(And perhaps the fact
that without copy offloading, we’ll never copy anything bigger than one cluster at a time anyway?)
This is a parameter for background copying. Look at block_copy_task_create(), if call_state has own max_chunk, it's used instead of common copy_size (derived from cluster_size). But at a moment of this patch background process through async block-copy is not yet implemented, and the parameter doesn't work, which is described in commit message.
Quoted 5 lines
Quoted 5 lines
block/backup.c +2
-0+return NULL ; + }/*
[..]
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 08/20] block/block-copy: add block_copy_cancel
In reply to Max ReitzQuoted 54 lines · Max Reitz
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Add function to cancel running async block-copy call. It will be used in backup.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- include/block/block-copy.h | 7 +++++++ block/block-copy.c | 22 +++++++++++++++++++--- 2 files changed, 26 insertions(+), 3 deletions(-)include/block/block-copy.h +7
-0@@ -67,6 +67,13 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,void block_copy_set_speed (BlockCopyState *s,BlockCopyCallState *call_state,uint64_t speed); +/* +* Cancel running block-copy call. +* Cancel leaves block-copy state valid: dirty bits are correct and you may use +* cancel + <run block_copy with same parameters> to emulate pause/resume. +*/ +void block_copy_cancel (BlockCopyCallState *call_state); +BdrvDirtyBitmap *block_copy_dirty_bitmap (BlockCopyState *s);void block_copy_set_skip_unallocated (BlockCopyState *s,bool skip);block/block-copy.c +3
-1@@ -44,6 +44,8 @@ typedef struct BlockCopyCallState {bool failed;bool finished;QemuCoSleepState *sleep_state; +bool cancelled; +Coroutine *canceller;/* OUT parameters */ bool error_is_read;@@ -582,7 +584,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state)assert (QEMU_IS_ALIGNED (offset, s->cluster_size));assert (QEMU_IS_ALIGNED (bytes, s->cluster_size)); -while (bytes &&aio_task_pool_status (aio) ==0 ) { +while (bytes &&aio_task_pool_status (aio) ==0 && !call_state->cancelled) {BlockCopyTask *task;int64_t status_bytes;@@ -693,7 +695,7 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) do { ret = block_copy_dirty_clusters(call_state);
Quoted 16 lines
block/block-copy.c static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) · line 712
Quoted 11 lines
block/block-copy.c +2
-2-if (ret ==0 ) { +if (ret ==0 && !call_state->cancelled) { ret =block_copy_wait_one (call_state->s, call_state->offset, call_state->bytes); }@@ -707,13 +709,18 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state)* 2. We have waited for some intersecting block-copy request * It may have failed and produced new dirty bits. */ - }while (ret >0 ); + }while (ret >0 && !call_state->cancelled);Would it be cleaner if
block_copy_dirty_cluster()just returned-ECANCELED? Or would that pose a problem for its callers or the async callback?
I'd prefer not to merge io ret with block-copy logic: who knows what underlying operations may return.. Can't it be _another_ ECANCELED? And it would be just a sugar for block_copy_dirty_clusters() call, I'll have to check ->cancelled after block_copy_wait_one() anyway. Also, for the next version I try to make it more obvious that finished block-copy call is in one of thee states:
- success
- failed
- cancelled
Hmm. Also, cancelled should be OK for copy-on-write operations in filter, it just mean that we don't need to care anymore.
Quoted 14 lines
Quoted 13 lines
block/block-copy.c +5
-0if (call_state->cb) { call_state->cb (ret, call_state->error_is_read, call_state->s->progress_opaque); } +if (call_state->canceller) { +aio_co_wake (call_state->canceller); + call_state->canceller =NULL ; + } + call_state->finished = true;return ret;
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 13/20] iotests: 129: prepare for backup over block-copy
In reply to Max ReitzQuoted 9 lines · Max Reitz
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, backup job may finish earlier than final assertion in
do_test_stop. Let's require slow backup explicitly by specifying speed parameter.Isn’t the problem really that
block_set_io_throttledoes absolutely nothing? (Which is a long-standing problem with 129. I personally just never run it, honestly.)
Hmm.. is it better to drop test_drive_backup() from here ?
Quoted 22 lines
Quoted 20 lines
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/129 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)tests/qemu-iotests/129 +1
-1@@ -76,7 +76,7 @@ class TestStopWithBlockJob(iotests.QMPTestCase): def test_drive_backup(self): self.do_test_stop("drive-backup", device="drive0", target=self.target_img, - sync="full") + sync="full", speed=1024) def test_block_commit(self): self.do_test_stop("block-commit", device="drive0")
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 14/20] iotests: 185: prepare for backup over block-copy
In reply to Max ReitzQuoted 4 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
The further change of moving backup to be a on block-copy call will
-on?
one :)
Quoted 24 lines
Quoted 2 lines
make copying chunk-size and cluster-size a separate things. So, even
s/a/two/
Quoted 8 lines
with 64k cluster sized qcow2 image, default chunk would be 1M. 185 test however assumes, that with speed limited to 64K, one iteration would result in offset=64K. It will change, as first iteration would result in offset=1M independently of speed.
So, let's explicitly specify, what test wants: set max-chunk to 64K, so that one iteration is 64K. Note, that we don't need to limit max-workers, as block-copy rate limitator will handle the situation and
*limitator
Quoted 8 lines
wouldn't start new workers when speed limit is obviously reached.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/185 | 3 ++- tests/qemu-iotests/185.out | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
Quoted 13 lines
tests/qemu-iotests/185 _send_qemu_cmd $h \ · line 186
Quoted 11 lines
tests/qemu-iotests/185 +2
-1@@ -182,7 +182,8 @@ _send_qemu_cmd $h \ 'target': '$TEST_IMG.copy', 'format': '$IMGFMT', 'sync': 'full', - 'speed': 65536 } }" \ + 'speed': 65536, + 'x-max-chunk': 65536 } }" \Out of curiosity, would it also suffice to disable copy offloading?
Note that x-max-chunk works even with copy offloading enabled, it sets maximum only for background copying, not for all operations.
Quoted 24 lines
But anyway:
Reviewed-by Max Reitz <[email protected]>Quoted 17 lines
tests/qemu-iotests/185 +0
-0"return" # If we don't sleep here 'quit' command races with disk I/Otests/qemu-iotests/185.out +1
-1@@ -61,7 +61,7 @@ Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 size=67108864 cluster_size=65536 l { 'execute': 'qmp_capabilities' } {"return": {}} -{ 'execute': 'drive-backup', 'arguments': { 'device': 'disk', 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', 'speed': 65536 } } +{ 'execute': 'drive-backup', 'arguments': { 'device': 'disk', 'target': 'TEST_DIR/t.IMGFMT.copy', 'format': 'IMGFMT', 'sync': 'full', 'speed': 65536, 'x-max-chunk': 65536 } } Formatting 'TEST_DIR/t.qcow2.copy', fmt=qcow2 size=67108864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "created", "id": "disk"}} {"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "JOB_STATUS_CHANGE", "data": {"status": "running", "id": "disk"}}
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 15/20] iotests: 219: prepare for backup over block-copy
In reply to Max ReitzQuoted 35 lines · Max Reitz
Quoted 1 line · Vladimir Sementsov-Ogievskiy
The further change of moving backup to be a on block-copy call will
-on?
Quoted 1 line
make copying chunk-size and cluster-size a separate things. So, even
s/a/two/
Quoted 26 lines
with 64k cluster sized qcow2 image, default chunk would be 1M. Test 219 depends on specified chunk-size. Update it for explicit chunk-size for backup as for mirror.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- tests/qemu-iotests/219 | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)tests/qemu-iotests/219 +2
-6@@ -203,13 +203,13 @@ with iotests.FilePath('disk.img') as disk_path, \ # but related to this also automatic state transitions like job # completion), but still get pause points often enough to avoid making this # test very slow, it's important to have the right ratio between speed and - # buf_size. + # copy-chunk-size. # - # For backup, buf_size is hard-coded to the source image cluster size (64k), - # so we'll pick the same for mirror. The slice time, i.e. the granularity - # of the rate limiting is 100ms. With a speed of 256k per second, we can - # get four pause points per second. This gives us 250ms per iteration, - # which should be enough to stay deterministic. + # Chose 64k copy-chunk-size both for mirror (by buf_size) and backup (by
Quoted 8 lines
tests/qemu-iotests/219 with iotests.FilePath('disk.img') as disk_path, \ · line 212
Quoted 4 lines
tests/qemu-iotests/219 +4
-0+ # x-max-chunk). The slice time, i.e. the granularity of the rate limiting + # is 100ms. With a speed of 256k per second, we can get four pause points + # per second. This gives us 250ms per iteration, which should be enough to + # stay deterministic.Don’t we also have to limit the number of workers to 1 so we actually keep 250 ms per iteration instead of just finishing four requests immediately, then pausing for a second?
Block-copy rate limiter works good enough: it will not start too much requests.
Quoted 14 lines
Quoted 12 lines
tests/qemu-iotests/219 +1
-0test_job_lifecycle(vm, 'drive-mirror', has_ready=True, job_args={ 'device': 'drive0-node',@@ -226,6 +226,7 @@ with iotests.FilePath('disk.img') as disk_path, \ 'target': copy_path, 'sync': 'full', 'speed': 262144, + 'x-max-chunk': 65536, 'auto-finalize': auto_finalize, 'auto-dismiss': auto_dismiss, })
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 08/20] block/block-copy: add block_copy_cancel
In reply to Vladimir Sementsov-OgievskiyQuoted 66 lines · Vladimir Sementsov-Ogievskiy
Quoted 55 lines · Max Reitz
Quoted 54 lines · Vladimir Sementsov-Ogievskiy
Add function to cancel running async block-copy call. It will be used in backup.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
include/block/block-copy.h| 7 +++++++ block/block-copy.c | 22 +++++++++++++++++++---2 files changed, 26 insertions(+), 3 deletions(-)include/block/block-copy.h +7
-0@@ -67,6 +67,13 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s, void block_copy_set_speed(BlockCopyState *s, BlockCopyCallState *call_state, uint64_t speed); +/* +* Cancel running block-copy call. +* Cancel leaves block-copy state valid: dirty bits are correct and you may use +* cancel + <run block_copy with same parameters> to emulate pause/resume. +*/ +void block_copy_cancel (BlockCopyCallState *call_state); + BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s); void block_copy_set_skip_unallocated(BlockCopyState *s, bool skip);block/block-copy.c +4
-2@@ -44,6 +44,8 @@ typedef struct BlockCopyCallState { bool failed; bool finished; QemuCoSleepState *sleep_state; + bool cancelled; + Coroutine *canceller; /* OUT parameters */ bool error_is_read;@@ -582,7 +584,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); - while (bytes && aio_task_pool_status(aio) == 0) { + while (bytes && aio_task_pool_status(aio) == 0 && !call_state->cancelled) { BlockCopyTask *task; int64_t status_bytes;@@ -693,7 +695,7 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) do { ret = block_copy_dirty_clusters(call_state); - if (ret == 0) { + if (ret == 0 && !call_state->cancelled) { ret = block_copy_wait_one(call_state->s, call_state->offset, call_state->bytes); }@@ -707,13 +709,18 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) * 2. We have waited for some intersecting block-copy requestQuoted 9 lines
block/block-copy.c static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) · line 712
Quoted 4 lines
block/block-copy.c +1
-1* It may have failed and produced new dirty bits. */ - } while (ret > 0); + } while (ret > 0 && !call_state->cancelled);Would it be cleaner if
block_copy_dirty_cluster()just returned -ECANCELED? Or would that pose a problem for its callers or the async callback?
Quoted 8 lines
I'd prefer not to merge io ret with block-copy logic: who knows what underlying operations may return.. Can't it be _another_ ECANCELED? And it would be just a sugar for
block_copy_dirty_clusters()call, I'll have to check ->cancelled afterblock_copy_wait_one()anyway. Also, for the next version I try to make it more obvious that finished block-copy call is in one of thee states:
- success
- failed
- cancelledHmm. Also, cancelled should be OK for copy-on-write operations in filter, it just mean that we don't need to care anymore.
This is unrelated: actually only async block-copy call may be cancelled.
Quoted 15 lines
Quoted 13 lines
Quoted 12 lines
block/block-copy.c +5
-0if (call_state->cb) { call_state->cb(ret, call_state->error_is_read, call_state->s->progress_opaque); } + if (call_state->canceller) { + aio_co_wake(call_state->canceller); + call_state->canceller = NULL; + } + call_state->finished = true; return ret;
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 17/20] backup: move to block-copy
In reply to Max ReitzQuoted 9 lines · Max Reitz
block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 284
Quoted 6 lines
block/backup.c +6
-0+static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) +{ +BackupBlockJob *s =container_of (job, BackupBlockJob, common); + +if (s->bcs) { +/* In block_job_create we yet don't have bcs */ Shouldn’t hurt to make it conditional, but how can we get here in
block_job_create()?
block_job_set_speed is called from block_job_create.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 11/20] qapi: backup: add x-max-chunk and x-max-workers parameters
In reply to Vladimir Sementsov-OgievskiyQuoted 34 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines · Max Reitz
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
qapi/block-core.json | 9 ++++++++-include/block/block_int.h| 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++5 files changed, 42 insertions(+), 2 deletions(-)[..]
Quoted 11 lines
Quoted 8 lines
block/backup.c +2
-0@@ -422,6 +436,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, if (cluster_size < 0) { goto error; } + if (max_chunk && max_chunk < cluster_size) { + error_setg(errp, "Required max-chunk (%" PRIi64") is lessthan backup "
(missing a space after PRIi64)
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
+ "cluster size (%" PRIi64 ")",
max_chunk,cluster_size);Should this be noted in the QAPI documentation?
Hmm.. It makes sense, but I don't know what to write: should be >= job
cluster_size? But then I'll have to describe the logic ofbackup_calculate_cluster_size()...
Isn’t the logic basically just “cluster size of the target image file (min 64k)”? The other cases just cover error cases, and they always return 64k, which would effectively be documented by stating that 64k is the minimum.
But in the common cases (qcow2 or raw), we’ll never get an error anyway.
I think it’d be good to describe the cluster size somewhere, because otherwise I find it a bit malicious to throw an error at the user that they could not have anticipated because they have no idea what valid values for the parameter are (until they try).
Quoted 10 lines
Quoted 3 lines
(And perhaps the fact
that without copy offloading, we’ll never copy anything bigger than one cluster at a time anyway?)This is a parameter for background copying. Look at
block_copy_task_create(), ifcall_statehas ownmax_chunk, it's used instead of commoncopy_size(derived fromcluster_size). But at a moment of this patch background process through async block-copy is not yet implemented, and the parameter doesn't work, which is described in commit message.
Ah, OK. Right.
Max
Re: [PATCH v2 08/20] block/block-copy: add block_copy_cancel
In reply to Vladimir Sementsov-OgievskiyQuoted 75 lines · Vladimir Sementsov-Ogievskiy
Quoted 64 lines · Max Reitz
Quoted 63 lines · Vladimir Sementsov-Ogievskiy
Add function to cancel running async block-copy call. It will be used in backup.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>---
include/block/block-copy.h| 7 +++++++ block/block-copy.c | 22 +++++++++++++++++++---2 files changed, 26 insertions(+), 3 deletions(-)include/block/block-copy.h +0
-0@@ -67,6 +67,13 @@ BlockCopyCallState *block_copy_async(BlockCopyState *s,void block_copy_set_speed(BlockCopyState *s, BlockCopyCallState
*call_state,
uint64_tspeed);
+/*
+ * Cancel running block-copy call.
+ * Cancel leaves block-copy state valid: dirty bits are correct and you may use
+ * cancel + <runblock_copywith same parameters> to emulate pause/resume.
+ */include/block/block-copy.h +2
-0+void block_copy_cancel (BlockCopyCallState *call_state); +BdrvDirtyBitmap *block_copy_dirty_bitmap(BlockCopyState *s); void
block_copy_set_skip_unallocated(BlockCopyState *s, bool skip); diff --gita/block/block-copy.cb/block/block-copy.cindex851d9c8aaf..b551feb6c2100644block/block-copy.c +3
-0@@ -44,6 +44,8 @@ typedef struct BlockCopyCallState { bool failed; bool finished; QemuCoSleepState *sleep_state; + bool cancelled; + Coroutine *canceller; /* OUT parameters */ bool error_is_read;@@ -582,7 +584,7 @@ block_copy_dirty_clusters(BlockCopyCallState *call_state) assert(QEMU_IS_ALIGNED(offset, s->cluster_size)); assert(QEMU_IS_ALIGNED(bytes, s->cluster_size)); - while (bytes && aio_task_pool_status(aio) == 0) { + while (bytes && aio_task_pool_status(aio) == 0 && !call_state->cancelled) { BlockCopyTask *task;
int64_tstatus_bytes;
@@ -693,7 +695,7 @@ static intcoroutine_fnblock_copy_common(BlockCopyCallState *call_state)do {
ret =block_copy_dirty_clusters(call_state); - if (ret == 0) {
+ if (ret == 0 && !call_state->cancelled) { ret = block_copy_wait_one(call_state->s,call_state->offset,
call_state->bytes); }block/block-copy.c +0
-0@@ -707,13 +709,18 @@ static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) * 2. We have waited for some intersecting block-copy requestQuoted 9 lines
block/block-copy.c static int coroutine_fn block_copy_common(BlockCopyCallState *call_state) · line 712
Quoted 4 lines
block/block-copy.c +1
-1* It may have failed and produced new dirty bits. */ - } while (ret > 0); + } while (ret > 0 && !call_state->cancelled);Would it be cleaner if
block_copy_dirty_cluster()just returned -ECANCELED? Or would that pose a problem for its callers or the async callback?
Quoted 9 lines
I'd prefer not to merge io ret with block-copy logic: who knows what underlying operations may return.. Can't it be _another_ ECANCELED? And it would be just a sugar for
block_copy_dirty_clusters()call, I'll have to check ->cancelled afterblock_copy_wait_one()anyway. Also, for the next version I try to make it more obvious that finished block-copy call is in one of thee states:
- success
- failed
- cancelled
OK.
Max
Re: [PATCH v2 13/20] iotests: 129: prepare for backup over block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 7 lines · Vladimir Sementsov-Ogievskiy
Quoted 6 lines · Max Reitz
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, backup job may finish earlier than final assertion in
do_test_stop. Let's require slow backup explicitly by specifying speed parameter.
Quoted 5 lines
Quoted 3 lines
Isn’t the problem really that
block_set_io_throttledoes absolutely nothing? (Which is a long-standing problem with 129. I personally just never run it, honestly.)Hmm.. is it better to drop
test_drive_backup()from here ?
I think the best would be to revisit this:
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html
Max
Re: [PATCH v2 17/20] backup: move to block-copy
In reply to Vladimir Sementsov-OgievskiyQuoted 11 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Max Reitz
block/backup.c static int coroutine_fn backup_run(Job *job, Error **errp) · line 284
Quoted 6 lines
block/backup.c +6
-0+static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) +{ + BackupBlockJob *s = container_of(job, BackupBlockJob, common); + + if (s->bcs) { + /* In block_job_create we yet don't have bcs */Shouldn’t hurt to make it conditional, but how can we get here in
block_job_create()?
Quoted 1 line
block_job_set_speedis called fromblock_job_create.
Ah, right.
Max
Re: [PATCH v2 11/20] qapi: backup: add x-max-chunk and x-max-workers parameters
In reply to Max ReitzQuoted 49 lines · Max Reitz
Quoted 34 lines · Vladimir Sementsov-Ogievskiy
Quoted 19 lines · Max Reitz
Quoted 18 lines · Vladimir Sementsov-Ogievskiy
Add new parameters to configure future backup features. The patch doesn't introduce aio backup requests (so we actually have only one worker) neither requests larger than one cluster. Still, formally we satisfy these maximums anyway, so add the parameters now, to facilitate further patch which will really change backup job behavior.
Options are added with x- prefixes, as the only use for them are some very conservative iotests which will be updated soon.
Signed-off-by Vladimir Sementsov-Ogievskiy <[email protected]>--- qapi/block-core.json | 9 ++++++++- include/block/block_int.h | 7 +++++++ block/backup.c | 21 +++++++++++++++++++++ block/replication.c | 2 +- blockdev.c | 5 +++++ 5 files changed, 42 insertions(+), 2 deletions(-)[..]
Quoted 11 lines
Quoted 8 lines
block/backup.c +2
-0@@ -422,6 +436,11 @@ BlockJob *backup_job_create(const char *job_id, BlockDriverState *bs, if (cluster_size < 0) { goto error; } + if (max_chunk && max_chunk < cluster_size) { + error_setg(errp, "Required max-chunk (%" PRIi64") is lessthan backup "
(missing a space after PRIi64)
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
+ "cluster size (%" PRIi64 ")",
max_chunk,cluster_size);Should this be noted in the QAPI documentation?
Hmm.. It makes sense, but I don't know what to write: should be >= job
cluster_size? But then I'll have to describe the logic ofbackup_calculate_cluster_size()...Isn’t the logic basically just “cluster size of the target image file (min 64k)”? The other cases just cover error cases, and they always return 64k, which would effectively be documented by stating that 64k is the minimum.
Quoted 6 lines
But in the common cases (qcow2 or raw), we’ll never get an error anyway.
I think it’d be good to describe the cluster size somewhere, because otherwise I find it a bit malicious to throw an error at the user that they could not have anticipated because they have no idea what valid values for the parameter are (until they try).
OK
Quoted 16 lines
Quoted 11 lines
Quoted 4 lines
(And perhaps the fact
that without copy offloading, we’ll never copy anything bigger than one cluster at a time anyway?)This is a parameter for background copying. Look at
block_copy_task_create(), ifcall_statehas ownmax_chunk, it's used instead of commoncopy_size(derived fromcluster_size). But at a moment of this patch background process through async block-copy is not yet implemented, and the parameter doesn't work, which is described in commit message.Ah, OK. Right.
Max
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH v2 13/20] iotests: 129: prepare for backup over block-copy
In reply to Max ReitzQuoted 12 lines · Max Reitz
Quoted 11 lines · Vladimir Sementsov-Ogievskiy
Quoted 9 lines · Max Reitz
Quoted 4 lines · Vladimir Sementsov-Ogievskiy
After introducing parallel async copy requests instead of plain cluster-by-cluster copying loop, backup job may finish earlier than final assertion in
do_test_stop. Let's require slow backup explicitly by specifying speed parameter.Isn’t the problem really that
block_set_io_throttledoes absolutely nothing? (Which is a long-standing problem with 129. I personally just never run it, honestly.)
Quoted 8 lines
Quoted 1 line
Hmm.. is it better to drop
test_drive_backup()from here ?I think the best would be to revisit this:
https://lists.nongnu.org/archive/html/qemu-block/2019-06/msg00499.html
Max
I've checked, no, that doesn't help. I suppose that throttling has same defect as original block-job speed: it calculates throttling after request, and if we issue simultaneously many requests they all will be handled (and after it we'll have long throttling pause). New solution for backup in these series works better with parallel requests (see patch [PATCH v2 07/20] block/block-copy: add ratelimit to block-copy).
So, I'd keep this patch for now.
Signature · 2 lines
Best regards, Vladimir