mirror: Handle errors after READY cancel
cover mirror: Handle errors after READY cancel
Hi,
This is a rather complex series with changes that aren’t exactly local to the mirror job, so maybe it’s too complex for 6.1.
However, it is a bug fix, and not an insignificant one, though probably not a regression of any kind.
Bug report:
https://gitlab.com/qemu-project/qemu/-/issues/462
(I didn’t put any “Fixes:” or “Resolves:” into the commit messages, because there is no single patch here that fixes the bug.)
The root of the problem is that if you cancel a mirror job during its READY phase, any kind of I/O error (with the error action 'stop') is likely to not be handled gracefully, which means that perhaps the job will just loop forever without pausing, doing nothing but emitting errors. There is no way to stop the job, or cancel it with force=true, and so you also cannot quit qemu normally, because, well, cancelling the job doesn’t do anything. So you have to kill qemu to stop the mess.
If you’re lucky, the error is transient. Then qemu will just kill itself with a failed assertion, because it’ll try a READY -> READY transition, which isn’t allowed.
There are a couple of problems contributing to it all:
(1) The READY -> READY transition comes from the fact that we will enter
the READY state whenever source and target are synced, and whenever
s->synced is false. I/O errors reset s->synced. I believe they
shouldn’t.
(Patch 1)(2) Quitting qemu doesn’t force-cancel jobs. I don’t understand why. If for all jobs but mirror we want them to be cancelled and not properly completed, why do we want mirror to get a consistent result? (Which is what cancel with force=false gives you.) I believe we actually don’t care, and so on many occasions where we invoke job_cancel_sync() and job_cancel_sync_all(), we want to force-cancel the job(s) in question.
(Patch 2)
(3) Cancelling mirror post-READY with force=false is actually not really cancelling the job. It’s a different completion mode. The job should run like any normal job, it shouldn’t be special-cased. However, we have a couple of places that special-case cancelled job because they believe that such jobs are on their way to definite termination. For example, we don’t allow pausing cancelled jobs. We definitely do want to allow pausing a mirror post-READY job that is being non-force-cancelled. The job may still take an arbitrary amount of time, so it absolutely should be pausable. (Patches 3, 4)
(4) Mirror only checks whether it’s been force-cancelled at the bottom of its main loop, after several continues. Therefore, if flushing fails (and it then continues), that check will be skipped. If flushing fails continuously, the job cannot be force-cancelled. (Patch 5)
Max Reitz (6):
mirror: Keep s->synced on error
job: @force parameter for job_cancel_sync{,_all}()
jobs: Give Job.force_cancel more meaning
job: Add job_cancel_requested()
mirror: Check job_is_cancelled() earlier
iotests: Add mirror-ready-cancel-error test
include/qemu/job.h | 29 +++- block/backup.c | 3 +- block/mirror.c | 35 +++-- block/replication.c | 4 +- blockdev.c | 4 +- job.c | 46 ++++-- qemu-nbd.c | 2 +- softmmu/runstate.c | 2 +- storage-daemon/qemu-storage-daemon.c | 2 +- tests/unit/test-block-iothread.c | 2 +- tests/unit/test-blockjob.c | 2 +- tests/qemu-iotests/109.out | 60 +++----- .../tests/mirror-ready-cancel-error | 143 ++++++++++++++++++ .../tests/mirror-ready-cancel-error.out | 5 + tests/qemu-iotests/tests/qsd-jobs.out | 2 +- 15 files changed, 262 insertions(+), 79 deletions(-) create mode 100755 tests/qemu-iotests/tests/mirror-ready-cancel-error create mode 100644 tests/qemu-iotests/tests/mirror-ready-cancel-error.out
Signature · 1 line
2.31.1
PATCH 1/6 mirror: Keep s->synced on error
An error does not take us out of the READY phase, which is what s->synced signifies. It does of course mean that source and target are no longer in sync, but that is what s->actively_sync is for -- s->synced never meant that source and target are in sync, only that they were at some point (and at that point we transitioned into the READY phase).
The tangible problem is that we transition to READY once we are in sync and s->synced is false. By resetting s->synced here, we will transition from READY to READY once the error is resolved (if the job keeps running), and that transition is not allowed.
--- block/mirror.c | 1 - 1 file changed, 1 deletion(-)
block/mirror.c +0 -1
@@ -121,7 +121,6 @@ typedef enum MirrorMethod {static BlockErrorAction mirror_error_action (MirrorBlockJob *s,bool read,int error) { - s->synced = false; s->actively_synced = false;if (read) { return block_job_error_action(&s->common, s->on_source_error,
Signature · 1 line
2.31.1
Re: [PATCH for-6.1? 1/6] mirror: Keep s->synced on error
Quoted 22 lines · Max Reitz
An error does not take us out of the READY phase, which is what
s->syncedsignifies. It does of course mean that source and target are no longer in sync, but that is whats->actively_syncis for --s->syncednever meant that source and target are in sync, only that they were at some point (and at that point we transitioned into the READY phase).The tangible problem is that we transition to READY once we are in sync and
s->syncedis false. By resettings->syncedhere, we will transition from READY to READY once the error is resolved (if the job keeps running), and that transition is not allowed.Signed-off-by Max Reitz <[email protected]>--- block/mirror.c | 1 - 1 file changed, 1 deletion(-)block/mirror.c +0
-0
block/mirror.c typedef enum MirrorMethod { · line 126
Quoted 7 lines
block/mirror.c +0
-1@@ -121,7 +121,6 @@ typedef enum MirrorMethod {static BlockErrorAction mirror_error_action (MirrorBlockJob *s,bool read,int error) { - s->synced = false; s->actively_synced = false;if (read) { return block_job_error_action(&s->common, s->on_source_error,
Looked through.. Yes, seems s->synced used as "is ready". Isn't it better to drop s->synced at all and use job_is_read() instead?
Hmm, s->actively_synced used only for assertion in active_write_settle().. That's not wrong, just interesting.
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-6.1? 1/6] mirror: Keep s->synced on error
Quoted 17 lines · Vladimir Sementsov-Ogievskiy
Quoted 16 lines · Max Reitz
An error does not take us out of the READY phase, which is what
s->syncedsignifies. It does of course mean that source and target are no longer in sync, but that is whats->actively_syncis for --s->syncednever meant that source and target are in sync, only that they were at some point (and at that point we transitioned into the READY phase).The tangible problem is that we transition to READY once we are in sync and
s->syncedis false. By resettings->syncedhere, we will transition from READY to READY once the error is resolved (if the job keeps running), and that transition is not allowed.Signed-off-by Max Reitz <[email protected]>---
block/mirror.c| 1 -1 file changed, 1 deletion(-)
Quoted 15 lines
block/mirror.c typedef enum MirrorMethod { · line 126
Quoted 13 lines
block/mirror.c +0
-1@@ -121,7 +121,6 @@ typedef enum MirrorMethod { static BlockErrorAction mirror_error_action(MirrorBlockJob *s, bool read, int error) { - s->synced = false; s->actively_synced = false; if (read) { return block_job_error_action(&s->common, s->on_source_error,Looked through.. Yes, seems
s->syncedused as "is ready". Isn't it better to drops->syncedat all and usejob_is_read()instead?
Sounds good, though I think for the change to be clear, I’d like to keep this patch and then drop s->synced on top.
Max
Quoted 1 line
Hmm,
s->actively_syncedused only for assertion inactive_write_settle().. That's not wrong, just interesting.
Re: [PATCH for-6.1? 1/6] mirror: Keep s->synced on error
In reply to Max ReitzQuoted 6 lines · Max Reitz
An error does not take us out of the READY phase, which is what
s->syncedsignifies. It does of course mean that source and target are no longer in sync, but that is whats->actively_syncis for --s->syncednever meant that source and target are in sync, only that they were at some point (and at that point we transitioned into the READY phase).
Quoted 6 lines
The tangible problem is that we transition to READY once we are in sync and
s->syncedis false. By resettings->syncedhere, we will transition from READY to READY once the error is resolved (if the job keeps running), and that transition is not allowed.Signed-off-by Max Reitz<[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH 2/6 job: @force parameter for job_cancel_sync{, _all}()
In reply to Max ReitzCallers should be able to specify whether they want job_cancel_sync() to force-cancel the job or not.
In fact, almost all invocations do not care about consistency of the result and just want the job to terminate as soon as possible, so they should pass force=true. The replication block driver is the exception.
This changes some iotest outputs, because quitting qemu while a mirror job is active will now lead to it being cancelled instead of completed, which is what we want. (Cancelling a READY mirror job with force=false may take an indefinite amount of time, which we do not want when quitting. If users want consistent results, they must have all jobs be done before they quit qemu.)
--- include/qemu/job.h | 10 ++--- block/replication.c | 4 +- blockdev.c | 4 +- job.c | 27 +++++++++--- qemu-nbd.c | 2 +- softmmu/runstate.c | 2 +- storage-daemon/qemu-storage-daemon.c | 2 +- tests/unit/test-block-iothread.c | 2 +- tests/unit/test-blockjob.c | 2 +- tests/qemu-iotests/109.out | 60 +++++++++++---------------- tests/qemu-iotests/tests/qsd-jobs.out | 2 +- 11 files changed, 61 insertions(+), 56 deletions(-)
include/qemu/job.h +5 -5
@@ -506,19 +506,19 @@ void job_user_cancel(Job *job, bool force, Error **errp);/** * Synchronously cancel the @job. The completion callback is called -* before the function returns. The job may actually complete -* instead of canceling itself; the circumstances under which this -* happens depend on the kind of job that is active. +* before the function returns. If @force is false, the job may +* actually complete instead of canceling itself; the circumstances +* under which this happens depend on the kind of job that is active. * * Returns the return value from the job if the job actually completed * during the call, or -ECANCELED if it was canceled. * * Callers must hold the AioContext lock of job->aio_context. */ -int job_cancel_sync (Job *job); +int job_cancel_sync (Job *job,bool force);/** Synchronously cancels all jobs using job_cancel_sync(). */ -void job_cancel_sync_all (void ); +void job_cancel_sync_all (bool force);/** * @job: The job to be completed.
block/replication.c +2 -2
@@ -149,7 +149,7 @@ static void replication_close(BlockDriverState *bs)if (s->stage ==BLOCK_REPLICATION_FAILOVER ) { commit_job = &s->commit_job->job;assert (commit_job->aio_context ==qemu_get_current_aio_context ()); -job_cancel_sync (commit_job); +job_cancel_sync (commit_job, false); }if (s->mode ==REPLICATION_MODE_SECONDARY ) {
@@ -726,7 +726,7 @@ static void replication_stop(ReplicationState *rs, bool failover, Error **errp)* disk, secondary disk in backup_job_completed(). */ if (s->backup_job) { -job_cancel_sync (&s->backup_job->job); +job_cancel_sync (&s->backup_job->job, false); }if (!failover) {
blockdev.c +2 -2
@@ -1848,7 +1848,7 @@ static void drive_backup_abort(BlkActionState *common) aio_context =bdrv_get_aio_context (state->bs);aio_context_acquire (aio_context); -job_cancel_sync (&state->job->job); +job_cancel_sync (&state->job->job, true);aio_context_release (aio_context); }
@@ -1949,7 +1949,7 @@ static void blockdev_backup_abort(BlkActionState *common) aio_context =bdrv_get_aio_context (state->bs);aio_context_acquire (aio_context); -job_cancel_sync (&state->job->job); +job_cancel_sync (&state->job->job, true);aio_context_release (aio_context); }
job.c +22 -5
@@ -763,7 +763,12 @@ static void job_completed_txn_abort(Job *job)if (other_job != job) { ctx = other_job->aio_context;aio_context_acquire (ctx); -job_cancel_async (other_job, false); +/* +* This is a transaction: If one job failed, no result will matter. +* Therefore, pass force=true to terminate all other jobs as quickly +* as possible. +*/ +job_cancel_async (other_job, true);aio_context_release (ctx); } }
@@ -964,12 +969,24 @@ static void job_cancel_err(Job *job, Error **errp)job_cancel (job, false); } -int job_cancel_sync (Job *job) +/** +* Same as job_cancel_err(), but force-cancel. +*/ +static void job_force_cancel_err (Job *job,Error **errp) { -return job_finish_sync (job, &job_cancel_err,NULL ); +job_cancel (job, true); +} + +int job_cancel_sync (Job *job,bool force) +{ +if (force) { +return job_finish_sync (job, &job_force_cancel_err,NULL ); + }else { +return job_finish_sync (job, &job_cancel_err,NULL ); + } } -void job_cancel_sync_all (void ) +void job_cancel_sync_all (bool force) {Job *job;AioContext *aio_context;
@@ -977,7 +994,7 @@ void job_cancel_sync_all(void)while ((job =job_next (NULL ))) { aio_context = job->aio_context;aio_context_acquire (aio_context); -job_cancel_sync (job); +job_cancel_sync (job, force);aio_context_release (aio_context); } }
qemu-nbd.c +1 -1
@@ -479,7 +479,7 @@ static const char *socket_activation_validate_opts(const char *device,static void qemu_nbd_shutdown (void ) { -job_cancel_sync_all (); +job_cancel_sync_all (true);blk_exp_close_all ();bdrv_close_all (); }
softmmu/runstate.c +1 -1
@@ -812,7 +812,7 @@ void qemu_cleanup(void)vm_shutdown ();replay_finish (); -job_cancel_sync_all (); +job_cancel_sync_all (true);bdrv_close_all ();/* vhost-user must be cleaned up before chardevs. */
storage-daemon/qemu-storage-daemon.c +1 -1
@@ -347,7 +347,7 @@ int main(int argc, char *argv[])blk_exp_close_all ();bdrv_drain_all_begin (); -job_cancel_sync_all (); +job_cancel_sync_all (true);bdrv_close_all ();monitor_cleanup ();
tests/unit/test-block-iothread.c +1 -1
@@ -662,7 +662,7 @@ static void test_propagate_mirror(void)g_assert (bdrv_get_aio_context (target) == ctx);g_assert (bdrv_get_aio_context (filter) == ctx); -job_cancel_sync_all (); +job_cancel_sync_all (true);aio_context_acquire (ctx);blk_set_aio_context (blk, main_ctx, &error_abort);
tests/unit/test-blockjob.c +1 -1
@@ -230,7 +230,7 @@ static void cancel_common(CancelJob *s) ctx = job->job.aio_context;aio_context_acquire (ctx); -job_cancel_sync (&job->job); +job_cancel_sync (&job->job, true);if (sts !=JOB_STATUS_CREATED && sts !=JOB_STATUS_CONCLUDED ) {Job *dummy = &job->job;job_dismiss (&dummy, &error_abort);
tests/qemu-iotests/109.out +24 -36
132 lines not on this page · raw message
tests/qemu-iotests/tests/qsd-jobs.out +1 -1
@@ -8,7 +8,7 @@ QMP_VERSION {"return": {}} {"return": {}} {"return": {}} -{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_COMPLETED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} +{"timestamp": {"seconds": TIMESTAMP, "microseconds": TIMESTAMP}, "event": "BLOCK_JOB_CANCELLED", "data": {"device": "job0", "len": 0, "offset": 0, "speed": 0, "type": "commit"}} === Streaming can't get permission on base node ===
Signature · 1 line
2.31.1
Re: [PATCH for-6.1? 2/6] job: @force parameter for job_cancel_sync{,_all}()
Quoted 14 lines · Max Reitz
Callers should be able to specify whether they want
job_cancel_sync()to force-cancel the job or not.In fact, almost all invocations do not care about consistency of the result and just want the job to terminate as soon as possible, so they should pass force=true. The replication block driver is the exception.
This changes some iotest outputs, because quitting qemu while a mirror job is active will now lead to it being cancelled instead of completed, which is what we want. (Cancelling a READY mirror job with force=false may take an indefinite amount of time, which we do not want when quitting. If users want consistent results, they must have all jobs be done before they quit qemu.)
Quoted 2 lines
Signed-off-by Max Reitz<[email protected]>
Signature · 2 lines
Best regards, Vladimir
PATCH 3/6 jobs: Give Job.force_cancel more meaning
In reply to Max ReitzWe largely have two cancel modes for jobs:
First, there is actual cancelling. The job is terminated as soon as possible, without trying to reach a consistent result.
Second, we have mirror in the READY state. Technically, the job is not really cancelled, but it just is a different completion mode. The job can still run for an indefinite amount of time while it tries to reach a consistent result.
We want to be able to clearly distinguish which cancel mode a job is in (when it has been cancelled). We can use Job.force_cancel for this, but right now it only reflects cancel requests from the user with force=true, but clearly, jobs that do not even distinguish between force=false and force=true are effectively always force-cancelled.
So this patch has Job.force_cancel signify whether the job will terminate as soon as possible (force_cancel=true) or whether it will effectively remain running despite being "cancelled" (force_cancel=false).
To this end, we let jobs that provide JobDriver.cancel() tell the generic job code whether they will terminate as soon as possible or not, and for jobs that do not provide that method we assume they will.
--- include/qemu/job.h | 11 ++++++++++- block/backup.c | 3 ++- block/mirror.c | 24 ++++++++++++++++++------ job.c | 6 +++++- 4 files changed, 35 insertions(+), 9 deletions(-)
include/qemu/job.h +10 -1
@@ -253,8 +253,17 @@ struct JobDriver {/** * If the callback is not NULL, it will be invoked in job_cancel_async +* +* This function must return true if the job will be cancelled +* immediately without any further I/O (mandatory if @force is +* true), and false otherwise. This lets the generic job layer +* know whether a job has been truly (force-)cancelled, or whether +* it is just in a special completion mode (like mirror after +* READY). +* (If the callback is NULL, the job is assumed to terminate +* without I/O.) */ -void (*cancel)(Job *job,bool force); +bool (*cancel)(Job *job,bool force);/** Called when the job is freed */
block/backup.c +2 -1
@@ -331,11 +331,12 @@ static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) } } -static void backup_cancel (Job *job,bool force) +static bool backup_cancel (Job *job,bool force) {BackupBlockJob *s =container_of (job, BackupBlockJob, common.job);bdrv_cancel_in_flight (s->target_bs); +return true; }static const BlockJobDriver backup_job_driver = {
block/mirror.c +18 -6
@@ -1089,9 +1089,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }trace_mirror_before_sleep (s, cnt, s->synced, delay_ns);job_sleep_ns (&s->common.job, delay_ns); -if (job_is_cancelled (&s->common.job) && - (!s->synced || s->common.job.force_cancel)) - { +if (job_is_cancelled (&s->common.job) && s->common.job.force_cancel) {break ; } s->last_pause_ns =qemu_clock_get_ns (QEMU_CLOCK_REALTIME );
@@ -1103,7 +1101,7 @@ immediate_exit:* or it was cancelled prematurely so that we do not guarantee that * the target is a copy of the source. */ -assert (ret <0 || ((s->common.job.force_cancel || !s->synced) && +assert (ret <0 || (s->common.job.force_cancel &&job_is_cancelled (&s->common.job)));assert (need_drain);mirror_wait_for_all_io (s);
@@ -1189,14 +1187,27 @@ static bool mirror_drained_poll(BlockJob *job)return !!s->in_flight; } -static void mirror_cancel (Job *job,bool force) +static bool mirror_cancel (Job *job,bool force) {MirrorBlockJob *s =container_of (job, MirrorBlockJob, common.job);BlockDriverState *target =blk_bs (s->target); -if (force || !job_is_ready (job)) { +/* +* Before the job is READY, we treat any cancellation like a +* force-cancellation. +*/ + force = force || !job_is_ready (job); + +if (force) {bdrv_cancel_in_flight (target); } +return force; +} + +static bool commit_active_cancel (Job *job,bool force) +{ +/* Same as above in mirror_cancel() */ +return force || !job_is_ready (job); }static const BlockJobDriver mirror_job_driver = {
@@ -1226,6 +1237,7 @@ static const BlockJobDriver commit_active_job_driver = { .abort = mirror_abort, .pause = mirror_pause, .complete = mirror_complete, + .cancel = commit_active_cancel, }, .drained_poll = mirror_drained_poll, };
job.c +5 -1
@@ -719,8 +719,12 @@ static int job_finalize_single(Job *job)static void job_cancel_async (Job *job,bool force) {if (job->driver->cancel) { - job->driver->cancel (job, force); + force = job->driver->cancel (job, force); + }else { +/* No .cancel() means the job will behave as if force-cancelled */ + force = true; } +if (job->user_paused) {/* Do not call job_enter here, the caller will handle it. */ if (job->driver->user_resume) {
Signature · 1 line
2.31.1
Re: [PATCH for-6.1? 3/6] jobs: Give Job.force_cancel more meaning
Quoted 21 lines · Max Reitz
We largely have two cancel modes for jobs:
First, there is actual cancelling. The job is terminated as soon as possible, without trying to reach a consistent result.
Second, we have mirror in the READY state. Technically, the job is not really cancelled, but it just is a different completion mode. The job can still run for an indefinite amount of time while it tries to reach a consistent result.
We want to be able to clearly distinguish which cancel mode a job is in (when it has been cancelled). We can use Job.force_cancel for this, but right now it only reflects cancel requests from the user with force=true, but clearly, jobs that do not even distinguish between force=false and force=true are effectively always force-cancelled.
So this patch has Job.force_cancel signify whether the job will terminate as soon as possible (force_cancel=true) or whether it will effectively remain running despite being "cancelled" (force_cancel=false).
Quoted 5 lines
To this end, we let jobs that provide
JobDriver.cancel()tell the generic job code whether they will terminate as soon as possible or not, and for jobs that do not provide that method we assume they will.Signed-off-by Max Reitz<[email protected]>
In isolation this patch is rather strange: force_cancel is used only by mirror. But we keep in generic job layer. And make a handler to set a value to this variable. So in generic layer we ask mirror which value it want to set to generic variable, which is used only by mirror.. This probably shows that this feature of mirror should be mirror only and generic layer shouldn't take care of it (see also my answer to next commit).
But at the end of the series the variable is not more used by mirror directly. So, technically the commit is not wrong, and it is a preparation for the following ones.
Signature · 2 lines
Best regards, Vladimir
PATCH 4/6 job: Add job_cancel_requested()
In reply to Max ReitzMost callers of job_is_cancelled() actually want to know whether the job is on its way to immediate termination. For example, we refuse to pause jobs that are cancelled; but this only makes sense for jobs that are really actually cancelled.
A mirror job that is cancelled during READY with force=false should absolutely be allowed to pause. This "cancellation" (which is actually a kind of completion) may take an indefinite amount of time, and so should behave like any job during normal operation. For example, with on-target-error=stop, the job should stop on write errors. (In contrast, force-cancelled jobs should not get write errors, as they should just terminate and not do further I/O.)
Therefore, redefine job_is_cancelled() to only return true for jobs that are force-cancelled (which as of HEAD^ means any job that interprets the cancellation request as a request for immediate termination), and add job_cancel_request() as the general variant, which returns true for any jobs which have been requested to be cancelled, whether it be immediately or after an arbitrarily long completion phase.
--- include/qemu/job.h | 8 +++++++- block/mirror.c | 9 ++++----- job.c | 13 +++++++++---- 3 files changed, 20 insertions(+), 10 deletions(-)
include/qemu/job.h +7 -1
@@ -436,9 +436,15 @@ const char *job_type_str(const Job *job);/** Returns true if the job should not be visible to the management layer. */ bool job_is_internal (Job *job); -/** Returns whether the job is scheduled for cancellation. */ +/** Returns whether the job is being cancelled. */ bool job_is_cancelled (Job *job); +/** +* Returns whether the job is scheduled for cancellation (at an +* indefinite point). +*/ +bool job_cancel_requested (Job *job); +/** Returns whether the job is in a completed state. */ bool job_is_completed (Job *job);
block/mirror.c +4 -5
@@ -938,7 +938,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)job_transition_to_ready (&s->common.job); s->synced = true; s->actively_synced = true; -while (!job_is_cancelled (&s->common.job) && !s->should_complete) { +while (!job_cancel_requested (&s->common.job) && !s->should_complete) {job_yield (&s->common.job); } s->common.job.cancelled = false;
@@ -1046,7 +1046,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) } should_complete = s->should_complete || -job_is_cancelled (&s->common.job); +job_cancel_requested (&s->common.job); cnt =bdrv_get_dirty_count (s->dirty_bitmap); }
@@ -1089,7 +1089,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp) }trace_mirror_before_sleep (s, cnt, s->synced, delay_ns);job_sleep_ns (&s->common.job, delay_ns); -if (job_is_cancelled (&s->common.job) && s->common.job.force_cancel) { +if (job_is_cancelled (&s->common.job)) {break ; } s->last_pause_ns =qemu_clock_get_ns (QEMU_CLOCK_REALTIME );
@@ -1101,8 +1101,7 @@ immediate_exit:* or it was cancelled prematurely so that we do not guarantee that * the target is a copy of the source. */ -assert (ret <0 || (s->common.job.force_cancel && -job_is_cancelled (&s->common.job))); +assert (ret <0 ||job_is_cancelled (&s->common.job));assert (need_drain);mirror_wait_for_all_io (s); }
job.c +9 -4
@@ -216,6 +216,11 @@ const char *job_type_str(const Job *job) }bool job_is_cancelled (Job *job) +{ +return job->cancelled && job->force_cancel; +} + +bool job_cancel_requested (Job *job) {return job->cancelled; }
@@ -650,7 +655,7 @@ static void job_conclude(Job *job)static void job_update_rc (Job *job) { -if (!job->ret &&job_is_cancelled (job)) { +if (!job->ret &&job_cancel_requested (job)) { job->ret = -ECANCELED ; }if (job->ret) {
@@ -704,7 +709,7 @@ static int job_finalize_single(Job *job)/* Emit events only if we actually started */ if (job_started (job)) { -if (job_is_cancelled (job)) { +if (job_cancel_requested (job)) {job_event_cancelled (job); }else {job_event_completed (job);
@@ -1015,7 +1020,7 @@ void job_complete(Job *job, Error **errp)if (job_apply_verb (job,JOB_VERB_COMPLETE , errp)) {return ; } -if (job_is_cancelled (job) || !job->driver->complete) { +if (job_cancel_requested (job) || !job->driver->complete) {error_setg (errp,"The active block job '%s' cannot be completed" , job->id);return ;
@@ -1043,7 +1048,7 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)AIO_WAIT_WHILE (job->aio_context, (job_enter (job), !job_is_completed (job))); - ret = (job_is_cancelled (job) && job->ret ==0 ) ? -ECANCELED : job->ret; + ret = (job_cancel_requested (job) && job->ret ==0 ) ? -ECANCELED : job->ret;job_unref (job);return ret; }
Signature · 1 line
2.31.1
Re: [PATCH for-6.1? 4/6] job: Add job_cancel_requested()
Quoted 8 lines · Max Reitz
Most callers of
job_is_cancelled()actually want to know whether the job is on its way to immediate termination. For example, we refuse to pause jobs that are cancelled; but this only makes sense for jobs that are really actually cancelled.A mirror job that is cancelled during READY with force=false should absolutely be allowed to pause. This "cancellation" (which is actually a kind of completion)
You have to repeat that this "cancel" is not "cancel".
So, the whole problem is that feature of mirror, on cancel in READY state do not cancel but do some specific kind of completion.
You try to make this thing correctly handled on generic layer..
Did you consider instead just drop the feature from generic layer? So that all *cancel* functions always do force-cancel. Then the internal implementation become a lot clearer.
But we have to support the qmp block-job-cancel of READY mirror (and commit) with force=false.
We can do it as an exclusion in qmp_block_job_cancel, something like:
if (job is mirror or commit AND it's ready AND force = false)
mirror_soft_cancel(...);
else
job_cancel(...);
Quoted 16 lines
may take an indefinite amount of time, and so
should behave like any job during normal operation. For example, with on-target-error=stop, the job should stop on write errors. (In contrast, force-cancelled jobs should not get write errors, as they should just terminate and not do further I/O.)Therefore, redefine
job_is_cancelled()to only return true for jobs that are force-cancelled (which as of HEAD^ means any job that interprets the cancellation request as a request for immediate termination), and addjob_cancel_request()as the general variant, which returns true for any jobs which have been requested to be cancelled, whether it be immediately or after an arbitrarily long completion phase.Signed-off-by Max Reitz <[email protected]>---
[..]
Quoted 15 lines
job.c +5
-0@@ -216,6 +216,11 @@ const char *job_type_str(const Job *job) }bool job_is_cancelled (Job *job) +{ +return job->cancelled && job->force_cancel; +} + +bool job_cancel_requested (Job *job) {return job->cancelled; }
job.c static void job_conclude(Job *job) · line 658
Quoted 5 lines
job.c +1
-1@@ -650,7 +655,7 @@ static void job_conclude(Job *job)static void job_update_rc (Job *job) { -if (!job->ret &&job_is_cancelled (job)) { +if (!job->ret &&job_cancel_requested (job)) {
Why not job_is_cancelled() here?
So in case of mirror other kind of completion we set ret to -ECANCELED?
job.c static int job_finalize_single(Job *job) · line 713
Quoted 10 lines
job.c +1
-1job->ret = -ECANCELED ; }if (job->ret) {@@ -704,7 +709,7 @@ static int job_finalize_single(Job *job)/* Emit events only if we actually started */ if (job_started (job)) { -if (job_is_cancelled (job)) { +if (job_cancel_requested (job)) {job_event_cancelled (job);
Same question here.. Shouldn't mirror report COMPLETED event in case of not-force cancelled in READY state?
Quoted 21 lines
job.c +2
-2}else {job_event_completed (job);@@ -1015,7 +1020,7 @@ void job_complete(Job *job, Error **errp)if (job_apply_verb (job,JOB_VERB_COMPLETE , errp)) {return ; } -if (job_is_cancelled (job) || !job->driver->complete) { +if (job_cancel_requested (job) || !job->driver->complete) {error_setg (errp,"The active block job '%s' cannot be completed" , job->id);return ;@@ -1043,7 +1048,7 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp)AIO_WAIT_WHILE (job->aio_context, (job_enter (job), !job_is_completed (job))); - ret = (job_is_cancelled (job) && job->ret ==0 ) ? -ECANCELED : job->ret; + ret = (job_cancel_requested (job) && job->ret ==0 ) ? -ECANCELED : job->ret;job_unref (job);return ret; }
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-6.1? 4/6] job: Add job_cancel_requested()
Quoted 10 lines · Vladimir Sementsov-Ogievskiy
Quoted 8 lines · Max Reitz
Most callers of
job_is_cancelled()actually want to know whether the job is on its way to immediate termination. For example, we refuse to pause jobs that are cancelled; but this only makes sense for jobs that are really actually cancelled.A mirror job that is cancelled during READY with force=false should absolutely be allowed to pause. This "cancellation" (which is actually a kind of completion)
Quoted 7 lines
You have to repeat that this "cancel" is not "cancel".
So, the whole problem is that feature of mirror, on cancel in READY state do not cancel but do some specific kind of completion.
You try to make this thing correctly handled on generic layer..
Did you consider instead just drop the feature from generic layer? So that all *cancel* functions always do force-cancel. Then the internal implementation become a lot clearer.
Yes, I considered that, and I’ve decided against it (for now), because such a change would obviously be an incompatible change. It would require a deprecation period, and so we would need to fix this bug now anyway.
Quoted 8 lines
But we have to support the qmp block-job-cancel of READY mirror (and commit) with force=false.
We can do it as an exclusion in
qmp_block_job_cancel, something like:if (job is mirror or commit AND it's ready AND force = false)
mirror_soft_cancel(...);
else
job_cancel(...);
I didn’t consider such a hack, though. I don’t like it. If we think that we should change our approach because mirror’s soft cancel is actually a completion mode, and the current situation is too confusing, such a change should be user-visible, too. (I think there was this idea of having job-specific flags or parameters you could change at runtime, and so you’d just change the “pivot” parameter between true or false.)
Also, I don’t know whether this would really make anything “a lot” easier. After this series job_is_cancelled() already tells the true story, so all we could really change is to drop force_cancel and unify the “s->should_complete || job_cancel_requested()” conditions in block/mirror.c into one variable. So when I considered making cancel exclusively force-cancel jobs, I thought it wouldn’t actually be worth it in practice.
Quoted 40 lines
Quoted 16 lines
may take an indefinite amount of time, and so
should behave like any job during normal operation. For example, with on-target-error=stop, the job should stop on write errors. (In contrast, force-cancelled jobs should not get write errors, as they should just terminate and not do further I/O.)Therefore, redefine
job_is_cancelled()to only return true for jobs that are force-cancelled (which as of HEAD^ means any job that interprets the cancellation request as a request for immediate termination), and addjob_cancel_request()as the general variant, which returns true for any jobs which have been requested to be cancelled, whether it be immediately or after an arbitrarily long completion phase.Signed-off-by Max Reitz <[email protected]>---
[..]
Quoted 14 lines
job.c +5
-0@@ -216,6 +216,11 @@ const char *job_type_str(const Job *job) } bool job_is_cancelled(Job *job) +{ + return job->cancelled && job->force_cancel; +} + +bool job_cancel_requested (Job *job) { return job->cancelled; }job.c static void job_conclude(Job *job) · line 657
Quoted 4 lines
job.c +1
-1@@ -650,7 +655,7 @@ static void job_conclude(Job *job) static void job_update_rc(Job *job) { - if (!job->ret && job_is_cancelled(job)) { + if (!job->ret && job_cancel_requested(job)) {Why not
job_is_cancelled()here?
Quoted 1 line
So in case of mirror other kind of completion we set ret to
-ECANCELED?
I thought the return value is a user-visible thing, so I left it as-is.
Seems I was wrong, more below.
Quoted 11 lines
job.c static int job_finalize_single(Job *job) · line 712
Quoted 9 lines
job.c +1
-1job->ret = -ECANCELED; } if (job->ret) {@@ -704,7 +709,7 @@ static int job_finalize_single(Job *job) /* Emit events only if we actually started */ if (job_started(job)) { - if (job_is_cancelled(job)) { + if (job_cancel_requested(job)) { job_event_cancelled(job);Same question here.. Shouldn't mirror report COMPLETED event in case of not-force cancelled in READY state?
Same here, I thought this is user-visible, nothing internal, so I should leave it as-is.
Now I see that cancelling mirror post-READY indeed should result in a COMPLETED event. So I’m actually not exactly sure how mirror does that, despite this code here (which functionally isn’t changed by this patch), but it’s absolutely true that job_is_cancelled() would be more appropriate here.
(No iotest failed, so I thought this change was right. Well.)
Quoted 10 lines
Quoted 10 lines
job.c +1
-1} else { job_event_completed(job);@@ -1015,7 +1020,7 @@ void job_complete(Job *job, Error **errp) if (job_apply_verb(job, JOB_VERB_COMPLETE, errp)) { return; } - if (job_is_cancelled(job) || !job->driver->complete) { + if (job_cancel_requested(job) || !job->driver->complete) { error_setg(errp, "The active block job '%s' cannot be completed", job->id);
Quoted 8 lines
job.c int job_finish_sync(Job *job, void · line 1053
Quoted 8 lines
job.c +1
-0return;@@ -1043,7 +1048,7 @@ int job_finish_sync(Job *job, void (*finish)(Job *, Error **errp), Error **errp) AIO_WAIT_WHILE(job->aio_context, (job_enter(job), !job_is_completed(job))); - ret = (job_is_cancelled(job) && job->ret == 0) ? -ECANCELED : job->ret; + ret = (job_cancel_requested(job) && job->ret == 0) ? -ECANCELED : job->ret;
So here it should probably stay a job_is_cancelled(), too.
Max
Quoted 6 lines
Quoted 4 lines
job.c +0
-0job_unref(job);return ret;
}
Re: [PATCH for-6.1? 4/6] job: Add job_cancel_requested()
Quoted 13 lines · Max Reitz
Quoted 12 lines
job.c static int job_finalize_single(Job *job) · line 712
Quoted 10 lines
job->ret=-ECANCELED;
}
if (job->ret) {job.c +1
-1@@ -704,7 +709,7 @@ static int job_finalize_single(Job *job) /* Emit events only if we actually started */ if (job_started(job)) { - if (job_is_cancelled(job)) { + if (job_cancel_requested(job)) { job_event_cancelled(job);Same question here.. Shouldn't mirror report COMPLETED event in case of not-force cancelled in READY state?
Quoted 3 lines
Same here, I thought this is user-visible, nothing internal, so I should leave it as-is.
Now I see that cancelling mirror post-READY indeed should result in a COMPLETED event. So I’m actually not exactly sure how mirror does that, despite this code here
Hmm. Now looking at mirror code, I see that it does "s->common.job.cancelled = false"
Signature · 2 lines
Best regards, Vladimir
Re: [PATCH for-6.1? 4/6] job: Add job_cancel_requested()
Quoted 14 lines · Vladimir Sementsov-Ogievskiy
Quoted 13 lines · Max Reitz
Quoted 12 lines
job.c static int job_finalize_single(Job *job) · line 712
Quoted 10 lines
job->ret=-ECANCELED;
}
if (job->ret) {job.c +1
-1@@ -704,7 +709,7 @@ static int job_finalize_single(Job *job) /* Emit events only if we actually started */ if (job_started(job)) { - if (job_is_cancelled(job)) { + if (job_cancel_requested(job)) { job_event_cancelled(job);Same question here.. Shouldn't mirror report COMPLETED event in case of not-force cancelled in READY state?
Quoted 6 lines
Quoted 3 lines
Same here, I thought this is user-visible, nothing internal, so I should leave it as-is.
Now I see that cancelling mirror post-READY indeed should result in a COMPLETED event. So I’m actually not exactly sure how mirror does that, despite this code here
Hmm. Now looking at mirror code, I see that it does "
s->common.job.cancelled= false"
*lol*, that’s nice.
So since we’ve missed the rc1 boat now, I think this is 6.2 material. I’ll look into whether we can drop that then, that would be nice.
Max
PATCH 5/6 mirror: Check job_is_cancelled() earlier
In reply to Max ReitzWe must check whether the job is force-cancelled early in our main loop, most importantly before any continue statement. For example, we used to have continues before our current checking location that are triggered by mirror_flush() failing. So, if mirror_flush() kept failing, force-cancelling the job would not terminate it.
A job being force-cancelled should be treated the same as the job having failed, so put the check in the same place where we check s->ret < 0.
--- block/mirror.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
block/mirror.c +1 -6
@@ -995,7 +995,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)mirror_wait_for_any_operation (s, true); } -if (s->ret <0 ) { +if (s->ret <0 ||job_is_cancelled (&s->common.job)) { ret = s->ret; goto immediate_exit; }
@@ -1081,17 +1081,12 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)break ; } - ret =0 ; -if (s->synced && !should_complete) { delay_ns = (s->in_flight ==0 && cnt ==0 ?BLOCK_JOB_SLICE_TIME :0 ); }trace_mirror_before_sleep (s, cnt, s->synced, delay_ns);job_sleep_ns (&s->common.job, delay_ns); -if (job_is_cancelled (&s->common.job)) { -break ; - } s->last_pause_ns =qemu_clock_get_ns (QEMU_CLOCK_REALTIME ); }
Signature · 1 line
2.31.1
PATCH 6/6 iotests: Add mirror-ready-cancel-error test
In reply to Max ReitzTest what happens when there is an I/O error after a mirror job in the READY phase has been cancelled.
--- .../tests/mirror-ready-cancel-error | 143 ++++++++++++++++++ .../tests/mirror-ready-cancel-error.out | 5 + 2 files changed, 148 insertions(+) create mode 100755 tests/qemu-iotests/tests/mirror-ready-cancel-error create mode 100644 tests/qemu-iotests/tests/mirror-ready-cancel-error.out
tests/qemu-iotests/tests/mirror-ready-cancel-error new file +143 -0
@@ -0,0 +1,143 @@ +#!/usr/bin/env python3 +# group: rw quick +# +# Test what happens when errors occur to a mirror job after it has +# been cancelled in the READY phase +# +# Copyright (C) 2021 Red Hat, Inc. +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. +# + +import os +import iotests + + +image_size = 1 * 1024 * 1024 +source = os.path.join(iotests.test_dir, 'source.img') +target = os.path.join(iotests.test_dir, 'target.img') + + +class TestMirrorReadyCancelError(iotests.QMPTestCase): + def setUp(self) -> None: + assert iotests.qemu_img_create('-f', iotests.imgfmt, source, + str(image_size)) == 0 + assert iotests.qemu_img_create('-f', iotests.imgfmt, target, + str(image_size)) == 0 + + self.vm = iotests.VM() + self.vm.launch() + + def tearDown(self) -> None: + self.vm.shutdown() + os.remove(source) + os.remove(target) + + def add_blockdevs(self, once: bool) -> None: + res = self.vm.qmp('blockdev-add', + **{'node-name': 'source', + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'file', + 'filename': source + }}) + self.assert_qmp(res, 'return', {}) + + # blkdebug notes: + # Enter state 2 on the first flush, which happens before the + # job enters the READY state. The second flush will happen + # when the job is about to complete, and we want that one to + # fail. + res = self.vm.qmp('blockdev-add', + **{'node-name': 'target', + 'driver': iotests.imgfmt, + 'file': { + 'driver': 'blkdebug', + 'image': { + 'driver': 'file', + 'filename': target + }, + 'set-state': [{ + 'event': 'flush_to_disk', + 'state': 1, + 'new_state': 2 + }], + 'inject-error': [{ + 'event': 'flush_to_disk', + 'once': once, + 'immediately': True, + 'state': 2 + }]}}) + self.assert_qmp(res, 'return', {}) + + def start_mirror(self) -> None: + res = self.vm.qmp('blockdev-mirror', + job_id='mirror', + device='source', + target='target', + filter_node_name='mirror-top', + sync='full', + on_target_error='stop') + self.assert_qmp(res, 'return', {}) + + def cancel_mirror_with_error(self) -> None: + self.vm.event_wait('BLOCK_JOB_READY') + + # Write something so will not leave the job immediately, but + # flush first (which will fail, thanks to blkdebug) + res = self.vm.qmp('human-monitor-command', + command_line='qemu-io mirror-top "write 0 64k"') + self.assert_qmp(res, 'return', '') + + # Drain status change events + while self.vm.event_wait('JOB_STATUS_CHANGE', timeout=0.0) is not None: + pass + + res = self.vm.qmp('block-job-cancel', device='mirror') + self.assert_qmp(res, 'return', {}) + + self.vm.event_wait('BLOCK_JOB_ERROR') + + def test_transient_error(self) -> None: + self.add_blockdevs(True) + self.start_mirror() + self.cancel_mirror_with_error() + + while True: + e = self.vm.event_wait('JOB_STATUS_CHANGE') + if e['data']['status'] == 'standby': + # Transient error, try again + self.vm.qmp('block-job-resume', device='mirror') + elif e['data']['status'] == 'null': + break + + def test_persistent_error(self) -> None: + self.add_blockdevs(False) + self.start_mirror() + self.cancel_mirror_with_error() + + while True: + e = self.vm.event_wait('JOB_STATUS_CHANGE') + if e['data']['status'] == 'standby': + # Persistent error, no point in continuing + self.vm.qmp('block-job-cancel', device='mirror', force=True) + elif e['data']['status'] == 'null': + break + + +if __name__ == '__main__': + # LUKS would require special key-secret handling in add_blockdevs() + iotests.main(supported_fmts=['generic'], + unsupported_fmts=['luks'], + supported_protocols=['file'])
tests/qemu-iotests/tests/mirror-ready-cancel-error.out new file +5 -0
@@ -0,0 +1,5 @@ +.. +---------------------------------------------------------------------- +Ran 2 tests + +OK
Signature · 1 line
2.31.1