job: replace AioContext lock with job_mutex
cover job: replace AioContext lock with job_mutex
This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.
This series aims to:
1) remove the aiocontext lock and substitute it with the already existing global job_mutex
2) fix what it looks like to be an oversight when moving the blockjob.c logic into the more generic job.c: job_mutex was introduced especially to protect job->busy flag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) use job_mutex instead of the aiocontext_lock
4) extend the reach of the job_mutex to protect all shared fields that the job structure has.
The reason why we propose to use the existing job_mutex and not make one for each job is to keep things as simple as possible for now, and because the jobs are not in the execution critical path, so we can affort some delays.
Having a lock per job would increase overall complexity and increase the chances of deadlocks (one good example could be the job transactions, where multiple jobs are grouped together). Anyways, the per-job mutex can always be added in the future.
Patch 1-4 are in preparation for patch 5. They try to simplify and clarify the job_mutex usage. Patch 5 tries to add proper syncronization to the job structure, replacing the AioContext lock when necessary.
Patch 6 just removes unnecessary AioContext locks that are now unneeded.
RFC: I am not sure the way I layed out the locks is ideal. But their usage should not make deadlocks. I also made sure the series passess all qemu_iotests.
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).
I also apologize for the amount of changes in patch 5, any suggestion on how to improve the patch layout is also very much appreciated.
Emanuele Giuseppe Esposito (6):
job: use getter/setters instead of accessing the Job fields directly
job: _locked functions and public job_lock/unlock for next patch
job: minor changes to simplify locking
job.h: categorize job fields
job: use global job_mutex to protect struct Job
jobs: remove unnecessary AioContext aquire/release pairs
include/block/blockjob_int.h | 1 + include/qemu/job.h | 159 ++++++++++-- block.c | 2 +- block/backup.c | 4 + block/commit.c | 4 +- block/mirror.c | 30 ++- block/monitor/block-hmp-cmds.c | 6 - block/replication.c | 3 +- blockdev.c | 235 ++++++------------ blockjob.c | 140 +++++++---- job-qmp.c | 65 +++-- job.c | 432 ++++++++++++++++++++++++++------- qemu-img.c | 19 +- 13 files changed, 724 insertions(+), 376 deletions(-)
Signature · 1 line
2.31.1
RFC PATCH 1/6 job: use getter/setters instead of accessing the Job fields directly
Using getters/setters we can have a more strict control on struct Job fields. The struct remains public, because it is also used as base class for BlockJobs and various, but replace all direct accesses to the fields we want to protect with getters/setters. This is in preparation to the locking patches.
No functional change intended.
--- include/qemu/job.h | 33 +++++++++++++++++++ block.c | 2 +- block/commit.c | 4 +-- block/mirror.c | 17 +++++----- block/replication.c | 3 +- blockdev.c | 2 +- blockjob.c | 78 ++++++++++++++++++++++++--------------------- job-qmp.c | 16 ++++++---- job.c | 52 +++++++++++++++++++++++++++++- qemu-img.c | 2 +- 10 files changed, 151 insertions(+), 58 deletions(-)
include/qemu/job.h +33 -0
@@ -303,6 +303,39 @@ void job_txn_unref(JobTxn *txn);*/ void job_txn_add_job (JobTxn *txn,Job *job); +/** Returns the @ret field of a given Job. */ +int job_get_ret (Job *job); + +/** Returns the AioContext of a given Job. */ +AioContext *job_get_aiocontext (Job *job); + +/** Sets the AioContext of a given Job. */ +void job_set_aiocontext (Job *job,AioContext *aio); + +/** Returns if a given Job is busy. */ +bool job_is_busy (Job *job); + +/** Returns the Error of a given Job. */ +Error *job_get_err (Job *job); + +/** Returns if a Job has a pause_count > 0. */ +bool job_should_pause (Job *job); + +/** Sets the user_paused flag of a given Job to true. */ +void job_set_user_paused (Job *job); + +/** Sets the cancelled flag of a given Job. */ +void job_set_cancelled (Job *job,bool cancel); + +/** Returns if a given Job is paused. */ +bool job_is_paused (Job *job); + +/** Returns if a given Job is force cancelled. */ +bool job_is_force_cancel (Job *job); + +/** Returns the statis of a given Job. */ +JobStatus job_get_status (Job *job); +/** * Create a new long-running job and return it. *
block.c +1 -1
@@ -5721,7 +5721,7 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)GSList *el;xdbg_graph_add_node (gr, job,X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB , - job->job.id); + job->job.id);for (el = job->nodes; el; el = el->next) {xdbg_graph_add_edge (gr, job, (BdrvChild *)el->data); }
block/commit.c +2 -2
@@ -367,7 +367,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, goto fail; } - s->base =blk_new (s->common.job.aio_context, + s->base =blk_new (job_get_aiocontext (&s->common.job), base_perms,BLK_PERM_CONSISTENT_READ | BLK_PERM_GRAPH_MOD
@@ -380,7 +380,7 @@ void commit_start(const char *job_id, BlockDriverState *bs, s->base_bs = base;/* Required permissions are already taken with block_job_add_bdrv() */ - s->top =blk_new (s->common.job.aio_context,0 ,BLK_PERM_ALL ); + s->top =blk_new (job_get_aiocontext (&s->common.job),0 ,BLK_PERM_ALL ); ret =blk_insert_bs (s->top, top, errp);if (ret <0 ) { goto fail;
block/mirror.c +9 -8
@@ -636,7 +636,7 @@ static int mirror_exit_common(Job *job)BlockDriverState *target_bs;BlockDriverState *mirror_top_bs;Error *local_err =NULL ; -bool abort = job->ret <0 ; +bool abort =job_get_ret (job) <0 ;int ret =0 ;if (s->prepared) {
@@ -930,7 +930,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)while (!job_is_cancelled (&s->common.job) && !s->should_complete) {job_yield (&s->common.job); } - s->common.job.cancelled = false; +job_set_cancelled (&s->common.job, false); goto immediate_exit; }
@@ -1065,7 +1065,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)* completion. */ assert (QLIST_EMPTY (&bs->tracked_requests)); - s->common.job.cancelled = false; +job_set_cancelled (&s->common.job, false); need_drain = false;break ; }
@@ -1079,7 +1079,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)trace_mirror_before_sleep (s, cnt, s->synced, delay_ns);job_sleep_ns (&s->common.job, delay_ns);if (job_is_cancelled (&s->common.job) && - (!s->synced || s->common.job.force_cancel)) + (!s->synced ||job_is_force_cancel (&s->common.job))) {break ; }
@@ -1092,8 +1092,8 @@ immediate_exit:* or it was cancelled prematurely so that we do not guarantee that * the target is a copy of the source. */ -assert (ret <0 || ((s->common.job.force_cancel || !s->synced) && -job_is_cancelled (&s->common.job))); +assert (ret <0 || ((job_is_force_cancel (&s->common.job) || !s->synced) + &&job_is_cancelled (&s->common.job)));assert (need_drain);mirror_wait_for_all_io (s); }
@@ -1150,7 +1150,7 @@ static void mirror_complete(Job *job, Error **errp) s->should_complete = true;/* If the job is paused, it will be re-entered when it is resumed */ -if (!job->paused) { +if (!job_is_paused (job)) {job_enter (job); } }
@@ -1171,7 +1171,8 @@ static bool mirror_drained_poll(BlockJob *job)* from one of our own drain sections, to avoid a deadlock waiting for * ourselves. */ -if (!s->common.job.paused && !s->common.job.cancelled && !s->in_drain) { +if (!job_is_paused (&s->common.job) && !job_is_cancelled (&s->common.job) && + !s->in_drain) {return true; }
block/replication.c +2 -1
@@ -149,7 +149,8 @@ static void replication_close(BlockDriverState *bs) }if (s->stage ==BLOCK_REPLICATION_FAILOVER ) { commit_job = &s->commit_job->job; -assert (commit_job->aio_context ==qemu_get_current_aio_context ()); +assert (job_get_aiocontext (commit_job) == +qemu_get_current_aio_context ());job_cancel_sync (commit_job); }
blockdev.c +1 -1
@@ -147,7 +147,7 @@ void blockdev_mark_auto_del(BlockBackend *blk)for (job =block_job_next (NULL ); job; job =block_job_next (job)) {if (block_job_has_bdrv (job,blk_bs (blk))) { -AioContext *aio_context = job->job.aio_context; +AioContext *aio_context =job_get_aiocontext (&job->job);aio_context_acquire (aio_context);job_cancel (&job->job, false);
blockjob.c +42 -36
161 lines not on this page · raw message
job-qmp.c +9 -7
@@ -42,7 +42,7 @@ static Job *find_job(const char *id, AioContext **aio_context, Error **errp)return NULL ; } - *aio_context = job->aio_context; + *aio_context =job_get_aiocontext (job);aio_context_acquire (*aio_context);return job;
@@ -122,7 +122,7 @@ void qmp_job_finalize(const char *id, Error **errp)* automatically acquires the new one), so make sure we release the correct * one. */ - aio_context = job->aio_context; + aio_context =job_get_aiocontext (job);job_unref (job);aio_context_release (aio_context); }
@@ -146,21 +146,23 @@ static JobInfo *job_query_single(Job *job, Error **errp)JobInfo *info;uint64_t progress_current;uint64_t progress_total; +Error *job_err;assert (!job_is_internal (job));progress_get_snapshot (&job->progress, &progress_current, &progress_total); + job_err =job_get_err (job); info =g_new (JobInfo,1 ); *info = (JobInfo ) { .id =g_strdup (job->id), .type =job_type (job), - .status = job->status, + .status =job_get_status (job), .current_progress = progress_current, .total_progress = progress_total, - .has_error = !!job->err, - .error = job->err ? \ -g_strdup (error_get_pretty (job->err)) :NULL , + .has_error = !!job_err, + .error = job_err ? \ +g_strdup (error_get_pretty (job_err)) :NULL , };return info;
@@ -178,7 +180,7 @@ JobInfoList *qmp_query_jobs(Error **errp)if (job_is_internal (job)) {continue ; } - aio_context = job->aio_context; + aio_context =job_get_aiocontext (job);aio_context_acquire (aio_context); value =job_query_single (job, errp);aio_context_release (aio_context);
job.c +51 -1
74 lines not on this page · raw message
qemu-img.c +1 -1
@@ -921,7 +921,7 @@ static void run_block_job(BlockJob *job, Error **errp)if (!job_is_completed (&job->job)) { ret =job_complete_sync (&job->job, errp); }else { - ret = job->job.ret; + ret =job_get_ret (&job->job); }job_unref (&job->job);aio_context_release (aio_context);
Signature · 1 line
2.31.1
RFC PATCH 2/6 job: _locked functions and public job_lock/unlock for next patch
In reply to Emanuele Giuseppe EspositoCreate _locked functions, to make next patch a little bit smaller. Also set the locking functions as public, so that they can be used also from structures using the Job struct.
--- include/qemu/job.h | 23 +++++++++++++ job.c | 85 ++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 93 insertions(+), 15 deletions(-)
include/qemu/job.h +23 -0
@@ -305,6 +305,7 @@ void job_txn_add_job(JobTxn *txn, Job *job);/** Returns the @ret field of a given Job. */ int job_get_ret (Job *job); +int job_get_ret_locked (Job *job);/** Returns the AioContext of a given Job. */ AioContext *job_get_aiocontext (Job *job);
@@ -336,6 +337,24 @@ bool job_is_force_cancel(Job *job);/** Returns the statis of a given Job. */ JobStatus job_get_status (Job *job); +/** +* job_lock: +* +* Take the mutex protecting the list of jobs and their status. +* Most functions called by the monitor need to call job_lock +* and job_unlock manually. On the other hand, function called +* by the block jobs themselves and by the block layer will take the +* lock for you. +*/ +void job_lock (void ); + +/** +* job_unlock: +* +* Release the mutex protecting the list of jobs and their status. +*/ +void job_unlock (void ); +/** * Create a new long-running job and return it. *
@@ -424,6 +443,7 @@ void job_start(Job *job);* Continue the specified job by entering the coroutine. */ void job_enter (Job *job); +void job_enter_locked (Job *job);/** * @job: The job that is ready to pause.
@@ -462,12 +482,15 @@ bool job_is_internal(Job *job);/** Returns whether the job is scheduled for cancellation. */ bool job_is_cancelled (Job *job); +bool job_is_cancelled_locked (Job *job);/** Returns whether the job is in a completed state. */ bool job_is_completed (Job *job); +bool job_is_completed_locked (Job *job);/** Returns whether the job is ready to be completed. */ bool job_is_ready (Job *job); +bool job_is_ready_locked (Job *job);/** * Request @job to pause at the next pause point. Must be paired with
job.c +70 -15
162 lines not on this page · raw message
Signature · 1 line
2.31.1
Re: [RFC PATCH 2/6] job: _locked functions and public job_lock/unlock for next patch
job.c line 36
Quoted 10 lines · Emanuele Giuseppe Esposito
job.c +2
-0@@ -32,6 +32,10 @@#include "trace/trace-root.h" #include "qapi/qapi-events-job.h" +/* job_mutex protexts the jobs list, but also the job operations. */ +static QemuMutex job_mutex;
It's unclear what protecting "job operations" means. I would prefer a fine-grained per-job lock that protects the job's fields instead of a global lock with an unclear scope.
job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 72
Quoted 12 lines
job.c +3
-1+ +/* Protected by job_mutex */ static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);/* Job State Transition Table */ @@ -64,27 +68,22 @@ bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = {/* Transactional group of jobs */ struct JobTxn { -/* Is this txn being cancelled? */ +/* Is this txn being cancelled? Atomic.*/ bool aborting;
The comment says atomic but this field is not accessed using atomic operations (at least at this point in the patch series)?
job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 78
Quoted 8 lines
job.c +2
-2-/* List of jobs */ +/* List of jobs. Protected by job_mutex. */ QLIST_HEAD(, Job) jobs; -/* Reference count */ +/* Reference count. Atomic. */ int refcnt;
Same.
Re: [RFC PATCH 2/6] job: _locked functions and public job_lock/unlock for next patch
Quoted 9 lines · Stefan Hajnoczi
Quoted 8 lines · Emanuele Giuseppe Esposito
job.c +0
-0@@ -32,6 +32,10 @@#include "trace/trace-root.h" #include "qapi/qapi-events-job.h"
Quoted 6 lines
job.c line 36
Quoted 2 lines
job.c +2
-0+/* job_mutex protexts the jobs list, but also the job operations. */ +static QemuMutex job_mutex;It's unclear what protecting "job operations" means. I would prefer a fine-grained per-job lock that protects the job's fields instead of a global lock with an unclear scope.
As I wrote in the cover letter, I wanted to try to keep things as simple as possible with a global lock. It is possible to try and have a per-job lock, but I don't know how complex will that be then. I will try and see what I can do.
Maybe "job_mutex protexts the jobs list, but also makes the job API thread-safe"?
Quoted 10 lines
Quoted 10 lines
job.c +2
-0+ +/* Protected by job_mutex */ static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);/* Job State Transition Table */ @@ -64,27 +68,22 @@ bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = {/* Transactional group of jobs */ struct JobTxn {
Quoted 6 lines
job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 72
Quoted 3 lines
job.c +1
-1-/* Is this txn being cancelled? */ +/* Is this txn being cancelled? Atomic.*/ bool aborting;The comment says atomic but this field is not accessed using atomic operations (at least at this point in the patch series)?
Yes sorry I messed up the hunks in one-two patches. These comments were supposed to be on patch 4 "job.h: categorize job fields". Even though that might also not be ideal, since that patch just introduces the comments, without applying the locking/protection yet. On the other side, if I merge everything together in patch 5, it will be even harder to read.
Emanuele
Quoted 12 lines
job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 78
Quoted 9 lines
job.c +2
-2-/* List of jobs */ +/* List of jobs. Protected by job_mutex. */ QLIST_HEAD(, Job) jobs; -/* Reference count */ +/* Reference count. Atomic. */ int refcnt;Same.
Re: [RFC PATCH 2/6] job: _locked functions and public job_lock/unlock for next patch
Quoted 18 lines · Emanuele Giuseppe Esposito
Quoted 14 lines · Stefan Hajnoczi
job.c line 35
Quoted 9 lines · Emanuele Giuseppe Esposito
job.c +2
-0@@ -32,6 +32,10 @@#include "trace/trace-root.h" #include "qapi/qapi-events-job.h" +/* job_mutex protexts the jobs list, but also the job operations. */ +static QemuMutex job_mutex;It's unclear what protecting "job operations" means. I would prefer a fine-grained per-job lock that protects the job's fields instead of a global lock with an unclear scope.
Quoted 7 lines
As I wrote in the cover letter, I wanted to try to keep things as simple as possible with a global lock. It is possible to try and have a per-job lock, but I don't know how complex will that be then.
I will try and see what I can do.Maybe "
job_mutexprotexts the jobs list, but also makes the job API thread-safe"?
That's clearer, thanks. I thought "job operations" meant the processing that the actual block jobs do (commit, mirror, stream, backup).
Quoted 16 lines
Quoted 15 lines
job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 71
Quoted 12 lines
job.c +3
-1+ +/* Protected by job_mutex */ static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);/* Job State Transition Table */ @@ -64,27 +68,22 @@ bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = {/* Transactional group of jobs */ struct JobTxn { -/* Is this txn being cancelled? */ +/* Is this txn being cancelled? Atomic.*/ bool aborting;The comment says atomic but this field is not accessed using atomic operations (at least at this point in the patch series)?
Quoted 6 lines
Yes sorry I messed up the hunks in one-two patches. These comments were supposed to be on patch 4 "
job.h: categorize job fields". Even though that might also not be ideal, since that patch just introduces the comments, without applying the locking/protection yet.
On the other side, if I merge everything together in patch 5, it will be even harder to read.
The commit description can describe changes that currently have no effect but are anticipating a later patch. That helps reviewers understand whether the change is intentional/correct.
Stefan
RFC PATCH 3/6 job: minor changes to simplify locking
In reply to Emanuele Giuseppe EspositoCheck for NULL id to job_get, so that in the next patch we can move job_get inside a single critical section of job_create.
Also add missing notifier_list_init for the on_idle NotifierList, which seems to have been forgot.
--- job.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
job.c +12 -4
@@ -375,6 +375,10 @@ Job *job_get(const char *id) {Job *job; +if (!id) { +return NULL ; + } + QLIST_FOREACH(job, &jobs, job_list) {if (job->id && !strcmp (id, job->id)) {return job;
@@ -406,15 +410,18 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,error_setg (errp,"Invalid job ID '%s'" , job_id);return NULL ; } -if (job_get (job_id)) { -error_setg (errp,"Job ID '%s' already in use" , job_id); -return NULL ; - } }else if (!(flags &JOB_INTERNAL )) {error_setg (errp,"An explicit job ID is required" );return NULL ; } +job_lock (); +if (job_get (job_id)) { +error_setg (errp,"Job ID '%s' already in use" , job_id); +job_unlock (); +return NULL ; + } + job =g_malloc0 (driver->instance_size); job->driver = driver; job->id =g_strdup (job_id);
@@ -434,6 +441,7 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,notifier_list_init (&job->on_finalize_completed);notifier_list_init (&job->on_pending);notifier_list_init (&job->on_ready); +notifier_list_init (&job->on_idle);job_state_transition (job,JOB_STATUS_CREATED ); aio_timer_init(qemu_get_aio_context(), &job->sleep_timer,
Signature · 1 line
2.31.1
Re: [RFC PATCH 3/6] job: minor changes to simplify locking
Quoted 10 lines · Emanuele Giuseppe Esposito
job.c +0
-4@@ -406,15 +410,18 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,error_setg (errp,"Invalid job ID '%s'" , job_id);return NULL ; } -if (job_get (job_id)) { -error_setg (errp,"Job ID '%s' already in use" , job_id); -return NULL ; - } }else if (!(flags &JOB_INTERNAL )) {error_setg (errp,"An explicit job ID is required" );
job.c void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn, · line 424
Quoted 10 lines
job.c +7
-0return NULL ; } +job_lock (); +if (job_get (job_id)) { +error_setg (errp,"Job ID '%s' already in use" , job_id); +job_unlock (); +return NULL ; + } +
Where is the matching job_unlock() in the success case? Please consider lock guard macros like QEMU_LOCK_GUARD()/WITH_QEMU_LOCK_GUARD() to prevent common errors.
Re: [RFC PATCH 3/6] job: minor changes to simplify locking
Quoted 14 lines · Stefan Hajnoczi
Quoted 13 lines · Emanuele Giuseppe Esposito
job.c +0
-4@@ -406,15 +410,18 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,error_setg (errp,"Invalid job ID '%s'" , job_id);return NULL ; } -if (job_get (job_id)) { -error_setg (errp,"Job ID '%s' already in use" , job_id); -return NULL ; - } }else if (!(flags &JOB_INTERNAL )) {error_setg (errp,"An explicit job ID is required" );return NULL ; }
Quoted 12 lines
job.c void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn, · line 424
Quoted 7 lines
job.c +7
-0+job_lock (); +if (job_get (job_id)) { +error_setg (errp,"Job ID '%s' already in use" , job_id); +job_unlock (); +return NULL ; + } +Where is the matching
job_unlock()in the success case? Please consider lock guard macros like QEMU_LOCK_GUARD()/WITH_QEMU_LOCK_GUARD() to prevent common errors.
Again this is a dumb mistake, the job_lock/unlock lines should go on patch 5, not here. Apologies.
I did not use QEMU_LOCK_GUARD()/WITH_QEMU_LOCK_GUARD() yet because I added some assertions to make sure I also didn't create nested locking situations. The final version will certainly use them.
Emanuele
Re: [RFC PATCH 3/6] job: minor changes to simplify locking
In reply to Emanuele Giuseppe EspositoQuoted 5 lines · Emanuele Giuseppe Esposito
Check for NULL id to
job_get, so that in the next patch we can movejob_getinside a single critical section ofjob_create.Also add missing
notifier_list_initfor theon_idleNotifierList, which seems to have been forgot.
forgotten
Quoted 5 lines
Signed-off-by Emanuele Giuseppe Esposito <[email protected]>--- job.c | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
RFC PATCH 4/6 job.h: categorize job fields
In reply to Emanuele Giuseppe EspositoThis makes it easier to understand what needs to be protected by a lock and what doesn't.
--- include/qemu/job.h | 101 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 82 insertions(+), 19 deletions(-)
include/qemu/job.h +82 -19
@@ -40,24 +40,40 @@ typedef struct JobTxn JobTxn;* Long-running operation. */ typedef struct Job { - /** The ID of the job. May be NULL for internal jobs. */ + /** + * The ID of the job. May be NULL for internal jobs. + * Set it in job_create and just read. + */ char *id; - /** The type of this job. */ + /** + * The type of this job. + * Set it in job_create and just read. + */ const JobDriver *driver; - /** Reference count of the block job */ + /** + * Reference count of the block job. + * Protected by job_mutex. + */ int refcnt; - /** Current state; See @JobStatus for details. */ + /** + * Current state; See @JobStatus for details. + * Protected by job_mutex. + */ JobStatus status; - /** AioContext to run the job coroutine in */ + /** + * AioContext to run the job coroutine in. + * Atomic. + */ AioContext *aio_context; /** * The coroutine that executes the job. If not NULL, it is reentered when * busy is false and the job is cancelled. + * Set it in job_create and just read. */ Coroutine *co;
@@ -70,13 +86,15 @@ typedef struct Job {/** * Counter for pause request. If non-zero, the block job is either paused, * or if busy == true will pause itself as soon as possible. +* Protected by job_mutex. */ int pause_count;/** * Set to false by the job while the coroutine has yielded and may be * re-entered by job_enter(). There may still be I/O or event loop activity -* pending. Accessed under block_job_mutex (in blockjob.c). +* pending. +* Protected by job_mutex. * * When the job is deferred to the main loop, busy is true as long as the * bottom half is still pending.
@@ -86,12 +104,14 @@ typedef struct Job {/** * Set to true by the job while it is in a quiescent state, where * no I/O or event loop activity is pending. +* Protected by job_mutex. */ bool paused;/** * Set to true if the job is paused by user. Can be unpaused with the * block-job-resume QMP command. +* Protected by job_mutex. */ bool user_paused;
@@ -100,22 +120,33 @@ typedef struct Job {* always be tested just before toggling the busy flag from false * to true. After a job has been cancelled, it should only yield * if #aio_poll will ("sooner or later") reenter the coroutine. +* Protected by job_mutex. */ bool cancelled;/** * Set to true if the job should abort immediately without waiting * for data to be in sync. +* Protected by job_mutex. */ bool force_cancel; -/** Set to true when the job has deferred work to the main loop. */ +/** +* Set to true when the job has deferred work to the main loop. +* Protected by job_mutex. +*/ bool deferred_to_main_loop; -/** True if this job should automatically finalize itself */ +/** +* True if this job should automatically finalize itself. +* Set it in job_create and just read. +*/ bool auto_finalize; -/** True if this job should automatically dismiss itself */ +/** +* True if this job should automatically dismiss itself. +* Set it in job_create and just read. +*/ bool auto_dismiss;ProgressMeter progress;
@@ -124,6 +155,7 @@ typedef struct Job {* Return code from @run and/or @prepare callback(s). * Not final until the job has reached the CONCLUDED status. * 0 on success, -errno on failure. +* Protected by job_mutex. */ int ret;
@@ -131,37 +163,68 @@ typedef struct Job {* Error object for a failed job. * If job->ret is nonzero and an error object was not set, it will be set * to strerror(-job->ret) during job_completed. +* Protected by job_mutex. */ Error *err; -/** The completion function that will be called when the job completes. */ +/** +* The completion function that will be called when the job completes. +* Set it in job_create and just read. +*/ BlockCompletionFunc *cb; -/** The opaque value that is passed to the completion function. */ +/** +* The opaque value that is passed to the completion function. +* Set it in job_create and just read. +*/ void *opaque; -/** Notifiers called when a cancelled job is finalised */ +/** +* Notifiers called when a cancelled job is finalised. +* Protected by job_mutex. +*/ NotifierList on_finalize_cancelled; -/** Notifiers called when a successfully completed job is finalised */ +/** +* Notifiers called when a successfully completed job is finalised. +* Protected by job_mutex. +*/ NotifierList on_finalize_completed; -/** Notifiers called when the job transitions to PENDING */ +/** +* Notifiers called when the job transitions to PENDING. +* Protected by job_mutex. +*/ NotifierList on_pending; -/** Notifiers called when the job transitions to READY */ +/** +* Notifiers called when the job transitions to READY. +* Protected by job_mutex. +*/ NotifierList on_ready; -/** Notifiers called when the job coroutine yields or terminates */ +/** +* Notifiers called when the job coroutine yields or terminates. +* Protected by job_mutex. +*/ NotifierList on_idle; -/** Element of the list of jobs */ +/** +* Element of the list of jobs. +* Protected by job_mutex. +*/ QLIST_ENTRY (Job) job_list; -/** Transaction this job is part of */ +/** +* Transaction this job is part of. +* Protected by job_mutex. +*/ JobTxn *txn; -/** Element of the list of jobs in a job transaction */ +/** +* Element of the list of jobs in a job transaction. +* Protected by job_mutex. +*/ QLIST_ENTRY (Job) txn_list; } Job;
Signature · 1 line
2.31.1
Re: [RFC PATCH 4/6] job.h: categorize job fields
include/qemu/job.h typedef struct JobTxn JobTxn; · line 71
Quoted 6 lines · Emanuele Giuseppe Esposito
include/qemu/job.h +4
-1-/** AioContext to run the job coroutine in */ +/** +* AioContext to run the job coroutine in. +* Atomic. +*/ AioContext *aio_context;
This isn't accessed using atomic operations, so I'm not sure why it's documented as atomic?
Re: [RFC PATCH 4/6] job.h: categorize job fields
Quoted 11 lines · Stefan Hajnoczi
include/qemu/job.h typedef struct JobTxn JobTxn; · line 71
Quoted 6 lines · Emanuele Giuseppe Esposito
include/qemu/job.h +4
-1-/** AioContext to run the job coroutine in */ +/** +* AioContext to run the job coroutine in. +* Atomic. +*/ AioContext *aio_context;This isn't accessed using atomic operations, so I'm not sure why it's documented as atomic?
Maybe this is unnecessary, but from what I understand right now when we want to change the AioContext of a child node, we need to acquire its AioContext lock, and then we try to set it. Without AioContext locks, my understanding is that this has to be protected somehow. Therefore I thought of setting this pointer atomically (actual code that does this is in patch 5).
Again same reasoning style applies here: this patch just adds a bounce of comments, but merging it with next one would just make it more unreadable.
Emanuele
RFC PATCH 5/6 job: use global job_mutex to protect struct Job
In reply to Emanuele Giuseppe EspositoThis lock is going to replace most of the AioContext locks in the job and blockjob, so that a Job can run in an arbitrary AioContext.
--- include/block/blockjob_int.h | 1 + include/qemu/job.h | 2 + block/backup.c | 4 + block/mirror.c | 11 +- blockdev.c | 62 ++++---- blockjob.c | 67 +++++++-- job-qmp.c | 55 +++---- job.c | 284 +++++++++++++++++++++++++++-------- qemu-img.c | 15 +- 9 files changed, 350 insertions(+), 151 deletions(-)
include/block/blockjob_int.h +1 -0
@@ -53,6 +53,7 @@ struct BlockJobDriver {*/ void (*attached_aio_context)(BlockJob *job,AioContext *new_context); +/* Called with job mutex *not* held. */ void (*set_speed)(BlockJob *job,int64_t speed); };
include/qemu/job.h +2 -0
@@ -49,6 +49,8 @@ typedef struct Job {/** * The type of this job. * Set it in job_create and just read. +* All calls to the driver function must be not locked by job_mutex, +* to avoid deadlocks. */ const JobDriver *driver;
block/backup.c +4 -0
@@ -315,6 +315,10 @@ static void coroutine_fn backup_pause(Job *job) } } +/* +* Called with job mutex *not* held (we don't want to call block_copy_kick +* with the lock held!) +*/ static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) {BackupBlockJob *s =container_of (job, BackupBlockJob, common);
block/mirror.c +8 -3
@@ -1150,9 +1150,11 @@ static void mirror_complete(Job *job, Error **errp) s->should_complete = true;/* If the job is paused, it will be re-entered when it is resumed */ +job_lock ();if (!job_is_paused (job)) { -job_enter (job); +job_enter_locked (job); } +job_unlock (); } static void coroutine_fn mirror_pause(Job *job)
@@ -1171,10 +1173,13 @@ static bool mirror_drained_poll(BlockJob *job)* from one of our own drain sections, to avoid a deadlock waiting for * ourselves. */ -if (!job_is_paused (&s->common.job) && !job_is_cancelled (&s->common.job) && - !s->in_drain) { +job_lock (); +if (!job_is_paused (&s->common.job) && + !job_is_cancelled_locked (&s->common.job) && !s->in_drain) { +job_unlock ();return true; } +job_unlock ();return !!s->in_flight; }
blockdev.c +27 -35
163 lines not on this page · raw message
blockjob.c +56 -11
266 lines not on this page · raw message
job-qmp.c +21 -34
146 lines not on this page · raw message
job.c +221 -63
895 lines not on this page · raw message
qemu-img.c +10 -5
@@ -898,17 +898,19 @@ static void common_block_job_cb(void *opaque, int ret) } } +/* Called with job_mutex held. Releases it temporarly */ static void run_block_job (BlockJob *job,Error **errp) {uint64_t progress_current, progress_total;AioContext *aio_context =blk_get_aio_context (job->blk);int ret =0 ; -aio_context_acquire (aio_context);job_ref (&job->job); do { float progress = 0.0f; + job_unlock(); aio_poll(aio_context, true); + job_lock(); progress_get_snapshot(&job->job.progress, &progress_current, &progress_total);
@@ -916,15 +918,15 @@ static void run_block_job(BlockJob *job, Error **errp) progress = (float )progress_current / progress_total *100.f ; }qemu_progress_print (progress,0 ); - }while (!job_is_ready (&job->job) && !job_is_completed (&job->job)); + }while (!job_is_ready_locked (&job->job) && + !job_is_completed_locked (&job->job)); -if (!job_is_completed (&job->job)) { +if (!job_is_completed_locked (&job->job)) { ret =job_complete_sync (&job->job, errp); }else { - ret =job_get_ret (&job->job); + ret =job_get_ret_locked (&job->job); }job_unref (&job->job); -aio_context_release (aio_context);/* publish completion progress only when success */ if (!ret) {
@@ -1076,9 +1078,12 @@ static int img_commit(int argc, char **argv)bdrv_ref (bs); } +job_lock (); job =block_job_get ("commit" );assert (job);run_block_job (job, &local_err); +job_unlock (); +if (local_err) { goto unref_backing; }
Signature · 1 line
2.31.1
Re: [RFC PATCH 5/6] job: use global job_mutex to protect struct Job
Quoted 95 lines · Emanuele Giuseppe Esposito
This lock is going to replace most of the AioContext locks in the job and blockjob, so that a Job can run in an arbitrary AioContext.
Signed-off-by Emanuele Giuseppe Esposito <[email protected]>--- include/block/blockjob_int.h | 1 + include/qemu/job.h | 2 + block/backup.c | 4 + block/mirror.c | 11 +- blockdev.c | 62 ++++---- blockjob.c | 67 +++++++-- job-qmp.c | 55 +++---- job.c | 284 +++++++++++++++++++++++++++-------- qemu-img.c | 15 +- 9 files changed, 350 insertions(+), 151 deletions(-)include/block/blockjob_int.h +1
-0@@ -53,6 +53,7 @@ struct BlockJobDriver {*/ void (*attached_aio_context)(BlockJob *job,AioContext *new_context); +/* Called with job mutex *not* held. */ void (*set_speed)(BlockJob *job,int64_t speed); };include/qemu/job.h +2
-0@@ -49,6 +49,8 @@ typedef struct Job {/** * The type of this job. * Set it in job_create and just read. +* All calls to the driver function must be not locked by job_mutex, +* to avoid deadlocks. */ const JobDriver *driver;block/backup.c +4
-0@@ -315,6 +315,10 @@ static void coroutine_fn backup_pause(Job *job) } } +/* +* Called with job mutex *not* held (we don't want to call block_copy_kick +* with the lock held!) +*/ static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) {BackupBlockJob *s =container_of (job, BackupBlockJob, common);block/mirror.c +8
-3@@ -1150,9 +1150,11 @@ static void mirror_complete(Job *job, Error **errp) s->should_complete = true;/* If the job is paused, it will be re-entered when it is resumed */ +job_lock ();if (!job_is_paused (job)) { -job_enter (job); +job_enter_locked (job); } +job_unlock (); } static void coroutine_fn mirror_pause(Job *job)@@ -1171,10 +1173,13 @@ static bool mirror_drained_poll(BlockJob *job)* from one of our own drain sections, to avoid a deadlock waiting for * ourselves. */ -if (!job_is_paused (&s->common.job) && !job_is_cancelled (&s->common.job) && - !s->in_drain) { +job_lock (); +if (!job_is_paused (&s->common.job) && + !job_is_cancelled_locked (&s->common.job) && !s->in_drain) { +job_unlock ();return true; } +job_unlock ();return !!s->in_flight; }blockdev.c +0
-0
blockdev.c void blockdev_mark_auto_del(BlockBackend *blk) · line 157
Quoted 8 lines
blockdev.c +2
-0@@ -150,9 +150,11 @@ void blockdev_mark_auto_del(BlockBackend *blk)AioContext *aio_context =job_get_aiocontext (&job->job);aio_context_acquire (aio_context); +job_lock ();job_cancel (&job->job, false);aio_context_release (aio_context); +job_unlock ();
This looks strange. The way it's written suggests there is a reason why job_unlock() has to be called after aio_context_release(). Can job_unlock() be called immediately after job_cancel()?
blockdev.c out: · line 3314
Quoted 11 lines
blockdev.c +1
-3} }@@ -3309,48 +3311,44 @@ out:aio_context_release (aio_context); } -/* Get a block job using its ID and acquire its AioContext */ -static BlockJob *find_block_job (const char *id,AioContext **aio_context, -Error **errp) +/* Get a block job using its ID and acquire its job_lock */
"its" suggests job_lock is per-Job. I suggest saying something like "Returns with job_lock held on success" instead.
Re: [RFC PATCH 5/6] job: use global job_mutex to protect struct Job
Quoted 102 lines · Stefan Hajnoczi
Quoted 101 lines · Emanuele Giuseppe Esposito
This lock is going to replace most of the AioContext locks in the job and blockjob, so that a Job can run in an arbitrary AioContext.
Signed-off-by Emanuele Giuseppe Esposito <[email protected]>--- include/block/blockjob_int.h | 1 + include/qemu/job.h | 2 + block/backup.c | 4 + block/mirror.c | 11 +- blockdev.c | 62 ++++---- blockjob.c | 67 +++++++-- job-qmp.c | 55 +++---- job.c | 284 +++++++++++++++++++++++++++-------- qemu-img.c | 15 +- 9 files changed, 350 insertions(+), 151 deletions(-)include/block/blockjob_int.h +1
-0@@ -53,6 +53,7 @@ struct BlockJobDriver {*/ void (*attached_aio_context)(BlockJob *job,AioContext *new_context); +/* Called with job mutex *not* held. */ void (*set_speed)(BlockJob *job,int64_t speed); };include/qemu/job.h +2
-0@@ -49,6 +49,8 @@ typedef struct Job {/** * The type of this job. * Set it in job_create and just read. +* All calls to the driver function must be not locked by job_mutex, +* to avoid deadlocks. */ const JobDriver *driver;block/backup.c +4
-0@@ -315,6 +315,10 @@ static void coroutine_fn backup_pause(Job *job) } } +/* +* Called with job mutex *not* held (we don't want to call block_copy_kick +* with the lock held!) +*/ static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed) {BackupBlockJob *s =container_of (job, BackupBlockJob, common);block/mirror.c +8
-3@@ -1150,9 +1150,11 @@ static void mirror_complete(Job *job, Error **errp) s->should_complete = true;/* If the job is paused, it will be re-entered when it is resumed */ +job_lock ();if (!job_is_paused (job)) { -job_enter (job); +job_enter_locked (job); } +job_unlock (); } static void coroutine_fn mirror_pause(Job *job)@@ -1171,10 +1173,13 @@ static bool mirror_drained_poll(BlockJob *job)* from one of our own drain sections, to avoid a deadlock waiting for * ourselves. */ -if (!job_is_paused (&s->common.job) && !job_is_cancelled (&s->common.job) && - !s->in_drain) { +job_lock (); +if (!job_is_paused (&s->common.job) && + !job_is_cancelled_locked (&s->common.job) && !s->in_drain) { +job_unlock ();return true; } +job_unlock ();return !!s->in_flight; }blockdev.c +1
-0@@ -150,9 +150,11 @@ void blockdev_mark_auto_del(BlockBackend *blk)AioContext *aio_context =job_get_aiocontext (&job->job);aio_context_acquire (aio_context); +job_lock ();job_cancel (&job->job, false);
Quoted 6 lines
blockdev.c void blockdev_mark_auto_del(BlockBackend *blk) · line 157
Quoted 2 lines
blockdev.c +1
-0aio_context_release (aio_context); +job_unlock ();This looks strange. The way it's written suggests there is a reason why
job_unlock()has to be called afteraio_context_release(). Canjob_unlock()be called immediately afterjob_cancel()?
Yes, another mistake I shouldn't have done.
Quoted 8 lines
Quoted 8 lines
blockdev.c +0
-0} }@@ -3309,48 +3311,44 @@ out:aio_context_release (aio_context); }
Quoted 8 lines
blockdev.c out: · line 3314
Quoted 4 lines
blockdev.c +1
-3-/* Get a block job using its ID and acquire its AioContext */ -static BlockJob *find_block_job (const char *id,AioContext **aio_context, -Error **errp) +/* Get a block job using its ID and acquire its job_lock */ "its" suggests
job_lockis per-Job. I suggest saying something like "Returns withjob_lockheld on success" instead.
Agree. Changed the same comment also for find_job() in job-qmp.c
Emanuele
RFC PATCH 6/6 jobs: remove unnecessary AioContext aquire/release pairs
In reply to Emanuele Giuseppe EspositoNow that we use the job_mutex, remove unnecessary aio_context_acquire/release pairs. However, some place still needs it, so try to reduce the aio_context critical section to the minimum.
This patch is separated from the one before because here we are removing locks without substituting it with aiocontext_acquire/release pairs.
These sections will also be removed in future, when the underlaying bdrv_* API will also be free of context locks.
--- block/mirror.c | 6 ++ block/monitor/block-hmp-cmds.c | 6 -- blockdev.c | 173 ++++++++------------------------- blockjob.c | 3 + job.c | 9 +- qemu-img.c | 4 - 6 files changed, 54 insertions(+), 147 deletions(-)
block/mirror.c +6 -0
@@ -1857,6 +1857,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs, {bool is_none_mode;BlockDriverState *base; +AioContext *aio_context;if ((mode ==MIRROR_SYNC_MODE_INCREMENTAL ) || (mode ==MIRROR_SYNC_MODE_BITMAP )) {
@@ -1866,11 +1867,16 @@ void mirror_start(const char *job_id, BlockDriverState *bs, } is_none_mode = mode ==MIRROR_SYNC_MODE_NONE ; base = mode ==MIRROR_SYNC_MODE_TOP ?bdrv_backing_chain_next (bs) :NULL ; + + aio_context =bdrv_get_aio_context (bs); +aio_context_acquire (aio_context);mirror_start_job (job_id, bs, creation_flags, target, replaces, speed, granularity, buf_size, backing_mode, zero_target, on_source_error, on_target_error, unmap,NULL ,NULL , &mirror_job_driver, is_none_mode, base, false, filter_node_name, true, copy_mode, errp); +aio_context_release (aio_context); + } BlockJob *commit_active_start(const char *job_id, BlockDriverState *bs,
block/monitor/block-hmp-cmds.c +0 -6
@@ -206,7 +206,6 @@ void hmp_commit(Monitor *mon, const QDict *qdict) ret =blk_commit_all (); }else {BlockDriverState *bs; -AioContext *aio_context; blk =blk_by_name (device);if (!blk) {
@@ -219,12 +218,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict) } bs =bdrv_skip_implicit_filters (blk_bs (blk)); - aio_context =bdrv_get_aio_context (bs); -aio_context_acquire (aio_context); - ret =bdrv_commit (bs); - -aio_context_release (aio_context); }if (ret <0 ) {error_report ("'commit' error for '%s': %s" , device,strerror (-ret));
blockdev.c +42 -131
522 lines not on this page · raw message
blockjob.c +3 -0
@@ -195,6 +195,7 @@ static const BdrvChildClass child_job = { .get_parent_aio_context = child_job_get_parent_aio_context, }; +/* Called with BQL held. */ void block_job_remove_all_bdrv (BlockJob *job) {/*
@@ -216,6 +217,7 @@ void block_job_remove_all_bdrv(BlockJob *job) } } +/* Called with BQL held. */ bool block_job_has_bdrv (BlockJob *job,BlockDriverState *bs) {GSList *el;
@@ -230,6 +232,7 @@ bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs)return false; } +/* Called with BQL held. */ int block_job_add_bdrv (BlockJob *job,const char *name,BlockDriverState *bs,uint64_t perm,uint64_t shared_perm,Error **errp) {
job.c +3 -6
@@ -220,7 +220,6 @@ static int job_txn_apply(Job *job, int fn(Job *))* break AIO_WAIT_WHILE from within fn. */ job_ref (job); -aio_context_release (job->aio_context); QLIST_FOREACH_SAFE(other_job, &txn->jobs, txn_list, next) { inner_ctx = other_job->aio_context;
@@ -232,11 +231,6 @@ static int job_txn_apply(Job *job, int fn(Job *)) } } -/* -* Note that job->aio_context might have been changed by calling fn, so we -* can't use a local variable to cache it. -*/ -aio_context_acquire (job->aio_context);job_unref (job);return rc; }
@@ -515,8 +509,11 @@ void job_unref(Job *job)assert (!job->txn);if (job->driver->free) { +AioContext *ctx =job_get_aiocontext (job);job_unlock (); +aio_context_acquire (ctx); job->driver->free (job); +aio_context_release (ctx);job_lock (); }
qemu-img.c +0 -4
@@ -946,7 +946,6 @@ static int img_commit(int argc, char **argv)Error *local_err =NULL ;CommonBlockJobCBInfo cbi;bool image_opts = false; -AioContext *aio_context;int64_t rate_limit =0 ; fmt =NULL ;
@@ -1060,12 +1059,9 @@ static int img_commit(int argc, char **argv) .bs = bs, }; - aio_context =bdrv_get_aio_context (bs); -aio_context_acquire (aio_context);commit_active_start ("commit" , bs, base_bs,JOB_DEFAULT , rate_limit,BLOCKDEV_ON_ERROR_REPORT ,NULL , common_block_job_cb, &cbi, false, &local_err); -aio_context_release (aio_context);if (local_err) { goto done; }
Signature · 1 line
2.31.1
Quoted 33 lines · Emanuele Giuseppe Esposito
This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.
This series aims to:
1) remove the aiocontext lock and substitute it with the already existing globaljob_mutex
2) fix what it looks like to be an oversight when moving theblockjob.clogic into the more genericjob.c:job_mutexwas introduced especially to protectjob->busyflag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) usejob_mutexinstead of theaiocontext_lock
4) extend the reach of thejob_mutexto protect all shared fields that the job structure has.The reason why we propose to use the existing
job_mutexand not make one for each job is to keep things as simple as possible for now, and because the jobs are not in the execution critical path, so we can affort some delays.
Having a lock per job would increase overall complexity and increase the chances of deadlocks (one good example could be the job transactions, where multiple jobs are grouped together). Anyways, the per-job mutex can always be added in the future.Patch 1-4 are in preparation for patch 5. They try to simplify and clarify the
job_mutexusage. Patch 5 tries to add proper syncronization to the job structure, replacing the AioContext lock when necessary.
Patch 6 just removes unnecessary AioContext locks that are now unneeded.RFC: I am not sure the way I layed out the locks is ideal. But their usage should not make deadlocks. I also made sure the series passess all
qemu_iotests.
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.
Stefan
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).
Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
The I/O blk_*()/bdrv_*() *should* not require the caller to hold the AioContext lock; all drivers use their own CoMutex or QemuMutex when needed, and generic code should also be ready (caveat emptor). Others, such as reopen, are a mess that requires a separate audit. Restricting acquire/release to be only around those seems like a good starting point.
Paolo
Am 08.07.2021 um 13:32 hat Paolo Bonzini geschrieben:
Quoted 23 lines · Stefan Hajnoczi
Quoted 8 lines
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 5 lines
The I/O blk_*()/bdrv_*() *should* not require the caller to hold the AioContext lock; all drivers use their own CoMutex or QemuMutex when needed, and generic code should also be ready (caveat emptor). Others, such as reopen, are a mess that requires a separate audit. Restricting acquire/release to be only around those seems like a good starting point.
Reopen isn't just a mess, but in fact buggy. After the following patch goes in, the rule is simple: Don't hold any AioContext locks when calling bdrv_reopen_multiple().
'block: Acquire AioContexts during bdrv_reopen_multiple()'
https://lists.gnu.org/archive/html/qemu-block/2021-07/msg00238.html
It still takes AioContext locks when it calls into other functions that currently expect it, but that should be the same as usual then.
And once callers don't even hold the lock in the first place, we'll also get rid of the ugly temporary lock release across reopen.
Kevin
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.
Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
If yes, then it's a risk worth taking. If no, then spending time developing interim code, reviewing those patches, and risking breakage doesn't seem worth it. I'd rather wait for blk_*()/bdrv_*() to be fully complete and then see patches that delete aio_context_acquire() in most places or add locks in the remaining places where the caller was relying on the AioContext lock.
Stefan
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 3 lines
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?
If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.
Emanuele
Quoted 10 lines
If yes, then it's a risk worth taking. If no, then spending time developing interim code, reviewing those patches, and risking breakage doesn't seem worth it. I'd rather wait for blk_*()/bdrv_*() to be fully complete and then see patches that delete
aio_context_acquire()in most places or add locks in the remaining places where the caller was relying on the AioContext lock.Stefan
Quoted 31 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 3 lines
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
Quoted 7 lines
Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?
If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.
Block graph manipulation (all_bdrv_states and friends) requires the BQL. It has always been this way.
This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why are aio_context_acquire() calls still needed in block jobs?
AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.
Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs. Maybe there's a benefit to this patch series that I've missed?
Stefan
Quoted 37 lines · Stefan Hajnoczi
Quoted 36 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 3 lines
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?
Quoted 8 lines
Quoted 2 lines
If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.
Block graph manipulation (
all_bdrv_statesand friends) requires the BQL. It has always been this way.This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why
I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety? I work mostly with nbd and qcow2 drivers, and they never seemd to be thread-safe to me. For example, qcow2 driver has enough operations that are done from non-coroutine context and therefore qcow2 co-mutex just not locked.
Quoted 8 lines
are
aio_context_acquire()calls still needed in block jobs?
AIO_WAIT_WHILE()requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used byAIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.
Quoted 4 lines
Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs.
I agree. Add my bit of criticism:
What I dislike about the whole thread-safe update you do:
1. There is no proof of concept - some good example of multiqueue, or something that uses mutliple threads and shows good performance. Something that works, and shows where are we going to and why it is good. That may be draft patches with a lot of "FIXME" and "TODO", but working. For now I feel that I've spent my time to reviewing and proving to myself thread-safety of two previous thread-safe series, but I don't have a hope to see a benefit of it in the near future..
1.1 If we have a proof of concept, that also gives a kind of plan: a list of subsystems (patch series) to do step by step and finally come to what we want. Do you have a kind of plan (of the whole feature) now?
2. There are no tests: something that doesn't work before the series and start to work after. Why it is important:
All these thread-safe primitives are complicated enough. They hard to review and prove correctness. And very simple to break by new code. Tests that runs by CI proves that we don't break subsystems that are already thread-safe. For example, you've recently updated block-copy and several related things. But we have no tests. So, assume, a year later you finish the work of updating all other subsystems to be thread-safe. You'll have no guarantee that block-copy is still thread-safe, and you'll have to start from the beginning.
3. As I said, I really doubt that block drivers are already thread safe. An audit and/or tests are needed at least.
Signature · 2 lines
Best regards, Vladimir
Quoted 41 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Stefan Hajnoczi
Quoted 38 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 3 lines
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?
If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.
Quoted 7 lines
Quoted 5 lines
Block graph manipulation (
all_bdrv_statesand friends) requires the BQL. It has always been this way.This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why
I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety?
Emanuele :)
FWIW I took a look at the stream, mirror, and backup jobs today and couldn't find anything that's unsafe after this series. I was expecting to find issues but I think Emanuele and Paolo have taken care of them.
Quoted 13 lines
Quoted 8 lines
are
aio_context_acquire()calls still needed in block jobs?
AIO_WAIT_WHILE()requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used byAIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.Quoted 4 lines
Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs.
Quoted 5 lines
I agree. Add my bit of criticism:
What I dislike about the whole thread-safe update you do:
1. There is no proof of concept - some good example of multiqueue, or something that uses mutliple threads and shows good performance. Something that works, and shows where are we going to and why it is good. That may be draft patches with a lot of "FIXME" and "TODO", but working. For now I feel that I've spent my time to reviewing and proving to myself thread-safety of two previous thread-safe series, but I don't have a hope to see a benefit of it in the near future..
The multi-queue block layer should improve performance in cases where the bottleneck is a single IOThread. It will allow users to assign more than one IOThread.
But I think the bigger impact of this work will be addressing long-standing problems with the block layer's programming model. We continue to have IOThread bugs because there are many undocumented assumptions. With the multi-queue block layer the code switches to a more well-understood multi-threaded programming model and hopefully fewer issues will arise because there is no problematic AioContext lock to worry about.
Stefan
Quoted 48 lines · Stefan Hajnoczi
Quoted 47 lines · Vladimir Sementsov-Ogievskiy
Quoted 45 lines · Stefan Hajnoczi
Quoted 31 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 3 lines
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
Quoted 7 lines
Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?
If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.
Block graph manipulation (
all_bdrv_statesand friends) requires the BQL. It has always been this way.This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why
Quoted 7 lines
Quoted 1 line
I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety?
Emanuele :)
FWIW I took a look at the stream, mirror, and backup jobs today and couldn't find anything that's unsafe after this series. I was expecting to find issues but I think Emanuele and Paolo have taken care of them.
Hmm, do you mean that all jobs are thread-safe?
Looking at the mirror, what protects s->ops_in_flight for example? It's accessed from job coroutines and from mirror_top filter write operation.
Quoted 34 lines
Quoted 14 lines
Quoted 9 lines
are
aio_context_acquire()calls still needed in block jobs?
AIO_WAIT_WHILE()requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used byAIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.Quoted 4 lines
Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs.
Quoted 5 lines
I agree. Add my bit of criticism:
What I dislike about the whole thread-safe update you do:
1. There is no proof of concept - some good example of multiqueue, or something that uses mutliple threads and shows good performance. Something that works, and shows where are we going to and why it is good. That may be draft patches with a lot of "FIXME" and "TODO", but working. For now I feel that I've spent my time to reviewing and proving to myself thread-safety of two previous thread-safe series, but I don't have a hope to see a benefit of it in the near future..
The multi-queue block layer should improve performance in cases where the bottleneck is a single IOThread. It will allow users to assign more than one IOThread.
But I think the bigger impact of this work will be addressing long-standing problems with the block layer's programming model. We continue to have IOThread bugs because there are many undocumented assumptions. With the multi-queue block layer the code switches to a more well-understood multi-threaded programming model and hopefully fewer issues will arise because there is no problematic AioContext lock to worry about.
Stefan
Signature · 2 lines
Best regards, Vladimir
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 52 lines · Stefan Hajnoczi
Quoted 41 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Stefan Hajnoczi
Quoted 38 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines
What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock()-lock()).Quoted 6 lines
Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those
aio_context_lock()calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.Quoted 6 lines
In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.
Quoted 3 lines
The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?
Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?
If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.
Quoted 7 lines
Quoted 5 lines
Block graph manipulation (
all_bdrv_statesand friends) requires the BQL. It has always been this way.This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why
I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety?
Emanuele :)
Quoted 8 lines
Quoted 3 lines
FWIW I took a look at the stream, mirror, and backup jobs today and couldn't find anything that's unsafe after this series. I was expecting to find issues but I think Emanuele and Paolo have taken care of them.
Hmm, do you mean that all jobs are thread-safe?
Looking at the mirror, what protects
s->ops_in_flightfor example? It's accessed from job coroutines and frommirror_topfilter write operation.
You're right. I missed the bdrv_mirror_top BlockDriver:
.pwrite_zeroes -> bdrv_mirror_top_pwrite_zeroes -> active_write_prepare -> QTAILQ_INSERT_TAIL(&s->ops_in_flight, op, next)
This is not thread-safe. A CoMutex is needed here to protect MirrorBSDOpaque fields.
Stefan
Am 13.07.2021 um 15:10 hat Stefan Hajnoczi geschrieben:
Quoted 5 lines
AIO_WAIT_WHILE()requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used byAIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.
Polling something in a different AioContext from the main thread still temporarily drops the lock, which crashes if it isn't locked. I'm not sure if the documentation claims that the lock is needed in more cases, I guess you could interpret it either way.
Kevin
Quoted 11 lines · Kevin Wolf
Am 13.07.2021 um 15:10 hat Stefan Hajnoczi geschrieben:
Quoted 5 lines
AIO_WAIT_WHILE()requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used byAIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.Polling something in a different AioContext from the main thread still temporarily drops the lock, which crashes if it isn't locked. I'm not sure if the documentation claims that the lock is needed in more cases, I guess you could interpret it either way.
I'm claiming that the lock doesn't need to be dropped in that case anymore - as long as the condition we're polling is thread-safe. :)
Have I missed something that still need locking?
We could temporarily introduce an AIO_WAIT_WHILE_UNLOCKED() macro so that callers can be converted individually.
Stefan
Am 19.07.2021 um 11:29 hat Stefan Hajnoczi geschrieben:
Quoted 13 lines · Kevin Wolf
Quoted 11 lines
Am 13.07.2021 um 15:10 hat Stefan Hajnoczi geschrieben:
Quoted 5 lines
AIO_WAIT_WHILE()requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used byAIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.Polling something in a different AioContext from the main thread still temporarily drops the lock, which crashes if it isn't locked. I'm not sure if the documentation claims that the lock is needed in more cases, I guess you could interpret it either way.
Quoted 4 lines
I'm claiming that the lock doesn't need to be dropped in that case anymore - as long as the condition we're polling is thread-safe. :)
Have I missed something that still need locking?
I'm not sure if AIO_WAIT_WHILE() actually ever needed the locking. I think it's more a convenience thing since the callers would already hold the lock, so dropping it temporarily in AIO_WAIT_WHILE() means that the callers don't have to duplicate the temporary unlock everywhere.
Quoted 2 lines
We could temporarily introduce an
AIO_WAIT_WHILE_UNLOCKED()macro so that callers can be converted individually.
Yes, this makes sense to me.
Kevin
Quoted 13 lines · Emanuele Giuseppe Esposito
This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.
This series aims to:
1) remove the aiocontext lock and substitute it with the already existing globaljob_mutex
2) fix what it looks like to be an oversight when moving theblockjob.clogic into the more genericjob.c:job_mutexwas introduced especially to protectjob->busyflag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) usejob_mutexinstead of theaiocontext_lock
4) extend the reach of thejob_mutexto protect all shared fields that the job structure has.
Can you explain the big picture:
1. What are the rules for JobDrivers? Imagine you are implementing a new JobDriver. What do you need to know in order to write correct code?
2. What are the rules for monitor? The main pattern is looking up a job, invoking a job API on it, and then calling job_unlock().
Stefan
Quoted 15 lines · Stefan Hajnoczi
Quoted 13 lines · Emanuele Giuseppe Esposito
This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.
This series aims to:
1) remove the aiocontext lock and substitute it with the already existing globaljob_mutex
2) fix what it looks like to be an oversight when moving theblockjob.clogic into the more genericjob.c:job_mutexwas introduced especially to protectjob->busyflag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) usejob_mutexinstead of theaiocontext_lock
4) extend the reach of thejob_mutexto protect all shared fields that the job structure has.
Quoted 4 lines
Can you explain the big picture:
1. What are the rules for JobDrivers? Imagine you are implementing a new JobDriver. What do you need to know in order to write correct code?
I think that in general, the rules for JobDrivers remain the same. The job_mutex lock should be invisible (or almost) from the point of view of a JobDriver, because the job API available for it should take care of the necessary locking/unlocking.
Quoted 3 lines
2. What are the rules for monitor? The main pattern is looking up a job, invoking a job API on it, and then calling
job_unlock().
The monitor instead is aware of this lock: the reason for that is exactly what you have described here.
Looking up + invoking a job API operation (for example calling find_job() and then job_pause() ) must be performed with the same lock hold all the time, otherwise other threads could modify the job while the monitor runs its command.
Please let me know if something is not clear and/or if you have additional comments on this!
Emanuele
Quoted 3 lines
Stefan
Quoted 30 lines · Emanuele Giuseppe Esposito
Quoted 15 lines · Stefan Hajnoczi
Quoted 13 lines · Emanuele Giuseppe Esposito
This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.
This series aims to:
1) remove the aiocontext lock and substitute it with the already existing globaljob_mutex
2) fix what it looks like to be an oversight when moving theblockjob.clogic into the more genericjob.c:job_mutexwas introduced especially to protectjob->busyflag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) usejob_mutexinstead of theaiocontext_lock
4) extend the reach of thejob_mutexto protect all shared fields that the job structure has.Quoted 4 lines
Can you explain the big picture:
1. What are the rules for JobDrivers? Imagine you are implementing a new JobDriver. What do you need to know in order to write correct code?
I think that in general, the rules for JobDrivers remain the same. The
job_mutexlock should be invisible (or almost) from the point of view of a JobDriver, because the job API available for it should take care of the necessary locking/unlocking.Quoted 3 lines
2. What are the rules for monitor? The main pattern is looking up a job, invoking a job API on it, and then calling
job_unlock().
Quoted 6 lines
The monitor instead is aware of this lock: the reason for that is exactly what you have described here.
Looking up + invoking a job API operation (for example callingfind_job()and thenjob_pause()) must be performed with the same lock hold all the time, otherwise other threads could modify the job while the monitor runs its command.
That helps, thanks!
Stefan