Skip to contentSkip to conversation

job: replace AioContext lock with job_mutex

RFC PATCH In review Emanuele Giuseppe Esposito 6 participants 33 messages Jul 7, 2021 – Jul 19, 2021 Block layer

lore
Share & tools
lore
#0 lore raw

cover job: replace AioContext lock with job_mutex

This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.

This series aims to:
1) remove the aiocontext lock and substitute it with the already existing global job_mutex
2) fix what it looks like to be an oversight when moving the blockjob.c logic into the more generic job.c: job_mutex was introduced especially to protect job->busy flag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) use job_mutex instead of the aiocontext_lock
4) extend the reach of the job_mutex to protect all shared fields that the job structure has.

The reason why we propose to use the existing job_mutex and not make one for each job is to keep things as simple as possible for now, and because the jobs are not in the execution critical path, so we can affort some delays.
Having a lock per job would increase overall complexity and increase the chances of deadlocks (one good example could be the job transactions, where multiple jobs are grouped together). Anyways, the per-job mutex can always be added in the future.

Patch 1-4 are in preparation for patch 5. They try to simplify and clarify the job_mutex usage. Patch 5 tries to add proper syncronization to the job structure, replacing the AioContext lock when necessary.
Patch 6 just removes unnecessary AioContext locks that are now unneeded.

RFC: I am not sure the way I layed out the locks is ideal. But their usage should not make deadlocks. I also made sure the series passess all qemu_iotests.

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

I also apologize for the amount of changes in patch 5, any suggestion on how to improve the patch layout is also very much appreciated.

Emanuele Giuseppe Esposito (6):
job: use getter/setters instead of accessing the Job fields directly
job: _locked functions and public job_lock/unlock for next patch
job: minor changes to simplify locking
job.h: categorize job fields
job: use global job_mutex to protect struct Job
jobs: remove unnecessary AioContext aquire/release pairs

 include/block/blockjob_int.h   |   1 +
 include/qemu/job.h             | 159 ++++++++++--
 block.c                        |   2 +-
 block/backup.c                 |   4 +
 block/commit.c                 |   4 +-
 block/mirror.c                 |  30 ++-
 block/monitor/block-hmp-cmds.c |   6 -
 block/replication.c            |   3 +-
 blockdev.c                     | 235 ++++++------------
 blockjob.c                     | 140 +++++++----
 job-qmp.c                      |  65 +++--
 job.c                          | 432 ++++++++++++++++++++++++++-------
 qemu-img.c                     |  19 +-
 13 files changed, 724 insertions(+), 376 deletions(-)
Signature · 1 line
2.31.1
#1 lore raw

RFC PATCH 1/6 job: use getter/setters instead of accessing the Job fields directly

Using getters/setters we can have a more strict control on struct Job fields. The struct remains public, because it is also used as base class for BlockJobs and various, but replace all direct accesses to the fields we want to protect with getters/setters. This is in preparation to the locking patches.

No functional change intended.

---
 include/qemu/job.h  | 33 +++++++++++++++++++
 block.c             |  2 +-
 block/commit.c      |  4 +--
 block/mirror.c      | 17 +++++-----
 block/replication.c |  3 +-
 blockdev.c          |  2 +-
 blockjob.c          | 78 ++++++++++++++++++++++++---------------------
 job-qmp.c           | 16 ++++++----
 job.c               | 52 +++++++++++++++++++++++++++++-
 qemu-img.c          |  2 +-
 10 files changed, 151 insertions(+), 58 deletions(-)
include/qemu/job.h +33 -0
index 41162ed494..72c7d0f69d 100644
@@ -303,6 +303,39 @@ void job_txn_unref(JobTxn *txn);
  */
 void job_txn_add_job(JobTxn *txn, Job *job);
 
+/** Returns the @ret field of a given Job. */
+int job_get_ret(Job *job);
+
+/** Returns the AioContext of a given Job. */
+AioContext *job_get_aiocontext(Job *job);
+
+/** Sets the AioContext of a given Job. */
+void job_set_aiocontext(Job *job, AioContext *aio);
+
+/** Returns if a given Job is busy. */
+bool job_is_busy(Job *job);
+
+/** Returns the Error of a given Job. */
+Error *job_get_err(Job *job);
+
+/** Returns if a Job has a pause_count > 0. */
+bool job_should_pause(Job *job);
+
+/** Sets the user_paused flag of a given Job to true. */
+void job_set_user_paused(Job *job);
+
+/** Sets the cancelled flag of a given Job. */
+void job_set_cancelled(Job *job, bool cancel);
+
+/** Returns if a given Job is paused. */
+bool job_is_paused(Job *job);
+
+/** Returns if a given Job is force cancelled. */
+bool job_is_force_cancel(Job *job);
+
+/** Returns the statis of a given Job. */
+JobStatus job_get_status(Job *job);
+
 /**
  * Create a new long-running job and return it.
  *
block.c +1 -1
index acd35cb0cb..1628db2550 100644
@@ -5721,7 +5721,7 @@ XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp)
         GSList *el;
 
         xdbg_graph_add_node(gr, job, X_DBG_BLOCK_GRAPH_NODE_TYPE_BLOCK_JOB,
-                           job->job.id);
+                            job->job.id);
         for (el = job->nodes; el; el = el->next) {
             xdbg_graph_add_edge(gr, job, (BdrvChild *)el->data);
         }
block/commit.c +2 -2
index 42792b4556..087865953e 100644
@@ -367,7 +367,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
         goto fail;
     }
 
-    s->base = blk_new(s->common.job.aio_context,
+    s->base = blk_new(job_get_aiocontext(&s->common.job),
                       base_perms,
                       BLK_PERM_CONSISTENT_READ
                       | BLK_PERM_GRAPH_MOD
@@ -380,7 +380,7 @@ void commit_start(const char *job_id, BlockDriverState *bs,
     s->base_bs = base;
 
     /* Required permissions are already taken with block_job_add_bdrv() */
-    s->top = blk_new(s->common.job.aio_context, 0, BLK_PERM_ALL);
+    s->top = blk_new(job_get_aiocontext(&s->common.job), 0, BLK_PERM_ALL);
     ret = blk_insert_bs(s->top, top, errp);
     if (ret < 0) {
         goto fail;
block/mirror.c +9 -8
index 019f6deaa5..49aaaafffa 100644
@@ -636,7 +636,7 @@ static int mirror_exit_common(Job *job)
     BlockDriverState *target_bs;
     BlockDriverState *mirror_top_bs;
     Error *local_err = NULL;
-    bool abort = job->ret < 0;
+    bool abort = job_get_ret(job) < 0;
     int ret = 0;
 
     if (s->prepared) {
@@ -930,7 +930,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
         while (!job_is_cancelled(&s->common.job) && !s->should_complete) {
             job_yield(&s->common.job);
         }
-        s->common.job.cancelled = false;
+        job_set_cancelled(&s->common.job, false);
         goto immediate_exit;
     }
 
@@ -1065,7 +1065,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
              * completion.
              */
             assert(QLIST_EMPTY(&bs->tracked_requests));
-            s->common.job.cancelled = false;
+            job_set_cancelled(&s->common.job, false);
             need_drain = false;
             break;
         }
@@ -1079,7 +1079,7 @@ static int coroutine_fn mirror_run(Job *job, Error **errp)
         trace_mirror_before_sleep(s, cnt, s->synced, delay_ns);
         job_sleep_ns(&s->common.job, delay_ns);
         if (job_is_cancelled(&s->common.job) &&
-            (!s->synced || s->common.job.force_cancel))
+            (!s->synced || job_is_force_cancel(&s->common.job)))
         {
             break;
         }
@@ -1092,8 +1092,8 @@ immediate_exit:
          * or it was cancelled prematurely so that we do not guarantee that
          * the target is a copy of the source.
          */
-        assert(ret < 0 || ((s->common.job.force_cancel || !s->synced) &&
-               job_is_cancelled(&s->common.job)));
+        assert(ret < 0 || ((job_is_force_cancel(&s->common.job) || !s->synced)
+               && job_is_cancelled(&s->common.job)));
         assert(need_drain);
         mirror_wait_for_all_io(s);
     }
@@ -1150,7 +1150,7 @@ static void mirror_complete(Job *job, Error **errp)
     s->should_complete = true;
 
     /* If the job is paused, it will be re-entered when it is resumed */
-    if (!job->paused) {
+    if (!job_is_paused(job)) {
         job_enter(job);
     }
 }
@@ -1171,7 +1171,8 @@ static bool mirror_drained_poll(BlockJob *job)
      * from one of our own drain sections, to avoid a deadlock waiting for
      * ourselves.
      */
-    if (!s->common.job.paused && !s->common.job.cancelled && !s->in_drain) {
+    if (!job_is_paused(&s->common.job) && !job_is_cancelled(&s->common.job) &&
+        !s->in_drain) {
         return true;
     }
 
block/replication.c +2 -1
index 52163f2d1f..3923761a54 100644
@@ -149,7 +149,8 @@ static void replication_close(BlockDriverState *bs)
     }
     if (s->stage == BLOCK_REPLICATION_FAILOVER) {
         commit_job = &s->commit_job->job;
-        assert(commit_job->aio_context == qemu_get_current_aio_context());
+        assert(job_get_aiocontext(commit_job) ==
+               qemu_get_current_aio_context());
         job_cancel_sync(commit_job);
     }
 
blockdev.c +1 -1
index f08192deda..8e2c15370e 100644
@@ -147,7 +147,7 @@ void blockdev_mark_auto_del(BlockBackend *blk)
 
     for (job = block_job_next(NULL); job; job = block_job_next(job)) {
         if (block_job_has_bdrv(job, blk_bs(blk))) {
-            AioContext *aio_context = job->job.aio_context;
+            AioContext *aio_context = job_get_aiocontext(&job->job);
             aio_context_acquire(aio_context);
 
             job_cancel(&job->job, false);
blockjob.c +42 -36

161 lines not on this page · raw message

job-qmp.c +9 -7
index 829a28aa70..12238a1643 100644
@@ -42,7 +42,7 @@ static Job *find_job(const char *id, AioContext **aio_context, Error **errp)
         return NULL;
     }
 
-    *aio_context = job->aio_context;
+    *aio_context = job_get_aiocontext(job);
     aio_context_acquire(*aio_context);
 
     return job;
@@ -122,7 +122,7 @@ void qmp_job_finalize(const char *id, Error **errp)
      * automatically acquires the new one), so make sure we release the correct
      * one.
      */
-    aio_context = job->aio_context;
+    aio_context = job_get_aiocontext(job);
     job_unref(job);
     aio_context_release(aio_context);
 }
@@ -146,21 +146,23 @@ static JobInfo *job_query_single(Job *job, Error **errp)
     JobInfo *info;
     uint64_t progress_current;
     uint64_t progress_total;
+    Error *job_err;
 
     assert(!job_is_internal(job));
     progress_get_snapshot(&job->progress, &progress_current,
                           &progress_total);
+    job_err = job_get_err(job);
 
     info = g_new(JobInfo, 1);
     *info = (JobInfo) {
         .id                 = g_strdup(job->id),
         .type               = job_type(job),
-        .status             = job->status,
+        .status             = job_get_status(job),
         .current_progress   = progress_current,
         .total_progress     = progress_total,
-        .has_error          = !!job->err,
-        .error              = job->err ? \
-                              g_strdup(error_get_pretty(job->err)) : NULL,
+        .has_error          = !!job_err,
+        .error              = job_err ? \
+                              g_strdup(error_get_pretty(job_err)) : NULL,
     };
 
     return info;
@@ -178,7 +180,7 @@ JobInfoList *qmp_query_jobs(Error **errp)
         if (job_is_internal(job)) {
             continue;
         }
-        aio_context = job->aio_context;
+        aio_context = job_get_aiocontext(job);
         aio_context_acquire(aio_context);
         value = job_query_single(job, errp);
         aio_context_release(aio_context);
job.c +51 -1

74 lines not on this page · raw message

qemu-img.c +1 -1
index 7956a89965..d16bd367d9 100644
@@ -921,7 +921,7 @@ static void run_block_job(BlockJob *job, Error **errp)
     if (!job_is_completed(&job->job)) {
         ret = job_complete_sync(&job->job, errp);
     } else {
-        ret = job->job.ret;
+        ret = job_get_ret(&job->job);
     }
     job_unref(&job->job);
     aio_context_release(aio_context);
Signature · 1 line
2.31.1
#2 lore raw

RFC PATCH 2/6 job: _locked functions and public job_lock/unlock for next patch

In reply to Emanuele Giuseppe Esposito

Create _locked functions, to make next patch a little bit smaller. Also set the locking functions as public, so that they can be used also from structures using the Job struct.

---
 include/qemu/job.h | 23 +++++++++++++
 job.c              | 85 ++++++++++++++++++++++++++++++++++++++--------
 2 files changed, 93 insertions(+), 15 deletions(-)
include/qemu/job.h +23 -0
index 72c7d0f69d..ba2f9b2660 100644
@@ -305,6 +305,7 @@ void job_txn_add_job(JobTxn *txn, Job *job);
 
 /** Returns the @ret field of a given Job. */
 int job_get_ret(Job *job);
+int job_get_ret_locked(Job *job);
 
 /** Returns the AioContext of a given Job. */
 AioContext *job_get_aiocontext(Job *job);
@@ -336,6 +337,24 @@ bool job_is_force_cancel(Job *job);
 /** Returns the statis of a given Job. */
 JobStatus job_get_status(Job *job);
 
+/**
+ * job_lock:
+ *
+ * Take the mutex protecting the list of jobs and their status.
+ * Most functions called by the monitor need to call job_lock
+ * and job_unlock manually.  On the other hand, function called
+ * by the block jobs themselves and by the block layer will take the
+ * lock for you.
+ */
+void job_lock(void);
+
+/**
+ * job_unlock:
+ *
+ * Release the mutex protecting the list of jobs and their status.
+ */
+void job_unlock(void);
+
 /**
  * Create a new long-running job and return it.
  *
@@ -424,6 +443,7 @@ void job_start(Job *job);
  * Continue the specified job by entering the coroutine.
  */
 void job_enter(Job *job);
+void job_enter_locked(Job *job);
 
 /**
  * @job: The job that is ready to pause.
@@ -462,12 +482,15 @@ bool job_is_internal(Job *job);
 
 /** Returns whether the job is scheduled for cancellation. */
 bool job_is_cancelled(Job *job);
+bool job_is_cancelled_locked(Job *job);
 
 /** Returns whether the job is in a completed state. */
 bool job_is_completed(Job *job);
+bool job_is_completed_locked(Job *job);
 
 /** Returns whether the job is ready to be completed. */
 bool job_is_ready(Job *job);
+bool job_is_ready_locked(Job *job);
 
 /**
  * Request @job to pause at the next pause point. Must be paired with
job.c +70 -15

162 lines not on this page · raw message

Signature · 1 line
2.31.1
#3 lore raw

Re: [RFC PATCH 2/6] job: _locked functions and public job_lock/unlock for next patch

job.c line 36

Quoted 10 lines · Emanuele Giuseppe Esposito
job.c +2 -0
index 872bbebb01..96fb8e9730 100644
@@ -32,6 +32,10 @@
 #include "trace/trace-root.h"
 #include "qapi/qapi-events-job.h"
 
+/* job_mutex protexts the jobs list, but also the job operations. */
+static QemuMutex job_mutex;

It's unclear what protecting "job operations" means. I would prefer a fine-grained per-job lock that protects the job's fields instead of a global lock with an unclear scope.

job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 72

Quoted 12 lines
job.c +3 -1
+
+/* Protected by job_mutex */
 static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);
 
 /* Job State Transition Table */
@@ -64,27 +68,22 @@ bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = {
 /* Transactional group of jobs */
 struct JobTxn {
 
-    /* Is this txn being cancelled? */
+    /* Is this txn being cancelled? Atomic.*/
     bool aborting;

The comment says atomic but this field is not accessed using atomic operations (at least at this point in the patch series)?

job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 78

Quoted 8 lines
job.c +2 -2
-    /* List of jobs */
+    /* List of jobs. Protected by job_mutex. */
     QLIST_HEAD(, Job) jobs;
 
-    /* Reference count */
+    /* Reference count. Atomic. */
     int refcnt;

Same.

#4 lore raw

Re: [RFC PATCH 2/6] job: _locked functions and public job_lock/unlock for next patch

Quoted 9 lines · Stefan Hajnoczi
Quoted 8 lines · Emanuele Giuseppe Esposito
job.c +0 -0
index 872bbebb01..96fb8e9730 100644
@@ -32,6 +32,10 @@
  #include "trace/trace-root.h"
  #include "qapi/qapi-events-job.h"
 
Quoted 6 lines

job.c line 36

Quoted 2 lines
job.c +2 -0
+/* job_mutex protexts the jobs list, but also the job operations. */
+static QemuMutex job_mutex;

It's unclear what protecting "job operations" means. I would prefer a fine-grained per-job lock that protects the job's fields instead of a global lock with an unclear scope.

As I wrote in the cover letter, I wanted to try to keep things as simple as possible with a global lock. It is possible to try and have a per-job lock, but I don't know how complex will that be then. I will try and see what I can do.

Maybe "job_mutex protexts the jobs list, but also makes the job API thread-safe"?

Quoted 10 lines
Quoted 10 lines
job.c +2 -0
+
+/* Protected by job_mutex */
  static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);
 
  /* Job State Transition Table */
@@ -64,27 +68,22 @@ bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = {
  /* Transactional group of jobs */
  struct JobTxn {
 
Quoted 6 lines

job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 72

Quoted 3 lines
job.c +1 -1
-    /* Is this txn being cancelled? */
+    /* Is this txn being cancelled? Atomic.*/
      bool aborting;

The comment says atomic but this field is not accessed using atomic operations (at least at this point in the patch series)?

Yes sorry I messed up the hunks in one-two patches. These comments were supposed to be on patch 4 "job.h: categorize job fields". Even though that might also not be ideal, since that patch just introduces the comments, without applying the locking/protection yet. On the other side, if I merge everything together in patch 5, it will be even harder to read.

Emanuele

Quoted 12 lines

job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 78

Quoted 9 lines
job.c +2 -2
-    /* List of jobs */
+    /* List of jobs. Protected by job_mutex. */
      QLIST_HEAD(, Job) jobs;
 
-    /* Reference count */
+    /* Reference count. Atomic. */
      int refcnt;

Same.

#5 lore raw

Re: [RFC PATCH 2/6] job: _locked functions and public job_lock/unlock for next patch

Quoted 18 lines · Emanuele Giuseppe Esposito
Quoted 14 lines · Stefan Hajnoczi

job.c line 35

Quoted 9 lines · Emanuele Giuseppe Esposito
job.c +2 -0
index 872bbebb01..96fb8e9730 100644
@@ -32,6 +32,10 @@
  #include "trace/trace-root.h"
  #include "qapi/qapi-events-job.h"
+/* job_mutex protexts the jobs list, but also the job operations. */
+static QemuMutex job_mutex;

It's unclear what protecting "job operations" means. I would prefer a fine-grained per-job lock that protects the job's fields instead of a global lock with an unclear scope.

Quoted 7 lines

As I wrote in the cover letter, I wanted to try to keep things as simple as possible with a global lock. It is possible to try and have a per-job lock, but I don't know how complex will that be then.
I will try and see what I can do.

Maybe "job_mutex protexts the jobs list, but also makes the job API thread-safe"?

That's clearer, thanks. I thought "job operations" meant the processing that the actual block jobs do (commit, mirror, stream, backup).

Quoted 16 lines
Quoted 15 lines

job.c bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = { · line 71

Quoted 12 lines
job.c +3 -1
+
+/* Protected by job_mutex */
  static QLIST_HEAD(, Job) jobs = QLIST_HEAD_INITIALIZER(jobs);
  /* Job State Transition Table */
@@ -64,27 +68,22 @@ bool JobVerbTable[JOB_VERB__MAX][JOB_STATUS__MAX] = {
  /* Transactional group of jobs */
  struct JobTxn {
-    /* Is this txn being cancelled? */
+    /* Is this txn being cancelled? Atomic.*/
      bool aborting;

The comment says atomic but this field is not accessed using atomic operations (at least at this point in the patch series)?

Quoted 6 lines

Yes sorry I messed up the hunks in one-two patches. These comments were supposed to be on patch 4 "job.h: categorize job fields". Even though that might also not be ideal, since that patch just introduces the comments, without applying the locking/protection yet.
On the other side, if I merge everything together in patch 5, it will be even harder to read.

The commit description can describe changes that currently have no effect but are anticipating a later patch. That helps reviewers understand whether the change is intentional/correct.

Stefan

#6 lore raw

RFC PATCH 3/6 job: minor changes to simplify locking

In reply to Emanuele Giuseppe Esposito

Check for NULL id to job_get, so that in the next patch we can move job_get inside a single critical section of job_create.

Also add missing notifier_list_init for the on_idle NotifierList, which seems to have been forgot.

---
 job.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
job.c +12 -4
index 96fb8e9730..48b304c3ff 100644
@@ -375,6 +375,10 @@ Job *job_get(const char *id)
 {
     Job *job;
 
+    if (!id) {
+        return NULL;
+    }
+
     QLIST_FOREACH(job, &jobs, job_list) {
         if (job->id && !strcmp(id, job->id)) {
             return job;
@@ -406,15 +410,18 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
             error_setg(errp, "Invalid job ID '%s'", job_id);
             return NULL;
         }
-        if (job_get(job_id)) {
-            error_setg(errp, "Job ID '%s' already in use", job_id);
-            return NULL;
-        }
     } else if (!(flags & JOB_INTERNAL)) {
         error_setg(errp, "An explicit job ID is required");
         return NULL;
     }
 
+    job_lock();
+    if (job_get(job_id)) {
+        error_setg(errp, "Job ID '%s' already in use", job_id);
+        job_unlock();
+        return NULL;
+    }
+
     job = g_malloc0(driver->instance_size);
     job->driver        = driver;
     job->id            = g_strdup(job_id);
@@ -434,6 +441,7 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
     notifier_list_init(&job->on_finalize_completed);
     notifier_list_init(&job->on_pending);
     notifier_list_init(&job->on_ready);
+    notifier_list_init(&job->on_idle);
 
     job_state_transition(job, JOB_STATUS_CREATED);
     aio_timer_init(qemu_get_aio_context(), &job->sleep_timer,
Signature · 1 line
2.31.1
#7 lore raw

Re: [RFC PATCH 3/6] job: minor changes to simplify locking

Quoted 10 lines · Emanuele Giuseppe Esposito
job.c +0 -4
@@ -406,15 +410,18 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
             error_setg(errp, "Invalid job ID '%s'", job_id);
             return NULL;
         }
-        if (job_get(job_id)) {
-            error_setg(errp, "Job ID '%s' already in use", job_id);
-            return NULL;
-        }
     } else if (!(flags & JOB_INTERNAL)) {
         error_setg(errp, "An explicit job ID is required");

job.c void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn, · line 424

Quoted 10 lines
job.c +7 -0
         return NULL;
     }
 
+    job_lock();
+    if (job_get(job_id)) {
+        error_setg(errp, "Job ID '%s' already in use", job_id);
+        job_unlock();
+        return NULL;
+    }
+

Where is the matching job_unlock() in the success case? Please consider lock guard macros like QEMU_LOCK_GUARD()/WITH_QEMU_LOCK_GUARD() to prevent common errors.

#8 lore raw

Re: [RFC PATCH 3/6] job: minor changes to simplify locking

Quoted 14 lines · Stefan Hajnoczi
Quoted 13 lines · Emanuele Giuseppe Esposito
job.c +0 -4
@@ -406,15 +410,18 @@ void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn,
              error_setg(errp, "Invalid job ID '%s'", job_id);
              return NULL;
          }
-        if (job_get(job_id)) {
-            error_setg(errp, "Job ID '%s' already in use", job_id);
-            return NULL;
-        }
      } else if (!(flags & JOB_INTERNAL)) {
          error_setg(errp, "An explicit job ID is required");
          return NULL;
      }
 
Quoted 12 lines

job.c void *job_create(const char *job_id, const JobDriver *driver, JobTxn *txn, · line 424

Quoted 7 lines
job.c +7 -0
+    job_lock();
+    if (job_get(job_id)) {
+        error_setg(errp, "Job ID '%s' already in use", job_id);
+        job_unlock();
+        return NULL;
+    }
+

Where is the matching job_unlock() in the success case? Please consider lock guard macros like QEMU_LOCK_GUARD()/WITH_QEMU_LOCK_GUARD() to prevent common errors.

Again this is a dumb mistake, the job_lock/unlock lines should go on patch 5, not here. Apologies.

I did not use QEMU_LOCK_GUARD()/WITH_QEMU_LOCK_GUARD() yet because I added some assertions to make sure I also didn't create nested locking situations. The final version will certainly use them.

Emanuele

#9 lore raw

Re: [RFC PATCH 3/6] job: minor changes to simplify locking

In reply to Emanuele Giuseppe Esposito
Quoted 5 lines · Emanuele Giuseppe Esposito

Check for NULL id to job_get, so that in the next patch we can move job_get inside a single critical section of job_create.

Also add missing notifier_list_init for the on_idle NotifierList, which seems to have been forgot.

forgotten

Quoted 5 lines
---
 job.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)
Signature · 3 lines
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
#10 lore raw

RFC PATCH 4/6 job.h: categorize job fields

In reply to Emanuele Giuseppe Esposito

This makes it easier to understand what needs to be protected by a lock and what doesn't.

---
 include/qemu/job.h | 101 ++++++++++++++++++++++++++++++++++++---------
 1 file changed, 82 insertions(+), 19 deletions(-)
include/qemu/job.h +82 -19
index ba2f9b2660..4421d08d93 100644
@@ -40,24 +40,40 @@ typedef struct JobTxn JobTxn;
  * Long-running operation.
  */
 typedef struct Job {
-    /** The ID of the job. May be NULL for internal jobs. */
+    /**
+     * The ID of the job. May be NULL for internal jobs.
+     * Set it in job_create and just read.
+     */
     char *id;
 
-    /** The type of this job. */
+    /**
+     * The type of this job.
+     * Set it in job_create and just read.
+     */
     const JobDriver *driver;
 
-    /** Reference count of the block job */
+    /**
+     * Reference count of the block job.
+     * Protected by job_mutex.
+     */
     int refcnt;
 
-    /** Current state; See @JobStatus for details. */
+    /**
+     * Current state; See @JobStatus for details.
+     * Protected by job_mutex.
+     */
     JobStatus status;
 
-    /** AioContext to run the job coroutine in */
+    /**
+     * AioContext to run the job coroutine in.
+     * Atomic.
+     */
     AioContext *aio_context;
 
     /**
      * The coroutine that executes the job.  If not NULL, it is reentered when
      * busy is false and the job is cancelled.
+     * Set it in job_create and just read.
      */
     Coroutine *co;
 
@@ -70,13 +86,15 @@ typedef struct Job {
     /**
      * Counter for pause request. If non-zero, the block job is either paused,
      * or if busy == true will pause itself as soon as possible.
+     * Protected by job_mutex.
      */
     int pause_count;
 
     /**
      * Set to false by the job while the coroutine has yielded and may be
      * re-entered by job_enter(). There may still be I/O or event loop activity
-     * pending. Accessed under block_job_mutex (in blockjob.c).
+     * pending.
+     * Protected by job_mutex.
      *
      * When the job is deferred to the main loop, busy is true as long as the
      * bottom half is still pending.
@@ -86,12 +104,14 @@ typedef struct Job {
     /**
      * Set to true by the job while it is in a quiescent state, where
      * no I/O or event loop activity is pending.
+     * Protected by job_mutex.
      */
     bool paused;
 
     /**
      * Set to true if the job is paused by user.  Can be unpaused with the
      * block-job-resume QMP command.
+     * Protected by job_mutex.
      */
     bool user_paused;
 
@@ -100,22 +120,33 @@ typedef struct Job {
      * always be tested just before toggling the busy flag from false
      * to true.  After a job has been cancelled, it should only yield
      * if #aio_poll will ("sooner or later") reenter the coroutine.
+     * Protected by job_mutex.
      */
     bool cancelled;
 
     /**
      * Set to true if the job should abort immediately without waiting
      * for data to be in sync.
+     * Protected by job_mutex.
      */
     bool force_cancel;
 
-    /** Set to true when the job has deferred work to the main loop. */
+    /**
+     * Set to true when the job has deferred work to the main loop.
+     * Protected by job_mutex.
+     */
     bool deferred_to_main_loop;
 
-    /** True if this job should automatically finalize itself */
+    /**
+     * True if this job should automatically finalize itself.
+     * Set it in job_create and just read.
+     */
     bool auto_finalize;
 
-    /** True if this job should automatically dismiss itself */
+    /**
+     * True if this job should automatically dismiss itself.
+     * Set it in job_create and just read.
+     */
     bool auto_dismiss;
 
     ProgressMeter progress;
@@ -124,6 +155,7 @@ typedef struct Job {
      * Return code from @run and/or @prepare callback(s).
      * Not final until the job has reached the CONCLUDED status.
      * 0 on success, -errno on failure.
+     * Protected by job_mutex.
      */
     int ret;
 
@@ -131,37 +163,68 @@ typedef struct Job {
      * Error object for a failed job.
      * If job->ret is nonzero and an error object was not set, it will be set
      * to strerror(-job->ret) during job_completed.
+     * Protected by job_mutex.
      */
     Error *err;
 
-    /** The completion function that will be called when the job completes.  */
+    /**
+     * The completion function that will be called when the job completes.
+     * Set it in job_create and just read.
+     */
     BlockCompletionFunc *cb;
 
-    /** The opaque value that is passed to the completion function.  */
+    /**
+     * The opaque value that is passed to the completion function.
+     * Set it in job_create and just read.
+     */
     void *opaque;
 
-    /** Notifiers called when a cancelled job is finalised */
+    /**
+     * Notifiers called when a cancelled job is finalised.
+     * Protected by job_mutex.
+     */
     NotifierList on_finalize_cancelled;
 
-    /** Notifiers called when a successfully completed job is finalised */
+    /**
+     * Notifiers called when a successfully completed job is finalised.
+     * Protected by job_mutex.
+     */
     NotifierList on_finalize_completed;
 
-    /** Notifiers called when the job transitions to PENDING */
+    /**
+     * Notifiers called when the job transitions to PENDING.
+     * Protected by job_mutex.
+     */
     NotifierList on_pending;
 
-    /** Notifiers called when the job transitions to READY */
+    /**
+     * Notifiers called when the job transitions to READY.
+     * Protected by job_mutex.
+     */
     NotifierList on_ready;
 
-    /** Notifiers called when the job coroutine yields or terminates */
+    /**
+     * Notifiers called when the job coroutine yields or terminates.
+     * Protected by job_mutex.
+     */
     NotifierList on_idle;
 
-    /** Element of the list of jobs */
+    /**
+     * Element of the list of jobs.
+     * Protected by job_mutex.
+     */
     QLIST_ENTRY(Job) job_list;
 
-    /** Transaction this job is part of */
+    /**
+     * Transaction this job is part of.
+     * Protected by job_mutex.
+     */
     JobTxn *txn;
 
-    /** Element of the list of jobs in a job transaction */
+    /**
+     * Element of the list of jobs in a job transaction.
+     * Protected by job_mutex.
+     */
     QLIST_ENTRY(Job) txn_list;
 } Job;
 
Signature · 1 line
2.31.1
#11 lore raw

Re: [RFC PATCH 4/6] job.h: categorize job fields

include/qemu/job.h typedef struct JobTxn JobTxn; · line 71

Quoted 6 lines · Emanuele Giuseppe Esposito
include/qemu/job.h +4 -1
-    /** AioContext to run the job coroutine in */
+    /**
+     * AioContext to run the job coroutine in.
+     * Atomic.
+     */
     AioContext *aio_context;

This isn't accessed using atomic operations, so I'm not sure why it's documented as atomic?

#12 lore raw

Re: [RFC PATCH 4/6] job.h: categorize job fields

Quoted 11 lines · Stefan Hajnoczi

include/qemu/job.h typedef struct JobTxn JobTxn; · line 71

Quoted 6 lines · Emanuele Giuseppe Esposito
include/qemu/job.h +4 -1
-    /** AioContext to run the job coroutine in */
+    /**
+     * AioContext to run the job coroutine in.
+     * Atomic.
+     */
      AioContext *aio_context;

This isn't accessed using atomic operations, so I'm not sure why it's documented as atomic?

Maybe this is unnecessary, but from what I understand right now when we want to change the AioContext of a child node, we need to acquire its AioContext lock, and then we try to set it. Without AioContext locks, my understanding is that this has to be protected somehow. Therefore I thought of setting this pointer atomically (actual code that does this is in patch 5).

Again same reasoning style applies here: this patch just adds a bounce of comments, but merging it with next one would just make it more unreadable.

Emanuele

#13 lore raw

RFC PATCH 5/6 job: use global job_mutex to protect struct Job

In reply to Emanuele Giuseppe Esposito

This lock is going to replace most of the AioContext locks in the job and blockjob, so that a Job can run in an arbitrary AioContext.

---
 include/block/blockjob_int.h |   1 +
 include/qemu/job.h           |   2 +
 block/backup.c               |   4 +
 block/mirror.c               |  11 +-
 blockdev.c                   |  62 ++++----
 blockjob.c                   |  67 +++++++--
 job-qmp.c                    |  55 +++----
 job.c                        | 284 +++++++++++++++++++++++++++--------
 qemu-img.c                   |  15 +-
 9 files changed, 350 insertions(+), 151 deletions(-)
include/block/blockjob_int.h +1 -0
index 6633d83da2..8b91126506 100644
@@ -53,6 +53,7 @@ struct BlockJobDriver {
      */
     void (*attached_aio_context)(BlockJob *job, AioContext *new_context);
 
+    /* Called with job mutex *not* held. */
     void (*set_speed)(BlockJob *job, int64_t speed);
 };
 
include/qemu/job.h +2 -0
index 4421d08d93..359f4e6b3a 100644
@@ -49,6 +49,8 @@ typedef struct Job {
     /**
      * The type of this job.
      * Set it in job_create and just read.
+     * All calls to the driver function must be not locked by job_mutex,
+     * to avoid deadlocks.
      */
     const JobDriver *driver;
 
block/backup.c +4 -0
index bd3614ce70..80ce956299 100644
@@ -315,6 +315,10 @@ static void coroutine_fn backup_pause(Job *job)
     }
 }
 
+/*
+ * Called with job mutex *not* held (we don't want to call block_copy_kick
+ * with the lock held!)
+ */
 static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed)
 {
     BackupBlockJob *s = container_of(job, BackupBlockJob, common);
block/mirror.c +8 -3
index 49aaaafffa..deefaa6a39 100644
@@ -1150,9 +1150,11 @@ static void mirror_complete(Job *job, Error **errp)
     s->should_complete = true;
 
     /* If the job is paused, it will be re-entered when it is resumed */
+    job_lock();
     if (!job_is_paused(job)) {
-        job_enter(job);
+        job_enter_locked(job);
     }
+    job_unlock();
 }
 
 static void coroutine_fn mirror_pause(Job *job)
@@ -1171,10 +1173,13 @@ static bool mirror_drained_poll(BlockJob *job)
      * from one of our own drain sections, to avoid a deadlock waiting for
      * ourselves.
      */
-    if (!job_is_paused(&s->common.job) && !job_is_cancelled(&s->common.job) &&
-        !s->in_drain) {
+    job_lock();
+    if (!job_is_paused(&s->common.job) &&
+        !job_is_cancelled_locked(&s->common.job) && !s->in_drain) {
+        job_unlock();
         return true;
     }
+    job_unlock();
 
     return !!s->in_flight;
 }
blockdev.c +27 -35

163 lines not on this page · raw message

blockjob.c +56 -11

266 lines not on this page · raw message

job-qmp.c +21 -34

146 lines not on this page · raw message

job.c +221 -63

895 lines not on this page · raw message

qemu-img.c +10 -5
index d16bd367d9..82debde038 100644
@@ -898,17 +898,19 @@ static void common_block_job_cb(void *opaque, int ret)
     }
 }
 
+/* Called with job_mutex held. Releases it temporarly */
 static void run_block_job(BlockJob *job, Error **errp)
 {
     uint64_t progress_current, progress_total;
     AioContext *aio_context = blk_get_aio_context(job->blk);
     int ret = 0;
 
-    aio_context_acquire(aio_context);
     job_ref(&job->job);
     do {
         float progress = 0.0f;
+        job_unlock();
         aio_poll(aio_context, true);
+        job_lock();
 
         progress_get_snapshot(&job->job.progress, &progress_current,
                               &progress_total);
@@ -916,15 +918,15 @@ static void run_block_job(BlockJob *job, Error **errp)
             progress = (float)progress_current / progress_total * 100.f;
         }
         qemu_progress_print(progress, 0);
-    } while (!job_is_ready(&job->job) && !job_is_completed(&job->job));
+    } while (!job_is_ready_locked(&job->job) &&
+             !job_is_completed_locked(&job->job));
 
-    if (!job_is_completed(&job->job)) {
+    if (!job_is_completed_locked(&job->job)) {
         ret = job_complete_sync(&job->job, errp);
     } else {
-        ret = job_get_ret(&job->job);
+        ret = job_get_ret_locked(&job->job);
     }
     job_unref(&job->job);
-    aio_context_release(aio_context);
 
     /* publish completion progress only when success */
     if (!ret) {
@@ -1076,9 +1078,12 @@ static int img_commit(int argc, char **argv)
         bdrv_ref(bs);
     }
 
+    job_lock();
     job = block_job_get("commit");
     assert(job);
     run_block_job(job, &local_err);
+    job_unlock();
+
     if (local_err) {
         goto unref_backing;
     }
Signature · 1 line
2.31.1
#14 lore raw

Re: [RFC PATCH 5/6] job: use global job_mutex to protect struct Job

Quoted 95 lines · Emanuele Giuseppe Esposito

This lock is going to replace most of the AioContext locks in the job and blockjob, so that a Job can run in an arbitrary AioContext.

---
 include/block/blockjob_int.h |   1 +
 include/qemu/job.h           |   2 +
 block/backup.c               |   4 +
 block/mirror.c               |  11 +-
 blockdev.c                   |  62 ++++----
 blockjob.c                   |  67 +++++++--
 job-qmp.c                    |  55 +++----
 job.c                        | 284 +++++++++++++++++++++++++++--------
 qemu-img.c                   |  15 +-
 9 files changed, 350 insertions(+), 151 deletions(-)
include/block/blockjob_int.h +1 -0
index 6633d83da2..8b91126506 100644
@@ -53,6 +53,7 @@ struct BlockJobDriver {
      */
     void (*attached_aio_context)(BlockJob *job, AioContext *new_context);
 
+    /* Called with job mutex *not* held. */
     void (*set_speed)(BlockJob *job, int64_t speed);
 };
 
include/qemu/job.h +2 -0
index 4421d08d93..359f4e6b3a 100644
@@ -49,6 +49,8 @@ typedef struct Job {
     /**
      * The type of this job.
      * Set it in job_create and just read.
+     * All calls to the driver function must be not locked by job_mutex,
+     * to avoid deadlocks.
      */
     const JobDriver *driver;
 
block/backup.c +4 -0
index bd3614ce70..80ce956299 100644
@@ -315,6 +315,10 @@ static void coroutine_fn backup_pause(Job *job)
     }
 }
 
+/*
+ * Called with job mutex *not* held (we don't want to call block_copy_kick
+ * with the lock held!)
+ */
 static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed)
 {
     BackupBlockJob *s = container_of(job, BackupBlockJob, common);
block/mirror.c +8 -3
index 49aaaafffa..deefaa6a39 100644
@@ -1150,9 +1150,11 @@ static void mirror_complete(Job *job, Error **errp)
     s->should_complete = true;
 
     /* If the job is paused, it will be re-entered when it is resumed */
+    job_lock();
     if (!job_is_paused(job)) {
-        job_enter(job);
+        job_enter_locked(job);
     }
+    job_unlock();
 }
 
 static void coroutine_fn mirror_pause(Job *job)
@@ -1171,10 +1173,13 @@ static bool mirror_drained_poll(BlockJob *job)
      * from one of our own drain sections, to avoid a deadlock waiting for
      * ourselves.
      */
-    if (!job_is_paused(&s->common.job) && !job_is_cancelled(&s->common.job) &&
-        !s->in_drain) {
+    job_lock();
+    if (!job_is_paused(&s->common.job) &&
+        !job_is_cancelled_locked(&s->common.job) && !s->in_drain) {
+        job_unlock();
         return true;
     }
+    job_unlock();
 
     return !!s->in_flight;
 }
blockdev.c +0 -0
index 8e2c15370e..9255aea6a2 100644

blockdev.c void blockdev_mark_auto_del(BlockBackend *blk) · line 157

Quoted 8 lines
blockdev.c +2 -0
@@ -150,9 +150,11 @@ void blockdev_mark_auto_del(BlockBackend *blk)
             AioContext *aio_context = job_get_aiocontext(&job->job);
             aio_context_acquire(aio_context);
 
+            job_lock();
             job_cancel(&job->job, false);
 
             aio_context_release(aio_context);
+            job_unlock();

This looks strange. The way it's written suggests there is a reason why job_unlock() has to be called after aio_context_release(). Can job_unlock() be called immediately after job_cancel()?

blockdev.c out: · line 3314

Quoted 11 lines
blockdev.c +1 -3
         }
     }
 
@@ -3309,48 +3311,44 @@ out:
     aio_context_release(aio_context);
 }
 
-/* Get a block job using its ID and acquire its AioContext */
-static BlockJob *find_block_job(const char *id, AioContext **aio_context,
-                                Error **errp)
+/* Get a block job using its ID and acquire its job_lock */

"its" suggests job_lock is per-Job. I suggest saying something like "Returns with job_lock held on success" instead.

#15 lore raw

Re: [RFC PATCH 5/6] job: use global job_mutex to protect struct Job

Quoted 102 lines · Stefan Hajnoczi
Quoted 101 lines · Emanuele Giuseppe Esposito

This lock is going to replace most of the AioContext locks in the job and blockjob, so that a Job can run in an arbitrary AioContext.

---
  include/block/blockjob_int.h |   1 +
  include/qemu/job.h           |   2 +
  block/backup.c               |   4 +
  block/mirror.c               |  11 +-
  blockdev.c                   |  62 ++++----
  blockjob.c                   |  67 +++++++--
  job-qmp.c                    |  55 +++----
  job.c                        | 284 +++++++++++++++++++++++++++--------
  qemu-img.c                   |  15 +-
  9 files changed, 350 insertions(+), 151 deletions(-)
include/block/blockjob_int.h +1 -0
index 6633d83da2..8b91126506 100644
@@ -53,6 +53,7 @@ struct BlockJobDriver {
       */
      void (*attached_aio_context)(BlockJob *job, AioContext *new_context);
 
+    /* Called with job mutex *not* held. */
      void (*set_speed)(BlockJob *job, int64_t speed);
  };
 
include/qemu/job.h +2 -0
index 4421d08d93..359f4e6b3a 100644
@@ -49,6 +49,8 @@ typedef struct Job {
      /**
       * The type of this job.
       * Set it in job_create and just read.
+     * All calls to the driver function must be not locked by job_mutex,
+     * to avoid deadlocks.
       */
      const JobDriver *driver;
 
block/backup.c +4 -0
index bd3614ce70..80ce956299 100644
@@ -315,6 +315,10 @@ static void coroutine_fn backup_pause(Job *job)
      }
  }
 
+/*
+ * Called with job mutex *not* held (we don't want to call block_copy_kick
+ * with the lock held!)
+ */
  static void coroutine_fn backup_set_speed(BlockJob *job, int64_t speed)
  {
      BackupBlockJob *s = container_of(job, BackupBlockJob, common);
block/mirror.c +8 -3
index 49aaaafffa..deefaa6a39 100644
@@ -1150,9 +1150,11 @@ static void mirror_complete(Job *job, Error **errp)
      s->should_complete = true;
 
      /* If the job is paused, it will be re-entered when it is resumed */
+    job_lock();
      if (!job_is_paused(job)) {
-        job_enter(job);
+        job_enter_locked(job);
      }
+    job_unlock();
  }
 
  static void coroutine_fn mirror_pause(Job *job)
@@ -1171,10 +1173,13 @@ static bool mirror_drained_poll(BlockJob *job)
       * from one of our own drain sections, to avoid a deadlock waiting for
       * ourselves.
       */
-    if (!job_is_paused(&s->common.job) && !job_is_cancelled(&s->common.job) &&
-        !s->in_drain) {
+    job_lock();
+    if (!job_is_paused(&s->common.job) &&
+        !job_is_cancelled_locked(&s->common.job) && !s->in_drain) {
+        job_unlock();
          return true;
      }
+    job_unlock();
 
      return !!s->in_flight;
  }
blockdev.c +1 -0
index 8e2c15370e..9255aea6a2 100644
@@ -150,9 +150,11 @@ void blockdev_mark_auto_del(BlockBackend *blk)
              AioContext *aio_context = job_get_aiocontext(&job->job);
              aio_context_acquire(aio_context);
 
+            job_lock();
              job_cancel(&job->job, false);
 
Quoted 6 lines

blockdev.c void blockdev_mark_auto_del(BlockBackend *blk) · line 157

Quoted 2 lines
blockdev.c +1 -0
              aio_context_release(aio_context);
+            job_unlock();

This looks strange. The way it's written suggests there is a reason why job_unlock() has to be called after aio_context_release(). Can job_unlock() be called immediately after job_cancel()?

Yes, another mistake I shouldn't have done.

Quoted 8 lines
Quoted 8 lines
blockdev.c +0 -0
          }
      }
 
@@ -3309,48 +3311,44 @@ out:
      aio_context_release(aio_context);
  }
 
Quoted 8 lines

blockdev.c out: · line 3314

Quoted 4 lines
blockdev.c +1 -3
-/* Get a block job using its ID and acquire its AioContext */
-static BlockJob *find_block_job(const char *id, AioContext **aio_context,
-                                Error **errp)
+/* Get a block job using its ID and acquire its job_lock */

"its" suggests job_lock is per-Job. I suggest saying something like "Returns with job_lock held on success" instead.

Agree. Changed the same comment also for find_job() in job-qmp.c

Emanuele

#16 lore raw

RFC PATCH 6/6 jobs: remove unnecessary AioContext aquire/release pairs

In reply to Emanuele Giuseppe Esposito

Now that we use the job_mutex, remove unnecessary aio_context_acquire/release pairs. However, some place still needs it, so try to reduce the aio_context critical section to the minimum.

This patch is separated from the one before because here we are removing locks without substituting it with aiocontext_acquire/release pairs.

These sections will also be removed in future, when the underlaying bdrv_* API will also be free of context locks.

---
 block/mirror.c                 |   6 ++
 block/monitor/block-hmp-cmds.c |   6 --
 blockdev.c                     | 173 ++++++++-------------------------
 blockjob.c                     |   3 +
 job.c                          |   9 +-
 qemu-img.c                     |   4 -
 6 files changed, 54 insertions(+), 147 deletions(-)
block/mirror.c +6 -0
index deefaa6a39..8d30c53690 100644
@@ -1857,6 +1857,7 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
 {
     bool is_none_mode;
     BlockDriverState *base;
+    AioContext *aio_context;
 
     if ((mode == MIRROR_SYNC_MODE_INCREMENTAL) ||
         (mode == MIRROR_SYNC_MODE_BITMAP)) {
@@ -1866,11 +1867,16 @@ void mirror_start(const char *job_id, BlockDriverState *bs,
     }
     is_none_mode = mode == MIRROR_SYNC_MODE_NONE;
     base = mode == MIRROR_SYNC_MODE_TOP ? bdrv_backing_chain_next(bs) : NULL;
+
+    aio_context = bdrv_get_aio_context(bs);
+    aio_context_acquire(aio_context);
     mirror_start_job(job_id, bs, creation_flags, target, replaces,
                      speed, granularity, buf_size, backing_mode, zero_target,
                      on_source_error, on_target_error, unmap, NULL, NULL,
                      &mirror_job_driver, is_none_mode, base, false,
                      filter_node_name, true, copy_mode, errp);
+    aio_context_release(aio_context);
+
 }
 
 BlockJob *commit_active_start(const char *job_id, BlockDriverState *bs,
block/monitor/block-hmp-cmds.c +0 -6
index 3e6670c963..99095afae7 100644
@@ -206,7 +206,6 @@ void hmp_commit(Monitor *mon, const QDict *qdict)
         ret = blk_commit_all();
     } else {
         BlockDriverState *bs;
-        AioContext *aio_context;
 
         blk = blk_by_name(device);
         if (!blk) {
@@ -219,12 +218,7 @@ void hmp_commit(Monitor *mon, const QDict *qdict)
         }
 
         bs = bdrv_skip_implicit_filters(blk_bs(blk));
-        aio_context = bdrv_get_aio_context(bs);
-        aio_context_acquire(aio_context);
-
         ret = bdrv_commit(bs);
-
-        aio_context_release(aio_context);
     }
     if (ret < 0) {
         error_report("'commit' error for '%s': %s", device, strerror(-ret));
blockdev.c +42 -131

522 lines not on this page · raw message

blockjob.c +3 -0
index e7b289089b..633abb3811 100644
@@ -195,6 +195,7 @@ static const BdrvChildClass child_job = {
     .get_parent_aio_context = child_job_get_parent_aio_context,
 };
 
+/* Called with BQL held.  */
 void block_job_remove_all_bdrv(BlockJob *job)
 {
     /*
@@ -216,6 +217,7 @@ void block_job_remove_all_bdrv(BlockJob *job)
     }
 }
 
+/* Called with BQL held.  */
 bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs)
 {
     GSList *el;
@@ -230,6 +232,7 @@ bool block_job_has_bdrv(BlockJob *job, BlockDriverState *bs)
     return false;
 }
 
+/* Called with BQL held.  */
 int block_job_add_bdrv(BlockJob *job, const char *name, BlockDriverState *bs,
                        uint64_t perm, uint64_t shared_perm, Error **errp)
 {
job.c +3 -6
index e2006532b5..b86fce3679 100644
@@ -220,7 +220,6 @@ static int job_txn_apply(Job *job, int fn(Job *))
      * break AIO_WAIT_WHILE from within fn.
      */
     job_ref(job);
-    aio_context_release(job->aio_context);
 
     QLIST_FOREACH_SAFE(other_job, &txn->jobs, txn_list, next) {
         inner_ctx = other_job->aio_context;
@@ -232,11 +231,6 @@ static int job_txn_apply(Job *job, int fn(Job *))
         }
     }
 
-    /*
-     * Note that job->aio_context might have been changed by calling fn, so we
-     * can't use a local variable to cache it.
-     */
-    aio_context_acquire(job->aio_context);
     job_unref(job);
     return rc;
 }
@@ -515,8 +509,11 @@ void job_unref(Job *job)
         assert(!job->txn);
 
         if (job->driver->free) {
+            AioContext *ctx = job_get_aiocontext(job);
             job_unlock();
+            aio_context_acquire(ctx);
             job->driver->free(job);
+            aio_context_release(ctx);
             job_lock();
         }
 
qemu-img.c +0 -4
index 82debde038..10bbe88b03 100644
@@ -946,7 +946,6 @@ static int img_commit(int argc, char **argv)
     Error *local_err = NULL;
     CommonBlockJobCBInfo cbi;
     bool image_opts = false;
-    AioContext *aio_context;
     int64_t rate_limit = 0;
 
     fmt = NULL;
@@ -1060,12 +1059,9 @@ static int img_commit(int argc, char **argv)
         .bs   = bs,
     };
 
-    aio_context = bdrv_get_aio_context(bs);
-    aio_context_acquire(aio_context);
     commit_active_start("commit", bs, base_bs, JOB_DEFAULT, rate_limit,
                         BLOCKDEV_ON_ERROR_REPORT, NULL, common_block_job_cb,
                         &cbi, false, &local_err);
-    aio_context_release(aio_context);
     if (local_err) {
         goto done;
     }
Signature · 1 line
2.31.1
#17 lore raw
In reply to Emanuele Giuseppe Esposito
Quoted 33 lines · Emanuele Giuseppe Esposito

This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.

This series aims to:
1) remove the aiocontext lock and substitute it with the already existing global job_mutex
2) fix what it looks like to be an oversight when moving the blockjob.c logic into the more generic job.c: job_mutex was introduced especially to protect job->busy flag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) use job_mutex instead of the aiocontext_lock
4) extend the reach of the job_mutex to protect all shared fields that the job structure has.

The reason why we propose to use the existing job_mutex and not make one for each job is to keep things as simple as possible for now, and because the jobs are not in the execution critical path, so we can affort some delays.
Having a lock per job would increase overall complexity and increase the chances of deadlocks (one good example could be the job transactions, where multiple jobs are grouped together). Anyways, the per-job mutex can always be added in the future.

Patch 1-4 are in preparation for patch 5. They try to simplify and clarify the job_mutex usage. Patch 5 tries to add proper syncronization to the job structure, replacing the AioContext lock when necessary.
Patch 6 just removes unnecessary AioContext locks that are now unneeded.

RFC: I am not sure the way I layed out the locks is ideal. But their usage should not make deadlocks. I also made sure the series passess all qemu_iotests.

Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Stefan

#18 lore raw
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

The I/O blk_*()/bdrv_*() *should* not require the caller to hold the AioContext lock; all drivers use their own CoMutex or QemuMutex when needed, and generic code should also be ready (caveat emptor). Others, such as reopen, are a mess that requires a separate audit. Restricting acquire/release to be only around those seems like a good starting point.

Paolo

#19 lore raw

Am 08.07.2021 um 13:32 hat Paolo Bonzini geschrieben:

Quoted 23 lines · Stefan Hajnoczi
Quoted 8 lines
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 5 lines

The I/O blk_*()/bdrv_*() *should* not require the caller to hold the AioContext lock; all drivers use their own CoMutex or QemuMutex when needed, and generic code should also be ready (caveat emptor). Others, such as reopen, are a mess that requires a separate audit. Restricting acquire/release to be only around those seems like a good starting point.

Reopen isn't just a mess, but in fact buggy. After the following patch goes in, the rule is simple: Don't hold any AioContext locks when calling bdrv_reopen_multiple().

'block: Acquire AioContexts during bdrv_reopen_multiple()'
https://lists.gnu.org/archive/html/qemu-block/2021-07/msg00238.html

It still takes AioContext locks when it calls into other functions that currently expect it, but that should be the same as usual then.

And once callers don't even hold the lock in the first place, we'll also get rid of the ugly temporary lock release across reopen.

Kevin

#20 lore raw
In reply to Paolo Bonzini
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

If yes, then it's a risk worth taking. If no, then spending time developing interim code, reviewing those patches, and risking breakage doesn't seem worth it. I'd rather wait for blk_*()/bdrv_*() to be fully complete and then see patches that delete aio_context_acquire() in most places or add locks in the remaining places where the caller was relying on the AioContext lock.

Stefan

#21 lore raw
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 3 lines

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?

If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.

Emanuele

Quoted 10 lines

If yes, then it's a risk worth taking. If no, then spending time developing interim code, reviewing those patches, and risking breakage doesn't seem worth it. I'd rather wait for blk_*()/bdrv_*() to be fully complete and then see patches that delete aio_context_acquire() in most places or add locks in the remaining places where the caller was relying on the AioContext lock.

Stefan

#22 lore raw
Quoted 31 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 3 lines

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

Quoted 7 lines

Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?

If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.

Block graph manipulation (all_bdrv_states and friends) requires the BQL. It has always been this way.

This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why are aio_context_acquire() calls still needed in block jobs?

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs. Maybe there's a benefit to this patch series that I've missed?

Stefan

#23 lore raw
Quoted 37 lines · Stefan Hajnoczi
Quoted 36 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 3 lines

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?

Quoted 8 lines
Quoted 2 lines

If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.

Block graph manipulation (all_bdrv_states and friends) requires the BQL. It has always been this way.

This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why

I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety? I work mostly with nbd and qcow2 drivers, and they never seemd to be thread-safe to me. For example, qcow2 driver has enough operations that are done from non-coroutine context and therefore qcow2 co-mutex just not locked.

Quoted 8 lines

are aio_context_acquire() calls still needed in block jobs?

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Quoted 4 lines

Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs.

I agree. Add my bit of criticism:

What I dislike about the whole thread-safe update you do:

1. There is no proof of concept - some good example of multiqueue, or something that uses mutliple threads and shows good performance. Something that works, and shows where are we going to and why it is good. That may be draft patches with a lot of "FIXME" and "TODO", but working. For now I feel that I've spent my time to reviewing and proving to myself thread-safety of two previous thread-safe series, but I don't have a hope to see a benefit of it in the near future..

1.1 If we have a proof of concept, that also gives a kind of plan: a list of subsystems (patch series) to do step by step and finally come to what we want. Do you have a kind of plan (of the whole feature) now?

2. There are no tests: something that doesn't work before the series and start to work after. Why it is important:

All these thread-safe primitives are complicated enough. They hard to review and prove correctness. And very simple to break by new code. Tests that runs by CI proves that we don't break subsystems that are already thread-safe. For example, you've recently updated block-copy and several related things. But we have no tests. So, assume, a year later you finish the work of updating all other subsystems to be thread-safe. You'll have no guarantee that block-copy is still thread-safe, and you'll have to start from the beginning.

3. As I said, I really doubt that block drivers are already thread safe. An audit and/or tests are needed at least.

Signature · 2 lines
Best regards,
Vladimir
#24 lore raw
Quoted 41 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Stefan Hajnoczi
Quoted 38 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 3 lines

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?

If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.

Quoted 7 lines
Quoted 5 lines

Block graph manipulation (all_bdrv_states and friends) requires the BQL. It has always been this way.

This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why

I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety?

Emanuele :)

FWIW I took a look at the stream, mirror, and backup jobs today and couldn't find anything that's unsafe after this series. I was expecting to find issues but I think Emanuele and Paolo have taken care of them.

Quoted 13 lines
Quoted 8 lines

are aio_context_acquire() calls still needed in block jobs?

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Quoted 4 lines

Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs.

Quoted 5 lines

I agree. Add my bit of criticism:

What I dislike about the whole thread-safe update you do:

1. There is no proof of concept - some good example of multiqueue, or something that uses mutliple threads and shows good performance. Something that works, and shows where are we going to and why it is good. That may be draft patches with a lot of "FIXME" and "TODO", but working. For now I feel that I've spent my time to reviewing and proving to myself thread-safety of two previous thread-safe series, but I don't have a hope to see a benefit of it in the near future..

The multi-queue block layer should improve performance in cases where the bottleneck is a single IOThread. It will allow users to assign more than one IOThread.

But I think the bigger impact of this work will be addressing long-standing problems with the block layer's programming model. We continue to have IOThread bugs because there are many undocumented assumptions. With the multi-queue block layer the code switches to a more well-understood multi-threaded programming model and hopefully fewer issues will arise because there is no problematic AioContext lock to worry about.

Stefan

#25 lore raw
Quoted 48 lines · Stefan Hajnoczi
Quoted 47 lines · Vladimir Sementsov-Ogievskiy
Quoted 45 lines · Stefan Hajnoczi
Quoted 31 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 3 lines

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

Quoted 7 lines

Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?

If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.

Block graph manipulation (all_bdrv_states and friends) requires the BQL. It has always been this way.

This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why

Quoted 7 lines
Quoted 1 line

I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety?

Emanuele :)

FWIW I took a look at the stream, mirror, and backup jobs today and couldn't find anything that's unsafe after this series. I was expecting to find issues but I think Emanuele and Paolo have taken care of them.

Hmm, do you mean that all jobs are thread-safe?

Looking at the mirror, what protects s->ops_in_flight for example? It's accessed from job coroutines and from mirror_top filter write operation.

Quoted 34 lines
Quoted 14 lines
Quoted 9 lines

are aio_context_acquire() calls still needed in block jobs?

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Quoted 4 lines

Back to my comment about unnecessary risk, pushing the lock down is a strategy for exploring the problem, but I'm not sure those intermediate commits need to be committed to qemu.git/master because of the time required to review them and the risk of introducing (temporary) bugs.

Quoted 5 lines

I agree. Add my bit of criticism:

What I dislike about the whole thread-safe update you do:

1. There is no proof of concept - some good example of multiqueue, or something that uses mutliple threads and shows good performance. Something that works, and shows where are we going to and why it is good. That may be draft patches with a lot of "FIXME" and "TODO", but working. For now I feel that I've spent my time to reviewing and proving to myself thread-safety of two previous thread-safe series, but I don't have a hope to see a benefit of it in the near future..

The multi-queue block layer should improve performance in cases where the bottleneck is a single IOThread. It will allow users to assign more than one IOThread.

But I think the bigger impact of this work will be addressing long-standing problems with the block layer's programming model. We continue to have IOThread bugs because there are many undocumented assumptions. With the multi-queue block layer the code switches to a more well-understood multi-threaded programming model and hopefully fewer issues will arise because there is no problematic AioContext lock to worry about.

Stefan

Signature · 2 lines
Best regards,
Vladimir
#26 lore raw
Quoted 53 lines · Vladimir Sementsov-Ogievskiy
Quoted 52 lines · Stefan Hajnoczi
Quoted 41 lines · Vladimir Sementsov-Ogievskiy
Quoted 40 lines · Stefan Hajnoczi
Quoted 38 lines · Emanuele Giuseppe Esposito
Quoted 24 lines · Stefan Hajnoczi
Quoted 16 lines · Paolo Bonzini
Quoted 8 lines · Stefan Hajnoczi
Quoted 7 lines

What is very clear from this patch is that it
is strictly related to the brdv_* and lower level calls, because they also internally check or even use the aiocontext lock. Therefore, in order to make it work, I temporarly added some aiocontext_acquire/release pair around the function that still assert for them or assume they are hold and temporarly unlock (unlock() - lock()).

Quoted 6 lines

Sounds like the issue is that this patch series assumes AioContext locks are no longer required for calling the blk_*()/bdrv_*() APIs? That is not the case yet, so you had to then add those aio_context_lock() calls back in elsewhere. This approach introduces unnecessary risk. I think we should wait until blk_*()/bdrv_*() no longer requires the caller to hold the AioContext lock before applying this series.

Quoted 6 lines

In general I'm in favor of pushing the lock further down into smaller and smaller critical sections; it's a good approach to make further audits easier until it's "obvious" that the lock is unnecessary. I haven't yet reviewed Emanuele's patches to see if this is what he's doing where he's adding the acquire/release calls, but that's my understanding of both his cover letter and your reply.

Quoted 3 lines

The problem is the unnecessary risk. We know what the goal is for blk_*()/bdrv_*() but it's not quite there yet. Does making changes in block jobs help solve the final issues with blk_*()/bdrv_*()?

Correct me if I am wrong, but it seems to me that the bdrv_*()/blk_*() operation mostly take care of building, modifying and walking the bds graph. So since graph nodes can have multiple AioContext, it makes sense that we have a lock when modifying the graph, right?

If so, we can simply try to replace the AioContext lock with a graph lock, or something like that. But I am not sure of this.

Quoted 7 lines
Quoted 5 lines

Block graph manipulation (all_bdrv_states and friends) requires the BQL. It has always been this way.

This raises the question: if block graph manipulation is already under the BQL and BlockDriver callbacks don't need the AioContext anymore, why

I don't believe that block drivers are thread-safe now. They have some mutexes.. But who make an audit of thread-safety?

Emanuele :)

Quoted 8 lines
Quoted 3 lines

FWIW I took a look at the stream, mirror, and backup jobs today and couldn't find anything that's unsafe after this series. I was expecting to find issues but I think Emanuele and Paolo have taken care of them.

Hmm, do you mean that all jobs are thread-safe?

Looking at the mirror, what protects s->ops_in_flight for example? It's accessed from job coroutines and from mirror_top filter write operation.

You're right. I missed the bdrv_mirror_top BlockDriver:

.pwrite_zeroes -> bdrv_mirror_top_pwrite_zeroes -> active_write_prepare -> QTAILQ_INSERT_TAIL(&s->ops_in_flight, op, next)

This is not thread-safe. A CoMutex is needed here to protect MirrorBSDOpaque fields.

Stefan

#27 lore raw
In reply to Stefan Hajnoczi

Am 13.07.2021 um 15:10 hat Stefan Hajnoczi geschrieben:

Quoted 5 lines

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Polling something in a different AioContext from the main thread still temporarily drops the lock, which crashes if it isn't locked. I'm not sure if the documentation claims that the lock is needed in more cases, I guess you could interpret it either way.

Kevin

#28 lore raw
Quoted 11 lines · Kevin Wolf

Am 13.07.2021 um 15:10 hat Stefan Hajnoczi geschrieben:

Quoted 5 lines

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Polling something in a different AioContext from the main thread still temporarily drops the lock, which crashes if it isn't locked. I'm not sure if the documentation claims that the lock is needed in more cases, I guess you could interpret it either way.

I'm claiming that the lock doesn't need to be dropped in that case anymore - as long as the condition we're polling is thread-safe. :)

Have I missed something that still need locking?

We could temporarily introduce an AIO_WAIT_WHILE_UNLOCKED() macro so that callers can be converted individually.

Stefan

#29 lore raw

Am 19.07.2021 um 11:29 hat Stefan Hajnoczi geschrieben:

Quoted 13 lines · Kevin Wolf
Quoted 11 lines

Am 13.07.2021 um 15:10 hat Stefan Hajnoczi geschrieben:

Quoted 5 lines

AIO_WAIT_WHILE() requires that AioContext is acquired according to its documentation, but I'm not sure that's true anymore. Thread-safe/atomic primitives are used by AIO_WAIT_WHILE(), so as long as the condition being waited for is thread-safe too it should work without the AioContext lock.

Polling something in a different AioContext from the main thread still temporarily drops the lock, which crashes if it isn't locked. I'm not sure if the documentation claims that the lock is needed in more cases, I guess you could interpret it either way.

Quoted 4 lines

I'm claiming that the lock doesn't need to be dropped in that case anymore - as long as the condition we're polling is thread-safe. :)

Have I missed something that still need locking?

I'm not sure if AIO_WAIT_WHILE() actually ever needed the locking. I think it's more a convenience thing since the callers would already hold the lock, so dropping it temporarily in AIO_WAIT_WHILE() means that the callers don't have to duplicate the temporary unlock everywhere.

Quoted 2 lines

We could temporarily introduce an AIO_WAIT_WHILE_UNLOCKED() macro so that callers can be converted individually.

Yes, this makes sense to me.

Kevin

#30 lore raw
In reply to Emanuele Giuseppe Esposito
Quoted 13 lines · Emanuele Giuseppe Esposito

This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.

This series aims to:
1) remove the aiocontext lock and substitute it with the already existing global job_mutex
2) fix what it looks like to be an oversight when moving the blockjob.c logic into the more generic job.c: job_mutex was introduced especially to protect job->busy flag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) use job_mutex instead of the aiocontext_lock
4) extend the reach of the job_mutex to protect all shared fields that the job structure has.

Can you explain the big picture:

1. What are the rules for JobDrivers? Imagine you are implementing a new JobDriver. What do you need to know in order to write correct code?

2. What are the rules for monitor? The main pattern is looking up a job, invoking a job API on it, and then calling job_unlock().

Stefan

#31 lore raw
Quoted 15 lines · Stefan Hajnoczi
Quoted 13 lines · Emanuele Giuseppe Esposito

This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.

This series aims to:
1) remove the aiocontext lock and substitute it with the already existing global job_mutex
2) fix what it looks like to be an oversight when moving the blockjob.c logic into the more generic job.c: job_mutex was introduced especially to protect job->busy flag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) use job_mutex instead of the aiocontext_lock
4) extend the reach of the job_mutex to protect all shared fields that the job structure has.

Quoted 4 lines

Can you explain the big picture:

1. What are the rules for JobDrivers? Imagine you are implementing a new JobDriver. What do you need to know in order to write correct code?

I think that in general, the rules for JobDrivers remain the same. The job_mutex lock should be invisible (or almost) from the point of view of a JobDriver, because the job API available for it should take care of the necessary locking/unlocking.

Quoted 3 lines

2. What are the rules for monitor? The main pattern is looking up a job, invoking a job API on it, and then calling job_unlock().

The monitor instead is aware of this lock: the reason for that is exactly what you have described here.
Looking up + invoking a job API operation (for example calling find_job() and then job_pause() ) must be performed with the same lock hold all the time, otherwise other threads could modify the job while the monitor runs its command.

Please let me know if something is not clear and/or if you have additional comments on this!

Emanuele

Quoted 3 lines

Stefan

#32 lore raw
Quoted 30 lines · Emanuele Giuseppe Esposito
Quoted 15 lines · Stefan Hajnoczi
Quoted 13 lines · Emanuele Giuseppe Esposito

This is a continuation on the work to reduce (and possibly get rid of) the usage of AioContext lock, by introducing smaller granularity locks to keep the thread safety.

This series aims to:
1) remove the aiocontext lock and substitute it with the already existing global job_mutex
2) fix what it looks like to be an oversight when moving the blockjob.c logic into the more generic job.c: job_mutex was introduced especially to protect job->busy flag, but it seems that it was not used in successive patches, because there are multiple code sections that directly access the field without any locking.
3) use job_mutex instead of the aiocontext_lock
4) extend the reach of the job_mutex to protect all shared fields that the job structure has.

Quoted 4 lines

Can you explain the big picture:

1. What are the rules for JobDrivers? Imagine you are implementing a new JobDriver. What do you need to know in order to write correct code?

I think that in general, the rules for JobDrivers remain the same. The job_mutex lock should be invisible (or almost) from the point of view of a JobDriver, because the job API available for it should take care of the necessary locking/unlocking.

Quoted 3 lines

2. What are the rules for monitor? The main pattern is looking up a job, invoking a job API on it, and then calling job_unlock().

Quoted 6 lines

The monitor instead is aware of this lock: the reason for that is exactly what you have described here.
Looking up + invoking a job API operation (for example calling find_job() and then job_pause() ) must be performed with the same lock hold all the time, otherwise other threads could modify the job while the monitor runs its command.

That helps, thanks!

Stefan