io_uring: avoid uring_lock for IORING_SETUP_SINGLE_ISSUER
Subsystems (2)
v6 cover io_uring: avoid uring_lock for IORING_SETUP_SINGLE_ISSUER
Setting IORING_SETUP_SINGLE_ISSUER when creating an io_uring doesn't actually enable any additional optimizations (aside from being a requirement for IORING_SETUP_DEFER_TASKRUN). This series leverages IORING_SETUP_SINGLE_ISSUER's guarantee that only one task submits SQEs to skip taking the uring_lock mutex for the issue and task work paths.
First, we need to disable this optimization for IORING_SETUP_SQPOLL by clearing the IORING_SETUP_SINGLE_ISSUER flag. For IORING_SETUP_SQPOLL, the SQ thread is the one taking the uring_lock mutex in the issue path. Since concurrent io_uring_register() syscalls are allowed on the thread that created/enabled the io_uring, some additional synchronization method would be required to synchronize the two threads. This is possible in principle by having io_uring_register() schedule a task work item to suspend the SQ thread, but seems complex for a niche use case.
Then we factor out helpers for interacting with uring_lock to centralize the logic.
Finally, we implement the optimization for IORING_SETUP_SINGLE_ISSUER. If the io_ring_ctx is setup with IORING_SETUP_SINGLE_ISSUER, skip the uring_lock mutex_lock() and mutex_unlock() on the submitter_task. On other tasks acquiring the ctx uring lock, use a task work item to suspend the submitter_task for the critical section.
If the io_ring_ctx is IORING_SETUP_R_DISABLED (possible during io_uring_setup(), io_uring_register(), or io_uring exit), submitter_task may be set concurrently, so acquire the uring_lock before checking it. If submitter_task isn't set yet, the uring_lock suffices to provide mutual exclusion. If task work can't be queued because submitter_task has exited, also use the uring_lock for mutual exclusion.
v6:
- Release submitter_task reference last in io_ring_ctx_free() (syzbot)
- Use the uring_lock to provide mutual exclusion if task_work_add() fails because submitter_task has exited
- Add Reviewed-by tag
v5:
- Ensure submitter_task is initialized in io_uring_create() before calling io_ring_ctx_wait_and_kill() (kernel test robot)
- Correct Fixes tag (Joanne)
- Add Reviewed-by tag
v4:
- Handle IORING_SETUP_SINGLE_ISSUER and IORING_SETUP_R_DISABLED correctly (syzbot)
- Remove separate set of helpers for io_uring_register()
- Add preliminary fix to prevent races between accessing ctx->flags and submitter_task
v3:
- Ensure mutual exclusion on threads other than submitter_task via a task work item to suspend submitter_task
- Drop patches already merged
v2:
- Don't enable these optimizations for IORING_SETUP_SQPOLL, as we still need to synchronize SQ thread submission with io_uring_register()
Caleb Sander Mateos (6):
io_uring: use release-acquire ordering for IORING_SETUP_R_DISABLED
io_uring: clear IORING_SETUP_SINGLE_ISSUER for IORING_SETUP_SQPOLL
io_uring: ensure submitter_task is valid for io_ring_ctx's lifetime
io_uring: use io_ring_submit_lock() in io_iopoll_req_issued()
io_uring: factor out uring_lock helpers
io_uring: avoid uring_lock for IORING_SETUP_SINGLE_ISSUER
include/linux/io_uring_types.h | 12 +- io_uring/cancel.c | 40 +++--- io_uring/cancel.h | 5 +- io_uring/eventfd.c | 5 +- io_uring/fdinfo.c | 8 +- io_uring/filetable.c | 8 +- io_uring/futex.c | 14 +- io_uring/io_uring.c | 232 ++++++++++++++++++++------------- io_uring/io_uring.h | 187 +++++++++++++++++++++++--- io_uring/kbuf.c | 32 +++-- io_uring/memmap.h | 2 +- io_uring/msg_ring.c | 33 +++-- io_uring/notif.c | 5 +- io_uring/notif.h | 3 +- io_uring/openclose.c | 14 +- io_uring/poll.c | 21 +-- io_uring/register.c | 81 ++++++------ io_uring/rsrc.c | 51 +++++--- io_uring/rsrc.h | 6 +- io_uring/rw.c | 2 +- io_uring/splice.c | 5 +- io_uring/sqpoll.c | 5 +- io_uring/tctx.c | 27 ++-- io_uring/tctx.h | 5 +- io_uring/uring_cmd.c | 13 +- io_uring/waitid.c | 13 +- io_uring/zcrx.c | 2 +- 27 files changed, 555 insertions(+), 276 deletions(-)
Signature · 1 line
2.45.2
PATCH v6 1/6 io_uring: use release-acquire ordering for IORING_SETUP_R_DISABLED
io_uring_enter() and io_msg_ring() read ctx->flags and ctx->submitter_task without holding the ctx's uring_lock. This means they may race with the assignment to ctx->submitter_task and the clearing of IORING_SETUP_R_DISABLED from ctx->flags in io_register_enable_rings(). Ensure the correct ordering of the ctx->flags and ctx->submitter_task memory accesses by storing to ctx->flags using release ordering and loading it using acquire ordering.
--- io_uring/io_uring.c | 2 +- io_uring/msg_ring.c | 4 ++-- io_uring/register.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
io_uring/io_uring.c +1 -1
@@ -3249,11 +3249,11 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, goto out; } ctx = file->private_data; ret = -EBADFD ; -if (unlikely (ctx->flags &IORING_SETUP_R_DISABLED )) +if (unlikely (smp_load_acquire (&ctx->flags) &IORING_SETUP_R_DISABLED )) goto out;/* * For SQ polling, the thread will do all submissions and completions. * Just return the requested submit count, and wake the thread if
io_uring/msg_ring.c +2 -2
@@ -123,11 +123,11 @@ static int __io_msg_ring_data(struct io_ring_ctx *target_ctx,if (msg->src_fd || msg->flags & ~IORING_MSG_RING_FLAGS_PASS )return -EINVAL ;if (!(msg->flags &IORING_MSG_RING_FLAGS_PASS ) && msg->dst_fd)return -EINVAL ; -if (target_ctx->flags &IORING_SETUP_R_DISABLED ) +if (smp_load_acquire (&target_ctx->flags) &IORING_SETUP_R_DISABLED )return -EBADFD ;if (io_msg_need_remote (target_ctx))return io_msg_data_remote (target_ctx, msg);
@@ -243,11 +243,11 @@ static int io_msg_send_fd(struct io_kiocb *req, unsigned int issue_flags)if (msg->len)return -EINVAL ;if (target_ctx == ctx)return -EINVAL ; -if (target_ctx->flags &IORING_SETUP_R_DISABLED ) +if (smp_load_acquire (&target_ctx->flags) &IORING_SETUP_R_DISABLED )return -EBADFD ;if (!msg->src_file) {int ret =io_msg_grab_file (req, issue_flags);if (unlikely (ret))return ret;
io_uring/register.c +1 -1
@@ -191,11 +191,11 @@ static int io_register_enable_rings(struct io_ring_ctx *ctx) }if (ctx->restrictions.registered) ctx->restricted =1 ; - ctx->flags &= ~IORING_SETUP_R_DISABLED ; +smp_store_release (&ctx->flags, ctx->flags & ~IORING_SETUP_R_DISABLED );if (ctx->sq_data &&wq_has_sleeper (&ctx->sq_data->wait))wake_up (&ctx->sq_data->wait);return 0 ; }
Signature · 1 line
2.45.2
PATCH v6 2/6 io_uring: clear IORING_SETUP_SINGLE_ISSUER for IORING_SETUP_SQPOLL
In reply to Caleb Sander MateosIORING_SETUP_SINGLE_ISSUER doesn't currently enable any optimizations, but it will soon be used to avoid taking io_ring_ctx's uring_lock when submitting from the single issuer task. If the IORING_SETUP_SQPOLL flag is set, the SQ thread is the sole task issuing SQEs. However, other tasks may make io_uring_register() syscalls, which must be synchronized with SQE submission. So it wouldn't be safe to skip the uring_lock around the SQ thread's submission even if IORING_SETUP_SINGLE_ISSUER is set. Therefore, clear IORING_SETUP_SINGLE_ISSUER from the io_ring_ctx flags if IORING_SETUP_SQPOLL is set.
--- io_uring/io_uring.c | 9 +++++++++ 1 file changed, 9 insertions(+)
io_uring/io_uring.c +9 -0
@@ -3478,10 +3478,19 @@ static int io_uring_sanitise_params(struct io_uring_params *p)*/ if ((flags & (IORING_SETUP_SQE128 |IORING_SETUP_SQE_MIXED )) == (IORING_SETUP_SQE128 |IORING_SETUP_SQE_MIXED ))return -EINVAL ; +/* +* If IORING_SETUP_SQPOLL is set, only the SQ thread issues SQEs, +* but other threads may call io_uring_register() concurrently. +* We still need ctx uring lock to synchronize these io_ring_ctx +* accesses, so disable the single issuer optimizations. +*/ +if (flags &IORING_SETUP_SQPOLL ) + p->flags &= ~IORING_SETUP_SINGLE_ISSUER ; +return 0 ; }static int io_uring_fill_params (struct io_uring_params *p) {
Signature · 1 line
2.45.2
PATCH v6 3/6 io_uring: ensure submitter_task is valid for io_ring_ctx's lifetime
In reply to Caleb Sander MateosIf io_uring_create() fails after allocating the struct io_ring_ctx, it may call io_ring_ctx_wait_and_kill() before submitter_task has been assigned. This is currently harmless, as the submit and register paths that check submitter_task aren't reachable until the io_ring_ctx has been successfully created. However, a subsequent commit will expect submitter_task to be set for every IORING_SETUP_SINGLE_ISSUER && !IORING_SETUP_R_DISABLED ctx. So assign ctx->submitter_task immediately after allocating the ctx in io_uring_create().
Similarly, the reference on submitter_task is currently released early in io_ring_ctx_free(). But it will soon be needed to acquire the uring lock during the later call to io_req_caches_free(). So release the submitter_task reference as the last thing before freeing the ctx.
--- io_uring/io_uring.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-)
io_uring/io_uring.c +13 -11
@@ -2852,12 +2852,10 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)io_destroy_buffers (ctx);io_free_region (ctx->user, &ctx->param_region);mutex_unlock (&ctx->uring_lock);if (ctx->sq_creds)put_cred (ctx->sq_creds); -if (ctx->submitter_task) -put_task_struct (ctx->submitter_task);WARN_ON_ONCE (!list_empty (&ctx->ltimeout_list));if (ctx->mm_account) {mmdrop (ctx->mm_account);
@@ -2877,10 +2875,13 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)if (ctx->hash_map)io_wq_put_hash (ctx->hash_map);io_napi_free (ctx);kvfree (ctx->cancel_table.hbs);xa_destroy (&ctx->io_bl_xa); +/* Release submitter_task last, as any io_ring_ctx_lock() may use it */ +if (ctx->submitter_task) +put_task_struct (ctx->submitter_task);kfree (ctx); }static __cold void io_activate_pollwq_cb (struct callback_head *cb) {
@@ -3594,10 +3595,20 @@ static __cold int io_uring_create(struct io_ctx_config *config) ctx =io_ring_ctx_alloc (p);if (!ctx)return -ENOMEM ; +/* Assign submitter_task first, as any io_ring_ctx_lock() may use it */ +if (ctx->flags &IORING_SETUP_SINGLE_ISSUER + && !(ctx->flags &IORING_SETUP_R_DISABLED )) { +/* +* Unlike io_register_enable_rings(), don't need WRITE_ONCE() +* since ctx isn't yet accessible from other tasks +*/ + ctx->submitter_task =get_task_struct (current); + } + ctx->clockid =CLOCK_MONOTONIC ; ctx->clock_offset =0 ;if (!(ctx->flags &IORING_SETUP_NO_SQARRAY ))static_branch_inc (&io_key_has_sqarray);
@@ -3662,19 +3673,10 @@ static __cold int io_uring_create(struct io_ctx_config *config)if (copy_to_user (config->uptr, p,sizeof (*p))) { ret = -EFAULT ; goto err; } -if (ctx->flags &IORING_SETUP_SINGLE_ISSUER - && !(ctx->flags &IORING_SETUP_R_DISABLED )) { -/* -* Unlike io_register_enable_rings(), don't need WRITE_ONCE() -* since ctx isn't yet accessible from other tasks -*/ - ctx->submitter_task =get_task_struct (current); - } - file =io_uring_get_file (ctx);if (IS_ERR (file)) { ret =PTR_ERR (file); goto err; }
Signature · 1 line
2.45.2
PATCH v6 4/6 io_uring: use io_ring_submit_lock() in io_iopoll_req_issued()
In reply to Caleb Sander MateosUse the io_ring_submit_lock() helper in io_iopoll_req_issued() instead of open-coding the logic. io_ring_submit_unlock() can't be used for the unlock, though, due to the extra logic before releasing the mutex.
--- io_uring/io_uring.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
io_uring/io_uring.c +2 -4
@@ -1670,15 +1670,13 @@ void io_req_task_complete(struct io_tw_req tw_req, io_tw_token_t tw)* accessing the kiocb cookie. */ static void io_iopoll_req_issued (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *ctx = req->ctx; -const bool needs_lock = issue_flags &IO_URING_F_UNLOCKED ;/* workqueue context doesn't hold uring_lock, grab it now */ -if (unlikely (needs_lock)) -mutex_lock (&ctx->uring_lock); +io_ring_submit_lock (ctx, issue_flags);/* * Track whether we have multiple files in our lists. This will impact * how we do polling eventually, not spinning if we're on potentially * different devices.
@@ -1701,11 +1699,11 @@ static void io_iopoll_req_issued(struct io_kiocb *req, unsigned int issue_flags)if (READ_ONCE (req->iopoll_completed))wq_list_add_head (&req->comp_list, &ctx->iopoll_list);else wq_list_add_tail (&req->comp_list, &ctx->iopoll_list); -if (unlikely (needs_lock)) { +if (unlikely (issue_flags &IO_URING_F_UNLOCKED )) {/* * If IORING_SETUP_SQPOLL is enabled, sqes are either handle * in sq thread task context or in io worker task context. If * current task context is sq thread, we don't need to check * whether should wake up sq thread.
Signature · 1 line
2.45.2
PATCH v6 5/6 io_uring: factor out uring_lock helpers
In reply to Caleb Sander MateosA subsequent commit will skip acquiring the io_ring_ctx uring_lock in io_uring_enter() and io_handle_tw_list() for IORING_SETUP_SINGLE_ISSUER. Prepare for this change by factoring out the uring_lock accesses under these functions into helpers. Aside from the helpers, the only remaining access of uring_lock is its mutex_init() call. Define a struct io_ring_ctx_lock_state to pass state from io_ring_ctx_lock() to io_ring_ctx_unlock(). It's currently empty but a subsequent commit will add fields.
Helpers:
- io_ring_ctx_lock() for mutex_lock()
- io_ring_ctx_lock_nested() for mutex_lock_nested()
- io_ring_ctx_trylock() for mutex_trylock()
- io_ring_ctx_unlock() for mutex_unlock()
- io_ring_ctx_lock_held() for lockdep_is_held()
- io_ring_ctx_assert_locked() for lockdep_assert_held()
--- include/linux/io_uring_types.h | 12 +-- io_uring/cancel.c | 40 ++++---- io_uring/cancel.h | 5 +- io_uring/eventfd.c | 5 +- io_uring/fdinfo.c | 8 +- io_uring/filetable.c | 8 +- io_uring/futex.c | 14 +-- io_uring/io_uring.c | 181 +++++++++++++++++++-------------- io_uring/io_uring.h | 75 +++++++++++--- io_uring/kbuf.c | 32 +++--- io_uring/memmap.h | 2 +- io_uring/msg_ring.c | 29 ++++-- io_uring/notif.c | 5 +- io_uring/notif.h | 3 +- io_uring/openclose.c | 14 +-- io_uring/poll.c | 21 ++-- io_uring/register.c | 79 +++++++------- io_uring/rsrc.c | 51 ++++++---- io_uring/rsrc.h | 6 +- io_uring/rw.c | 2 +- io_uring/splice.c | 5 +- io_uring/sqpoll.c | 5 +- io_uring/tctx.c | 27 +++-- io_uring/tctx.h | 5 +- io_uring/uring_cmd.c | 13 ++- io_uring/waitid.c | 13 +-- io_uring/zcrx.c | 2 +- 27 files changed, 404 insertions(+), 258 deletions(-)
include/linux/io_uring_types.h +6 -6
@@ -86,11 +86,11 @@ struct io_mapped_region {/* * Return value from io_buffer_list selection, to avoid stashing it in * struct io_kiocb. For legacy/classic provided buffers, keeping a reference * across execution contexts are fine. But for ring provided buffers, the -* list may go away as soon as ->uring_lock is dropped. As the io_kiocb +* list may go away as soon as the ctx uring lock is dropped. As the io_kiocb * persists, it's better to just keep the buffer local for those cases. */ struct io_br_sel {struct io_buffer_list *buf_list;/*
@@ -231,11 +231,11 @@ struct io_submit_link {struct io_kiocb *head;struct io_kiocb *last; };struct io_submit_state { -/* inline/task_work completion list, under ->uring_lock */ +/* inline/task_work completion list, under ctx uring lock */ struct io_wq_work_node free_list;/* batch completion logic */ struct io_wq_work_list compl_reqs;struct io_submit_link link;
@@ -303,16 +303,16 @@ struct io_ring_ctx {unsigned cached_sq_head;unsigned sq_entries;/* * Fixed resources fast path, should be accessed only under -* uring_lock, and updated through io_uring_register(2) +* ctx uring lock, and updated through io_uring_register(2) */ atomic_t cancel_seq;/* -* ->iopoll_list is protected by the ctx->uring_lock for +* ->iopoll_list is protected by the ctx uring lock for * io_uring instances that don't use IORING_SETUP_SQPOLL. * For SQPOLL, only the single threaded io_sq_thread() will * manipulate the list, hence no extra locking is needed there. */ bool poll_multi_queue;
@@ -324,11 +324,11 @@ struct io_ring_ctx {struct io_alloc_cache imu_cache;struct io_submit_state submit_state;/* -* Modifications are protected by ->uring_lock and ->mmap_lock. +* Modifications protected by ctx uring lock and ->mmap_lock. * The buffer list's io mapped region should be stable once * published. */ struct xarray io_bl_xa;
@@ -467,11 +467,11 @@ struct io_ring_ctx {struct io_mapped_region param_region; };/* * Token indicating function is called in task work context: -* ctx->uring_lock is held and any completions generated will be flushed. +* ctx uring lock is held and any completions generated will be flushed. * ONLY core io_uring.c should instantiate this struct. */ struct io_tw_state {bool cancel; };
io_uring/cancel.c +23 -17
@@ -168,10 +168,11 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)static int __io_async_cancel (struct io_cancel_data *cd,struct io_uring_task *tctx,unsigned int issue_flags) {bool all = cd->flags & (IORING_ASYNC_CANCEL_ALL |IORING_ASYNC_CANCEL_ANY ); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = cd->ctx;struct io_tctx_node *node;int ret, nr =0 ; do {
@@ -182,21 +183,21 @@ static int __io_async_cancel(struct io_cancel_data *cd,return ret; nr++; }while (1 );/* slow path, try all io-wq's */ -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ret = -ENOENT ;list_for_each_entry (node, &ctx->tctx_list, ctx_node) { ret =io_async_cancel_one (node->task->io_uring, cd);if (ret != -ENOENT ) {if (!all)break ; nr++; } } -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return all ? nr : ret; }int io_async_cancel (struct io_kiocb *req,unsigned int issue_flags) {
@@ -238,11 +239,11 @@ int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags)static int __io_sync_cancel (struct io_uring_task *tctx,struct io_cancel_data *cd,int fd) {struct io_ring_ctx *ctx = cd->ctx; -/* fixed must be grabbed every time since we drop the uring_lock */ +/* fixed must be grabbed every time since we drop the ctx uring lock */ if ((cd->flags &IORING_ASYNC_CANCEL_FD ) && (cd->flags &IORING_ASYNC_CANCEL_FD_FIXED )) {struct io_rsrc_node *node; node =io_rsrc_node_lookup (&ctx->file_table.data, fd);
@@ -254,12 +255,12 @@ static int __io_sync_cancel(struct io_uring_task *tctx, }return __io_async_cancel (cd, tctx,0 ); } -int io_sync_cancel (struct io_ring_ctx *ctx,void __user *arg) - __must_hold(&ctx->uring_lock) +int io_sync_cancel (struct io_ring_ctx *ctx,void __user *arg, +struct io_ring_ctx_lock_state *lock_state) {struct io_cancel_data cd = { .ctx = ctx, .seq =atomic_inc_return (&ctx->cancel_seq), };
@@ -267,10 +268,12 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg)struct io_uring_sync_cancel_reg sc;struct file *file =NULL ;DEFINE_WAIT (wait);int ret, i; +io_ring_ctx_assert_locked (ctx); +if (copy_from_user (&sc, arg,sizeof (sc)))return -EFAULT ;if (sc.flags & ~CANCEL_FLAGS )return -EINVAL ;for (i =0 ; i <ARRAY_SIZE (sc.pad); i++)
@@ -317,11 +320,11 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg)prepare_to_wait (&ctx->cq_wait, &wait,TASK_INTERRUPTIBLE ); ret =__io_sync_cancel (current->io_uring, &cd, sc.fd); -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, lock_state);if (ret != -EALREADY )break ; ret =io_run_task_work_sig (ctx);if (ret <0 )
@@ -329,15 +332,15 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg) ret =schedule_hrtimeout (&timeout,HRTIMER_MODE_ABS );if (!ret) { ret = -ETIME ;break ; } -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, lock_state); }while (1 );finish_wait (&ctx->cq_wait, &wait); -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, lock_state);if (ret == -ENOENT || ret >0 ) ret =0 ; out:if (file)
@@ -351,11 +354,11 @@ bool io_cancel_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx, {struct hlist_node *tmp;struct io_kiocb *req;bool found = false; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);hlist_for_each_entry_safe (req, tmp, list, hash_node) {if (!io_match_task_safe (req, tctx, cancel_all))continue ;hlist_del_init (&req->hash_node);
@@ -368,24 +371,25 @@ bool io_cancel_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx,int io_cancel_remove (struct io_ring_ctx *ctx,struct io_cancel_data *cd,unsigned int issue_flags,struct hlist_head *list,bool (*cancel)(struct io_kiocb *)) { +struct io_ring_ctx_lock_state lock_state;struct hlist_node *tmp;struct io_kiocb *req;int nr =0 ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state);hlist_for_each_entry_safe (req, tmp, list, hash_node) {if (!io_cancel_req_match (req, cd))continue ;if (cancel (req)) nr++;if (!(cd->flags &IORING_ASYNC_CANCEL_ALL ))break ; } -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return nr ?: -ENOENT ; }static bool io_match_linked (struct io_kiocb *head) {
@@ -477,37 +481,39 @@ __cold bool io_cancel_ctx_cb(struct io_wq_work *work, void *data)return req->ctx == data; }static __cold bool io_uring_try_cancel_iowq (struct io_ring_ctx *ctx) { +struct io_ring_ctx_lock_state lock_state;struct io_tctx_node *node;enum io_wq_cancel cret;bool ret = false; -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, &lock_state);list_for_each_entry (node, &ctx->tctx_list, ctx_node) {struct io_uring_task *tctx = node->task->io_uring;/* -* io_wq will stay alive while we hold uring_lock, because it's -* killed after ctx nodes, which requires to take the lock. +* io_wq will stay alive while we hold ctx uring lock, because +* it's killed after ctx nodes, which requires to take the lock. */ if (!tctx || !tctx->io_wq)continue ; cret =io_wq_cancel_cb (tctx->io_wq, io_cancel_ctx_cb, ctx, true); ret |= (cret !=IO_WQ_CANCEL_NOTFOUND ); } -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state);return ret; }__cold bool io_uring_try_cancel_requests (struct io_ring_ctx *ctx,struct io_uring_task *tctx,bool cancel_all,bool is_sqpoll_thread) {struct io_task_cancel cancel = { .tctx = tctx, .all = cancel_all, }; +struct io_ring_ctx_lock_state lock_state;enum io_wq_cancel cret;bool ret = false;/* set it so io_req_local_work_add() would wake us up */ if (ctx->flags &IORING_SETUP_DEFER_TASKRUN ) {
@@ -542,17 +548,17 @@ __cold bool io_uring_try_cancel_requests(struct io_ring_ctx *ctx, }if ((ctx->flags &IORING_SETUP_DEFER_TASKRUN ) &&io_allowed_defer_tw_run (ctx)) ret |=io_run_local_work (ctx,INT_MAX ,INT_MAX ) >0 ; -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, &lock_state); ret |=io_cancel_defer_files (ctx, tctx, cancel_all); ret |=io_poll_remove_all (ctx, tctx, cancel_all); ret |=io_waitid_remove_all (ctx, tctx, cancel_all); ret |=io_futex_remove_all (ctx, tctx, cancel_all); ret |=io_uring_try_cancel_uring_cmd (ctx, tctx, cancel_all); -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state); ret |=io_kill_timeouts (ctx, tctx, cancel_all);if (tctx) ret |=io_run_task_work () >0 ;else ret |=flush_delayed_work (&ctx->fallback_work);
io_uring/cancel.h +4 -1
@@ -2,10 +2,12 @@ #ifndef IORING_CANCEL_H #define IORING_CANCEL_H #include <linux/io_uring_types.h> +#include "io_uring.h" + struct io_cancel_data { struct io_ring_ctx *ctx; union { u64 data; struct file *file;
@@ -19,11 +21,12 @@ int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);int io_async_cancel (struct io_kiocb *req,unsigned int issue_flags);int io_try_cancel (struct io_uring_task *tctx,struct io_cancel_data *cd,unsigned int issue_flags); -int io_sync_cancel (struct io_ring_ctx *ctx,void __user *arg); +int io_sync_cancel (struct io_ring_ctx *ctx,void __user *arg, +struct io_ring_ctx_lock_state *lock_state);bool io_cancel_req_match (struct io_kiocb *req,struct io_cancel_data *cd);bool io_match_task_safe (struct io_kiocb *head,struct io_uring_task *tctx,bool cancel_all); bool io_cancel_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tctx,
io_uring/eventfd.c +3 -2
@@ -6,10 +6,11 @@#include <linux/eventfd.h> #include <linux/eventpoll.h> #include <linux/io_uring.h> #include <linux/io_uring_types.h> +#include "io_uring.h" #include "io-wq.h" #include "eventfd.h" struct io_ev_fd {struct eventfd_ctx *cq_ev_fd;
@@ -118,11 +119,11 @@ int io_eventfd_register(struct io_ring_ctx *ctx, void __user *arg,struct io_ev_fd *ev_fd;__s32 __user *fds = arg;int fd; ev_fd =rcu_dereference_protected (ctx->io_ev_fd, -lockdep_is_held (&ctx->uring_lock)); +io_ring_ctx_lock_held (ctx));if (ev_fd)return -EBUSY ;if (copy_from_user (&fd, fds,sizeof (*fds)))return -EFAULT ;
@@ -154,11 +155,11 @@ int io_eventfd_register(struct io_ring_ctx *ctx, void __user *arg,int io_eventfd_unregister (struct io_ring_ctx *ctx) {struct io_ev_fd *ev_fd; ev_fd =rcu_dereference_protected (ctx->io_ev_fd, -lockdep_is_held (&ctx->uring_lock)); +io_ring_ctx_lock_held (ctx));if (ev_fd) { ctx->has_evfd = false;rcu_assign_pointer (ctx->io_ev_fd,NULL );io_eventfd_put (ev_fd);return 0 ;
io_uring/fdinfo.c +5 -3
@@ -9,10 +9,11 @@#include <linux/io_uring.h> #include <uapi/linux/io_uring.h> #include "filetable.h" +#include "io_uring.h" #include "sqpoll.h" #include "fdinfo.h" #include "cancel.h" #include "rsrc.h" #include "opdef.h"
@@ -75,11 +76,11 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)if (ctx->flags &IORING_SETUP_SQE128 ) sq_shift =1 ;/* * we may get imprecise sqe and cqe info if uring is actively running -* since we get cached_sq_head and cached_cq_tail without uring_lock +* since we get cached_sq_head and cached_cq_tail without ctx uring lock * and sq_tail and cq_head are changed by userspace. But it's ok since * we usually use these info when it is stuck. */ seq_printf (m,"SqMask:\t0x%x\n" , sq_mask);seq_printf (m,"SqHead:\t%u\n" , sq_head);
@@ -249,16 +250,17 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)* anything else to get an extra reference. */ __cold void io_uring_show_fdinfo (struct seq_file *m,struct file *file) {struct io_ring_ctx *ctx = file->private_data; +struct io_ring_ctx_lock_state lock_state;/* * Avoid ABBA deadlock between the seq lock and the io_uring mutex, * since fdinfo case grabs it in the opposite direction of normal use * cases. */ -if (mutex_trylock (&ctx->uring_lock)) { +if (io_ring_ctx_trylock (ctx, &lock_state)) {__io_uring_show_fdinfo (ctx, m); -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state); } }
io_uring/filetable.c +5 -3
@@ -55,14 +55,15 @@ void io_free_file_tables(struct io_ring_ctx *ctx, struct io_file_table *table) table->bitmap =NULL ; }static int io_install_fixed_file (struct io_ring_ctx *ctx,struct file *file,u32 slot_index) - __must_hold(&ctx->uring_lock) {struct io_rsrc_node *node; +io_ring_ctx_assert_locked (ctx); +if (io_is_uring_fops (file))return -EBADF ;if (!ctx->file_table.data.nr)return -ENXIO ;if (slot_index >= ctx->file_table.data.nr)
@@ -105,16 +106,17 @@ int __io_fixed_fd_install(struct io_ring_ctx *ctx, struct file *file,* fput() is called correspondingly. */ int io_fixed_fd_install (struct io_kiocb *req,unsigned int issue_flags,struct file *file,unsigned int file_slot) { +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;int ret; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ret =__io_fixed_fd_install (ctx, file, file_slot); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);if (unlikely (ret <0 ))fput (file);return ret; }
io_uring/futex.c +8 -6
@@ -220,22 +220,23 @@ static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q)int io_futexv_wait (struct io_kiocb *req,unsigned int issue_flags) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex);struct io_futexv_data *ifd = req->async_data; +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;int ret, woken =-1 ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ret =futex_wait_multiple_setup (ifd->futexv, iof->futex_nr, &woken);/* * Error case, ret is < 0. Mark the request as failed. */ if (unlikely (ret <0 )) { -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);req_set_fail (req);io_req_set_res (req, ret,0 );io_req_async_data_free (req);return IOU_COMPLETE ; }
@@ -265,27 +266,28 @@ int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags) iof->futexv_unqueued =1 ;if (woken !=-1 )io_req_set_res (req, woken,0 ); } -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return IOU_ISSUE_SKIP_COMPLETE ; }int io_futex_wait (struct io_kiocb *req,unsigned int issue_flags) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_futex_data *ifd =NULL ;int ret;if (!iof->futex_mask) { ret = -EINVAL ; goto done; } -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ifd =io_cache_alloc (&ctx->futex_cache,GFP_NOWAIT );if (!ifd) { ret = -ENOMEM ; goto done_unlock; }
@@ -299,17 +301,17 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) ret =futex_wait_setup (iof->uaddr, iof->futex_val, iof->futex_flags, &ifd->q,NULL ,NULL );if (!ret) {hlist_add_head (&req->hash_node, &ctx->futex_list); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return IOU_ISSUE_SKIP_COMPLETE ; } done_unlock: -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state); done:if (ret <0 )req_set_fail (req);io_req_set_res (req, ret,0 );io_req_async_data_free (req);
io_uring/io_uring.c +107 -74
760 lines not on this page · raw message
io_uring/io_uring.h +60 -15
134 lines not on this page · raw message
io_uring/kbuf.c +18 -14
@@ -72,22 +72,22 @@ bool io_kbuf_commit(struct io_kiocb *req, }static inline struct io_buffer_list *io_buffer_get_list (struct io_ring_ctx *ctx,unsigned int bgid) { -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);return xa_load (&ctx->io_bl_xa, bgid); }static int io_buffer_add_list (struct io_ring_ctx *ctx,struct io_buffer_list *bl,unsigned int bgid) {/* * Store buffer group ID and finally mark the list as visible. * The normal lookup doesn't care about the visibility as we're -* always under the ->uring_lock, but lookups from mmap do. +* always under the ctx uring lock, but lookups from mmap do. */ bl->bgid = bgid;guard (mutex)(&ctx->mmap_lock);return xa_err (xa_store (&ctx->io_bl_xa, bgid, bl,GFP_KERNEL )); }
@@ -101,23 +101,24 @@ void io_kbuf_drop_legacy(struct io_kiocb *req) req->kbuf =NULL ; }bool io_kbuf_recycle_legacy (struct io_kiocb *req,unsigned issue_flags) { +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_buffer_list *bl;struct io_buffer *buf; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); buf = req->kbuf; bl =io_buffer_get_list (ctx, buf->bgid);list_add (&buf->list, &bl->buf_list); bl->nbufs++; req->flags &= ~REQ_F_BUFFER_SELECTED ; -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return true; }static void __user *io_provided_buffer_select (struct io_kiocb *req,size_t *len,struct io_buffer_list *bl)
@@ -210,24 +211,25 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, }struct io_br_sel io_buffer_select (struct io_kiocb *req,size_t *len,unsigned buf_group,unsigned int issue_flags) { +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_br_sel sel = { };struct io_buffer_list *bl; -io_ring_submit_lock (req->ctx, issue_flags); +io_ring_submit_lock (req->ctx, issue_flags, &lock_state); bl =io_buffer_get_list (ctx, buf_group);if (likely (bl)) {if (bl->flags &IOBL_BUF_RING ) sel =io_ring_buffer_select (req, len, bl, issue_flags);else sel.addr =io_provided_buffer_select (req, len, bl); } -io_ring_submit_unlock (req->ctx, issue_flags); +io_ring_submit_unlock (req->ctx, issue_flags, &lock_state);return sel; }/* cap it at a reasonable 256, will be one page even for 4K */ #define PEEK_MAX_IMPORT 256
@@ -315,14 +317,15 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, }int io_buffers_select (struct io_kiocb *req,struct buf_sel_arg *arg,struct io_br_sel *sel,unsigned int issue_flags) { +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;int ret = -ENOENT ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); sel->buf_list =io_buffer_get_list (ctx, arg->buf_group);if (unlikely (!sel->buf_list)) goto out_unlock;if (sel->buf_list->flags &IOBL_BUF_RING ) {
@@ -342,11 +345,11 @@ int io_buffers_select(struct io_kiocb *req, struct buf_sel_arg *arg, ret =io_provided_buffers_select (req, &arg->out_len, sel->buf_list, arg->iovs); } out_unlock:if (issue_flags &IO_URING_F_UNLOCKED ) { sel->buf_list =NULL ; -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state); }return ret; } int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg,
@@ -354,11 +357,11 @@ int io_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, {struct io_ring_ctx *ctx = req->ctx;struct io_buffer_list *bl;int ret; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx); bl =io_buffer_get_list (ctx, arg->buf_group);if (unlikely (!bl))return -ENOENT ;
@@ -410,11 +413,11 @@ static int io_remove_buffers_legacy(struct io_ring_ctx *ctx, {unsigned long i =0 ;struct io_buffer *nxt;/* protects io_buffers_cache */ -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);WARN_ON_ONCE (bl->flags &IOBL_BUF_RING );for (i =0 ; i < nbufs && !list_empty (&bl->buf_list); i++) { nxt = list_first_entry(&bl->buf_list, struct io_buffer, list);list_del (&nxt->list);
@@ -579,18 +582,19 @@ static int __io_manage_buffers_legacy(struct io_kiocb *req, }int io_manage_buffers_legacy (struct io_kiocb *req,unsigned int issue_flags) { struct io_provide_buf *p = io_kiocb_to_cmd(req, struct io_provide_buf); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_buffer_list *bl;int ret; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); bl =io_buffer_get_list (ctx, p->bgid); ret =__io_manage_buffers_legacy (req, bl); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);if (ret <0 )req_set_fail (req);io_req_set_res (req, ret,0 );return IOU_COMPLETE ;
@@ -604,11 +608,11 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)struct io_uring_buf_ring *br;unsigned long mmap_offset;unsigned long ring_size;int ret; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);if (copy_from_user (®, arg,sizeof (reg)))return -EFAULT ;if (!mem_is_zero (reg.resv,sizeof (reg.resv)))return -EINVAL ;
@@ -680,11 +684,11 @@ int io_register_pbuf_ring(struct io_ring_ctx *ctx, void __user *arg)int io_unregister_pbuf_ring (struct io_ring_ctx *ctx,void __user *arg) {struct io_uring_buf_reg reg;struct io_buffer_list *bl; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);if (copy_from_user (®, arg,sizeof (reg)))return -EFAULT ;if (!mem_is_zero (reg.resv,sizeof (reg.resv)) || reg.flags)return -EINVAL ;
io_uring/memmap.h +1 -1
@@ -35,11 +35,11 @@ static inline void io_region_publish(struct io_ring_ctx *ctx,struct io_mapped_region *src_region, struct io_mapped_region *dst_region) {/* * Once published mmap can find it without holding only the ->mmap_lock -* and not ->uring_lock. +* and not the ctx uring lock. */ guard(mutex)(&ctx->mmap_lock); *dst_region = *src_region; }
io_uring/msg_ring.c +18 -11
@@ -30,29 +30,31 @@ struct io_msg {u32 cqe_flags; };u32 flags; }; -static void io_double_unlock_ctx (struct io_ring_ctx *octx) +static void io_double_unlock_ctx (struct io_ring_ctx *octx, +struct io_ring_ctx_lock_state *lock_state) { -mutex_unlock (&octx->uring_lock); +io_ring_ctx_unlock (octx, lock_state); }static int io_lock_external_ctx (struct io_ring_ctx *octx, -unsigned int issue_flags) +unsigned int issue_flags, +struct io_ring_ctx_lock_state *lock_state) {/* * To ensure proper ordering between the two ctxs, we can only * attempt a trylock on the target. If that fails and we already have * the source ctx lock, punt to io-wq. */ if (!(issue_flags &IO_URING_F_UNLOCKED )) { -if (!mutex_trylock (&octx->uring_lock)) +if (!io_ring_ctx_trylock (octx, lock_state))return -EAGAIN ;return 0 ; } -mutex_lock (&octx->uring_lock); +io_ring_ctx_lock (octx, lock_state);return 0 ; }void io_msg_ring_cleanup (struct io_kiocb *req) {
@@ -116,10 +118,11 @@ static int io_msg_data_remote(struct io_ring_ctx *target_ctx, }static int __io_msg_ring_data (struct io_ring_ctx *target_ctx,struct io_msg *msg,unsigned int issue_flags) { +struct io_ring_ctx_lock_state lock_state;u32 flags =0 ;int ret;if (msg->src_fd || msg->flags & ~IORING_MSG_RING_FLAGS_PASS )return -EINVAL ;
@@ -134,17 +137,18 @@ static int __io_msg_ring_data(struct io_ring_ctx *target_ctx,if (msg->flags &IORING_MSG_RING_FLAGS_PASS ) flags = msg->cqe_flags; ret = -EOVERFLOW ;if (target_ctx->flags &IORING_SETUP_IOPOLL ) { -if (unlikely (io_lock_external_ctx (target_ctx, issue_flags))) +if (unlikely (io_lock_external_ctx (target_ctx, issue_flags, + &lock_state)))return -EAGAIN ; }if (io_post_aux_cqe (target_ctx, msg->user_data, msg->len, flags)) ret =0 ;if (target_ctx->flags &IORING_SETUP_IOPOLL ) -io_double_unlock_ctx (target_ctx); +io_double_unlock_ctx (target_ctx, &lock_state);return ret; }static int io_msg_ring_data (struct io_kiocb *req,unsigned int issue_flags) {
@@ -155,35 +159,38 @@ static int io_msg_ring_data(struct io_kiocb *req, unsigned int issue_flags) }static int io_msg_grab_file (struct io_kiocb *req,unsigned int issue_flags) { struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_rsrc_node *node;int ret = -EBADF ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); node =io_rsrc_node_lookup (&ctx->file_table.data, msg->src_fd);if (node) { msg->src_file =io_slot_file (node);if (msg->src_file)get_file (msg->src_file); req->flags |=REQ_F_NEED_CLEANUP ; ret =0 ; } -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return ret; }static int io_msg_install_complete (struct io_kiocb *req,unsigned int issue_flags) {struct io_ring_ctx *target_ctx = req->file->private_data; struct io_msg *msg = io_kiocb_to_cmd(req, struct io_msg); +struct io_ring_ctx_lock_state lock_state;struct file *src_file = msg->src_file;int ret; -if (unlikely (io_lock_external_ctx (target_ctx, issue_flags))) +if (unlikely (io_lock_external_ctx (target_ctx, issue_flags, + &lock_state)))return -EAGAIN ; ret =__io_fixed_fd_install (target_ctx, src_file, msg->dst_fd);if (ret <0 ) goto out_unlock;
@@ -200,11 +207,11 @@ static int io_msg_install_complete(struct io_kiocb *req, unsigned int issue_flag* later IORING_OP_MSG_RING delivers the message. */ if (!io_post_aux_cqe (target_ctx, msg->user_data, ret,0 )) ret = -EOVERFLOW ; out_unlock: -io_double_unlock_ctx (target_ctx); +io_double_unlock_ctx (target_ctx, &lock_state);return ret; }static void io_msg_tw_fd_complete (struct callback_head *head) {
io_uring/notif.c +3 -2
@@ -15,11 +15,11 @@ static void io_notif_tw_complete(struct io_tw_req tw_req, io_tw_token_t tw) {struct io_kiocb *notif = tw_req.req;struct io_notif_data *nd =io_notif_to_data (notif);struct io_ring_ctx *ctx = notif->ctx; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx); do { notif = cmd_to_io_kiocb(nd); if (WARN_ON_ONCE(ctx != notif->ctx))
@@ -109,15 +109,16 @@ static const struct ubuf_info_ops io_ubuf_ops = { .complete = io_tx_ubuf_complete, .link_skb = io_link_skb, };struct io_kiocb *io_alloc_notif (struct io_ring_ctx *ctx) - __must_hold(&ctx->uring_lock) {struct io_kiocb *notif;struct io_notif_data *nd; +io_ring_ctx_assert_locked (ctx); +if (unlikely (!io_alloc_req (ctx, ¬if)))return NULL ; notif->ctx = ctx; notif->opcode =IORING_OP_NOP ; notif->flags =0 ;
io_uring/notif.h +2 -1
@@ -31,14 +31,15 @@ static inline struct io_notif_data *io_notif_to_data(struct io_kiocb *notif) { return io_kiocb_to_cmd(notif, struct io_notif_data); }static inline void io_notif_flush (struct io_kiocb *notif) - __must_hold(¬if->ctx->uring_lock) {struct io_notif_data *nd =io_notif_to_data (notif); +io_ring_ctx_assert_locked (notif->ctx); +io_tx_ubuf_complete (NULL , &nd->uarg, true); }static inline int io_notif_account_mem (struct io_kiocb *notif,unsigned len) {
io_uring/openclose.c +8 -6
@@ -189,15 +189,16 @@ void io_open_cleanup(struct io_kiocb *req) }int __io_close_fixed (struct io_ring_ctx *ctx,unsigned int issue_flags,unsigned int offset) { +struct io_ring_ctx_lock_state lock_state;int ret; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ret =io_fixed_fd_remove (ctx, offset); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return ret; }static inline int io_close_fixed (struct io_kiocb *req,unsigned int issue_flags)
@@ -333,18 +334,19 @@ int io_pipe_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)static int io_pipe_fixed (struct io_kiocb *req,struct file **files,unsigned int issue_flags) { struct io_pipe *p = io_kiocb_to_cmd(req, struct io_pipe); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;int ret, fds[2 ] = {-1 ,-1 };int slot = p->file_slot;if (p->flags &O_CLOEXEC )return -EINVAL ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ret =__io_fixed_fd_install (ctx, files[0 ], slot);if (ret <0 ) goto err; fds[0 ] = ret;
@@ -361,23 +363,23 @@ static int io_pipe_fixed(struct io_kiocb *req, struct file **files,if (ret <0 ) goto err; fds[1 ] = ret; files[1 ] =NULL ; -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);if (!copy_to_user (p->fds, fds,sizeof (fds)))return 0 ; ret = -EFAULT ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); err:if (fds[0 ] !=-1 )io_fixed_fd_remove (ctx, fds[0 ]);if (fds[1 ] !=-1 )io_fixed_fd_remove (ctx, fds[1 ]); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return ret; }static int io_pipe_fd (struct io_kiocb *req,struct file **files) {
io_uring/poll.c +12 -9
@@ -121,11 +121,11 @@ static struct io_poll *io_poll_get_single(struct io_kiocb *req)static void io_poll_req_insert (struct io_kiocb *req) {struct io_hash_table *table = &req->ctx->cancel_table;u32 index =hash_long (req->cqe.user_data, table->hash_bits); -lockdep_assert_held (&req->ctx->uring_lock); +io_ring_ctx_assert_locked (req->ctx);hlist_add_head (&req->hash_node, &table->hbs[index].list); }static void io_init_poll_iocb (struct io_poll *poll,__poll_t events)
@@ -339,11 +339,11 @@ void io_poll_task_func(struct io_tw_req tw_req, io_tw_token_t tw) }else if (ret ==IOU_POLL_REQUEUE ) {__io_poll_execute (req,0 );return ; }io_poll_remove_entries (req); -/* task_work always has ->uring_lock held */ +/* task_work always holds ctx uring lock */ hash_del (&req->hash_node);if (req->opcode ==IORING_OP_POLL_ADD ) {if (ret ==IOU_POLL_DONE ) {struct io_poll *poll;
@@ -525,15 +525,16 @@ static bool io_poll_can_finish_inline(struct io_kiocb *req,return pt->owning ||io_poll_get_ownership (req); }static void io_poll_add_hash (struct io_kiocb *req,unsigned int issue_flags) { +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state);io_poll_req_insert (req); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state); }/* * Returns 0 when it's handed over for polling. The caller owns the requests if * it returns non-zero, but otherwise should not touch it. Negative values
@@ -728,11 +729,11 @@ __cold bool io_poll_remove_all(struct io_ring_ctx *ctx, struct io_uring_task *tcstruct hlist_node *tmp;struct io_kiocb *req;bool found = false;int i; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);for (i =0 ; i < nr_buckets; i++) {struct io_hash_bucket *hb = &ctx->cancel_table.hbs[i];hlist_for_each_entry_safe (req, tmp, &hb->list, hash_node) {
@@ -814,15 +815,16 @@ static int __io_poll_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) }int io_poll_cancel (struct io_ring_ctx *ctx,struct io_cancel_data *cd,unsigned issue_flags) { +struct io_ring_ctx_lock_state lock_state;int ret; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); ret =__io_poll_cancel (ctx, cd); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return ret; }static __poll_t io_poll_parse_events (const struct io_uring_sqe *sqe,unsigned int flags)
@@ -905,16 +907,17 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) }int io_poll_remove (struct io_kiocb *req,unsigned int issue_flags) { struct io_poll_update *poll_update = io_kiocb_to_cmd(req, struct io_poll_update); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_cancel_data cd = { .ctx = ctx, .data = poll_update->old_user_data, };struct io_kiocb *preq;int ret2, ret =0 ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); preq =io_poll_find (ctx, true, &cd); ret2 =io_poll_disarm (preq);if (ret2) { ret = ret2; goto out;
@@ -950,11 +953,11 @@ int io_poll_remove(struct io_kiocb *req, unsigned int issue_flags)if (preq->cqe.res <0 )req_set_fail (preq); preq->io_task_work.func = io_req_task_complete;io_req_task_work_add (preq); out: -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);if (ret <0 ) {req_set_fail (req);return ret; }/* complete update request, we're done with it */
io_uring/register.c +43 -36
268 lines not on this page · raw message
io_uring/rsrc.c +31 -20
199 lines not on this page · raw message
io_uring/rsrc.h +4 -2
@@ -2,10 +2,11 @@#ifndef IOU_RSRC_H #define IOU_RSRC_H #include <linux/io_uring_types.h> #include <linux/lockdep.h> +#include "io_uring.h" #define IO_VEC_CACHE_SOFT_CAP 256enum { IORING_RSRC_FILE = 0,
@@ -68,11 +69,12 @@ int io_import_reg_vec(int ddir, struct iov_iter *iter, struct io_kiocb *req, struct iou_vec *vec, unsigned nr_iovs, unsigned issue_flags);int io_prep_reg_iovec (struct io_kiocb *req,struct iou_vec *iv,const struct iovec __user *uvec,size_t uvec_segs); -int io_register_clone_buffers (struct io_ring_ctx *ctx,void __user *arg); +int io_register_clone_buffers (struct io_ring_ctx *ctx,void __user *arg, +struct io_ring_ctx_lock_state *lock_state);int io_sqe_buffers_unregister (struct io_ring_ctx *ctx);int io_sqe_buffers_register (struct io_ring_ctx *ctx,void __user *arg,unsigned int nr_args,u64 __user *tags);int io_sqe_files_unregister (struct io_ring_ctx *ctx); int io_sqe_files_register(struct io_ring_ctx *ctx, void __user *arg,
@@ -97,11 +99,11 @@ static inline struct io_rsrc_node *io_rsrc_node_lookup(struct io_rsrc_data *datareturn NULL ; }static inline void io_put_rsrc_node (struct io_ring_ctx *ctx,struct io_rsrc_node *node) { -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);if (!--node->refs)io_free_rsrc_node (ctx, node); } static inline bool io_reset_rsrc_node(struct io_ring_ctx *ctx,
io_uring/rw.c +1 -1
@@ -462,11 +462,11 @@ int io_read_mshot_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)void io_readv_writev_cleanup (struct io_kiocb *req) {struct io_async_rw *rw = req->async_data; -lockdep_assert_held (&req->ctx->uring_lock); +io_ring_ctx_assert_locked (req->ctx);io_vec_free (&rw->vec);io_rw_recycle (req,0 ); }static inline loff_t *io_kiocb_update_pos (struct io_kiocb *req)
io_uring/splice.c +3 -2
@@ -58,26 +58,27 @@ void io_splice_cleanup(struct io_kiocb *req)static struct file *io_splice_get_file (struct io_kiocb *req,unsigned int issue_flags) { struct io_splice *sp = io_kiocb_to_cmd(req, struct io_splice); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;struct io_rsrc_node *node;struct file *file =NULL ;if (!(sp->flags &SPLICE_F_FD_IN_FIXED ))return io_file_get_normal (req, sp->splice_fd_in); -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state); node =io_rsrc_node_lookup (&ctx->file_table.data, sp->splice_fd_in);if (node) { node->refs++; sp->rsrc_node = node; file =io_slot_file (node); req->flags |=REQ_F_NEED_CLEANUP ; } -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return file; }int io_tee (struct io_kiocb *req,unsigned int issue_flags) {
io_uring/sqpoll.c +3 -2
@@ -211,29 +211,30 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, struct io_sq_data *sqd,/* if we're handling multiple rings, cap submit size for fairness */ if (cap_entries && to_submit >IORING_SQPOLL_CAP_ENTRIES_VALUE ) to_submit =IORING_SQPOLL_CAP_ENTRIES_VALUE ;if (to_submit || !wq_list_empty (&ctx->iopoll_list)) { +struct io_ring_ctx_lock_state lock_state;const struct cred *creds =NULL ;io_sq_start_worktime (ist);if (ctx->sq_creds !=current_cred ()) creds =override_creds (ctx->sq_creds); -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, &lock_state);if (!wq_list_empty (&ctx->iopoll_list))io_do_iopoll (ctx, true);/* * Don't submit if refs are dying, good for io_uring_register(), * but also it is relied upon by io_ring_exit_work() */ if (to_submit &&likely (!percpu_ref_is_dying (&ctx->refs)) && !(ctx->flags &IORING_SETUP_R_DISABLED )) ret =io_submit_sqes (ctx, to_submit); -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state);if (to_submit &&wq_has_sleeper (&ctx->sqo_sq_wait))wake_up (&ctx->sqo_sq_wait);if (creds)revert_creds (creds);
io_uring/tctx.c +16 -11
@@ -13,27 +13,28 @@#include "tctx.h" static struct io_wq *io_init_wq_offload (struct io_ring_ctx *ctx,struct task_struct *task) { +struct io_ring_ctx_lock_state lock_state;struct io_wq_hash *hash;struct io_wq_data data;unsigned int concurrency; -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, &lock_state); hash = ctx->hash_map;if (!hash) { hash =kzalloc (sizeof (*hash),GFP_KERNEL );if (!hash) { -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state);return ERR_PTR (-ENOMEM ); }refcount_set (&hash->refs,1 );init_waitqueue_head (&hash->wait); ctx->hash_map = hash; } -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state); data.hash = hash; data.task = task;/* Do QD, or 4 * CPUS, whatever is smallest */
@@ -121,10 +122,12 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx)if (ret)return ret; } }if (!xa_load (&tctx->xa, (unsigned long )ctx)) { +struct io_ring_ctx_lock_state lock_state; + node =kmalloc (sizeof (*node),GFP_KERNEL );if (!node)return -ENOMEM ; node->ctx = ctx; node->task = current;
@@ -134,13 +137,13 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx)if (ret) {kfree (node);return ret; } -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, &lock_state);list_add (&node->ctx_node, &ctx->tctx_list); -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, &lock_state); }return 0 ; }int __io_uring_add_tctx_node_from_submit (struct io_ring_ctx *ctx)
@@ -163,10 +166,11 @@ int __io_uring_add_tctx_node_from_submit(struct io_ring_ctx *ctx)* Remove this io_uring_file -> task mapping. */ __cold void io_uring_del_tctx_node (unsigned long index) {struct io_uring_task *tctx = current->io_uring; +struct io_ring_ctx_lock_state lock_state;struct io_tctx_node *node;if (!tctx)return ; node =xa_erase (&tctx->xa, index);
@@ -174,13 +178,13 @@ __cold void io_uring_del_tctx_node(unsigned long index)return ;WARN_ON_ONCE (current != node->task);WARN_ON_ONCE (list_empty (&node->ctx_node)); -mutex_lock (&node->ctx->uring_lock); +io_ring_ctx_lock (node->ctx, &lock_state);list_del (&node->ctx_node); -mutex_unlock (&node->ctx->uring_lock); +io_ring_ctx_unlock (node->ctx, &lock_state);if (tctx->last == node->ctx) tctx->last =NULL ;kfree (node); }
@@ -196,11 +200,11 @@ __cold void io_uring_clean_tctx(struct io_uring_task *tctx)cond_resched (); }if (wq) {/* * Must be after io_uring_del_tctx_node() (removes nodes under -* uring_lock) to avoid race with io_uring_try_cancel_iowq(). +* ctx uring lock) to avoid race with io_uring_try_cancel_iowq() */ io_wq_put_and_exit (wq); tctx->io_wq =NULL ; } }
@@ -259,23 +263,24 @@ static int io_ring_add_registered_fd(struct io_uring_task *tctx, int fd,* index. If no index is desired, application may set ->offset == -1U * and we'll find an available index. Returns number of entries * successfully processed, or < 0 on error if none were processed. */ int io_ringfd_register (struct io_ring_ctx *ctx,void __user *__arg, -unsigned nr_args) +unsigned nr_args, +struct io_ring_ctx_lock_state *lock_state) {struct io_uring_rsrc_update __user *arg = __arg;struct io_uring_rsrc_update reg;struct io_uring_task *tctx;int ret, i;if (!nr_args || nr_args >IO_RINGFD_REG_MAX )return -EINVAL ; -mutex_unlock (&ctx->uring_lock); +io_ring_ctx_unlock (ctx, lock_state); ret =__io_uring_add_tctx_node (ctx); -mutex_lock (&ctx->uring_lock); +io_ring_ctx_lock (ctx, lock_state);if (ret)return ret; tctx = current->io_uring;for (i =0 ; i < nr_args; i++) {
io_uring/tctx.h +4 -1
@@ -1,7 +1,9 @@// SPDX-License-Identifier: GPL-2.0 +#include "io_uring.h" +struct io_tctx_node {struct list_head ctx_node;struct task_struct *task;struct io_ring_ctx *ctx; };
@@ -13,11 +15,12 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx);int __io_uring_add_tctx_node_from_submit (struct io_ring_ctx *ctx);void io_uring_clean_tctx (struct io_uring_task *tctx);void io_uring_unreg_ringfd (void );int io_ringfd_register (struct io_ring_ctx *ctx,void __user *__arg, -unsigned nr_args); +unsigned nr_args, +struct io_ring_ctx_lock_state *lock_state);int io_ringfd_unregister (struct io_ring_ctx *ctx,void __user *__arg,unsigned nr_args);/* * Note that this task has used io_uring. We use it for cancelation purposes.
io_uring/uring_cmd.c +8 -5
@@ -51,11 +51,11 @@ bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx, {struct hlist_node *tmp;struct io_kiocb *req;bool ret = false; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);hlist_for_each_entry_safe (req, tmp, &ctx->cancelable_uring_cmd, hash_node) {struct io_uring_cmd *cmd =io_kiocb_to_cmd (req, struct io_uring_cmd);
@@ -76,19 +76,20 @@ bool io_uring_try_cancel_uring_cmd(struct io_ring_ctx *ctx,static void io_uring_cmd_del_cancelable (struct io_uring_cmd *cmd,unsigned int issue_flags) {struct io_kiocb *req =cmd_to_io_kiocb (cmd); +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;if (!(cmd->flags &IORING_URING_CMD_CANCELABLE ))return ; cmd->flags &= ~IORING_URING_CMD_CANCELABLE ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state);hlist_del (&req->hash_node); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state); }/* * Mark this command as concelable, then io_uring_try_cancel_uring_cmd() * will try to cancel this issued command by sending ->uring_cmd() with
@@ -103,14 +104,16 @@ void io_uring_cmd_mark_cancelable(struct io_uring_cmd *cmd, {struct io_kiocb *req =cmd_to_io_kiocb (cmd);struct io_ring_ctx *ctx = req->ctx;if (!(cmd->flags &IORING_URING_CMD_CANCELABLE )) { +struct io_ring_ctx_lock_state lock_state; + cmd->flags |=IORING_URING_CMD_CANCELABLE ; -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state);hlist_add_head (&req->hash_node, &ctx->cancelable_uring_cmd); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state); } }EXPORT_SYMBOL_GPL (io_uring_cmd_mark_cancelable); void __io_uring_cmd_do_in_task(struct io_uring_cmd *ioucmd,
io_uring/waitid.c +7 -6
@@ -130,11 +130,11 @@ static void io_waitid_complete(struct io_kiocb *req, int ret) struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid);/* anyone completing better be holding a reference */ WARN_ON_ONCE (!(atomic_read (&iw->refs) &IO_WAITID_REF_MASK )); -lockdep_assert_held (&req->ctx->uring_lock); +io_ring_ctx_assert_locked (req->ctx);hlist_del_init (&req->hash_node);io_waitid_remove_wq (req); ret =io_waitid_finish (req, ret);
@@ -145,11 +145,11 @@ static void io_waitid_complete(struct io_kiocb *req, int ret)static bool __io_waitid_cancel (struct io_kiocb *req) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid); -lockdep_assert_held (&req->ctx->uring_lock); +io_ring_ctx_assert_locked (req->ctx);/* * Mark us canceled regardless of ownership. This will prevent a * potential retry from a spurious wakeup. */
@@ -280,10 +280,11 @@ int io_waitid_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)int io_waitid (struct io_kiocb *req,unsigned int issue_flags) { struct io_waitid *iw = io_kiocb_to_cmd(req, struct io_waitid);struct io_waitid_async *iwa = req->async_data; +struct io_ring_ctx_lock_state lock_state;struct io_ring_ctx *ctx = req->ctx;int ret; ret =kernel_waitid_prepare (&iwa->wo, iw->which, iw->upid, &iw->info, iw->options,NULL );
@@ -301,11 +302,11 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags)* Cancel must hold the ctx lock, so there's no risk of cancelation * finding us until a) we remain on the list, and b) the lock is * dropped. We only need to worry about racing with the wakeup * callback. */ -io_ring_submit_lock (ctx, issue_flags); +io_ring_submit_lock (ctx, issue_flags, &lock_state);/* * iw->head is valid under the ring lock, and as long as the request * is on the waitid_list where cancelations may find it. */
@@ -321,27 +322,27 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags)/* * Nobody else grabbed a reference, it'll complete when we get * a waitqueue callback, or if someone cancels it. */ if (!io_waitid_drop_issue_ref (req)) { -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return IOU_ISSUE_SKIP_COMPLETE ; }/* * Wakeup triggered, racing with us. It was prevented from * completing because of that, queue up the tw to do that. */ -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state);return IOU_ISSUE_SKIP_COMPLETE ; }hlist_del_init (&req->hash_node);io_waitid_remove_wq (req); ret =io_waitid_finish (req, ret); -io_ring_submit_unlock (ctx, issue_flags); +io_ring_submit_unlock (ctx, issue_flags, &lock_state); done:if (ret <0 )req_set_fail (req);io_req_set_res (req, ret,0 );return IOU_COMPLETE ;
io_uring/zcrx.c +1 -1
@@ -851,11 +851,11 @@ static struct net_iov *__io_zcrx_get_free_niov(struct io_zcrx_area *area)void io_unregister_zcrx_ifqs (struct io_ring_ctx *ctx) {struct io_zcrx_ifq *ifq; -lockdep_assert_held (&ctx->uring_lock); +io_ring_ctx_assert_locked (ctx);while (1 ) { scoped_guard(mutex, &ctx->mmap_lock) {unsigned long id =0 ;
Signature · 1 line
2.45.2
PATCH v6 6/6 io_uring: avoid uring_lock for IORING_SETUP_SINGLE_ISSUER
In reply to Caleb Sander Mateosio_ring_ctx's mutex uring_lock can be quite expensive in high-IOPS workloads. Even when only one thread pinned to a single CPU is accessing the io_ring_ctx, the atomic CASes required to lock and unlock the mutex are very hot instructions. The mutex's primary purpose is to prevent concurrent io_uring system calls on the same io_ring_ctx. However, there is already a flag IORING_SETUP_SINGLE_ISSUER that promises only one task will make io_uring_enter() and io_uring_register() system calls on the io_ring_ctx once it's enabled.
So if the io_ring_ctx is setup with IORING_SETUP_SINGLE_ISSUER, skip the uring_lock mutex_lock() and mutex_unlock() on the submitter_task. On other tasks acquiring the ctx uring lock, use a task work item to suspend the submitter_task for the critical section.
If the io_ring_ctx is IORING_SETUP_R_DISABLED (possible during io_uring_setup(), io_uring_register(), or io_uring exit), submitter_task may be set concurrently, so acquire the uring_lock before checking it. If submitter_task isn't set yet, the uring_lock suffices to provide mutual exclusion. If task work can't be queued because submitter_task has exited, also use the uring_lock for mutual exclusion.
--- io_uring/io_uring.c | 12 +++++ io_uring/io_uring.h | 118 ++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 127 insertions(+), 3 deletions(-)
io_uring/io_uring.c +12 -0
@@ -363,10 +363,22 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)xa_destroy (&ctx->io_bl_xa);kfree (ctx);return NULL ; } +void io_ring_suspend_work (struct callback_head *cb_head) +{ +struct io_ring_suspend_work *suspend_work = + container_of(cb_head, struct io_ring_suspend_work, cb_head); +DECLARE_COMPLETION_ONSTACK (suspend_end); + + *suspend_work->suspend_end = &suspend_end; +complete (&suspend_work->suspend_start); + +wait_for_completion (&suspend_end); +} +static void io_clean_op (struct io_kiocb *req) {if (unlikely (req->flags &REQ_F_BUFFER_SELECTED ))io_kbuf_drop_legacy (req);
io_uring/io_uring.h +115 -3
171 lines not on this page · raw message
Signature · 1 line
2.45.2
syzbot ci has tested the following series
syzbot ci has tested the following series
[v6] io_uring: avoid uring_lock for IORING_SETUP_SINGLE_ISSUER
https://lore.kernel.org/all/[email protected]
* [PATCH v6 1/6] io_uring: use release-acquire ordering for IORING_SETUP_R_DISABLED
* [PATCH v6 2/6] io_uring: clear IORING_SETUP_SINGLE_ISSUER for IORING_SETUP_SQPOLL
* [PATCH v6 3/6] io_uring: ensure submitter_task is valid for io_ring_ctx's lifetime
* [PATCH v6 4/6] io_uring: use io_ring_submit_lock() in io_iopoll_req_issued()
* [PATCH v6 5/6] io_uring: factor out uring_lock helpers
* [PATCH v6 6/6] io_uring: avoid uring_lock for IORING_SETUP_SINGLE_ISSUER
and found the following issue:
INFO: task hung in io_wq_put_and_exit
Full report is available here:
https://ci.syzbot.org/series/21eac721-670b-4f34-9696-66f9b28233ac
***
INFO: task hung in io_wq_put_and_exit
tree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: d358e5254674b70f34c847715ca509e46eb81e6f arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/1710cffe-7d78-4489-9aa1-823b8c2532ed/config syz repro: https://ci.syzbot.org/findings/74ae8703-9484-4d82-aa78-84cc37dcb1ef/syz_repro
INFO: task syz.1.18:6046 blocked for more than 143 seconds.
Not tainted syzkaller #0
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.1.18 state:D stack:25672 pid:6046 tgid:6045 ppid:5971 task_flags:0x400548 flags:0x00080004
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5256 [inline]
__schedule+0x14bc/0x5000 kernel/sched/core.c:6863
__schedule_loop kernel/sched/core.c:6945 [inline]
schedule+0x165/0x360 kernel/sched/core.c:6960
schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75
do_wait_for_common kernel/sched/completion.c:100 [inline]
__wait_for_common kernel/sched/completion.c:121 [inline]
wait_for_common kernel/sched/completion.c:132 [inline]
wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153
io_wq_exit_workers io_uring/io-wq.c:1328 [inline]
io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356
io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207
io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652
io_uring_files_cancel include/linux/io_uring.h:19 [inline]
do_exit+0x345/0x2310 kernel/exit.c:911
do_group_exit+0x21c/0x2d0 kernel/exit.c:1112
get_signal+0x1285/0x1340 kernel/signal.c:3034
arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337
__exit_to_user_mode_loop kernel/entry/common.c:41 [inline]
exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75
__exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline]
syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline]
syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline]
syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline]
do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f6a8b58f7c9
RSP: 002b:00007f6a8c4a00e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000001 RBX: 00007f6a8b7e5fa8 RCX: 00007f6a8b58f7c9
RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f6a8b7e5fac
RBP: 00007f6a8b7e5fa0 R08: 3fffffffffffffff R09: 0000000000000000
R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f6a8b7e6038 R14: 00007ffcac96d220 R15: 00007ffcac96d308
</TASK>
INFO: task iou-wrk-6046:6047 blocked for more than 143 seconds.
Not tainted syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-6046 state:D stack:27760 pid:6047 tgid:6045 ppid:5971 task_flags:0x404050 flags:0x00080002
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5256 [inline]
__schedule+0x14bc/0x5000 kernel/sched/core.c:6863
__schedule_loop kernel/sched/core.c:6945 [inline]
schedule+0x165/0x360 kernel/sched/core.c:6960
schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75
do_wait_for_common kernel/sched/completion.c:100 [inline]
__wait_for_common kernel/sched/completion.c:121 [inline]
wait_for_common kernel/sched/completion.c:132 [inline]
wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153
io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283
io_ring_ctx_lock io_uring/io_uring.h:290 [inline]
io_ring_submit_lock io_uring/io_uring.h:554 [inline]
io_files_update+0x677/0x7f0 io_uring/rsrc.c:504
__io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818
io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841
io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953
io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650
io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704
ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
</TASK>
INFO: task syz.0.17:6049 blocked for more than 143 seconds.
Not tainted syzkaller #0
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.0.17 state:D stack:25592 pid:6049 tgid:6048 ppid:5967 task_flags:0x400548 flags:0x00080004
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5256 [inline]
__schedule+0x14bc/0x5000 kernel/sched/core.c:6863
__schedule_loop kernel/sched/core.c:6945 [inline]
schedule+0x165/0x360 kernel/sched/core.c:6960
schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75
do_wait_for_common kernel/sched/completion.c:100 [inline]
__wait_for_common kernel/sched/completion.c:121 [inline]
wait_for_common kernel/sched/completion.c:132 [inline]
wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153
io_wq_exit_workers io_uring/io-wq.c:1328 [inline]
io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356
io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207
io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652
io_uring_files_cancel include/linux/io_uring.h:19 [inline]
do_exit+0x345/0x2310 kernel/exit.c:911
do_group_exit+0x21c/0x2d0 kernel/exit.c:1112
get_signal+0x1285/0x1340 kernel/signal.c:3034
arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337
__exit_to_user_mode_loop kernel/entry/common.c:41 [inline]
exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75
__exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline]
syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline]
syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline]
syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline]
do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fa96a98f7c9
RSP: 002b:00007fa96b7430e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000001 RBX: 00007fa96abe5fa8 RCX: 00007fa96a98f7c9
RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007fa96abe5fac
RBP: 00007fa96abe5fa0 R08: 3fffffffffffffff R09: 0000000000000000
R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000
R13: 00007fa96abe6038 R14: 00007ffd9fcc00d0 R15: 00007ffd9fcc01b8
</TASK>
INFO: task iou-wrk-6049:6050 blocked for more than 143 seconds.
Not tainted syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-6049 state:D stack:27760 pid:6050 tgid:6048 ppid:5967 task_flags:0x404050 flags:0x00080002
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5256 [inline]
__schedule+0x14bc/0x5000 kernel/sched/core.c:6863
__schedule_loop kernel/sched/core.c:6945 [inline]
schedule+0x165/0x360 kernel/sched/core.c:6960
schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75
do_wait_for_common kernel/sched/completion.c:100 [inline]
__wait_for_common kernel/sched/completion.c:121 [inline]
wait_for_common kernel/sched/completion.c:132 [inline]
wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153
io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283
io_ring_ctx_lock io_uring/io_uring.h:290 [inline]
io_ring_submit_lock io_uring/io_uring.h:554 [inline]
io_files_update+0x677/0x7f0 io_uring/rsrc.c:504
__io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818
io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841
io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953
io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650
io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704
ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
</TASK>
INFO: task syz.2.19:6052 blocked for more than 144 seconds.
Not tainted syzkaller #0
Blocked by coredump.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:syz.2.19 state:D stack:26208 pid:6052 tgid:6051 ppid:5972 task_flags:0x400548 flags:0x00080004
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5256 [inline]
__schedule+0x14bc/0x5000 kernel/sched/core.c:6863
__schedule_loop kernel/sched/core.c:6945 [inline]
schedule+0x165/0x360 kernel/sched/core.c:6960
schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75
do_wait_for_common kernel/sched/completion.c:100 [inline]
__wait_for_common kernel/sched/completion.c:121 [inline]
wait_for_common kernel/sched/completion.c:132 [inline]
wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153
io_wq_exit_workers io_uring/io-wq.c:1328 [inline]
io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356
io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207
io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652
io_uring_files_cancel include/linux/io_uring.h:19 [inline]
do_exit+0x345/0x2310 kernel/exit.c:911
do_group_exit+0x21c/0x2d0 kernel/exit.c:1112
get_signal+0x1285/0x1340 kernel/signal.c:3034
arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337
__exit_to_user_mode_loop kernel/entry/common.c:41 [inline]
exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75
__exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline]
syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline]
syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline]
syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline]
do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100
entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7f4b5cb8f7c9
RSP: 002b:00007f4b5d9a80e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: 0000000000000001 RBX: 00007f4b5cde5fa8 RCX: 00007f4b5cb8f7c9
RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f4b5cde5fac
RBP: 00007f4b5cde5fa0 R08: 3fffffffffffffff R09: 0000000000000000
R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000
R13: 00007f4b5cde6038 R14: 00007ffcdd64aed0 R15: 00007ffcdd64afb8
</TASK>
INFO: task iou-wrk-6052:6053 blocked for more than 144 seconds.
Not tainted syzkaller #0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:iou-wrk-6052 state:D stack:27760 pid:6053 tgid:6051 ppid:5972 task_flags:0x404050 flags:0x00080006
Call Trace:
<TASK>
context_switch kernel/sched/core.c:5256 [inline]
__schedule+0x14bc/0x5000 kernel/sched/core.c:6863
__schedule_loop kernel/sched/core.c:6945 [inline]
schedule+0x165/0x360 kernel/sched/core.c:6960
schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75
do_wait_for_common kernel/sched/completion.c:100 [inline]
__wait_for_common kernel/sched/completion.c:121 [inline]
wait_for_common kernel/sched/completion.c:132 [inline]
wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153
io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283
io_ring_ctx_lock io_uring/io_uring.h:290 [inline]
io_ring_submit_lock io_uring/io_uring.h:554 [inline]
io_files_update+0x677/0x7f0 io_uring/rsrc.c:504
__io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818
io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841
io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953
io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650
io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704
ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158
ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246
</TASK>
Showing all locks held in the system:
1 lock held by khungtaskd/35:
#0: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:331 [inline]
#0: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:867 [inline]
#0: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x2e/0x180 kernel/locking/lockdep.c:6775
5 locks held by kworker/u10:8/1120:
#0: ffff88823c63a918 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:639
#1: ffff88823c624588 (psi_seq){-.-.}-{0:0}, at: psi_task_switch+0x53/0x880 kernel/sched/psi.c:933
#2: ffff88810ac50788 (&rdev->wiphy.mtx){+.+.}-{4:4}, at: class_wiphy_constructor include/net/cfg80211.h:6363 [inline]
#2: ffff88810ac50788 (&rdev->wiphy.mtx){+.+.}-{4:4}, at: cfg80211_wiphy_work+0xb4/0x450 net/wireless/core.c:424
#3: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:331 [inline]
#3: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:867 [inline]
#3: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: ieee80211_sta_active_ibss+0xc3/0x330 net/mac80211/ibss.c:635
#4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:331 [inline]
#4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:867 [inline]
#4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: class_rcu_constructor include/linux/rcupdate.h:1195 [inline]
#4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: unwind_next_frame+0xa5/0x2390 arch/x86/kernel/unwind_orc.c:479
2 locks held by getty/5656:
#0: ffff8881133040a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243
#1: ffffc900035732f0 (&ldata->atomic_read_lock){+.+.}-{4:4}, at: n_tty_read+0x449/0x1460 drivers/tty/n_tty.c:2211
3 locks held by kworker/0:9/6480:
#0: ffff888100075948 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3232 [inline]
#0: ffff888100075948 ((wq_completion)events){+.+.}-{0:0}, at: process_scheduled_works+0x9b4/0x1770 kernel/workqueue.c:3340
#1: ffffc9000546fb80 (deferred_process_work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3233 [inline]
#1: ffffc9000546fb80 (deferred_process_work){+.+.}-{0:0}, at: process_scheduled_works+0x9ef/0x1770 kernel/workqueue.c:3340
#2: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: switchdev_deferred_process_work+0xe/0x20 net/switchdev/switchdev.c:104
1 lock held by syz-executor/6649:
#0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock net/core/rtnetlink.c:80 [inline]
#0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_nets_lock net/core/rtnetlink.c:341 [inline]
#0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x8ec/0x1c90 net/core/rtnetlink.c:4071
2 locks held by syz-executor/6651:
#0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock net/core/rtnetlink.c:80 [inline]
#0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_nets_lock net/core/rtnetlink.c:341 [inline]
#0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x8ec/0x1c90 net/core/rtnetlink.c:4071
#1: ffff88823c63a918 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:639
4 locks held by syz-executor/6653:=============================================
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 35 Comm: khungtaskd Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x189/0x250 lib/dump_stack.c:120 nmi_cpu_backtrace+0x39e/0x3d0 lib/nmi_backtrace.c:113 nmi_trigger_cpumask_backtrace+0x17a/0x300 lib/nmi_backtrace.c:62 trigger_all_cpu_backtrace include/linux/nmi.h:160 [inline] __sys_info lib/sys_info.c:157 [inline] sys_info+0x135/0x170 lib/sys_info.c:165 check_hung_uninterruptible_tasks kernel/hung_task.c:346 [inline] watchdog+0xf95/0xfe0 kernel/hung_task.c:515 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK> Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 UID: 0 PID: 6653 Comm: syz-executor Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:io_serial_out+0x7c/0xc0 drivers/tty/serial/8250/8250_port.c:407 Code: 3f a6 fc 44 89 f9 d3 e5 49 83 c6 40 4c 89 f0 48 c1 e8 03 42 80 3c 20 00 74 08 4c 89 f7 e8 ec 91 0c fd 41 03 2e 89 d8 89 ea ee <5b> 41 5c 41 5e 41 5f 5d c3 cc cc cc cc cc 44 89 f9 80 e1 07 38 c1 RSP: 0018:ffffc90008156590 EFLAGS: 00000002 RAX: 000000000000005b RBX: 000000000000005b RCX: 0000000000000000 RDX: 00000000000003f8 RSI: 0000000000000000 RDI: 0000000000000020 RBP: 00000000000003f8 R08: ffff888102f08237 R09: 1ffff110205e1046 R10: dffffc0000000000 R11: ffffffff851b9060 R12: dffffc0000000000 R13: ffffffff998dd9e1 R14: ffffffff99bf2420 R15: 0000000000000000 FS: 0000555595186500(0000) GS:ffff8882a9e37000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055599f9c9018 CR3: 0000000112ed8000 CR4: 00000000000006f0 Call Trace: <TASK> serial_port_out include/linux/serial_core.h:811 [inline] serial8250_console_putchar drivers/tty/serial/8250/8250_port.c:3192 [inline] serial8250_console_fifo_write drivers/tty/serial/8250/8250_port.c:-1 [inline] serial8250_console_write+0x1410/0x1ba0 drivers/tty/serial/8250/8250_port.c:3342 console_emit_next_record kernel/printk/printk.c:3129 [inline] console_flush_one_record kernel/printk/printk.c:3215 [inline] console_flush_all+0x745/0xb60 kernel/printk/printk.c:3289 __console_flush_and_unlock kernel/printk/printk.c:3319 [inline] console_unlock+0xbb/0x190 kernel/printk/printk.c:3359 vprintk_emit+0x4f8/0x5f0 kernel/printk/printk.c:2426 _printk+0xcf/0x120 kernel/printk/printk.c:2451 br_set_state+0x475/0x710 net/bridge/br_stp.c:57 br_init_port+0x99/0x200 net/bridge/br_stp_if.c:39 new_nbp+0x2f9/0x440 net/bridge/br_if.c:443 br_add_if+0x283/0xeb0 net/bridge/br_if.c:586 do_set_master+0x533/0x6d0 net/core/rtnetlink.c:2963 do_setlink+0xcf0/0x41c0 net/core/rtnetlink.c:3165 rtnl_changelink net/core/rtnetlink.c:3776 [inline] __rtnl_newlink net/core/rtnetlink.c:3935 [inline] rtnl_newlink+0x161c/0x1c90 net/core/rtnetlink.c:4072 rtnetlink_rcv_msg+0x7cf/0xb70 net/core/rtnetlink.c:6958 netlink_rcv_skb+0x208/0x470 net/netlink/af_netlink.c:2550 netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline] netlink_unicast+0x82f/0x9e0 net/netlink/af_netlink.c:1344 netlink_sendmsg+0x805/0xb30 net/netlink/af_netlink.c:1894 sock_sendmsg_nosec net/socket.c:727 [inline] __sock_sendmsg+0x21c/0x270 net/socket.c:742 __sys_sendto+0x3bd/0x520 net/socket.c:2206 __do_sys_sendto net/socket.c:2213 [inline] __se_sys_sendto net/socket.c:2209 [inline] __x64_sys_sendto+0xde/0x100 net/socket.c:2209 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xf80 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f780c39165c Code: 2a 5f 02 00 44 8b 4c 24 2c 4c 8b 44 24 20 89 c5 44 8b 54 24 28 48 8b 54 24 18 b8 2c 00 00 00 48 8b 74 24 10 8b 7c 24 08 0f 05 <48> 3d 00 f0 ff ff 77 34 89 ef 48 89 44 24 08 e8 70 5f 02 00 48 8b RSP: 002b:00007ffcecb618b0 EFLAGS: 00000293 ORIG_RAX: 000000000000002c RAX: ffffffffffffffda RBX: 00007f780d114620 RCX: 00007f780c39165c RDX: 0000000000000028 RSI: 00007f780d114670 RDI: 0000000000000003 RBP: 0000000000000000 R08: 00007ffcecb61904 R09: 000000000000000c R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000003 R13: 0000000000000000 R14: 00007f780d114670 R15: 0000000000000000 </TASK>
***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
Quoted 90 lines · syzbot ci
syzbot ci has tested the following series
[v6]
io_uring: avoiduring_lockforIORING_SETUP_SINGLE_ISSUER
https://lore.kernel.org/all/[email protected]
* [PATCH v6 1/6]io_uring: use release-acquire ordering forIORING_SETUP_R_DISABLED
* [PATCH v6 2/6]io_uring: clearIORING_SETUP_SINGLE_ISSUERforIORING_SETUP_SQPOLL
* [PATCH v6 3/6]io_uring: ensuresubmitter_taskis valid for io_ring_ctx's lifetime
* [PATCH v6 4/6]io_uring: useio_ring_submit_lock()inio_iopoll_req_issued()
* [PATCH v6 5/6]io_uring: factor outuring_lockhelpers
* [PATCH v6 6/6]io_uring: avoiduring_lockforIORING_SETUP_SINGLE_ISSUERand found the following issue:
INFO: task hung in io_wq_put_and_exitFull report is available here:
https://ci.syzbot.org/series/21eac721-670b-4f34-9696-66f9b28233ac***
INFO: task hung in io_wq_put_and_exittree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: d358e5254674b70f34c847715ca509e46eb81e6f arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/1710cffe-7d78-4489-9aa1-823b8c2532ed/config syz repro: https://ci.syzbot.org/findings/74ae8703-9484-4d82-aa78-84cc37dcb1ef/syz_reproINFO: task syz.1.18:6046 blocked for more than 143 seconds. Not tainted syzkaller #0 Blocked by coredump. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz.1.18 state:D stack:25672 pid:6046 tgid:6045 ppid:5971 task_flags:0x400548 flags:0x00080004 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_wq_exit_workers io_uring/io-wq.c:1328 [inline] io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356 io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207 io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652 io_uring_files_cancel include/linux/io_uring.h:19 [inline] do_exit+0x345/0x2310 kernel/exit.c:911 do_group_exit+0x21c/0x2d0 kernel/exit.c:1112 get_signal+0x1285/0x1340 kernel/signal.c:3034 arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337 __exit_to_user_mode_loop kernel/entry/common.c:41 [inline] exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline] syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline] syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline] syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline] do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f6a8b58f7c9 RSP: 002b:00007f6a8c4a00e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000001 RBX: 00007f6a8b7e5fa8 RCX: 00007f6a8b58f7c9 RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f6a8b7e5fac RBP: 00007f6a8b7e5fa0 R08: 3fffffffffffffff R09: 0000000000000000 R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f6a8b7e6038 R14: 00007ffcac96d220 R15: 00007ffcac96d308 </TASK> INFO: task iou-wrk-6046:6047 blocked for more than 143 seconds. Not tainted syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:iou-wrk-6046 state:D stack:27760 pid:6047 tgid:6045 ppid:5971 task_flags:0x404050 flags:0x00080002 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283 io_ring_ctx_lock io_uring/io_uring.h:290 [inline] io_ring_submit_lock io_uring/io_uring.h:554 [inline] io_files_update+0x677/0x7f0 io_uring/rsrc.c:504
Quoted 8 lines
__io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818 io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841 io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953 io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650 io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK>
Interesting, a deadlock between io_wq_exit_workers() on submitter_task (which is exiting) and io_ring_ctx_lock() on an io_uring worker thread. io_ring_ctx_lock() is blocked until submitter_task runs task work, but that will never happen because it's waiting on the completion. Not sure what the best approach is here. Maybe have the submitter_task alternate between running task work and waiting on the completion? Or have some way for submitter_task to indicate that it's exiting and disable the IORING_SETUP_SINGLE_ISSUER optimization in io_ring_ctx_lock()?
Thanks,
Caleb
Quoted 264 lines
INFO: task syz.0.17:6049 blocked for more than 143 seconds. Not tainted syzkaller #0 Blocked by coredump. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz.0.17 state:D stack:25592 pid:6049 tgid:6048 ppid:5967 task_flags:0x400548 flags:0x00080004 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_wq_exit_workers io_uring/io-wq.c:1328 [inline] io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356 io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207 io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652 io_uring_files_cancel include/linux/io_uring.h:19 [inline] do_exit+0x345/0x2310 kernel/exit.c:911 do_group_exit+0x21c/0x2d0 kernel/exit.c:1112 get_signal+0x1285/0x1340 kernel/signal.c:3034 arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337 __exit_to_user_mode_loop kernel/entry/common.c:41 [inline] exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline] syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline] syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline] syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline] do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7fa96a98f7c9 RSP: 002b:00007fa96b7430e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000001 RBX: 00007fa96abe5fa8 RCX: 00007fa96a98f7c9 RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007fa96abe5fac RBP: 00007fa96abe5fa0 R08: 3fffffffffffffff R09: 0000000000000000 R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fa96abe6038 R14: 00007ffd9fcc00d0 R15: 00007ffd9fcc01b8 </TASK> INFO: task iou-wrk-6049:6050 blocked for more than 143 seconds. Not tainted syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:iou-wrk-6049 state:D stack:27760 pid:6050 tgid:6048 ppid:5967 task_flags:0x404050 flags:0x00080002 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283 io_ring_ctx_lock io_uring/io_uring.h:290 [inline] io_ring_submit_lock io_uring/io_uring.h:554 [inline] io_files_update+0x677/0x7f0 io_uring/rsrc.c:504 __io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818 io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841 io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953 io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650 io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK> INFO: task syz.2.19:6052 blocked for more than 144 seconds. Not tainted syzkaller #0 Blocked by coredump. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz.2.19 state:D stack:26208 pid:6052 tgid:6051 ppid:5972 task_flags:0x400548 flags:0x00080004 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_wq_exit_workers io_uring/io-wq.c:1328 [inline] io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356 io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207 io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652 io_uring_files_cancel include/linux/io_uring.h:19 [inline] do_exit+0x345/0x2310 kernel/exit.c:911 do_group_exit+0x21c/0x2d0 kernel/exit.c:1112 get_signal+0x1285/0x1340 kernel/signal.c:3034 arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337 __exit_to_user_mode_loop kernel/entry/common.c:41 [inline] exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline] syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline] syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline] syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline] do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f4b5cb8f7c9 RSP: 002b:00007f4b5d9a80e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000001 RBX: 00007f4b5cde5fa8 RCX: 00007f4b5cb8f7c9 RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f4b5cde5fac RBP: 00007f4b5cde5fa0 R08: 3fffffffffffffff R09: 0000000000000000 R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f4b5cde6038 R14: 00007ffcdd64aed0 R15: 00007ffcdd64afb8 </TASK> INFO: task iou-wrk-6052:6053 blocked for more than 144 seconds. Not tainted syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:iou-wrk-6052 state:D stack:27760 pid:6053 tgid:6051 ppid:5972 task_flags:0x404050 flags:0x00080006 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283 io_ring_ctx_lock io_uring/io_uring.h:290 [inline] io_ring_submit_lock io_uring/io_uring.h:554 [inline] io_files_update+0x677/0x7f0 io_uring/rsrc.c:504 __io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818 io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841 io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953 io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650 io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK> Showing all locks held in the system: 1 lock held by khungtaskd/35: #0: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:331 [inline] #0: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:867 [inline] #0: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: debug_show_all_locks+0x2e/0x180 kernel/locking/lockdep.c:6775 5 locks held by kworker/u10:8/1120: #0: ffff88823c63a918 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:639 #1: ffff88823c624588 (psi_seq){-.-.}-{0:0}, at: psi_task_switch+0x53/0x880 kernel/sched/psi.c:933 #2: ffff88810ac50788 (&rdev->wiphy.mtx){+.+.}-{4:4}, at: class_wiphy_constructor include/net/cfg80211.h:6363 [inline] #2: ffff88810ac50788 (&rdev->wiphy.mtx){+.+.}-{4:4}, at: cfg80211_wiphy_work+0xb4/0x450 net/wireless/core.c:424 #3: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:331 [inline] #3: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:867 [inline] #3: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: ieee80211_sta_active_ibss+0xc3/0x330 net/mac80211/ibss.c:635 #4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire include/linux/rcupdate.h:331 [inline] #4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock include/linux/rcupdate.h:867 [inline] #4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: class_rcu_constructor include/linux/rcupdate.h:1195 [inline] #4: ffffffff8df419e0 (rcu_read_lock){....}-{1:3}, at: unwind_next_frame+0xa5/0x2390 arch/x86/kernel/unwind_orc.c:479 2 locks held by getty/5656: #0: ffff8881133040a0 (&tty->ldisc_sem){++++}-{0:0}, at: tty_ldisc_ref_wait+0x25/0x70 drivers/tty/tty_ldisc.c:243 #1: ffffc900035732f0 (&ldata->atomic_read_lock){+.+.}-{4:4}, at: n_tty_read+0x449/0x1460 drivers/tty/n_tty.c:2211 3 locks held by kworker/0:9/6480: #0: ffff888100075948 ((wq_completion)events){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3232 [inline] #0: ffff888100075948 ((wq_completion)events){+.+.}-{0:0}, at: process_scheduled_works+0x9b4/0x1770 kernel/workqueue.c:3340 #1: ffffc9000546fb80 (deferred_process_work){+.+.}-{0:0}, at: process_one_work kernel/workqueue.c:3233 [inline] #1: ffffc9000546fb80 (deferred_process_work){+.+.}-{0:0}, at: process_scheduled_works+0x9ef/0x1770 kernel/workqueue.c:3340 #2: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: switchdev_deferred_process_work+0xe/0x20 net/switchdev/switchdev.c:104 1 lock held by syz-executor/6649: #0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock net/core/rtnetlink.c:80 [inline] #0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_nets_lock net/core/rtnetlink.c:341 [inline] #0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x8ec/0x1c90 net/core/rtnetlink.c:4071 2 locks held by syz-executor/6651: #0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_lock net/core/rtnetlink.c:80 [inline] #0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_nets_lock net/core/rtnetlink.c:341 [inline] #0: ffffffff8f30ffc8 (rtnl_mutex){+.+.}-{4:4}, at: rtnl_newlink+0x8ec/0x1c90 net/core/rtnetlink.c:4071 #1: ffff88823c63a918 (&rq->__lock){-.-.}-{2:2}, at: raw_spin_rq_lock_nested+0x2a/0x140 kernel/sched/core.c:639 4 locks held by syz-executor/6653:=============================================
NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 35 Comm: khungtaskd Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x189/0x250 lib/dump_stack.c:120 nmi_cpu_backtrace+0x39e/0x3d0 lib/nmi_backtrace.c:113 nmi_trigger_cpumask_backtrace+0x17a/0x300 lib/nmi_backtrace.c:62 trigger_all_cpu_backtrace include/linux/nmi.h:160 [inline] __sys_info lib/sys_info.c:157 [inline] sys_info+0x135/0x170 lib/sys_info.c:165 check_hung_uninterruptible_tasks kernel/hung_task.c:346 [inline] watchdog+0xf95/0xfe0 kernel/hung_task.c:515 kthread+0x711/0x8a0 kernel/kthread.c:463 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK> Sending NMI from CPU 0 to CPUs 1: NMI backtrace for cpu 1 CPU: 1 UID: 0 PID: 6653 Comm: syz-executor Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:io_serial_out+0x7c/0xc0 drivers/tty/serial/8250/8250_port.c:407 Code: 3f a6 fc 44 89 f9 d3 e5 49 83 c6 40 4c 89 f0 48 c1 e8 03 42 80 3c 20 00 74 08 4c 89 f7 e8 ec 91 0c fd 41 03 2e 89 d8 89 ea ee <5b> 41 5c 41 5e 41 5f 5d c3 cc cc cc cc cc 44 89 f9 80 e1 07 38 c1 RSP: 0018:ffffc90008156590 EFLAGS: 00000002 RAX: 000000000000005b RBX: 000000000000005b RCX: 0000000000000000 RDX: 00000000000003f8 RSI: 0000000000000000 RDI: 0000000000000020 RBP: 00000000000003f8 R08: ffff888102f08237 R09: 1ffff110205e1046 R10: dffffc0000000000 R11: ffffffff851b9060 R12: dffffc0000000000 R13: ffffffff998dd9e1 R14: ffffffff99bf2420 R15: 0000000000000000 FS: 0000555595186500(0000) GS:ffff8882a9e37000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055599f9c9018 CR3: 0000000112ed8000 CR4: 00000000000006f0 Call Trace: <TASK> serial_port_out include/linux/serial_core.h:811 [inline] serial8250_console_putchar drivers/tty/serial/8250/8250_port.c:3192 [inline] serial8250_console_fifo_write drivers/tty/serial/8250/8250_port.c:-1 [inline] serial8250_console_write+0x1410/0x1ba0 drivers/tty/serial/8250/8250_port.c:3342 console_emit_next_record kernel/printk/printk.c:3129 [inline] console_flush_one_record kernel/printk/printk.c:3215 [inline] console_flush_all+0x745/0xb60 kernel/printk/printk.c:3289 __console_flush_and_unlock kernel/printk/printk.c:3319 [inline] console_unlock+0xbb/0x190 kernel/printk/printk.c:3359 vprintk_emit+0x4f8/0x5f0 kernel/printk/printk.c:2426 _printk+0xcf/0x120 kernel/printk/printk.c:2451 br_set_state+0x475/0x710 net/bridge/br_stp.c:57 br_init_port+0x99/0x200 net/bridge/br_stp_if.c:39 new_nbp+0x2f9/0x440 net/bridge/br_if.c:443 br_add_if+0x283/0xeb0 net/bridge/br_if.c:586 do_set_master+0x533/0x6d0 net/core/rtnetlink.c:2963 do_setlink+0xcf0/0x41c0 net/core/rtnetlink.c:3165 rtnl_changelink net/core/rtnetlink.c:3776 [inline] __rtnl_newlink net/core/rtnetlink.c:3935 [inline] rtnl_newlink+0x161c/0x1c90 net/core/rtnetlink.c:4072 rtnetlink_rcv_msg+0x7cf/0xb70 net/core/rtnetlink.c:6958 netlink_rcv_skb+0x208/0x470 net/netlink/af_netlink.c:2550 netlink_unicast_kernel net/netlink/af_netlink.c:1318 [inline] netlink_unicast+0x82f/0x9e0 net/netlink/af_netlink.c:1344 netlink_sendmsg+0x805/0xb30 net/netlink/af_netlink.c:1894 sock_sendmsg_nosec net/socket.c:727 [inline] __sock_sendmsg+0x21c/0x270 net/socket.c:742 __sys_sendto+0x3bd/0x520 net/socket.c:2206 __do_sys_sendto net/socket.c:2213 [inline] __se_sys_sendto net/socket.c:2209 [inline] __x64_sys_sendto+0xde/0x100 net/socket.c:2209 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline] do_syscall_64+0xfa/0xf80 arch/x86/entry/syscall_64.c:94 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f780c39165c Code: 2a 5f 02 00 44 8b 4c 24 2c 4c 8b 44 24 20 89 c5 44 8b 54 24 28 48 8b 54 24 18 b8 2c 00 00 00 48 8b 74 24 10 8b 7c 24 08 0f 05 <48> 3d 00 f0 ff ff 77 34 89 ef 48 89 44 24 08 e8 70 5f 02 00 48 8b RSP: 002b:00007ffcecb618b0 EFLAGS: 00000293 ORIG_RAX: 000000000000002c RAX: ffffffffffffffda RBX: 00007f780d114620 RCX: 00007f780c39165c RDX: 0000000000000028 RSI: 00007f780d114670 RDI: 0000000000000003 RBP: 0000000000000000 R08: 00007ffcecb61904 R09: 000000000000000c R10: 0000000000000000 R11: 0000000000000293 R12: 0000000000000003 R13: 0000000000000000 R14: 00007f780d114670 R15: 0000000000000000 </TASK>***
If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message:
Tested-by [email protected]---
This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at [email protected].
Jens Axboe
maintainerQuoted 101 lines · Caleb Sander Mateos
Quoted 90 lines · On Thu, Dec 18, 2025 at 3:01?AM syzbot ci…
syzbot ci has tested the following series
[v6]
io_uring: avoiduring_lockforIORING_SETUP_SINGLE_ISSUER
https://lore.kernel.org/all/[email protected]
* [PATCH v6 1/6]io_uring: use release-acquire ordering forIORING_SETUP_R_DISABLED
* [PATCH v6 2/6]io_uring: clearIORING_SETUP_SINGLE_ISSUERforIORING_SETUP_SQPOLL
* [PATCH v6 3/6]io_uring: ensuresubmitter_taskis valid for io_ring_ctx's lifetime
* [PATCH v6 4/6]io_uring: useio_ring_submit_lock()inio_iopoll_req_issued()
* [PATCH v6 5/6]io_uring: factor outuring_lockhelpers
* [PATCH v6 6/6]io_uring: avoiduring_lockforIORING_SETUP_SINGLE_ISSUERand found the following issue:
INFO: task hung in io_wq_put_and_exitFull report is available here:
https://ci.syzbot.org/series/21eac721-670b-4f34-9696-66f9b28233ac***
INFO: task hung in io_wq_put_and_exittree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: d358e5254674b70f34c847715ca509e46eb81e6f arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/1710cffe-7d78-4489-9aa1-823b8c2532ed/config syz repro: https://ci.syzbot.org/findings/74ae8703-9484-4d82-aa78-84cc37dcb1ef/syz_reproINFO: task syz.1.18:6046 blocked for more than 143 seconds. Not tainted syzkaller #0 Blocked by coredump. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz.1.18 state:D stack:25672 pid:6046 tgid:6045 ppid:5971 task_flags:0x400548 flags:0x00080004 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_wq_exit_workers io_uring/io-wq.c:1328 [inline] io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356 io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207 io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652 io_uring_files_cancel include/linux/io_uring.h:19 [inline] do_exit+0x345/0x2310 kernel/exit.c:911 do_group_exit+0x21c/0x2d0 kernel/exit.c:1112 get_signal+0x1285/0x1340 kernel/signal.c:3034 arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337 __exit_to_user_mode_loop kernel/entry/common.c:41 [inline] exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline] syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline] syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline] syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline] do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f6a8b58f7c9 RSP: 002b:00007f6a8c4a00e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000001 RBX: 00007f6a8b7e5fa8 RCX: 00007f6a8b58f7c9 RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f6a8b7e5fac RBP: 00007f6a8b7e5fa0 R08: 3fffffffffffffff R09: 0000000000000000 R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f6a8b7e6038 R14: 00007ffcac96d220 R15: 00007ffcac96d308 </TASK> INFO: task iou-wrk-6046:6047 blocked for more than 143 seconds. Not tainted syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:iou-wrk-6046 state:D stack:27760 pid:6047 tgid:6045 ppid:5971 task_flags:0x404050 flags:0x00080002 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283 io_ring_ctx_lock io_uring/io_uring.h:290 [inline] io_ring_submit_lock io_uring/io_uring.h:554 [inline] io_files_update+0x677/0x7f0 io_uring/rsrc.c:504Quoted 8 lines
__io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818 io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841 io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953 io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650 io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK>
Quoted 9 lines
Interesting, a deadlock between
io_wq_exit_workers()onsubmitter_task(which is exiting) andio_ring_ctx_lock()on anio_uringworker thread.io_ring_ctx_lock()is blocked untilsubmitter_taskruns task work, but that will never happen because it's waiting on the completion. Not sure what the best approach is here. Maybe have thesubmitter_taskalternate between running task work and waiting on the completion? Or have some way forsubmitter_taskto indicate that it's exiting and disable theIORING_SETUP_SINGLE_ISSUERoptimization inio_ring_ctx_lock()?
Finally got around to taking a look at this patchset today, and it does look sound to me. For cases that have zero expected io-wq activity, then it seems like a no-brainer. For cases that have a lot of expected io-wq activity, which are basically only things like fs/storage workloads on suboptimal configurations, the then the suspend/resume mechanism may be troublesome. But not quite sure what to do about that, or if it's evne noticable?
For the case in question, yes I think we'll need the completion wait cases to break for running task_work.
Signature · 1 line
Jens Axboe
Quoted 113 lines · Jens Axboe
Quoted 101 lines · Caleb Sander Mateos
Quoted 90 lines · On Thu, Dec 18, 2025 at 3:01?AM syzbot ci…
syzbot ci has tested the following series
[v6]
io_uring: avoiduring_lockforIORING_SETUP_SINGLE_ISSUER
https://lore.kernel.org/all/[email protected]
* [PATCH v6 1/6]io_uring: use release-acquire ordering forIORING_SETUP_R_DISABLED
* [PATCH v6 2/6]io_uring: clearIORING_SETUP_SINGLE_ISSUERforIORING_SETUP_SQPOLL
* [PATCH v6 3/6]io_uring: ensuresubmitter_taskis valid for io_ring_ctx's lifetime
* [PATCH v6 4/6]io_uring: useio_ring_submit_lock()inio_iopoll_req_issued()
* [PATCH v6 5/6]io_uring: factor outuring_lockhelpers
* [PATCH v6 6/6]io_uring: avoiduring_lockforIORING_SETUP_SINGLE_ISSUERand found the following issue:
INFO: task hung in io_wq_put_and_exitFull report is available here:
https://ci.syzbot.org/series/21eac721-670b-4f34-9696-66f9b28233ac***
INFO: task hung in io_wq_put_and_exittree: torvalds URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux base: d358e5254674b70f34c847715ca509e46eb81e6f arch: amd64 compiler: Debian clang version 20.1.8 (++20250708063551+0c9f909b7976-1~exp1~20250708183702.136), Debian LLD 20.1.8 config: https://ci.syzbot.org/builds/1710cffe-7d78-4489-9aa1-823b8c2532ed/config syz repro: https://ci.syzbot.org/findings/74ae8703-9484-4d82-aa78-84cc37dcb1ef/syz_reproINFO: task syz.1.18:6046 blocked for more than 143 seconds. Not tainted syzkaller #0 Blocked by coredump. "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:syz.1.18 state:D stack:25672 pid:6046 tgid:6045 ppid:5971 task_flags:0x400548 flags:0x00080004 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_wq_exit_workers io_uring/io-wq.c:1328 [inline] io_wq_put_and_exit+0x316/0x650 io_uring/io-wq.c:1356 io_uring_clean_tctx+0x11f/0x1a0 io_uring/tctx.c:207 io_uring_cancel_generic+0x6ca/0x7d0 io_uring/cancel.c:652 io_uring_files_cancel include/linux/io_uring.h:19 [inline] do_exit+0x345/0x2310 kernel/exit.c:911 do_group_exit+0x21c/0x2d0 kernel/exit.c:1112 get_signal+0x1285/0x1340 kernel/signal.c:3034 arch_do_signal_or_restart+0x9a/0x7a0 arch/x86/kernel/signal.c:337 __exit_to_user_mode_loop kernel/entry/common.c:41 [inline] exit_to_user_mode_loop+0x87/0x4f0 kernel/entry/common.c:75 __exit_to_user_mode_prepare include/linux/irq-entry-common.h:226 [inline] syscall_exit_to_user_mode_prepare include/linux/irq-entry-common.h:256 [inline] syscall_exit_to_user_mode_work include/linux/entry-common.h:159 [inline] syscall_exit_to_user_mode include/linux/entry-common.h:194 [inline] do_syscall_64+0x2e3/0xf80 arch/x86/entry/syscall_64.c:100 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f6a8b58f7c9 RSP: 002b:00007f6a8c4a00e8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca RAX: 0000000000000001 RBX: 00007f6a8b7e5fa8 RCX: 00007f6a8b58f7c9 RDX: 00000000000f4240 RSI: 0000000000000081 RDI: 00007f6a8b7e5fac RBP: 00007f6a8b7e5fa0 R08: 3fffffffffffffff R09: 0000000000000000 R10: 0000000000000800 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f6a8b7e6038 R14: 00007ffcac96d220 R15: 00007ffcac96d308 </TASK> INFO: task iou-wrk-6046:6047 blocked for more than 143 seconds. Not tainted syzkaller #0 "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. task:iou-wrk-6046 state:D stack:27760 pid:6047 tgid:6045 ppid:5971 task_flags:0x404050 flags:0x00080002 Call Trace: <TASK> context_switch kernel/sched/core.c:5256 [inline] __schedule+0x14bc/0x5000 kernel/sched/core.c:6863 __schedule_loop kernel/sched/core.c:6945 [inline] schedule+0x165/0x360 kernel/sched/core.c:6960 schedule_timeout+0x9a/0x270 kernel/time/sleep_timeout.c:75 do_wait_for_common kernel/sched/completion.c:100 [inline] __wait_for_common kernel/sched/completion.c:121 [inline] wait_for_common kernel/sched/completion.c:132 [inline] wait_for_completion+0x2bf/0x5d0 kernel/sched/completion.c:153 io_ring_ctx_lock_nested+0x2b3/0x380 io_uring/io_uring.h:283 io_ring_ctx_lock io_uring/io_uring.h:290 [inline] io_ring_submit_lock io_uring/io_uring.h:554 [inline] io_files_update+0x677/0x7f0 io_uring/rsrc.c:504Quoted 8 lines
__io_issue_sqe+0x181/0x4b0 io_uring/io_uring.c:1818 io_issue_sqe+0x1de/0x1190 io_uring/io_uring.c:1841 io_wq_submit_work+0x6e9/0xb90 io_uring/io_uring.c:1953 io_worker_handle_work+0x7cd/0x1180 io_uring/io-wq.c:650 io_wq_worker+0x42f/0xeb0 io_uring/io-wq.c:704 ret_from_fork+0x599/0xb30 arch/x86/kernel/process.c:158 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:246 </TASK>Quoted 9 lines
Interesting, a deadlock between
io_wq_exit_workers()onsubmitter_task(which is exiting) andio_ring_ctx_lock()on anio_uringworker thread.io_ring_ctx_lock()is blocked untilsubmitter_taskruns task work, but that will never happen because it's waiting on the completion. Not sure what the best approach is here. Maybe have thesubmitter_taskalternate between running task work and waiting on the completion? Or have some way forsubmitter_taskto indicate that it's exiting and disable theIORING_SETUP_SINGLE_ISSUERoptimization inio_ring_ctx_lock()?
Quoted 1 line
Finally got around to taking a look at this patchset today, and it does
Appreciate you taking a look!
Quoted 6 lines
look sound to me. For cases that have zero expected io-wq activity, then it seems like a no-brainer. For cases that have a lot of expected io-wq activity, which are basically only things like fs/storage workloads on suboptimal configurations, the then the suspend/resume mechanism may be troublesome. But not quite sure what to do about that, or if it's evne noticable?
Yes, this is a good point. I was hoping io_uring worker threads wouldn't need to acquire the ctx uring lock in most cases, as fixed buffer and file lookup will likely have already happened during the initial non-blocking submission. That clearly wouldn't be true for IOSQE_ASYNC requests, though. And maybe other cases I haven't thought of?
One possibility would be to introduce a new setup flag ("IORING_SETUP_TRULY_SINGLE_ISSUER" or something) to explicitly opt into this behavior to avoid regressing any existing applications. Or we could try to optimize IORING_SETUP_SINGLE_ISSUER + IOSQE_ASYNC to import any fixed file, fixed buffer, or provided buffer on submitter_task before punting the request to the worker thread.
Quoted 3 lines
For the case in question, yes I think we'll need the completion wait cases to break for running
task_work.
Thanks for taking a look. Are you thinking something like changing the wait_for_completion() in io_wq_exit_workers() to wait_for_completion_interruptible() and call io_run_task_work() whenever the wait is interrupted?
Thanks,
Caleb