Skip to contentSkip to conversation

Cancel and wait for all requests on exit

RFC PATCH Superseded Jens Axboe 2 participants 10 messages Mar 21, 2025 – Apr 9, 2025

Subsystems (6)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

v2 cover Cancel and wait for all requests on exit

Hi,

Currently, when a ring is being shut down, some cancelations may happen out-of-line. This means that an application cannot rely on the ring exit meaning that any IO has fully completed, or someone else waiting on an application (which has a ring with pending IO) being terminated will mean that all requests are done. This has also manifested itself as various testing sometimes finding a mount point busy after a test has exited, because it may take a brief period of time for things to quiesce and be fully done.

This patchset makes the task wait on the cancelations, if any, when the io_uring file fd is being put. That has the effect of ensuring that pending IO has fully completed, and files closed, before the ring exit returns.

I did post a previous version of this - fundamentally this one is the same, with the main difference being that rather than invent our own type of references for the ring, a basic atomic_long_t is used. io_uring batches the reference gets and puts on the ring, so this should not be noticeable. The only potential outlier is setting up a ring without DEFER_TASKRUN, where running task_work will result in an atomic dec and inc per ring in running the task_work. We can probably do something about that, but I don't consider it pressing.

The switch away from percpu reference counts is done mostly because exiting those references will cost us an RCU grace period. That will noticeably slow down the ring tear down.

The changes can also be found here:

https://git.kernel.dk/cgit/linux/log/?h=io_uring-exit-cancel.2

 fs/file_table.c                |  2 +-
 include/linux/io_uring_types.h |  4 +-
 include/linux/sched.h          |  2 +-
 io_uring/io_uring.c            | 79 +++++++++++++++++++++++-----------
 io_uring/io_uring.h            |  3 +-
 io_uring/msg_ring.c            |  4 +-
 io_uring/refs.h                | 43 ++++++++++++++++++
 io_uring/register.c            |  2 +-
 io_uring/rw.c                  |  2 +-
 io_uring/sqpoll.c              |  2 +-
 io_uring/zcrx.c                |  4 +-
 kernel/fork.c                  |  2 +-
 12 files changed, 111 insertions(+), 38 deletions(-)
Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH v2 1/5 fs: gate final fput task_work on PF_NO_TASKWORK

Rather than hardwire this to kernel threads, add a task flag that tells us whether the task in question runs task_work or not. At fork time, this flag is set for kernel threads. This is in preparation for allowing kernel threads to signal that they will run deferred task_work.

No functional changes in this patch.

---
 fs/file_table.c       | 2 +-
 include/linux/sched.h | 2 +-
 kernel/fork.c         | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)
fs/file_table.c +1 -1
index 5c00dc38558d..d824f1330d6e 100644
@@ -520,7 +520,7 @@ void fput(struct file *file)
 			file_free(file);
 			return;
 		}
-		if (likely(!in_interrupt() && !(task->flags & PF_KTHREAD))) {
+		if (likely(!in_interrupt() && !(task->flags & PF_NO_TASKWORK))) {
 			init_task_work(&file->f_task_work, ____fput);
 			if (!task_work_add(task, &file->f_task_work, TWA_RESUME))
 				return;
include/linux/sched.h +1 -1
index 9c15365a30c0..301f5dda6a06 100644
@@ -1709,7 +1709,7 @@ extern struct pid *cad_pid;
 						 * I am cleaning dirty pages from some other bdi. */
 #define PF_KTHREAD		0x00200000	/* I am a kernel thread */
 #define PF_RANDOMIZE		0x00400000	/* Randomize virtual address space */
-#define PF__HOLE__00800000	0x00800000
+#define PF_NO_TASKWORK		0x00800000	/* task doesn't run task_work */
 #define PF__HOLE__01000000	0x01000000
 #define PF__HOLE__02000000	0x02000000
 #define PF_NO_SETAFFINITY	0x04000000	/* Userland is not allowed to meddle with cpus_mask */
kernel/fork.c +1 -1
index 735405a9c5f3..3745407624c7 100644
@@ -2235,7 +2235,7 @@ __latent_entropy struct task_struct *copy_process(
 		goto fork_out;
 	p->flags &= ~PF_KTHREAD;
 	if (args->kthread)
-		p->flags |= PF_KTHREAD;
+		p->flags |= PF_KTHREAD | PF_NO_TASKWORK;
 	if (args->user_worker) {
 		/*
 		 * Mark us a user worker, and block any signal that isn't
Signature · 1 line
2.49.0
author maintainer
#2 lore raw

PATCH v2 2/5 io_uring: mark exit side kworkers as task_work capable

In reply to Jens Axboe

There are two types of work here:

1) Fallback work, if the task is exiting
2) The exit side cancelations

and both of them may do the final fput() of a file. When this happens, fput() will schedule delayed work. This slows down exits when io_uring needs to wait for that work to finish. It is possible to flush this via flush_delayed_fput(), but that's a big hammer as other unrelated files could be involved, and from other tasks as well.

Add two io_uring helpers to temporarily clear PF_NO_TASKWORK for the worker threads, and run any queued task_work before setting the flag again. Then we can ensure we only flush related items that received their final fput as part of work cancelation and flushing.

For now these are io_uring private, but could obviously be made generically available, should there be a need to do so.

---
 io_uring/io_uring.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
io_uring/io_uring.c +21 -0
index 5f625be52e52..2b9dae588f04 100644
@@ -238,6 +238,20 @@ static inline void io_req_add_to_cache(struct io_kiocb *req, struct io_ring_ctx
 	wq_stack_add_head(&req->comp_list, &ctx->submit_state.free_list);
 }
 
+static __cold void io_kworker_tw_start(void)
+{
+	if (WARN_ON_ONCE(!(current->flags & PF_NO_TASKWORK)))
+		return;
+	current->flags &= ~PF_NO_TASKWORK;
+}
+
+static __cold void io_kworker_tw_end(void)
+{
+	while (task_work_pending(current))
+		task_work_run();
+	current->flags |= PF_NO_TASKWORK;
+}
+
 static __cold void io_ring_ctx_ref_free(struct percpu_ref *ref)
 {
 	struct io_ring_ctx *ctx = container_of(ref, struct io_ring_ctx, refs);
@@ -253,6 +267,8 @@ static __cold void io_fallback_req_func(struct work_struct *work)
 	struct io_kiocb *req, *tmp;
 	struct io_tw_state ts = {};
 
+	io_kworker_tw_start();
+
 	percpu_ref_get(&ctx->refs);
 	mutex_lock(&ctx->uring_lock);
 	llist_for_each_entry_safe(req, tmp, node, io_task_work.node)
@@ -260,6 +276,7 @@ static __cold void io_fallback_req_func(struct work_struct *work)
 	io_submit_flush_completions(ctx);
 	mutex_unlock(&ctx->uring_lock);
 	percpu_ref_put(&ctx->refs);
+	io_kworker_tw_end();
 }
 
 static int io_alloc_hash_table(struct io_hash_table *table, unsigned bits)
@@ -2879,6 +2896,8 @@ static __cold void io_ring_exit_work(struct work_struct *work)
 	struct io_tctx_node *node;
 	int ret;
 
+	io_kworker_tw_start();
+
 	/*
 	 * If we're doing polled IO and end up having requests being
 	 * submitted async (out-of-line), then completions can come in while
@@ -2935,6 +2954,8 @@ static __cold void io_ring_exit_work(struct work_struct *work)
 		 */
 	} while (!wait_for_completion_interruptible_timeout(&ctx->ref_comp, interval));
 
+	io_kworker_tw_end();
+
 	init_completion(&exit.completion);
 	init_task_work(&exit.task_work, io_tctx_exit_cb);
 	exit.ctx = ctx;
Signature · 1 line
2.49.0
author maintainer
#3 lore raw

PATCH v2 3/5 io_uring: consider ring dead once the ref is marked dying

In reply to Jens Axboe

Don't gate this on the task exiting flag. It's generally not a good idea to gate it on the task PF_EXITING flag anyway. Once the ring is starting to go through ring teardown, the ref is marked as dying. Use that as our fallback/cancel mechanism.

---
 io_uring/io_uring.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
io_uring/io_uring.c +4 -2
index 2b9dae588f04..984db01f5184 100644
@@ -555,7 +555,8 @@ static void io_queue_iowq(struct io_kiocb *req)
 	 * procedure rather than attempt to run this request (or create a new
 	 * worker for it).
 	 */
-	if (WARN_ON_ONCE(!same_thread_group(tctx->task, current)))
+	if (WARN_ON_ONCE(!same_thread_group(tctx->task, current) &&
+			 !percpu_ref_is_dying(&req->ctx->refs)))
 		atomic_or(IO_WQ_WORK_CANCEL, &req->work.flags);
 
 	trace_io_uring_queue_async_work(req, io_wq_is_hashed(&req->work));
@@ -1254,7 +1255,8 @@ static void io_req_normal_work_add(struct io_kiocb *req)
 		return;
 	}
 
-	if (likely(!task_work_add(tctx->task, &tctx->task_work, ctx->notify_method)))
+	if (!percpu_ref_is_dying(&ctx->refs) &&
+	    !task_work_add(tctx->task, &tctx->task_work, ctx->notify_method))
 		return;
 
 	io_fallback_tw(tctx, false);
Signature · 1 line
2.49.0
maintainer
#4 lore raw

Re: [PATCH 3/5] io_uring: consider ring dead once the ref is marked dying

Quoted 1 line · Jens Axboe

Don't gate this on the task exiting flag. It's generally not a good idea

Do you refer to tw add and the PF_EXITING logic inside? We can't gate it solely on dying refs as it's not sync'ed (and the patch doesn't). And task is dying is not same as ring is closed. E.g. a task can exit(2) but leave the ring intact to other tasks.

Quoted 14 lines

to gate it on the task PF_EXITING flag anyway. Once the ring is starting to go through ring teardown, the ref is marked as dying. Use that as our fallback/cancel mechanism.

---
  io_uring/io_uring.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
io_uring/io_uring.c +0 -0
index 2b9dae588f04..984db01f5184 100644

io_uring/io_uring.c static void io_queue_iowq(struct io_kiocb *req) · line 559

Quoted 6 lines
io_uring/io_uring.c +2 -1
@@ -555,7 +555,8 @@ static void io_queue_iowq(struct io_kiocb *req)
  	 * procedure rather than attempt to run this request (or create a new
  	 * worker for it).
  	 */
-	if (WARN_ON_ONCE(!same_thread_group(tctx->task, current)))
+	if (WARN_ON_ONCE(!same_thread_group(tctx->task, current) &&
+			 !percpu_ref_is_dying(&req->ctx->refs)))

Should it be "||"? Otherwise I don't understand the purpose of it.

Quoted 13 lines
io_uring/io_uring.c +2 -1
  		atomic_or(IO_WQ_WORK_CANCEL, &req->work.flags);
 
  	trace_io_uring_queue_async_work(req, io_wq_is_hashed(&req->work));
@@ -1254,7 +1255,8 @@ static void io_req_normal_work_add(struct io_kiocb *req)
  		return;
  	}
 
-	if (likely(!task_work_add(tctx->task, &tctx->task_work, ctx->notify_method)))
+	if (!percpu_ref_is_dying(&ctx->refs) &&
+	    !task_work_add(tctx->task, &tctx->task_work, ctx->notify_method))
  		return;
 
  	io_fallback_tw(tctx, false);
Signature · 1 line
Pavel Begunkov
author maintainer
#5 lore raw

Re: [PATCH 3/5] io_uring: consider ring dead once the ref is marked dying

(getting back to this post the merge window)

Quoted 7 lines · Pavel Begunkov
Quoted 1 line · Jens Axboe

Don't gate this on the task exiting flag. It's generally not a good idea

Do you refer to tw add and the PF_EXITING logic inside? We can't gate it solely on dying refs as it's not sync'ed (and the patch doesn't). And task is dying is not same as ring is closed. E.g. a task can exit(2) but leave the ring intact to other tasks.

It's not gated solely on dying refs, it's an addition.

Quoted 9 lines
Quoted 9 lines

to gate it on the task PF_EXITING flag anyway. Once the ring is starting to go through ring teardown, the ref is marked as dying. Use that as our fallback/cancel mechanism.

---
  io_uring/io_uring.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
Quoted 13 lines

io_uring/io_uring.c static void io_queue_iowq(struct io_kiocb *req) · line 559

Quoted 11 lines
io_uring/io_uring.c +2 -1
index 2b9dae588f04..984db01f5184 100644
@@ -555,7 +555,8 @@ static void io_queue_iowq(struct io_kiocb *req)
       * procedure rather than attempt to run this request (or create a new
       * worker for it).
       */
-    if (WARN_ON_ONCE(!same_thread_group(tctx->task, current)))
+    if (WARN_ON_ONCE(!same_thread_group(tctx->task, current) &&
+             !percpu_ref_is_dying(&req->ctx->refs)))

Should it be "||"? Otherwise I don't understand the purpose of it.

Yep that's a braino, it should be:

if (!same_thread_group(req->task, current) ||

    io_ring_ref_is_dying(req->ctx))
	atomic_or(IO_WQ_WORK_CANCEL, &req->work.flags);
Signature · 1 line
Jens Axboe
maintainer
#6 lore raw

Re: [PATCH 3/5] io_uring: consider ring dead once the ref is marked dying

Quoted 4 lines · Jens Axboe

(getting back to this post the merge window)

Quoted 1 line · Pavel Begunkov

On 3/21/25 19:24, Jens Axboe wrote:

Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Don't gate this on the task exiting flag. It's generally not a good idea

Do you refer to tw add and the PF_EXITING logic inside? We can't gate it solely on dying refs as it's not sync'ed (and the patch doesn't). And task is dying is not same as ring is closed. E.g. a task can exit(2) but leave the ring intact to other tasks.

It's not gated solely on dying refs, it's an addition.

Right, which is why the commit message stating "don't gate on the task exiting flag" and leaving it gated on the task exiting flag is confusing.

Signature · 1 line
Pavel Begunkov
author maintainer
#7 lore raw

Re: [PATCH 3/5] io_uring: consider ring dead once the ref is marked dying

Quoted 12 lines · Pavel Begunkov
Quoted 11 lines · Jens Axboe

(getting back to this post the merge window)

Quoted 7 lines · Pavel Begunkov
Quoted 1 line · Jens Axboe

Don't gate this on the task exiting flag. It's generally not a good idea

Do you refer to tw add and the PF_EXITING logic inside? We can't gate it solely on dying refs as it's not sync'ed (and the patch doesn't). And task is dying is not same as ring is closed. E.g. a task can exit(2) but leave the ring intact to other tasks.

Quoted 5 lines
Quoted 1 line

It's not gated solely on dying refs, it's an addition.

Right, which is why the commit message stating "don't gate on the task exiting flag" and leaving it gated on the task exiting flag is confusing.

Ah yes, the commit message is garbage... I'll send out a new revision.

Signature · 1 line
Jens Axboe
author maintainer
#8 lore raw

PATCH v2 4/5 io_uring: wait for cancelations on final ring put

In reply to Jens Axboe

We still offload the cancelation to a workqueue, as not to introduce dependencies between the exiting task waiting on cleanup, and that task needing to run task_work to complete the process.

This means that once the final ring put is done, any request that was inflight and needed cancelation will be done as well. Notably requests that hold references to files - once the ring fd close is done, we will have dropped any of those references too.

---
 include/linux/io_uring_types.h |  2 ++
 io_uring/io_uring.c            | 17 +++++++++++++++++
 2 files changed, 19 insertions(+)
include/linux/io_uring_types.h +2 -0
index c17d2eedf478..79e223fd4733 100644
@@ -450,6 +450,8 @@ struct io_ring_ctx {
 	struct io_mapped_region		param_region;
 	/* just one zcrx per ring for now, will move to io_zcrx_ifq eventually */
 	struct io_mapped_region		zcrx_region;
+
+	struct completion		*exit_comp;
 };
 
 /*
io_uring/io_uring.c +17 -0
index 984db01f5184..d9b65a322ae1 100644
@@ -2894,6 +2894,7 @@ static __cold void io_ring_exit_work(struct work_struct *work)
 	struct io_ring_ctx *ctx = container_of(work, struct io_ring_ctx, exit_work);
 	unsigned long timeout = jiffies + HZ * 60 * 5;
 	unsigned long interval = HZ / 20;
+	struct completion *exit_comp;
 	struct io_tctx_exit exit;
 	struct io_tctx_node *node;
 	int ret;
@@ -2958,6 +2959,10 @@ static __cold void io_ring_exit_work(struct work_struct *work)
 
 	io_kworker_tw_end();
 
+	exit_comp = READ_ONCE(ctx->exit_comp);
+	if (exit_comp)
+		complete(exit_comp);
+
 	init_completion(&exit.completion);
 	init_task_work(&exit.task_work, io_tctx_exit_cb);
 	exit.ctx = ctx;
@@ -3020,9 +3025,21 @@ static __cold void io_ring_ctx_wait_and_kill(struct io_ring_ctx *ctx)
 static int io_uring_release(struct inode *inode, struct file *file)
 {
 	struct io_ring_ctx *ctx = file->private_data;
+	DECLARE_COMPLETION_ONSTACK(exit_comp);
 
 	file->private_data = NULL;
+	WRITE_ONCE(ctx->exit_comp, &exit_comp);
 	io_ring_ctx_wait_and_kill(ctx);
+
+	/*
+	 * Wait for cancel to run before exiting task
+	 */
+	do {
+		if (current->io_uring)
+			io_fallback_tw(current->io_uring, false);
+		cond_resched();
+	} while (wait_for_completion_interruptible(&exit_comp));
+
 	return 0;
 }
 
Signature · 1 line
2.49.0
author maintainer
#9 lore raw

PATCH v2 5/5 io_uring: switch away from percpu refcounts

In reply to Jens Axboe

For the common cases, the io_uring ref counts are all batched and hence need not be a percpu reference. This saves some memory on systems, but outside of that, it gets rid of needing a full RCU grace period on tearing down the reference. With io_uring now waiting on cancelations and IO during exit, this slows down the tear down a lot, up to 100x as slow.

---
 include/linux/io_uring_types.h |  2 +-
 io_uring/io_uring.c            | 47 ++++++++++++----------------------
 io_uring/io_uring.h            |  3 ++-
 io_uring/msg_ring.c            |  4 +--
 io_uring/refs.h                | 43 +++++++++++++++++++++++++++++++
 io_uring/register.c            |  2 +-
 io_uring/rw.c                  |  2 +-
 io_uring/sqpoll.c              |  2 +-
 io_uring/zcrx.c                |  4 +--
 9 files changed, 70 insertions(+), 39 deletions(-)
include/linux/io_uring_types.h +1 -1
index 79e223fd4733..8894b0639a3a 100644
@@ -256,7 +256,7 @@ struct io_ring_ctx {
 
 		struct task_struct	*submitter_task;
 		struct io_rings		*rings;
-		struct percpu_ref	refs;
+		atomic_long_t		refs;
 
 		clockid_t		clockid;
 		enum tk_offsets		clock_offset;
io_uring/io_uring.c +17 -30

158 lines not on this page · raw message

io_uring/io_uring.h +2 -1
index 87f883130286..67e5921771be 100644
@@ -13,6 +13,7 @@
 #include "slist.h"
 #include "filetable.h"
 #include "opdef.h"
+#include "refs.h"
 
 #ifndef CREATE_TRACE_POINTS
 #include <trace/events/io_uring.h>
@@ -143,7 +144,7 @@ static inline void io_lockdep_assert_cq_locked(struct io_ring_ctx *ctx)
 		 * Not from an SQE, as those cannot be submitted, but via
 		 * updating tagged resources.
 		 */
-		if (!percpu_ref_is_dying(&ctx->refs))
+		if (!io_ring_ref_is_dying(ctx))
 			lockdep_assert(current == ctx->submitter_task);
 	}
 #endif
io_uring/msg_ring.c +2 -2
index 0bbcbbcdebfd..30d4cabb66d6 100644
@@ -83,7 +83,7 @@ static void io_msg_tw_complete(struct io_kiocb *req, io_tw_token_t tw)
 	}
 	if (req)
 		kmem_cache_free(req_cachep, req);
-	percpu_ref_put(&ctx->refs);
+	io_ring_ref_put(ctx);
 }
 
 static int io_msg_remote_post(struct io_ring_ctx *ctx, struct io_kiocb *req,
@@ -95,7 +95,7 @@ static int io_msg_remote_post(struct io_ring_ctx *ctx, struct io_kiocb *req,
 	}
 	req->cqe.user_data = user_data;
 	io_req_set_res(req, res, cflags);
-	percpu_ref_get(&ctx->refs);
+	io_ring_ref_get(ctx);
 	req->ctx = ctx;
 	req->tctx = NULL;
 	req->io_task_work.func = io_msg_tw_complete;
io_uring/refs.h +43 -0

47 lines not on this page · raw message

io_uring/register.c +1 -1
index cc23a4c205cd..54fe94a0101b 100644
@@ -637,7 +637,7 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 	 * We don't quiesce the refs for register anymore and so it can't be
 	 * dying as we're holding a file ref here.
 	 */
-	if (WARN_ON_ONCE(percpu_ref_is_dying(&ctx->refs)))
+	if (WARN_ON_ONCE(io_ring_ref_is_dying(ctx)))
 		return -ENXIO;
 
 	if (ctx->submitter_task && ctx->submitter_task != current)
io_uring/rw.c +1 -1
index 039e063f7091..e010d548edea 100644
@@ -496,7 +496,7 @@ static bool io_rw_should_reissue(struct io_kiocb *req)
 	 * Don't attempt to reissue from that path, just let it fail with
 	 * -EAGAIN.
 	 */
-	if (percpu_ref_is_dying(&ctx->refs))
+	if (io_ring_ref_is_dying(ctx))
 		return false;
 
 	io_meta_restore(io, &rw->kiocb);
io_uring/sqpoll.c +1 -1
index d037cc68e9d3..b71f8d52386e 100644
@@ -184,7 +184,7 @@ static int __io_sq_thread(struct io_ring_ctx *ctx, bool cap_entries)
 		 * Don't submit if refs are dying, good for io_uring_register(),
 		 * but also it is relied upon by io_ring_exit_work()
 		 */
-		if (to_submit && likely(!percpu_ref_is_dying(&ctx->refs)) &&
+		if (to_submit && likely(!io_ring_ref_is_dying(ctx)) &&
 		    !(ctx->flags & IORING_SETUP_R_DISABLED))
 			ret = io_submit_sqes(ctx, to_submit);
 		mutex_unlock(&ctx->uring_lock);
io_uring/zcrx.c +2 -2
index 9c95b5b6ec4e..07719e3bf1b3 100644
@@ -629,7 +629,7 @@ static int io_pp_zc_init(struct page_pool *pp)
 	if (pp->p.dma_dir != DMA_FROM_DEVICE)
 		return -EOPNOTSUPP;
 
-	percpu_ref_get(&ifq->ctx->refs);
+	io_ring_ref_get(ifq->ctx);
 	return 0;
 }
 
@@ -640,7 +640,7 @@ static void io_pp_zc_destroy(struct page_pool *pp)
 
 	if (WARN_ON_ONCE(area->free_count != area->nia.num_niovs))
 		return;
-	percpu_ref_put(&ifq->ctx->refs);
+	io_ring_ref_put(ifq->ctx);
 }
 
 static int io_pp_nl_fill(void *mp_priv, struct sk_buff *rsp,
Signature · 1 line
2.49.0