Skip to contentSkip to conversation

io_uring: defer task work to when it is needed

PATCH Applied Dylan Yudaken 2 participants 9 messages Aug 30, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v4 cover io_uring: defer task work to when it is needed

We have seen workloads which suffer due to the way task work is currently scheduled. This scheduling can cause non-trivial tasks to run interrupting useful work on the workload. For example in network servers, a large async recv may run, calling memcpy on a large packet, interrupting a send. Which would add latency.

This series adds an option to defer async work until user space calls io_uring_enter with the GETEVENTS flag. This allows the workload to choose when to schedule async work and have finer control (at the expense of complexity of managing this) of scheduling.

Patches 1,2 are prep patches
Patch 3 changes io_uring_enter to not pre-run task work Patch 4/5/6 adds the new flag and functionality Patch 7 adds tracing for the local task work running

Changes since v3:
- Remove optimisation to save a single unlock. Can readd this later but it definitely made the code significantly harder to understand.
- Thread actual error code back through io_run* functions

Changes since v2:
- add a patch to trace local task work run
- return -EEXIST if calling from the wrong task
- properly handle shutting down due to an exec
- remove 'all' parameter from io_run_task_work_ctx

Changes since v1:
- Removed the first patch (using ctx variable) which was broken
- Require IORING_SETUP_SINGLE_ISSUER and make sure waiter task is the same as the submitter task
- Just don't run task work at the start of io_uring_enter (Pavel's suggestion)
- Remove io_move_task_work_from_local
- Fix locking bugs

Dylan Yudaken (7):
io_uring: remove unnecessary variable
io_uring: introduce io_has_work
io_uring: do not run task work at the start of io_uring_enter
io_uring: add IORING_SETUP_DEFER_TASKRUN
io_uring: move io_eventfd_put
io_uring: signal registered eventfd to process deferred task work
io_uring: trace local task work run

 include/linux/io_uring_types.h  |   3 +
 include/trace/events/io_uring.h |  29 ++++
 include/uapi/linux/io_uring.h   |   7 +
 io_uring/cancel.c               |   2 +-
 io_uring/io_uring.c             | 253 +++++++++++++++++++++++++-------
 io_uring/io_uring.h             |  29 +++-
 io_uring/rsrc.c                 |   2 +-
 7 files changed, 269 insertions(+), 56 deletions(-)

base-commit: b90cb1053190353cc30f0fef0ef1f378ccc063c5
prerequisite-patch-id: cb1d024945aa728d09a131156140a33d30bc268b

Signature · 1 line
2.30.2
#1 lore raw

PATCH v4 1/7 io_uring: remove unnecessary variable

'running' is set once and read once, so can easily just remove it

---
 io_uring/io_uring.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
io_uring/io_uring.c +1 -4
index 77616279000b..41eaf5ec70df 100644
@@ -1052,12 +1052,9 @@ void io_req_task_work_add(struct io_kiocb *req)
 	struct io_uring_task *tctx = req->task->io_uring;
 	struct io_ring_ctx *ctx = req->ctx;
 	struct llist_node *node;
-	bool running;
-
-	running = !llist_add(&req->io_task_work.node, &tctx->task_list);
 
 	/* task_work already pending, we're done */
-	if (running)
+	if (!llist_add(&req->io_task_work.node, &tctx->task_list))
 		return;
 
 	if (ctx->flags & IORING_SETUP_TASKRUN_FLAG)
Signature · 1 line
2.30.2
#2 lore raw

PATCH v4 2/7 io_uring: introduce io_has_work

In reply to Dylan Yudaken

This will be used later to know if the ring has outstanding work. Right now just if there is overflow CQEs to copy to the main CQE ring, but later will include deferred tasks

---
 io_uring/io_uring.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
io_uring/io_uring.c +9 -4
index 41eaf5ec70df..7998dc23360f 100644
@@ -2145,6 +2145,11 @@ struct io_wait_queue {
 	unsigned nr_timeouts;
 };
 
+static inline bool io_has_work(struct io_ring_ctx *ctx)
+{
+	return test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq);
+}
+
 static inline bool io_should_wake(struct io_wait_queue *iowq)
 {
 	struct io_ring_ctx *ctx = iowq->ctx;
@@ -2163,13 +2168,13 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode,
 {
 	struct io_wait_queue *iowq = container_of(curr, struct io_wait_queue,
 							wq);
+	struct io_ring_ctx *ctx = iowq->ctx;
 
 	/*
 	 * Cannot safely flush overflowed CQEs from here, ensure we wake up
 	 * the task, and the next invocation will do it.
 	 */
-	if (io_should_wake(iowq) ||
-	    test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &iowq->ctx->check_cq))
+	if (io_should_wake(iowq) || io_has_work(ctx))
 		return autoremove_wake_function(curr, mode, wake_flags, key);
 	return -1;
 }
@@ -2505,8 +2510,8 @@ static __poll_t io_uring_poll(struct file *file, poll_table *wait)
 	 * Users may get EPOLLIN meanwhile seeing nothing in cqring, this
 	 * pushs them to do the flush.
 	 */
-	if (io_cqring_events(ctx) ||
-	    test_bit(IO_CHECK_CQ_OVERFLOW_BIT, &ctx->check_cq))
+
+	if (io_cqring_events(ctx) || io_has_work(ctx))
 		mask |= EPOLLIN | EPOLLRDNORM;
 
 	return mask;
Signature · 1 line
2.30.2
#3 lore raw

PATCH v4 3/7 io_uring: do not run task work at the start of io_uring_enter

In reply to Dylan Yudaken

This is not needed, and it is normally better to wait for task work until after submissions. This will allow greater batching if either work arrives in the meanwhile, or if the submissions cause task work to be queued up.

For SQPOLL this also no longer runs task work, but this is handled inside the SQPOLL loop anyway.

For IOPOLL io_iopoll_check will run task work anyway

And otherwise io_cqring_wait will run task work

---
 io_uring/io_uring.c | 2 --
 1 file changed, 2 deletions(-)
io_uring/io_uring.c +0 -2
index 7998dc23360f..329d5b9d448e 100644
@@ -2991,8 +2991,6 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
 	struct fd f;
 	long ret;
 
-	io_run_task_work();
-
 	if (unlikely(flags & ~(IORING_ENTER_GETEVENTS | IORING_ENTER_SQ_WAKEUP |
 			       IORING_ENTER_SQ_WAIT | IORING_ENTER_EXT_ARG |
 			       IORING_ENTER_REGISTERED_RING)))
Signature · 1 line
2.30.2
#4 lore raw

PATCH v4 4/7 io_uring: add IORING_SETUP_DEFER_TASKRUN

In reply to Dylan Yudaken

Allow deferring async tasks until the user calls io_uring_enter(2) with the IORING_ENTER_GETEVENTS flag. Enable this mode with a flag at io_uring_setup time. This functionality requires that the later io_uring_enter will be called from the same submission task, and therefore restrict this flag to work only when IORING_SETUP_SINGLE_ISSUER is also set.

Being able to hand pick when tasks are run prevents the problem where there is current work to be done, however task work runs anyway.

For example, a common workload would obtain a batch of CQEs, and process each one. Interrupting this to additional taskwork would add latency but not gain anything. If instead task work is deferred to just before more CQEs are obtained then no additional latency is added.

The way this is implemented is by trying to keep task work local to a io_ring_ctx, rather than to the submission task. This is required, as the application will want to wake up only a single io_ring_ctx at a time to process work, and so the lists of work have to be kept separate.

This has some other benefits like not having to check the task continually in handle_tw_list (and potentially unlocking/locking those), and reducing locks in the submit & process completions path.

There are networking cases where using this option can reduce request latency by 50%. For example a contrived example using [1] where the client sends 2k data and receives the same data back while doing some system calls (to trigger task work) shows this reduction. The reason ends up being that if sending responses is delayed by processing task work, then the client side sits idle. Whereas reordering the sends first means that the client runs it's workload in parallel with the local task work.

[1]:
Using https://github.com/DylanZA/netbench/tree/defer_run
Client:
./netbench --client_only 1 --control_port 10000 --host <host> --tx "epoll --threads 16 --per_thread 1 --size 2048 --resp 2048 --workload 1000" Server:
./netbench --server_only 1 --control_port 10000 --rx "io_uring --defer_taskrun 0 --workload 100" --rx "io_uring --defer_taskrun 1 --workload 100"

---
 include/linux/io_uring_types.h |   2 +
 include/uapi/linux/io_uring.h  |   7 ++
 io_uring/cancel.c              |   2 +-
 io_uring/io_uring.c            | 147 +++++++++++++++++++++++++++++----
 io_uring/io_uring.h            |  29 ++++++-
 io_uring/rsrc.c                |   2 +-
 6 files changed, 168 insertions(+), 21 deletions(-)
include/linux/io_uring_types.h +2 -0
index 677a25d44d7f..d56ff2185168 100644
@@ -301,6 +301,8 @@ struct io_ring_ctx {
 		struct io_hash_table	cancel_table;
 		bool			poll_multi_queue;
 
+		struct llist_head	work_llist;
+
 		struct list_head	io_buffers_comp;
 	} ____cacheline_aligned_in_smp;
 
include/uapi/linux/io_uring.h +7 -0
index 9e0b5c8d92ce..48e5c70e0baf 100644
@@ -157,6 +157,13 @@ enum {
  */
 #define IORING_SETUP_SINGLE_ISSUER	(1U << 12)
 
+/*
+ * Defer running task work to get events.
+ * Rather than running bits of task work whenever the task transitions
+ * try to do it just before it is needed.
+ */
+#define IORING_SETUP_DEFER_TASKRUN	(1U << 13)
+
 enum io_uring_op {
 	IORING_OP_NOP,
 	IORING_OP_READV,
io_uring/cancel.c +1 -1
index 5fc5d3e80fcb..2291a53cdabd 100644
@@ -292,7 +292,7 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg)
 			break;
 
 		mutex_unlock(&ctx->uring_lock);
-		ret = io_run_task_work_sig();
+		ret = io_run_task_work_sig(ctx);
 		if (ret < 0) {
 			mutex_lock(&ctx->uring_lock);
 			break;
io_uring/io_uring.c +132 -15

264 lines not on this page · raw message

io_uring/io_uring.h +25 -4
index 2f73f83af960..f417d75d7bc1 100644
@@ -26,7 +26,8 @@ enum {
 
 struct io_uring_cqe *__io_get_cqe(struct io_ring_ctx *ctx);
 bool io_req_cqe_overflow(struct io_kiocb *req);
-int io_run_task_work_sig(void);
+int io_run_task_work_sig(struct io_ring_ctx *ctx);
+int io_run_local_work(struct io_ring_ctx *ctx);
 void io_req_complete_failed(struct io_kiocb *req, s32 res);
 void __io_req_complete(struct io_kiocb *req, unsigned issue_flags);
 void io_req_complete_post(struct io_kiocb *req);
@@ -221,17 +222,37 @@ static inline unsigned int io_sqring_entries(struct io_ring_ctx *ctx)
 	return smp_load_acquire(&rings->sq.tail) - ctx->cached_sq_head;
 }
 
-static inline bool io_run_task_work(void)
+static inline int io_run_task_work(void)
 {
 	if (test_thread_flag(TIF_NOTIFY_SIGNAL)) {
 		__set_current_state(TASK_RUNNING);
 		clear_notify_signal();
 		if (task_work_pending(current))
 			task_work_run();
-		return true;
+		return 1;
 	}
 
-	return false;
+	return 0;
+}
+
+static inline int io_run_task_work_ctx(struct io_ring_ctx *ctx)
+{
+	int ret = 0;
+	int ret2;
+
+	if (ctx->flags & IORING_SETUP_DEFER_TASKRUN)
+		ret = io_run_local_work(ctx);
+
+	/* want to run this after in case more is added */
+	ret2 = io_run_task_work();
+
+	/* Try propagate error in favour of if tasks were run,
+	 * but still make sure to run them if requested
+	 */
+	if (ret >= 0)
+		ret += ret2;
+
+	return ret;
 }
 
 static inline void io_tw_lock(struct io_ring_ctx *ctx, bool *locked)
io_uring/rsrc.c +1 -1
index 71359a4d0bd4..80cda6e2067f 100644
@@ -343,7 +343,7 @@ __cold static int io_rsrc_ref_quiesce(struct io_rsrc_data *data,
 		flush_delayed_work(&ctx->rsrc_put_work);
 		reinit_completion(&data->done);
 
-		ret = io_run_task_work_sig();
+		ret = io_run_task_work_sig(ctx);
 		mutex_lock(&ctx->uring_lock);
 	} while (ret >= 0);
 	data->quiesce = false;
Signature · 1 line
2.30.2
#5 lore raw

PATCH v4 5/7 io_uring: move io_eventfd_put

In reply to Dylan Yudaken

Non functional change: move this function above io_eventfd_signal so it can be used from there

---
 io_uring/io_uring.c | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)
io_uring/io_uring.c +8 -8
index 9f90b0633de7..c6e416be7ed3 100644
@@ -478,6 +478,14 @@ static __cold void io_queue_deferred(struct io_ring_ctx *ctx)
 	}
 }
 
+static void io_eventfd_put(struct rcu_head *rcu)
+{
+	struct io_ev_fd *ev_fd = container_of(rcu, struct io_ev_fd, rcu);
+
+	eventfd_ctx_put(ev_fd->cq_ev_fd);
+	kfree(ev_fd);
+}
+
 static void io_eventfd_signal(struct io_ring_ctx *ctx)
 {
 	struct io_ev_fd *ev_fd;
@@ -2468,14 +2476,6 @@ static int io_eventfd_register(struct io_ring_ctx *ctx, void __user *arg,
 	return 0;
 }
 
-static void io_eventfd_put(struct rcu_head *rcu)
-{
-	struct io_ev_fd *ev_fd = container_of(rcu, struct io_ev_fd, rcu);
-
-	eventfd_ctx_put(ev_fd->cq_ev_fd);
-	kfree(ev_fd);
-}
-
 static int io_eventfd_unregister(struct io_ring_ctx *ctx)
 {
 	struct io_ev_fd *ev_fd;
Signature · 1 line
2.30.2
#6 lore raw

PATCH v4 6/7 io_uring: signal registered eventfd to process deferred task work

In reply to Dylan Yudaken

Some workloads rely on a registered eventfd (via io_uring_register_eventfd(3)) in order to wake up and process the io_uring.

In the case of a ring setup with IORING_SETUP_DEFER_TASKRUN, that eventfd also needs to be signalled when there are tasks to run.

This changes an old behaviour which assumed 1 eventfd signal implied at least 1 CQE, however only when this new flag is set (and so old users will not notice). This should be expected with the IORING_SETUP_DEFER_TASKRUN flag as it is not guaranteed that every task will result in a CQE.

---
 include/linux/io_uring_types.h |  1 +
 io_uring/io_uring.c            | 75 ++++++++++++++++++++++++----------
 2 files changed, 55 insertions(+), 21 deletions(-)
include/linux/io_uring_types.h +1 -0
index d56ff2185168..42494176434a 100644
@@ -184,6 +184,7 @@ struct io_ev_fd {
 	struct eventfd_ctx	*cq_ev_fd;
 	unsigned int		eventfd_async: 1;
 	struct rcu_head		rcu;
+	atomic_t		refs;
 };
 
 struct io_alloc_cache {
io_uring/io_uring.c +54 -21

119 lines not on this page · raw message

Signature · 1 line
2.30.2
#7 lore raw

PATCH v4 7/7 io_uring: trace local task work run

In reply to Dylan Yudaken

Add tracing for io_run_local_task_work

---
 include/trace/events/io_uring.h | 29 +++++++++++++++++++++++++++++
 io_uring/io_uring.c             |  3 +++
 2 files changed, 32 insertions(+)
include/trace/events/io_uring.h +29 -0
index c5b21ff0ac85..936fd41bf147 100644
@@ -655,6 +655,35 @@ TRACE_EVENT(io_uring_short_write,
 			  __entry->wanted, __entry->got)
 );
 
+/*
+ * io_uring_local_work_run - ran ring local task work
+ *
+ * @tctx:		pointer to a io_uring_ctx
+ * @count:		how many functions it ran
+ * @loops:		how many loops it ran
+ *
+ */
+TRACE_EVENT(io_uring_local_work_run,
+
+	TP_PROTO(void *ctx, int count, unsigned int loops),
+
+	TP_ARGS(ctx, count, loops),
+
+	TP_STRUCT__entry (
+		__field(void *,		ctx	)
+		__field(int,		count	)
+		__field(unsigned int,	loops	)
+	),
+
+	TP_fast_assign(
+		__entry->ctx		= ctx;
+		__entry->count		= count;
+		__entry->loops		= loops;
+	),
+
+	TP_printk("ring %p, count %d, loops %u", __entry->ctx, __entry->count, __entry->loops)
+);
+
 #endif /* _TRACE_IO_URING_H */
 
 /* This part must be outside protection */
io_uring/io_uring.c +3 -0
index fffa81e498a4..cdd8d10e9638 100644
@@ -1165,6 +1165,7 @@ int io_run_local_work(struct io_ring_ctx *ctx)
 	struct llist_node fake;
 	struct llist_node *current_final = NULL;
 	int ret;
+	unsigned int loops = 1;
 
 	if (unlikely(ctx->submitter_task != current)) {
 		/* maybe this is before any submissions */
@@ -1194,6 +1195,7 @@ int io_run_local_work(struct io_ring_ctx *ctx)
 
 	node = io_llist_cmpxchg(&ctx->work_llist, &fake, NULL);
 	if (node != &fake) {
+		loops++;
 		current_final = &fake;
 		node = io_llist_xchg(&ctx->work_llist, &fake);
 		goto again;
@@ -1203,6 +1205,7 @@ int io_run_local_work(struct io_ring_ctx *ctx)
 		io_submit_flush_completions(ctx);
 		mutex_unlock(&ctx->uring_lock);
 	}
+	trace_io_uring_local_work_run(ctx, ret, loops);
 	return ret;
 }
 
Signature · 1 line
2.30.2
maintainer Applied
#8 lore raw
In reply to Dylan Yudaken
Quoted 6 lines · Dylan Yudaken

We have seen workloads which suffer due to the way task work is currently scheduled. This scheduling can cause non-trivial tasks to run interrupting useful work on the workload. For example in network servers, a large async recv may run, calling memcpy on a large packet, interrupting a send. Which would add latency.

Quoted 6 lines

This series adds an option to defer async work until user space calls io_uring_enter with the GETEVENTS flag. This allows the workload to choose when to schedule async work and have finer control (at the expense of complexity of managing this) of scheduling.

[...]

Applied, thanks!

[1/7] io_uring: remove unnecessary variable

      commit: e2f73811084cb6f4821cbe6e53bb0d9407af52f5
[2/7] io_uring: introduce io_has_work
      commit: cb0101159452eb63d0facf806e0a868963a11035
[3/7] io_uring: do not run task work at the start of io_uring_enter
      commit: 415f872f5f3aedb62bf857c6e315a88e26c08aaa
[4/7] io_uring: add IORING_SETUP_DEFER_TASKRUN
      commit: 2a23c7a52405c17183f36ef4aefb1fcf44425c2d
[5/7] io_uring: move io_eventfd_put
      commit: 488a41100f4b9d7e10f05edead4e4d6d1c0c85a9
[6/7] io_uring: signal registered eventfd to process deferred task work
      commit: 3403692af0c6d5ae826f536a2c08f4e7c0b8b163
[7/7] io_uring: trace local task work run
      commit: 73100da58e5b1f6d8ba85e134463fb2aaca59978

Best regards,

Signature · 1 line
Jens Axboe