Skip to contentSkip to conversation

Add ability to ignore inline completions

RFC PATCH New Jens Axboe 1 participant 4 messages Oct 14, 2024 io_uring

lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

cover Add ability to ignore inline completions

Hi,

For request/response type scenarios, it's not uncommon to have the send/write be of less interesting than new incoming recv/reads. Yet it's not quite possible to wait for only recv/reads when an application does:

io_uring_submit_and_wait(ring, nr_to_wait_for);

or any of the other variants of submit_and_wait() helpers. The application may attempt to count up the sends manually and include them in the 'nr_to_wait_for', however even that isn't infallible as that would also ignore failed/short sends/writes.

This adds support for ignoring inline completions. Note that while this doesn't catch all scenarios where a send/write can complete, it does catch the interesting ones - the ones that complete fully as part of normal submission. With that, it's possible to simply ignore the expected inline completions that naturally happen as part of a submit and wait scenario.

Patch 1 adds the general simple infrastructure for this, and patch 2+3 adds support for send/sendmsg/sendzc. For those, they have to set a send specific flag, IORING_SEND_IGNORE_INLINE, to have it be enabled for those requests.

Comments welcome!

 include/linux/io_uring_types.h |  4 ++++
 include/uapi/linux/io_uring.h  |  8 ++++++++
 io_uring/io_uring.c            | 12 +++++++++---
 io_uring/io_uring.h            |  2 ++
 io_uring/net.c                 | 16 ++++++++++++++--
 5 files changed, 37 insertions(+), 5 deletions(-)
Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH 1/3 io_uring: add support for ignoring inline completions for waits

io_uring treats all completions the same - they post a completion event, or more, and anyone waiting on event completions will see each event as it gets posted.

However, some events may be more interesting that others. For a request and response type model, it's not uncommon to have send/write events that are submitted with a recv/read type of request. While the app does want to see a successful send/write completion eventually, it need not handle it upfront as it would want to do with a recv/read, as it isn't time sensitive. Generally, a send/write completion will just mean that a buffer can get recycled/reused, whereas a recv/read completion needs acting upon (and a response sent).

This can be somewhat tricky to handle if many requests and responses are being handled, and the app generally needs to track the number of pending sends/writes to be able to sanely wait on just new incoming recv/read requests. And even with that, an application would still like to see a completion for a short/failed send/write immediately.

Add infrastructure to account inline completions, such that they can be deducted from the 'wait_nr' being passed in via a submit_and_wait() type of situation. Inline completions are ones that complete directly inline from submission, such as a send to a socket where there's enough space to accomodate the data being sent.

No functional changes in this patch, as no opcode supports setting REQ_F_IGNORE_INLINE just yet.

---
 include/linux/io_uring_types.h |  4 ++++
 include/uapi/linux/io_uring.h  |  1 +
 io_uring/io_uring.c            | 12 +++++++++---
 io_uring/io_uring.h            |  2 ++
 4 files changed, 16 insertions(+), 3 deletions(-)
include/linux/io_uring_types.h +4 -0
index 9c7e1d3f06e5..6eb8b739ea0d 100644
@@ -206,6 +206,7 @@ struct io_submit_state {
 	bool			need_plug;
 	bool			cq_flush;
 	unsigned short		submit_nr;
+	unsigned short		inline_completions;
 	struct blk_plug		plug;
 };
 
@@ -465,6 +466,7 @@ enum {
 	REQ_F_BL_EMPTY_BIT,
 	REQ_F_BL_NO_RECYCLE_BIT,
 	REQ_F_BUFFERS_COMMIT_BIT,
+	REQ_F_IGNORE_INLINE_BIT,
 
 	/* not a real bit, just to check we're not overflowing the space */
 	__REQ_F_LAST_BIT,
@@ -541,6 +543,8 @@ enum {
 	REQ_F_BL_NO_RECYCLE	= IO_REQ_FLAG(REQ_F_BL_NO_RECYCLE_BIT),
 	/* buffer ring head needs incrementing on put */
 	REQ_F_BUFFERS_COMMIT	= IO_REQ_FLAG(REQ_F_BUFFERS_COMMIT_BIT),
+	/* if set, ignore these completions for when waiting on events */
+	REQ_F_IGNORE_INLINE	= IO_REQ_FLAG(REQ_F_IGNORE_INLINE_BIT),
 };
 
 typedef void (*io_req_tw_func_t)(struct io_kiocb *req, struct io_tw_state *ts);
include/uapi/linux/io_uring.h +1 -0
index 86cb385fe0b5..1967f5ab2317 100644
@@ -554,6 +554,7 @@ struct io_uring_params {
 #define IORING_FEAT_REG_REG_RING	(1U << 13)
 #define IORING_FEAT_RECVSEND_BUNDLE	(1U << 14)
 #define IORING_FEAT_MIN_TIMEOUT		(1U << 15)
+#define IORING_FEAT_IGNORE_INLINE	(1U << 16)
 
 /*
  * io_uring_register(2) opcodes and arguments
io_uring/io_uring.c +9 -3
index d7ad4ea5f40b..706822db7447 100644
@@ -2239,6 +2239,7 @@ static void io_submit_state_start(struct io_submit_state *state,
 	state->plug_started = false;
 	state->need_plug = max_ios > 2;
 	state->submit_nr = max_ios;
+	state->inline_completions = 0;
 	/* set only head, no need to init link_last in advance */
 	state->link.head = NULL;
 }
@@ -3285,6 +3286,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
 		size_t, argsz)
 {
 	struct io_ring_ctx *ctx;
+	int inline_complete = 0;
 	struct file *file;
 	long ret;
 
@@ -3349,6 +3351,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
 			mutex_unlock(&ctx->uring_lock);
 			goto out;
 		}
+		inline_complete = ctx->submit_state.inline_completions;
 		if (flags & IORING_ENTER_GETEVENTS) {
 			if (ctx->syscall_iopoll)
 				goto iopoll_locked;
@@ -3386,8 +3389,10 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
 
 			ret2 = io_get_ext_arg(flags, argp, &ext_arg);
 			if (likely(!ret2)) {
-				min_complete = min(min_complete,
-						   ctx->cq_entries);
+				if (min_complete > ctx->cq_entries)
+					min_complete = ctx->cq_entries;
+				else
+					min_complete += inline_complete;
 				ret2 = io_cqring_wait(ctx, min_complete, flags,
 						      &ext_arg);
 			}
@@ -3674,7 +3679,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,
 			IORING_FEAT_EXT_ARG | IORING_FEAT_NATIVE_WORKERS |
 			IORING_FEAT_RSRC_TAGS | IORING_FEAT_CQE_SKIP |
 			IORING_FEAT_LINKED_FILE | IORING_FEAT_REG_REG_RING |
-			IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT;
+			IORING_FEAT_RECVSEND_BUNDLE | IORING_FEAT_MIN_TIMEOUT |
+			IORING_FEAT_IGNORE_INLINE;
 
 	if (copy_to_user(params, p, sizeof(*p))) {
 		ret = -EFAULT;
io_uring/io_uring.h +2 -0
index 9d70b2cf7b1e..bd1d4b6e46f0 100644
@@ -357,6 +357,8 @@ static inline void io_req_complete_defer(struct io_kiocb *req)
 	lockdep_assert_held(&req->ctx->uring_lock);
 
 	wq_list_add_tail(&req->comp_list, &state->compl_reqs);
+	if (req->flags & REQ_F_IGNORE_INLINE)
+		state->inline_completions++;
 }
 
 static inline void io_commit_cqring_flush(struct io_ring_ctx *ctx)
Signature · 1 line
2.45.2
author maintainer
#2 lore raw

PATCH 2/3 io_uring/net: add IORING_SEND_IGNORE_INLINE support to send/sendmsg

In reply to Jens Axboe

If IORING_SEND_IGNORE_INLINE is set for a send/sendmsg request, then a successful inline completion of such a request will be ignored for a submit_and_wait() type of submissions. In other words, if an application submits a send for socketA with a recv for socketB, it can now do:

io_uring_submit_and_wait(ring, 1);

and have the inline send completion be ignored from the number of items to wait for.

---
 include/uapi/linux/io_uring.h | 7 +++++++
 io_uring/net.c                | 9 ++++++++-
 2 files changed, 15 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +7 -0
index 1967f5ab2317..e026ade027c1 100644
@@ -362,12 +362,19 @@ enum io_uring_op {
  *				the starting buffer ID in cqe->flags as per
  *				usual for provided buffer usage. The buffers
  *				will be	contigious from the starting buffer ID.
+ *
+ * IORING_SEND_IGNORE_INLINE	If set for a send[msg] request, then the
+ *				completion will NOT be included in the count
+ *				waited for by an application, if completed
+ *				inline as part of submission. It will still
+ *				generate a CQE.
  */
 #define IORING_RECVSEND_POLL_FIRST	(1U << 0)
 #define IORING_RECV_MULTISHOT		(1U << 1)
 #define IORING_RECVSEND_FIXED_BUF	(1U << 2)
 #define IORING_SEND_ZC_REPORT_USAGE	(1U << 3)
 #define IORING_RECVSEND_BUNDLE		(1U << 4)
+#define IORING_SEND_IGNORE_INLINE	(1U << 5)
 
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
io_uring/net.c +8 -1
index 18507658a921..11ff58a5c145 100644
@@ -400,7 +400,8 @@ static int io_sendmsg_prep_setup(struct io_kiocb *req, int is_msg)
 	return ret;
 }
 
-#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE)
+#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | \
+			IORING_SEND_IGNORE_INLINE)
 
 int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
@@ -434,6 +435,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		sr->buf_group = req->buf_index;
 		req->buf_list = NULL;
 	}
+	if (sr->flags & IORING_SEND_IGNORE_INLINE)
+		req->flags |= REQ_F_IGNORE_INLINE;
 
 #ifdef CONFIG_COMPAT
 	if (req->ctx->compat)
@@ -550,6 +553,8 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags)
 	ret = __sys_sendmsg_sock(sock, &kmsg->msg, flags);
 
 	if (ret < min_ret) {
+		req->flags &= ~REQ_F_IGNORE_INLINE;
+
 		if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK))
 			return -EAGAIN;
 		if (ret > 0 && io_net_retry(sock, flags)) {
@@ -647,6 +652,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags)
 	kmsg->msg.msg_flags = flags;
 	ret = sock_sendmsg(sock, &kmsg->msg);
 	if (ret < min_ret) {
+		req->flags &= ~REQ_F_IGNORE_INLINE;
+
 		if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK))
 			return -EAGAIN;
 
Signature · 1 line
2.45.2
author maintainer
#3 lore raw

PATCH 3/3 io_uring/net: add IORING_SEND_IGNORE_INLINE support to zerocopy send

In reply to Jens Axboe

If IORING_SEND_IGNORE_INLINE is set for a send zerocopy request, then a successful inline completion of such a request will be ignored for a submit_and_wait() type of submissions. In other words, if an application submits a send for socketA with a recv for socketB, it can now do:

io_uring_submit_and_wait(ring, 1);

and have the inline send completion be ignored from the number of items to wait for. Note that this only applies to the direct zerocopy send completion, it does not include the notification when it's safe to reuse the buffer. Those happen out-of-line anyway.

---
 io_uring/net.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
io_uring/net.c +6 -1
index 11ff58a5c145..79f980182a10 100644
@@ -1223,7 +1223,8 @@ void io_send_zc_cleanup(struct io_kiocb *req)
 }
 
 #define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF)
-#define IO_ZC_FLAGS_VALID  (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE)
+#define IO_ZC_FLAGS_VALID  (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE | \
+				IORING_SEND_IGNORE_INLINE)
 
 int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
@@ -1259,6 +1260,8 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 			nd->zc_used = false;
 			nd->zc_copied = false;
 		}
+		if (zc->flags & IORING_SEND_IGNORE_INLINE)
+			req->flags |= REQ_F_IGNORE_INLINE;
 	}
 
 	if (zc->flags & IORING_RECVSEND_FIXED_BUF) {
@@ -1406,6 +1409,8 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags)
 	ret = sock_sendmsg(sock, &kmsg->msg);
 
 	if (unlikely(ret < min_ret)) {
+		req->flags &= ~REQ_F_IGNORE_INLINE;
+
 		if (ret == -EAGAIN && (issue_flags & IO_URING_F_NONBLOCK))
 			return -EAGAIN;
 
Signature · 1 line
2.45.2