Add ability to ignore inline completions
Jens Axboe
maintainercover Add ability to ignore inline completions
Hi,
For request/response type scenarios, it's not uncommon to have the send/write be of less interesting than new incoming recv/reads. Yet it's not quite possible to wait for only recv/reads when an application does:
io_uring_submit_and_wait(ring, nr_to_wait_for);
or any of the other variants of submit_and_wait() helpers. The application may attempt to count up the sends manually and include them in the 'nr_to_wait_for', however even that isn't infallible as that would also ignore failed/short sends/writes.
This adds support for ignoring inline completions. Note that while this doesn't catch all scenarios where a send/write can complete, it does catch the interesting ones - the ones that complete fully as part of normal submission. With that, it's possible to simply ignore the expected inline completions that naturally happen as part of a submit and wait scenario.
Patch 1 adds the general simple infrastructure for this, and patch 2+3 adds support for send/sendmsg/sendzc. For those, they have to set a send specific flag, IORING_SEND_IGNORE_INLINE, to have it be enabled for those requests.
Comments welcome!
include/linux/io_uring_types.h | 4 ++++ include/uapi/linux/io_uring.h | 8 ++++++++ io_uring/io_uring.c | 12 +++++++++--- io_uring/io_uring.h | 2 ++ io_uring/net.c | 16 ++++++++++++++-- 5 files changed, 37 insertions(+), 5 deletions(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/3 io_uring: add support for ignoring inline completions for waits
io_uring treats all completions the same - they post a completion event, or more, and anyone waiting on event completions will see each event as it gets posted.
However, some events may be more interesting that others. For a request and response type model, it's not uncommon to have send/write events that are submitted with a recv/read type of request. While the app does want to see a successful send/write completion eventually, it need not handle it upfront as it would want to do with a recv/read, as it isn't time sensitive. Generally, a send/write completion will just mean that a buffer can get recycled/reused, whereas a recv/read completion needs acting upon (and a response sent).
This can be somewhat tricky to handle if many requests and responses are being handled, and the app generally needs to track the number of pending sends/writes to be able to sanely wait on just new incoming recv/read requests. And even with that, an application would still like to see a completion for a short/failed send/write immediately.
Add infrastructure to account inline completions, such that they can be deducted from the 'wait_nr' being passed in via a submit_and_wait() type of situation. Inline completions are ones that complete directly inline from submission, such as a send to a socket where there's enough space to accomodate the data being sent.
No functional changes in this patch, as no opcode supports setting REQ_F_IGNORE_INLINE just yet.
--- include/linux/io_uring_types.h | 4 ++++ include/uapi/linux/io_uring.h | 1 + io_uring/io_uring.c | 12 +++++++++--- io_uring/io_uring.h | 2 ++ 4 files changed, 16 insertions(+), 3 deletions(-)
include/linux/io_uring_types.h +4 -0
@@ -206,6 +206,7 @@ struct io_submit_state {bool need_plug;bool cq_flush;unsigned short submit_nr; +unsigned short inline_completions;struct blk_plug plug; };
@@ -465,6 +466,7 @@ enum {REQ_F_BL_EMPTY_BIT ,REQ_F_BL_NO_RECYCLE_BIT ,REQ_F_BUFFERS_COMMIT_BIT , +REQ_F_IGNORE_INLINE_BIT ,/* not a real bit, just to check we're not overflowing the space */ __REQ_F_LAST_BIT,
@@ -541,6 +543,8 @@ enum {REQ_F_BL_NO_RECYCLE =IO_REQ_FLAG (REQ_F_BL_NO_RECYCLE_BIT ),/* buffer ring head needs incrementing on put */ REQ_F_BUFFERS_COMMIT =IO_REQ_FLAG (REQ_F_BUFFERS_COMMIT_BIT ), +/* if set, ignore these completions for when waiting on events */ + REQ_F_IGNORE_INLINE = IO_REQ_FLAG(REQ_F_IGNORE_INLINE_BIT), };typedef void (*io_req_tw_func_t )(struct io_kiocb *req,struct io_tw_state *ts);
include/uapi/linux/io_uring.h +1 -0
@@ -554,6 +554,7 @@ struct io_uring_params {#define IORING_FEAT_REG_REG_RING (1U << 13)#define IORING_FEAT_RECVSEND_BUNDLE (1U << 14)#define IORING_FEAT_MIN_TIMEOUT (1U << 15) +#define IORING_FEAT_IGNORE_INLINE (1U << 16)/* * io_uring_register(2) opcodes and arguments
io_uring/io_uring.c +9 -3
@@ -2239,6 +2239,7 @@ static void io_submit_state_start(struct io_submit_state *state, state->plug_started = false; state->need_plug = max_ios >2 ; state->submit_nr = max_ios; + state->inline_completions =0 ;/* set only head, no need to init link_last in advance */ state->link.head =NULL ; }
@@ -3285,6 +3286,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, size_t, argsz) {struct io_ring_ctx *ctx; +int inline_complete =0 ;struct file *file;long ret;
@@ -3349,6 +3351,7 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,mutex_unlock (&ctx->uring_lock); goto out; } + inline_complete = ctx->submit_state.inline_completions;if (flags &IORING_ENTER_GETEVENTS ) {if (ctx->syscall_iopoll) goto iopoll_locked;
@@ -3386,8 +3389,10 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit, ret2 =io_get_ext_arg (flags, argp, &ext_arg);if (likely (!ret2)) { - min_complete =min (min_complete, - ctx->cq_entries); +if (min_complete > ctx->cq_entries) + min_complete = ctx->cq_entries; +else + min_complete += inline_complete; ret2 =io_cqring_wait (ctx, min_complete, flags, &ext_arg); }
@@ -3674,7 +3679,8 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p,IORING_FEAT_EXT_ARG |IORING_FEAT_NATIVE_WORKERS |IORING_FEAT_RSRC_TAGS |IORING_FEAT_CQE_SKIP |IORING_FEAT_LINKED_FILE |IORING_FEAT_REG_REG_RING | -IORING_FEAT_RECVSEND_BUNDLE |IORING_FEAT_MIN_TIMEOUT ; +IORING_FEAT_RECVSEND_BUNDLE |IORING_FEAT_MIN_TIMEOUT | +IORING_FEAT_IGNORE_INLINE ;if (copy_to_user (params, p,sizeof (*p))) { ret = -EFAULT ;
io_uring/io_uring.h +2 -0
@@ -357,6 +357,8 @@ static inline void io_req_complete_defer(struct io_kiocb *req)lockdep_assert_held (&req->ctx->uring_lock);wq_list_add_tail (&req->comp_list, &state->compl_reqs); +if (req->flags &REQ_F_IGNORE_INLINE ) + state->inline_completions++; }static inline void io_commit_cqring_flush (struct io_ring_ctx *ctx)
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 2/3 io_uring/net: add IORING_SEND_IGNORE_INLINE support to send/sendmsg
In reply to Jens AxboeIf IORING_SEND_IGNORE_INLINE is set for a send/sendmsg request, then a successful inline completion of such a request will be ignored for a submit_and_wait() type of submissions. In other words, if an application submits a send for socketA with a recv for socketB, it can now do:
io_uring_submit_and_wait(ring, 1);
and have the inline send completion be ignored from the number of items to wait for.
--- include/uapi/linux/io_uring.h | 7 +++++++ io_uring/net.c | 9 ++++++++- 2 files changed, 15 insertions(+), 1 deletion(-)
include/uapi/linux/io_uring.h +7 -0
@@ -362,12 +362,19 @@ enum io_uring_op {* the starting buffer ID in cqe->flags as per * usual for provided buffer usage. The buffers * will be contigious from the starting buffer ID. +* +* IORING_SEND_IGNORE_INLINE If set for a send[msg] request, then the +* completion will NOT be included in the count +* waited for by an application, if completed +* inline as part of submission. It will still +* generate a CQE. */ #define IORING_RECVSEND_POLL_FIRST (1U << 0)#define IORING_RECV_MULTISHOT (1U << 1)#define IORING_RECVSEND_FIXED_BUF (1U << 2)#define IORING_SEND_ZC_REPORT_USAGE (1U << 3)#define IORING_RECVSEND_BUNDLE (1U << 4) +#define IORING_SEND_IGNORE_INLINE (1U << 5)/* * cqe.res for IORING_CQE_F_NOTIF if
io_uring/net.c +8 -1
@@ -400,7 +400,8 @@ static int io_sendmsg_prep_setup(struct io_kiocb *req, int is_msg)return ret; } -#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE) +#define SENDMSG_FLAGS (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_BUNDLE | \ + IORING_SEND_IGNORE_INLINE)int io_sendmsg_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -434,6 +435,8 @@ int io_sendmsg_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) sr->buf_group = req->buf_index; req->buf_list =NULL ; } +if (sr->flags &IORING_SEND_IGNORE_INLINE ) + req->flags |=REQ_F_IGNORE_INLINE ;#ifdef CONFIG_COMPAT if (req->ctx->compat)
@@ -550,6 +553,8 @@ int io_sendmsg(struct io_kiocb *req, unsigned int issue_flags) ret =__sys_sendmsg_sock (sock, &kmsg->msg, flags);if (ret < min_ret) { + req->flags &= ~REQ_F_IGNORE_INLINE ; +if (ret == -EAGAIN && (issue_flags &IO_URING_F_NONBLOCK ))return -EAGAIN ;if (ret >0 &&io_net_retry (sock, flags)) {
@@ -647,6 +652,8 @@ int io_send(struct io_kiocb *req, unsigned int issue_flags) kmsg->msg.msg_flags = flags; ret =sock_sendmsg (sock, &kmsg->msg);if (ret < min_ret) { + req->flags &= ~REQ_F_IGNORE_INLINE ; +if (ret == -EAGAIN && (issue_flags &IO_URING_F_NONBLOCK ))return -EAGAIN ;
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 3/3 io_uring/net: add IORING_SEND_IGNORE_INLINE support to zerocopy send
In reply to Jens AxboeIf IORING_SEND_IGNORE_INLINE is set for a send zerocopy request, then a successful inline completion of such a request will be ignored for a submit_and_wait() type of submissions. In other words, if an application submits a send for socketA with a recv for socketB, it can now do:
io_uring_submit_and_wait(ring, 1);
and have the inline send completion be ignored from the number of items to wait for. Note that this only applies to the direct zerocopy send completion, it does not include the notification when it's safe to reuse the buffer. Those happen out-of-line anyway.
--- io_uring/net.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
io_uring/net.c +6 -1
@@ -1223,7 +1223,8 @@ void io_send_zc_cleanup(struct io_kiocb *req) }#define IO_ZC_FLAGS_COMMON (IORING_RECVSEND_POLL_FIRST | IORING_RECVSEND_FIXED_BUF) -#define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE) +#define IO_ZC_FLAGS_VALID (IO_ZC_FLAGS_COMMON | IORING_SEND_ZC_REPORT_USAGE | \ + IORING_SEND_IGNORE_INLINE)int io_send_zc_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -1259,6 +1260,8 @@ int io_send_zc_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) nd->zc_used = false; nd->zc_copied = false; } +if (zc->flags &IORING_SEND_IGNORE_INLINE ) + req->flags |=REQ_F_IGNORE_INLINE ; }if (zc->flags &IORING_RECVSEND_FIXED_BUF ) {
@@ -1406,6 +1409,8 @@ int io_send_zc(struct io_kiocb *req, unsigned int issue_flags) ret =sock_sendmsg (sock, &kmsg->msg);if (unlikely (ret < min_ret)) { + req->flags &= ~REQ_F_IGNORE_INLINE ; +if (ret == -EAGAIN && (issue_flags &IO_URING_F_NONBLOCK ))return -EAGAIN ;
Signature · 1 line
2.45.2