Add support for mixed sized CQEs
Subsystems (5)
Jens Axboe
maintainercover Add support for mixed sized CQEs
Hi,
Currently io_uring supports two modes for CQEs:
1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b
Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.
This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.
This is mostly trivial to support, with the corner case being attempting to post a 32b CQE when the ring is a single 16b CQE away from wrapping. As CQEs must be contigious in memory, that's simply not possible. The solution taken by this patchset is to add a special CQE type, which has IORING_CQE_F_SKIP set. This is a pad/nop CQE, which should simply be ignored, as it carries no information and serves no other purpose than to re-align the posted CQEs for ring wrap.
If used with liburing, then both the 32b vs 16b postings and the skip are transparent.
liburing support and a few basic test cases can be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/liburing.git/log/?h=cqe-mixed
and these patches can also be found here:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=io_uring-cqe-mix
Patch 1 is just a prep patch, and patch 2 adds the cqe flags so that the core can be adapted before support is actually there. Patches 3 and 4 are exactly that, and patch 5 finally adds support for the mixed mode. Patch 6 adds support for NOP testing of this, and patches 7/8 allow IORING_SETUP_CQE_MIXED for uring_cmd/zcrx which previously required IORING_SETUP_CQE32 to work.
Documentation/networking/iou-zcrx.rst | 2 +- include/linux/io_uring_types.h | 6 --- include/trace/events/io_uring.h | 4 +- include/uapi/linux/io_uring.h | 17 +++++++ io_uring/cmd_net.c | 3 +- io_uring/fdinfo.c | 22 +++++---- io_uring/io_uring.c | 71 +++++++++++++++++++++------ io_uring/io_uring.h | 49 ++++++++++++------ io_uring/nop.c | 17 ++++++- io_uring/register.c | 3 +- io_uring/uring_cmd.c | 2 +- io_uring/zcrx.c | 5 +- 12 files changed, 146 insertions(+), 55 deletions(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/8 io_uring: remove io_ctx_cqe32() helper
It's pretty pointless and only used for the tracing helper, get rid of it.
--- include/linux/io_uring_types.h | 6 ------ include/trace/events/io_uring.h | 4 ++-- 2 files changed, 2 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +0 -6
@@ -727,10 +727,4 @@ struct io_overflow_cqe {struct list_head list; struct io_uring_cqe cqe; }; - -static inline bool io_ctx_cqe32 (struct io_ring_ctx *ctx) -{ -return ctx->flags &IORING_SETUP_CQE32 ; -} -#endif
include/trace/events/io_uring.h +2 -2
@@ -340,8 +340,8 @@ TP_PROTO(struct io_ring_ctx *ctx, void *req, struct io_uring_cqe *cqe), __entry->user_data = cqe->user_data; __entry->res = cqe->res; __entry->cflags = cqe->flags; - __entry->extra1 =io_ctx_cqe32 (ctx) ? cqe->big_cqe[0 ] :0 ; - __entry->extra2 =io_ctx_cqe32 (ctx) ? cqe->big_cqe[1 ] :0 ; + __entry->extra1 = ctx->flags &IORING_SETUP_CQE32 ? cqe->big_cqe[0 ] :0 ; + __entry->extra2 = ctx->flags &IORING_SETUP_CQE32 ? cqe->big_cqe[1 ] :0 ; ), TP_printk("ring %p, req %p, user_data 0x%llx, result %d, cflags 0x%x "
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 2/8 io_uring: add UAPI definitions for mixed CQE postings
In reply to Jens AxboeThis adds the CQE flags related to supporting a mixed CQ ring mode, where both normal (16b) and big (32b) CQEs may be posted.
No functional changes in this patch.
--- include/uapi/linux/io_uring.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
include/uapi/linux/io_uring.h +10 -0
@@ -487,12 +487,22 @@ struct io_uring_cqe {* other provided buffer type, all completions with a * buffer passed back is automatically returned to the * application. +* IORING_CQE_F_SKIP If set, then the application/liburing must ignore this +* CQE. It's only purpose is to fill a gap in the ring, +* if a large CQE is attempted posted when the ring has +* just a single small CQE worth of space left before +* wrapping. +* IORING_CQE_F_32 If set, this is a 32b/big-cqe posting. Use with rings +* setup in a mixed CQE mode, where both 16b and 32b +* CQEs may be posted to the CQ ring. */ #define IORING_CQE_F_BUFFER (1U << 0)#define IORING_CQE_F_MORE (1U << 1)#define IORING_CQE_F_SOCK_NONEMPTY (1U << 2)#define IORING_CQE_F_NOTIF (1U << 3)#define IORING_CQE_F_BUF_MORE (1U << 4) +#define IORING_CQE_F_SKIP (1U << 5) +#define IORING_CQE_F_32 (1U << 15)#define IORING_CQE_BUFFER_SHIFT 16
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 3/8 io_uring/fdinfo: handle mixed sized CQEs
In reply to Jens AxboeEnsure that the CQ ring iteration handles differently sized CQEs, not just a fixed 16b or 32b size per ring. These CQEs aren't possible just yet, but prepare the fdinfo CQ ring dumping for handling them.
--- io_uring/fdinfo.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-)
io_uring/fdinfo.c +12 -10
@@ -65,15 +65,12 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)unsigned int sq_tail =READ_ONCE (r->sq.tail);unsigned int cq_head =READ_ONCE (r->cq.head);unsigned int cq_tail =READ_ONCE (r->cq.tail); -unsigned int cq_shift =0 ;unsigned int sq_shift =0 ; -unsigned int sq_entries, cq_entries; +unsigned int sq_entries;int sq_pid =-1 , sq_cpu =-1 ;u64 sq_total_time =0 , sq_work_time =0 ;unsigned int i; -if (ctx->flags &IORING_SETUP_CQE32 ) - cq_shift =1 ;if (ctx->flags &IORING_SETUP_SQE128 ) sq_shift =1 ;
@@ -125,18 +122,23 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)seq_printf (m,"\n" ); }seq_printf (m,"CQEs:\t%u\n" , cq_tail - cq_head); - cq_entries =min (cq_tail - cq_head, ctx->cq_entries); -for (i =0 ; i < cq_entries; i++) { -unsigned int entry = i + cq_head; -struct io_uring_cqe *cqe = &r->cqes[(entry & cq_mask) << cq_shift]; +while (cq_head < cq_tail) { +struct io_uring_cqe *cqe; +bool cqe32 = false; + cqe = &r->cqes[(cq_head & cq_mask)]; +if (cqe->flags &IORING_CQE_F_32 || ctx->flags &IORING_SETUP_CQE32 ) + cqe32 = true;seq_printf (m,"%5u: user_data:%llu, res:%d, flag:%x" , - entry & cq_mask, cqe->user_data, cqe->res, + cq_head & cq_mask, cqe->user_data, cqe->res, cqe->flags); -if (cq_shift) +if (cqe32)seq_printf (m,", extra1:%llu, extra2:%llu\n" , cqe->big_cqe[0 ], cqe->big_cqe[1 ]);seq_printf (m,"\n" ); + cq_head++; +if (cqe32) + cq_head++; }if (ctx->flags &IORING_SETUP_SQPOLL ) {
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 4/8 io_uring/trace: support completion tracing of mixed 32b CQEs
In reply to Jens AxboeCheck for IORING_CQE_F_32 as well, not just if the ring was setup with IORING_SETUP_CQE32 to only support big CQEs.
--- include/trace/events/io_uring.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
include/trace/events/io_uring.h +2 -2
@@ -340,8 +340,8 @@ TP_PROTO(struct io_ring_ctx *ctx, void *req, struct io_uring_cqe *cqe), __entry->user_data = cqe->user_data; __entry->res = cqe->res; __entry->cflags = cqe->flags; - __entry->extra1 = ctx->flags &IORING_SETUP_CQE32 ? cqe->big_cqe[0 ] :0 ; - __entry->extra2 = ctx->flags &IORING_SETUP_CQE32 ? cqe->big_cqe[1 ] :0 ; + __entry->extra1 = ctx->flags &IORING_SETUP_CQE32 || cqe->flags &IORING_CQE_F_32 ? cqe->big_cqe[0 ] :0 ; + __entry->extra2 = ctx->flags &IORING_SETUP_CQE32 || cqe->flags &IORING_CQE_F_32 ? cqe->big_cqe[1 ] :0 ; ), TP_printk("ring %p, req %p, user_data 0x%llx, result %d, cflags 0x%x "
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 5/8 io_uring: add support for IORING_SETUP_CQE_MIXED
In reply to Jens AxboeNormal rings support 16b CQEs for posting completions, while certain features require the ring to be configured with IORING_SETUP_CQE32, as they need to convey more information per completion. This, in turn, makes ALL the CQEs be 32b in size. This is somewhat wasteful and inefficient, particularly when only certain CQEs need to be of the bigger variant.
This adds support for setting up a ring with mixed CQE sizes, using IORING_SETUP_CQE_MIXED. When setup in this mode, CQEs posted to the ring may be either 16b or 32b in size. If a CQE is 32b in size, then IORING_CQE_F_32 is set in the CQE flags to indicate that this is the case. If this flag isn't set, the CQE is the normal 16b variant.
CQEs on these types of mixed rings may also have IORING_CQE_F_SKIP set. This can happen if the ring is one (small) CQE entry away from wrapping, and an attempt is made to post a 32b CQE. As CQEs must be contigious in the CQ ring, a 32b CQE cannot wrap the ring. For this case, a single dummy CQE is posted with the SKIP flag set. The application should simply ignore those.
--- include/uapi/linux/io_uring.h | 6 +++ io_uring/io_uring.c | 71 +++++++++++++++++++++++++++-------- io_uring/io_uring.h | 49 +++++++++++++++++------- io_uring/register.c | 3 +- 4 files changed, 99 insertions(+), 30 deletions(-)
include/uapi/linux/io_uring.h +6 -0
@@ -225,6 +225,12 @@ enum io_uring_sqe_flags_bit {/* Use hybrid poll in iopoll process */ #define IORING_SETUP_HYBRID_IOPOLL (1U << 17) +/* +* Allow both 16b and 32b CQEs. If a 32b CQE is posted, it will have +* IORING_CQE_F_32 set in cqe->flags. +*/ +#define IORING_SETUP_CQE_MIXED (1U << 18) +enum io_uring_op {IORING_OP_NOP ,IORING_OP_READV ,
io_uring/io_uring.c +56 -15
159 lines not on this page · raw message
io_uring/io_uring.h +35 -14
98 lines not on this page · raw message
io_uring/register.c +2 -1
@@ -396,7 +396,8 @@ static void io_register_free_rings(struct io_ring_ctx *ctx,#define RESIZE_FLAGS (IORING_SETUP_CQSIZE | IORING_SETUP_CLAMP)#define COPY_FLAGS (IORING_SETUP_NO_SQARRAY | IORING_SETUP_SQE128 | \ - IORING_SETUP_CQE32 | IORING_SETUP_NO_MMAP) + IORING_SETUP_CQE32 | IORING_SETUP_NO_MMAP | \ + IORING_SETUP_CQE_MIXED)static int io_register_resize_rings (struct io_ring_ctx *ctx,void __user *arg) {
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 6/8 io_uring/nop: add support for IORING_SETUP_CQE_MIXED
In reply to Jens AxboeThis adds support for setting IORING_NOP_CQE32 as a flag for a NOP command, in which case a 32b CQE will be posted rather than a regular one. This is the default if the ring has been setup with IORING_SETUP_CQE32. If the ring has been setup with IORING_SETUP_CQE_MIXED, then 16b CQEs will be posted without this flag set, and 32b CQEs if this flag is set. For the latter case, sqe->off is what will be posted as cqe->big_cqe[0] and sqe->addr is what will be posted as cqe->big_cqe[1].
--- include/uapi/linux/io_uring.h | 1 + io_uring/nop.c | 17 +++++++++++++++-- 2 files changed, 16 insertions(+), 2 deletions(-)
include/uapi/linux/io_uring.h +1 -0
@@ -460,6 +460,7 @@ enum io_uring_msg_ring_flags {#define IORING_NOP_FIXED_FILE (1U << 2)#define IORING_NOP_FIXED_BUFFER (1U << 3)#define IORING_NOP_TW (1U << 4) +#define IORING_NOP_CQE32 (1U << 5)/* * IO completion data structure (Completion Queue Entry)
io_uring/nop.c +15 -2
@@ -17,11 +17,13 @@ struct io_nop {int result;int fd;unsigned int flags; +__u64 extra1; +__u64 extra2; };#define NOP_FLAGS (IORING_NOP_INJECT_RESULT | IORING_NOP_FIXED_FILE | \ IORING_NOP_FIXED_BUFFER | IORING_NOP_FILE | \ - IORING_NOP_TW) + IORING_NOP_TW | IORING_NOP_CQE32)int io_nop_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) {
@@ -41,6 +43,14 @@ int io_nop_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) nop->fd =-1 ;if (nop->flags &IORING_NOP_FIXED_BUFFER ) req->buf_index =READ_ONCE (sqe->buf_index); +if (nop->flags &IORING_NOP_CQE32 ) { +struct io_ring_ctx *ctx = req->ctx; + +if (!(ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED ))) +return -EINVAL ; + nop->extra1 =READ_ONCE (sqe->off); + nop->extra2 =READ_ONCE (sqe->addr); + }return 0 ; }
@@ -68,7 +78,10 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags) done:if (ret <0 )req_set_fail (req); -io_req_set_res (req, nop->result,0 ); +if (nop->flags &IORING_NOP_CQE32 ) +io_req_set_res32 (req, nop->result,0 , nop->extra1, nop->extra2); +else +io_req_set_res (req, nop->result,0 );if (nop->flags &IORING_NOP_TW ) { req->io_task_work.func = io_req_task_complete;io_req_task_work_add (req);
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 7/8 io_uring/uring_cmd: add support for IORING_SETUP_CQE_MIXED
In reply to Jens AxboeCertain users of uring_cmd currently require fixed 32b CQE support, which is propagated through IO_URING_F_CQE32. Allow IORING_SETUP_CQE_MIXED to cover that case as well, so not all CQEs posted need to be 32b in size.
--- io_uring/cmd_net.c | 3 ++- io_uring/uring_cmd.c | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-)
io_uring/cmd_net.c +2 -1
@@ -4,6 +4,7 @@#include <net/sock.h> #include "uring_cmd.h" +#include "io_uring.h" static inline int io_uring_cmd_getsockopt(struct socket *sock, struct io_uring_cmd *cmd,
@@ -73,7 +74,7 @@ static bool io_process_timestamp_skb(struct io_uring_cmd *cmd, struct sock *sk, cqe->user_data =0 ; cqe->res = tskey; - cqe->flags =IORING_CQE_F_MORE ; + cqe->flags =IORING_CQE_F_MORE |ctx_cqe32_flags (cmd_to_io_kiocb (cmd)->ctx); cqe->flags |= tstype <<IORING_TIMESTAMP_TYPE_SHIFT ;if (ret ==SOF_TIMESTAMPING_TX_HARDWARE ) cqe->flags |=IORING_CQE_F_TSTAMP_HW ;
io_uring/uring_cmd.c +1 -1
@@ -234,7 +234,7 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags)if (ctx->flags &IORING_SETUP_SQE128 ) issue_flags |=IO_URING_F_SQE128 ; -if (ctx->flags &IORING_SETUP_CQE32 ) +if (ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED )) issue_flags |=IO_URING_F_CQE32 ;if (io_is_compat (ctx)) issue_flags |=IO_URING_F_COMPAT ;
Signature · 1 line
2.50.1
Jens Axboe
maintainerPATCH 8/8 io_uring/zcrx: add support for IORING_SETUP_CQE_MIXED
In reply to Jens Axboezcrx currently requires the ring to be set up with fixed 32b CQEs, allow it to use IORING_SETUP_CQE_MIXED as well.
--- Documentation/networking/iou-zcrx.rst | 2 +- io_uring/zcrx.c | 5 +++-- 2 files changed, 4 insertions(+), 3 deletions(-)
Documentation/networking/iou-zcrx.rst +1 -1
@@ -75,7 +75,7 @@ Create an io_uring instance with the following required setup flags:: IORING_SETUP_SINGLE_ISSUER IORING_SETUP_DEFER_TASKRUN - IORING_SETUP_CQE32 + IORING_SETUP_CQE32 or IORING_SETUP_CQE_MIXED Create memory area ------------------
io_uring/zcrx.c +3 -2
@@ -554,8 +554,9 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx,return -EPERM ;/* mandatory io_uring features for zc rx */ -if (!(ctx->flags &IORING_SETUP_DEFER_TASKRUN && - ctx->flags &IORING_SETUP_CQE32 )) +if (!(ctx->flags &IORING_SETUP_DEFER_TASKRUN )) +return -EINVAL ; +if (!(ctx->flags & (IORING_SETUP_CQE32 |IORING_SETUP_CQE_MIXED )))return -EINVAL ;if (copy_from_user (®, arg,sizeof (reg)))return -EFAULT ;
Signature · 1 line
2.50.1