Skip to contentSkip to conversation

Add support for mixed sized CQEs

PATCH In review Jens Axboe 3 participants 16 messages Aug 21, 2025

Subsystems (5)
lore
Share & tools
lore

Jens Axboe

author maintainer
#0 lore raw

v2 cover Add support for mixed sized CQEs

Hi,

Currently io_uring supports two modes for CQEs:

1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b

Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.

This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.

This is mostly trivial to support, with the corner case being attempting to post a 32b CQE when the ring is a single 16b CQE away from wrapping. As CQEs must be contigious in memory, that's simply not possible. The solution taken by this patchset is to add a special CQE type, which has IORING_CQE_F_SKIP set. This is a pad/nop CQE, which should simply be ignored, as it carries no information and serves no other purpose than to re-align the posted CQEs for ring wrap.

If used with liburing, then both the 32b vs 16b postings and the skip are transparent.

liburing support and a few basic test cases can be found here:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/liburing.git/log/?h=cqe-mixed

including man page updates for the newly added setup and CQE flags, and the patches posted here can also be found at:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=io_uring-cqe-mix

Patch 1 is just a prep patch, and patch 2 adds the cqe flags so that the core can be adapted before support is actually there. Patches 3 and 4 are exactly that, and patch 5 finally adds support for the mixed mode. Patch 6 adds support for NOP testing of this, and patches 7/8 allow IORING_SETUP_CQE_MIXED for uring_cmd/zcrx which previously required IORING_SETUP_CQE32 to work.

 Documentation/networking/iou-zcrx.rst |  2 +-
 include/linux/io_uring_types.h        |  6 ---
 include/trace/events/io_uring.h       |  4 +-
 include/uapi/linux/io_uring.h         | 17 ++++++
 io_uring/cmd_net.c                    |  3 +-
 io_uring/fdinfo.c                     | 22 ++++----
 io_uring/io_uring.c                   | 78 +++++++++++++++++++++------
 io_uring/io_uring.h                   | 49 ++++++++++++-----
 io_uring/nop.c                        | 17 +++++-
 io_uring/register.c                   |  3 +-
 io_uring/uring_cmd.c                  |  2 +-
 io_uring/zcrx.c                       |  5 +-
 12 files changed, 152 insertions(+), 56 deletions(-)

Changes since v1:
- Various little cleanups
- Rebase on for-6.18/io_uring

Signature · 1 line
Jens Axboe
author maintainer
#1 lore raw

PATCH v2 1/8 io_uring: remove io_ctx_cqe32() helper

It's pretty pointless and only used for the tracing helper, get rid of it.

---
 include/linux/io_uring_types.h  | 6 ------
 include/trace/events/io_uring.h | 4 ++--
 2 files changed, 2 insertions(+), 8 deletions(-)
include/linux/io_uring_types.h +0 -6
index 9c6c548f43f5..d1e25f3fe0b3 100644
@@ -740,10 +740,4 @@ struct io_overflow_cqe {
 	struct list_head list;
 	struct io_uring_cqe cqe;
 };
-
-static inline bool io_ctx_cqe32(struct io_ring_ctx *ctx)
-{
-	return ctx->flags & IORING_SETUP_CQE32;
-}
-
 #endif
include/trace/events/io_uring.h +2 -2
index 178ab6f611be..6a970625a3ea 100644
@@ -340,8 +340,8 @@ TP_PROTO(struct io_ring_ctx *ctx, void *req, struct io_uring_cqe *cqe),
 		__entry->user_data	= cqe->user_data;
 		__entry->res		= cqe->res;
 		__entry->cflags		= cqe->flags;
-		__entry->extra1		= io_ctx_cqe32(ctx) ? cqe->big_cqe[0] : 0;
-		__entry->extra2		= io_ctx_cqe32(ctx) ? cqe->big_cqe[1] : 0;
+		__entry->extra1		= ctx->flags & IORING_SETUP_CQE32 ? cqe->big_cqe[0] : 0;
+		__entry->extra2		= ctx->flags & IORING_SETUP_CQE32 ? cqe->big_cqe[1] : 0;
 	),
 
 	TP_printk("ring %p, req %p, user_data 0x%llx, result %d, cflags 0x%x "
Signature · 1 line
2.50.1
author maintainer
#2 lore raw

PATCH v2 2/8 io_uring: add UAPI definitions for mixed CQE postings

In reply to Jens Axboe

This adds the CQE flags related to supporting a mixed CQ ring mode, where both normal (16b) and big (32b) CQEs may be posted.

No functional changes in this patch.

---
 include/uapi/linux/io_uring.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
include/uapi/linux/io_uring.h +10 -0
index 1e935f8901c5..7af8d10b3aba 100644
@@ -491,12 +491,22 @@ struct io_uring_cqe {
  *			other provided buffer type, all completions with a
  *			buffer passed back is automatically returned to the
  *			application.
+ * IORING_CQE_F_SKIP	If set, then the application/liburing must ignore this
+ *			CQE. It's only purpose is to fill a gap in the ring,
+ *			if a large CQE is attempted posted when the ring has
+ *			just a single small CQE worth of space left before
+ *			wrapping.
+ * IORING_CQE_F_32	If set, this is a 32b/big-cqe posting. Use with rings
+ *			setup in a mixed CQE mode, where both 16b and 32b
+ *			CQEs may be posted to the CQ ring.
  */
 #define IORING_CQE_F_BUFFER		(1U << 0)
 #define IORING_CQE_F_MORE		(1U << 1)
 #define IORING_CQE_F_SOCK_NONEMPTY	(1U << 2)
 #define IORING_CQE_F_NOTIF		(1U << 3)
 #define IORING_CQE_F_BUF_MORE		(1U << 4)
+#define IORING_CQE_F_SKIP		(1U << 5)
+#define IORING_CQE_F_32			(1U << 15)
 
 #define IORING_CQE_BUFFER_SHIFT		16
 
Signature · 1 line
2.50.1
author maintainer
#3 lore raw

PATCH v2 3/8 io_uring/fdinfo: handle mixed sized CQEs

In reply to Jens Axboe

Ensure that the CQ ring iteration handles differently sized CQEs, not just a fixed 16b or 32b size per ring. These CQEs aren't possible just yet, but prepare the fdinfo CQ ring dumping for handling them.

---
 io_uring/fdinfo.c | 22 ++++++++++++----------
 1 file changed, 12 insertions(+), 10 deletions(-)
io_uring/fdinfo.c +12 -10
index 9798d6fb4ec7..5c7339838769 100644
@@ -65,15 +65,12 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)
 	unsigned int sq_tail = READ_ONCE(r->sq.tail);
 	unsigned int cq_head = READ_ONCE(r->cq.head);
 	unsigned int cq_tail = READ_ONCE(r->cq.tail);
-	unsigned int cq_shift = 0;
 	unsigned int sq_shift = 0;
-	unsigned int sq_entries, cq_entries;
+	unsigned int sq_entries;
 	int sq_pid = -1, sq_cpu = -1;
 	u64 sq_total_time = 0, sq_work_time = 0;
 	unsigned int i;
 
-	if (ctx->flags & IORING_SETUP_CQE32)
-		cq_shift = 1;
 	if (ctx->flags & IORING_SETUP_SQE128)
 		sq_shift = 1;
 
@@ -125,18 +122,23 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m)
 		seq_printf(m, "\n");
 	}
 	seq_printf(m, "CQEs:\t%u\n", cq_tail - cq_head);
-	cq_entries = min(cq_tail - cq_head, ctx->cq_entries);
-	for (i = 0; i < cq_entries; i++) {
-		unsigned int entry = i + cq_head;
-		struct io_uring_cqe *cqe = &r->cqes[(entry & cq_mask) << cq_shift];
+	while (cq_head < cq_tail) {
+		struct io_uring_cqe *cqe;
+		bool cqe32 = false;
 
+		cqe = &r->cqes[(cq_head & cq_mask)];
+		if (cqe->flags & IORING_CQE_F_32 || ctx->flags & IORING_SETUP_CQE32)
+			cqe32 = true;
 		seq_printf(m, "%5u: user_data:%llu, res:%d, flag:%x",
-			   entry & cq_mask, cqe->user_data, cqe->res,
+			   cq_head & cq_mask, cqe->user_data, cqe->res,
 			   cqe->flags);
-		if (cq_shift)
+		if (cqe32)
 			seq_printf(m, ", extra1:%llu, extra2:%llu\n",
 					cqe->big_cqe[0], cqe->big_cqe[1]);
 		seq_printf(m, "\n");
+		cq_head++;
+		if (cqe32)
+			cq_head++;
 	}
 
 	if (ctx->flags & IORING_SETUP_SQPOLL) {
Signature · 1 line
2.50.1
author maintainer
#4 lore raw

PATCH v2 4/8 io_uring/trace: support completion tracing of mixed 32b CQEs

In reply to Jens Axboe

Check for IORING_CQE_F_32 as well, not just if the ring was setup with IORING_SETUP_CQE32 to only support big CQEs.

---
 include/trace/events/io_uring.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/trace/events/io_uring.h +2 -2
index 6a970625a3ea..45d15460b495 100644
@@ -340,8 +340,8 @@ TP_PROTO(struct io_ring_ctx *ctx, void *req, struct io_uring_cqe *cqe),
 		__entry->user_data	= cqe->user_data;
 		__entry->res		= cqe->res;
 		__entry->cflags		= cqe->flags;
-		__entry->extra1		= ctx->flags & IORING_SETUP_CQE32 ? cqe->big_cqe[0] : 0;
-		__entry->extra2		= ctx->flags & IORING_SETUP_CQE32 ? cqe->big_cqe[1] : 0;
+		__entry->extra1		= ctx->flags & IORING_SETUP_CQE32 || cqe->flags & IORING_CQE_F_32 ? cqe->big_cqe[0] : 0;
+		__entry->extra2		= ctx->flags & IORING_SETUP_CQE32 || cqe->flags & IORING_CQE_F_32 ? cqe->big_cqe[1] : 0;
 	),
 
 	TP_printk("ring %p, req %p, user_data 0x%llx, result %d, cflags 0x%x "
Signature · 1 line
2.50.1
author maintainer
#5 lore raw

PATCH v2 5/8 io_uring: add support for IORING_SETUP_CQE_MIXED

In reply to Jens Axboe

Normal rings support 16b CQEs for posting completions, while certain features require the ring to be configured with IORING_SETUP_CQE32, as they need to convey more information per completion. This, in turn, makes ALL the CQEs be 32b in size. This is somewhat wasteful and inefficient, particularly when only certain CQEs need to be of the bigger variant.

This adds support for setting up a ring with mixed CQE sizes, using IORING_SETUP_CQE_MIXED. When setup in this mode, CQEs posted to the ring may be either 16b or 32b in size. If a CQE is 32b in size, then IORING_CQE_F_32 is set in the CQE flags to indicate that this is the case. If this flag isn't set, the CQE is the normal 16b variant.

CQEs on these types of mixed rings may also have IORING_CQE_F_SKIP set. This can happen if the ring is one (small) CQE entry away from wrapping, and an attempt is made to post a 32b CQE. As CQEs must be contigious in the CQ ring, a 32b CQE cannot wrap the ring. For this case, a single dummy CQE is posted with the SKIP flag set. The application should simply ignore those.

---
 include/uapi/linux/io_uring.h |  6 +++
 io_uring/io_uring.c           | 78 ++++++++++++++++++++++++++++-------
 io_uring/io_uring.h           | 49 +++++++++++++++-------
 io_uring/register.c           |  3 +-
 4 files changed, 105 insertions(+), 31 deletions(-)
include/uapi/linux/io_uring.h +6 -0
index 7af8d10b3aba..5135e1be0390 100644
@@ -225,6 +225,12 @@ enum io_uring_sqe_flags_bit {
 /* Use hybrid poll in iopoll process */
 #define IORING_SETUP_HYBRID_IOPOLL	(1U << 17)
 
+/*
+ * Allow both 16b and 32b CQEs. If a 32b CQE is posted, it will have
+ * IORING_CQE_F_32 set in cqe->flags.
+ */
+#define IORING_SETUP_CQE_MIXED		(1U << 18)
+
 enum io_uring_op {
 	IORING_OP_NOP,
 	IORING_OP_READV,
io_uring/io_uring.c +62 -16

165 lines not on this page · raw message

io_uring/io_uring.h +35 -14

98 lines not on this page · raw message

io_uring/register.c +2 -1
index a59589249fce..a1a9b2884eae 100644
@@ -396,7 +396,8 @@ static void io_register_free_rings(struct io_ring_ctx *ctx,
 
 #define RESIZE_FLAGS	(IORING_SETUP_CQSIZE | IORING_SETUP_CLAMP)
 #define COPY_FLAGS	(IORING_SETUP_NO_SQARRAY | IORING_SETUP_SQE128 | \
-			 IORING_SETUP_CQE32 | IORING_SETUP_NO_MMAP)
+			 IORING_SETUP_CQE32 | IORING_SETUP_NO_MMAP | \
+			 IORING_SETUP_CQE_MIXED)
 
 static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg)
 {
Signature · 1 line
2.50.1
author maintainer
#6 lore raw

PATCH v2 6/8 io_uring/nop: add support for IORING_SETUP_CQE_MIXED

In reply to Jens Axboe

This adds support for setting IORING_NOP_CQE32 as a flag for a NOP command, in which case a 32b CQE will be posted rather than a regular one. This is the default if the ring has been setup with IORING_SETUP_CQE32. If the ring has been setup with IORING_SETUP_CQE_MIXED, then 16b CQEs will be posted without this flag set, and 32b CQEs if this flag is set. For the latter case, sqe->off is what will be posted as cqe->big_cqe[0] and sqe->addr is what will be posted as cqe->big_cqe[1].

---
 include/uapi/linux/io_uring.h |  1 +
 io_uring/nop.c                | 17 +++++++++++++++--
 2 files changed, 16 insertions(+), 2 deletions(-)
include/uapi/linux/io_uring.h +1 -0
index 5135e1be0390..04ebff33d0e6 100644
@@ -464,6 +464,7 @@ enum io_uring_msg_ring_flags {
 #define IORING_NOP_FIXED_FILE		(1U << 2)
 #define IORING_NOP_FIXED_BUFFER		(1U << 3)
 #define IORING_NOP_TW			(1U << 4)
+#define IORING_NOP_CQE32		(1U << 5)
 
 /*
  * IO completion data structure (Completion Queue Entry)
io_uring/nop.c +15 -2
index 20ed0f85b1c2..3caf07878f8a 100644
@@ -17,11 +17,13 @@ struct io_nop {
 	int             result;
 	int		fd;
 	unsigned int	flags;
+	__u64		extra1;
+	__u64		extra2;
 };
 
 #define NOP_FLAGS	(IORING_NOP_INJECT_RESULT | IORING_NOP_FIXED_FILE | \
 			 IORING_NOP_FIXED_BUFFER | IORING_NOP_FILE | \
-			 IORING_NOP_TW)
+			 IORING_NOP_TW | IORING_NOP_CQE32)
 
 int io_nop_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
@@ -41,6 +43,14 @@ int io_nop_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		nop->fd = -1;
 	if (nop->flags & IORING_NOP_FIXED_BUFFER)
 		req->buf_index = READ_ONCE(sqe->buf_index);
+	if (nop->flags & IORING_NOP_CQE32) {
+		struct io_ring_ctx *ctx = req->ctx;
+
+		if (!(ctx->flags & (IORING_SETUP_CQE32|IORING_SETUP_CQE_MIXED)))
+			return -EINVAL;
+		nop->extra1 = READ_ONCE(sqe->off);
+		nop->extra2 = READ_ONCE(sqe->addr);
+	}
 	return 0;
 }
 
@@ -68,7 +78,10 @@ int io_nop(struct io_kiocb *req, unsigned int issue_flags)
 done:
 	if (ret < 0)
 		req_set_fail(req);
-	io_req_set_res(req, nop->result, 0);
+	if (nop->flags & IORING_NOP_CQE32)
+		io_req_set_res32(req, nop->result, 0, nop->extra1, nop->extra2);
+	else
+		io_req_set_res(req, nop->result, 0);
 	if (nop->flags & IORING_NOP_TW) {
 		req->io_task_work.func = io_req_task_complete;
 		io_req_task_work_add(req);
Signature · 1 line
2.50.1
author maintainer
#7 lore raw

PATCH v2 7/8 io_uring/uring_cmd: add support for IORING_SETUP_CQE_MIXED

In reply to Jens Axboe

Certain users of uring_cmd currently require fixed 32b CQE support, which is propagated through IO_URING_F_CQE32. Allow IORING_SETUP_CQE_MIXED to cover that case as well, so not all CQEs posted need to be 32b in size.

---
 io_uring/cmd_net.c   | 3 ++-
 io_uring/uring_cmd.c | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)
io_uring/cmd_net.c +2 -1
index 3866fe6ff541..27a09aa4c9d0 100644
@@ -4,6 +4,7 @@
 #include <net/sock.h>
 
 #include "uring_cmd.h"
+#include "io_uring.h"
 
 static inline int io_uring_cmd_getsockopt(struct socket *sock,
 					  struct io_uring_cmd *cmd,
@@ -73,7 +74,7 @@ static bool io_process_timestamp_skb(struct io_uring_cmd *cmd, struct sock *sk,
 
 	cqe->user_data = 0;
 	cqe->res = tskey;
-	cqe->flags = IORING_CQE_F_MORE;
+	cqe->flags = IORING_CQE_F_MORE | ctx_cqe32_flags(cmd_to_io_kiocb(cmd)->ctx);
 	cqe->flags |= tstype << IORING_TIMESTAMP_TYPE_SHIFT;
 	if (ret == SOF_TIMESTAMPING_TX_HARDWARE)
 		cqe->flags |= IORING_CQE_F_TSTAMP_HW;
io_uring/uring_cmd.c +1 -1
index 3cfb5d51b88a..90d3239df6bd 100644
@@ -248,7 +248,7 @@ int io_uring_cmd(struct io_kiocb *req, unsigned int issue_flags)
 
 	if (ctx->flags & IORING_SETUP_SQE128)
 		issue_flags |= IO_URING_F_SQE128;
-	if (ctx->flags & IORING_SETUP_CQE32)
+	if (ctx->flags & (IORING_SETUP_CQE32 | IORING_SETUP_CQE_MIXED))
 		issue_flags |= IO_URING_F_CQE32;
 	if (io_is_compat(ctx))
 		issue_flags |= IO_URING_F_COMPAT;
Signature · 1 line
2.50.1
author maintainer
#8 lore raw

PATCH v2 8/8 io_uring/zcrx: add support for IORING_SETUP_CQE_MIXED

In reply to Jens Axboe

zcrx currently requires the ring to be set up with fixed 32b CQEs, allow it to use IORING_SETUP_CQE_MIXED as well.

---
 Documentation/networking/iou-zcrx.rst | 2 +-
 io_uring/zcrx.c                       | 5 +++--
 2 files changed, 4 insertions(+), 3 deletions(-)
Documentation/networking/iou-zcrx.rst +1 -1
index 0127319b30bb..54a72e172bdc 100644
@@ -75,7 +75,7 @@ Create an io_uring instance with the following required setup flags::
 
   IORING_SETUP_SINGLE_ISSUER
   IORING_SETUP_DEFER_TASKRUN
-  IORING_SETUP_CQE32
+  IORING_SETUP_CQE32 or IORING_SETUP_CQE_MIXED
 
 Create memory area
 ------------------
io_uring/zcrx.c +3 -2
index e5ff49f3425e..f1da852c496b 100644
@@ -554,8 +554,9 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx,
 		return -EPERM;
 
 	/* mandatory io_uring features for zc rx */
-	if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN &&
-	      ctx->flags & IORING_SETUP_CQE32))
+	if (!(ctx->flags & IORING_SETUP_DEFER_TASKRUN))
+		return -EINVAL;
+	if (!(ctx->flags & (IORING_SETUP_CQE32|IORING_SETUP_CQE_MIXED)))
 		return -EINVAL;
 	if (copy_from_user(&reg, arg, sizeof(reg)))
 		return -EFAULT;
Signature · 1 line
2.50.1
#9 lore raw
In reply to Jens Axboe
Quoted 13 lines · Jens Axboe

Hi,

Currently io_uring supports two modes for CQEs:

1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b

Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.

Quoted 7 lines

This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.

This makes a lot of sense. Have you considered something analogous for SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for a mix of 64-byte and 128-byte SQEs also wastes memory, probably even more since SQEs are 4x larger than CQEs.

Best,
Caleb

Quoted 50 lines

This is mostly trivial to support, with the corner case being attempting to post a 32b CQE when the ring is a single 16b CQE away from wrapping. As CQEs must be contigious in memory, that's simply not possible. The solution taken by this patchset is to add a special CQE type, which has IORING_CQE_F_SKIP set. This is a pad/nop CQE, which should simply be ignored, as it carries no information and serves no other purpose than to re-align the posted CQEs for ring wrap.

If used with liburing, then both the 32b vs 16b postings and the skip are transparent.

liburing support and a few basic test cases can be found here:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/liburing.git/log/?h=cqe-mixed

including man page updates for the newly added setup and CQE flags, and the patches posted here can also be found at:

https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/log/?h=io_uring-cqe-mix

Patch 1 is just a prep patch, and patch 2 adds the cqe flags so that the core can be adapted before support is actually there. Patches 3 and 4 are exactly that, and patch 5 finally adds support for the mixed mode. Patch 6 adds support for NOP testing of this, and patches 7/8 allow IORING_SETUP_CQE_MIXED for uring_cmd/zcrx which previously required IORING_SETUP_CQE32 to work.

 Documentation/networking/iou-zcrx.rst |  2 +-
 include/linux/io_uring_types.h        |  6 ---
 include/trace/events/io_uring.h       |  4 +-
 include/uapi/linux/io_uring.h         | 17 ++++++
 io_uring/cmd_net.c                    |  3 +-
 io_uring/fdinfo.c                     | 22 ++++----
 io_uring/io_uring.c                   | 78 +++++++++++++++++++++------
 io_uring/io_uring.h                   | 49 ++++++++++++-----
 io_uring/nop.c                        | 17 +++++-
 io_uring/register.c                   |  3 +-
 io_uring/uring_cmd.c                  |  2 +-
 io_uring/zcrx.c                       |  5 +-
 12 files changed, 152 insertions(+), 56 deletions(-)

Changes since v1:
- Various little cleanups
- Rebase on for-6.18/io_uring

--
Jens Axboe

author maintainer
#10 lore raw
Quoted 22 lines · Caleb Sander Mateos
Quoted 13 lines · On Thu, Aug 21, 2025 at 7:28?AM Jens Axboe <[email protected]> wrote:

Hi,

Currently io_uring supports two modes for CQEs:

1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b

Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.

Quoted 7 lines

This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.

Quoted 4 lines

This makes a lot of sense. Have you considered something analogous for SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for a mix of 64-byte and 128-byte SQEs also wastes memory, probably even more since SQEs are 4x larger than CQEs.

Adding Keith, as he and I literally just talked about that. My answer was that the case is a bit different in that 32b CQEs can be useful in cases that are predominately 16b in the first place. For example, networking workload doing send/recv/etc and the occassional get/setsockopt kind of thing. Or maybe a mix of normal recv and zero copy rx.

For the SQE case, I think it's a bit different. At least the cases I know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing to be told otherwise! Because that is kind of the key question that needs answering before even thinking about doing that kind of work.

But yes, it could be supported, and Keith (kind of) signed himself up to do that. One oddity I see on that side is that while with CQE32 the kernel can manage the potential wrap-around gap, for SQEs that's obviously on the application to do. That could just be a NOP or something like that, but you do need something to fill/skip that space. I guess that could be as simple as having an opcode that is simply "skip me", so on the kernel side it'd be easy as it'd just drop it on the floor. You still need to app side to fill one, however, and then deal with "oops SQ ring is now full" too.

Probably won't be too bad at all, however.

Signature · 1 line
Jens Axboe
#11 lore raw
Quoted 5 lines · Jens Axboe

For the SQE case, I think it's a bit different. At least the cases I know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing to be told otherwise! Because that is kind of the key question that needs answering before even thinking about doing that kind of work.

The main use case I can think of is if an application allocates one ring for uring_cmd with the 128b SQEs, and then a separate ring for normal file and network stuff. Mixed SQE's would allow that application to have just one ring without being wasteful, but I'm just not sure if the separate rings is undesirable enough to make the effort worth it.

Quoted 9 lines

But yes, it could be supported, and Keith (kind of) signed himself up to do that. One oddity I see on that side is that while with CQE32 the kernel can manage the potential wrap-around gap, for SQEs that's obviously on the application to do. That could just be a NOP or something like that, but you do need something to fill/skip that space. I guess that could be as simple as having an opcode that is simply "skip me", so on the kernel side it'd be easy as it'd just drop it on the floor. You still need to app side to fill one, however, and then deal with "oops SQ ring is now full" too.

Yep, I think it's doable, and your implementation for mixed CQEs provides a great reference. I trust we can get liburing using it correctly, but would be afraid for anyone not using the library.

author maintainer
#12 lore raw
Quoted 7 lines · Keith Busch
Quoted 5 lines · Jens Axboe

For the SQE case, I think it's a bit different. At least the cases I know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing to be told otherwise! Because that is kind of the key question that needs answering before even thinking about doing that kind of work.

Quoted 5 lines

The main use case I can think of is if an application allocates one ring for uring_cmd with the 128b SQEs, and then a separate ring for normal file and network stuff. Mixed SQE's would allow that application to have just one ring without being wasteful, but I'm just not sure if the separate rings is undesirable enough to make the effort worth it.

Indeed! And like Caleb mentioned, their use case already does this in fact, just passthrough with housekeeping buffer commands.

Quoted 10 lines
Quoted 9 lines

But yes, it could be supported, and Keith (kind of) signed himself up to do that. One oddity I see on that side is that while with CQE32 the kernel can manage the potential wrap-around gap, for SQEs that's obviously on the application to do. That could just be a NOP or something like that, but you do need something to fill/skip that space. I guess that could be as simple as having an opcode that is simply "skip me", so on the kernel side it'd be easy as it'd just drop it on the floor. You still need to app side to fill one, however, and then deal with "oops SQ ring is now full" too.

Quoted 3 lines

Yep, I think it's doable, and your implementation for mixed CQEs provides a great reference. I trust we can get liburing using it correctly, but would be afraid for anyone not using the library.

That's the same with the mixed CQEs though - as you can see from the liburing changes, it's really not hard to support. If you're using the raw interface, well then things are already a bit more complicated for you. Not too worried about that use case.

Signature · 1 line
Jens Axboe
#13 lore raw
In reply to Jens Axboe
Quoted 36 lines · Jens Axboe
Quoted 22 lines · Caleb Sander Mateos
Quoted 13 lines · On Thu, Aug 21, 2025 at 7:28?AM Jens Axboe <[email protected]> wrote:

Hi,

Currently io_uring supports two modes for CQEs:

1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b

Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.

Quoted 7 lines

This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.

Quoted 4 lines

This makes a lot of sense. Have you considered something analogous for SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for a mix of 64-byte and 128-byte SQEs also wastes memory, probably even more since SQEs are 4x larger than CQEs.

Adding Keith, as he and I literally just talked about that. My answer was that the case is a bit different in that 32b CQEs can be useful in cases that are predominately 16b in the first place. For example, networking workload doing send/recv/etc and the occassional get/setsockopt kind of thing. Or maybe a mix of normal recv and zero copy rx.

Quoted 4 lines

For the SQE case, I think it's a bit different. At least the cases I know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing to be told otherwise! Because that is kind of the key question that needs answering before even thinking about doing that kind of work.

We certainly have a use case that mixes the two on the same io_uring:
ublk commit/buffer register/unregister commands (64 byte SQEs) and NVMe passthru commands (128 byte SQEs). I could also imagine an application issuing both normal read/write commands and NVMe passthru commands. But you're probably right that this isn't a super common use case.

Quoted 10 lines

But yes, it could be supported, and Keith (kind of) signed himself up to do that. One oddity I see on that side is that while with CQE32 the kernel can manage the potential wrap-around gap, for SQEs that's obviously on the application to do. That could just be a NOP or something like that, but you do need something to fill/skip that space. I guess that could be as simple as having an opcode that is simply "skip me", so on the kernel side it'd be easy as it'd just drop it on the floor. You still need to app side to fill one, however, and then deal with "oops SQ ring is now full" too.

Sure, of course userspace would need to handle a misaligned big SQE at the end of the SQ analogously to mixed CQE sizes. I assume liburing should be able to do that mostly transparently, that logic could all be encapsulated by io_uring_get_sqe().

Best,
Caleb

Quoted 5 lines

Probably won't be too bad at all, however.

--
Jens Axboe

author maintainer
#14 lore raw
Quoted 42 lines · Caleb Sander Mateos
Quoted 36 lines · On Thu, Aug 21, 2025 at 10:12?AM Jens Axboe <[email protected]> wrote:
Quoted 22 lines · Caleb Sander Mateos
Quoted 13 lines · On Thu, Aug 21, 2025 at 7:28?AM Jens Axboe <[email protected]> wrote:

Hi,

Currently io_uring supports two modes for CQEs:

1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b

Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.

Quoted 7 lines

This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.

Quoted 4 lines

This makes a lot of sense. Have you considered something analogous for SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for a mix of 64-byte and 128-byte SQEs also wastes memory, probably even more since SQEs are 4x larger than CQEs.

Adding Keith, as he and I literally just talked about that. My answer was that the case is a bit different in that 32b CQEs can be useful in cases that are predominately 16b in the first place. For example, networking workload doing send/recv/etc and the occassional get/setsockopt kind of thing. Or maybe a mix of normal recv and zero copy rx.

Quoted 4 lines

For the SQE case, I think it's a bit different. At least the cases I know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing to be told otherwise! Because that is kind of the key question that needs answering before even thinking about doing that kind of work.

Quoted 6 lines

We certainly have a use case that mixes the two on the same io_uring:
ublk commit/buffer register/unregister commands (64 byte SQEs) and NVMe passthru commands (128 byte SQEs). I could also imagine an application issuing both normal read/write commands and NVMe passthru commands. But you're probably right that this isn't a super common use case.

Yes that's a good point, and that would roughly be 50/50 in terms of 64b vs 128b SQEs?

And yes, I can imagine other uses cases too, but I'm also finding a hard time justifying those as likely. On the other hand, people do the weirdest things...

Quoted 10 lines
Quoted 9 lines

But yes, it could be supported, and Keith (kind of) signed himself up to do that. One oddity I see on that side is that while with CQE32 the kernel can manage the potential wrap-around gap, for SQEs that's obviously on the application to do. That could just be a NOP or something like that, but you do need something to fill/skip that space. I guess that could be as simple as having an opcode that is simply "skip me", so on the kernel side it'd be easy as it'd just drop it on the floor. You still need to app side to fill one, however, and then deal with "oops SQ ring is now full" too.

Quoted 4 lines

Sure, of course userspace would need to handle a misaligned big SQE at the end of the SQ analogously to mixed CQE sizes. I assume liburing should be able to do that mostly transparently, that logic could all be encapsulated by io_uring_get_sqe().

Yep I think so, we'd need a new helper to return the kind of SQE you want, and it'd just need to get a 64b one and mark it with the SKIP opcode first if being asked for a 128b one and we're one off from wrapping around.

Signature · 1 line
Jens Axboe
#15 lore raw
Quoted 51 lines · Jens Axboe
Quoted 42 lines · Caleb Sander Mateos
Quoted 36 lines · On Thu, Aug 21, 2025 at 10:12?AM Jens Axboe <[email protected]> wrote:
Quoted 22 lines · Caleb Sander Mateos
Quoted 13 lines · On Thu, Aug 21, 2025 at 7:28?AM Jens Axboe <[email protected]> wrote:

Hi,

Currently io_uring supports two modes for CQEs:

1) The standard mode, where 16b CQEs are used
2) Setting IORING_SETUP_CQE32, which makes all CQEs posted 32b

Certain features need to pass more information back than just a single 32-bit res field, and hence mandate the use of CQE32 to be able to work. Examples of that include passthrough or other uses of ->uring_cmd() like socket option getting and setting, including timestamps.

Quoted 7 lines

This patchset adds support for IORING_SETUP_CQE_MIXED, which allows posting both 16b and 32b CQEs on the same CQ ring. The idea here is that we need not waste twice the space for CQ rings, or use twice the space per CQE posted, if only some of the CQEs posted require the use of 32b CQEs. On a ring setup in CQE mixed mode, 32b posted CQEs will have IORING_CQE_F_32 set in cqe->flags to tell the application (or liburing) about this fact.

Quoted 4 lines

This makes a lot of sense. Have you considered something analogous for SQEs? Requiring all SQEs to be 128 bytes when an io_uring is used for a mix of 64-byte and 128-byte SQEs also wastes memory, probably even more since SQEs are 4x larger than CQEs.

Adding Keith, as he and I literally just talked about that. My answer was that the case is a bit different in that 32b CQEs can be useful in cases that are predominately 16b in the first place. For example, networking workload doing send/recv/etc and the occassional get/setsockopt kind of thing. Or maybe a mix of normal recv and zero copy rx.

Quoted 4 lines

For the SQE case, I think it's a bit different. At least the cases I know of, it's mostly 100% 64b SQEs or 128b SQEs. I'm certainly willing to be told otherwise! Because that is kind of the key question that needs answering before even thinking about doing that kind of work.

Quoted 6 lines

We certainly have a use case that mixes the two on the same io_uring:
ublk commit/buffer register/unregister commands (64 byte SQEs) and NVMe passthru commands (128 byte SQEs). I could also imagine an application issuing both normal read/write commands and NVMe passthru commands. But you're probably right that this isn't a super common use case.

Quoted 2 lines

Yes that's a good point, and that would roughly be 50/50 in terms of 64b vs 128b SQEs?

For our application, the ratio between 64 and 128 bytes SQEs depends on the ublk workload. Small ublk I/Os are translated 1-1 into NVMe passthru I/Os, so there can be as many as 3 64-byte ublk SQEs (register buffer, unregister buffer, and commit) for each 128-byte NVMe passthru SQE. Larger I/Os are sharded into more NVMe passthru commands, so there are relatively more 128-byte SQEs. And some workloads can't use ublk zero-copy (since the data needs to go through a RAID computation), in which case the only 64-byte SQE is the ublk commit.

Best,
Caleb

Quoted 27 lines

And yes, I can imagine other uses cases too, but I'm also finding a hard time justifying those as likely. On the other hand, people do the weirdest things...

Quoted 10 lines
Quoted 9 lines

But yes, it could be supported, and Keith (kind of) signed himself up to do that. One oddity I see on that side is that while with CQE32 the kernel can manage the potential wrap-around gap, for SQEs that's obviously on the application to do. That could just be a NOP or something like that, but you do need something to fill/skip that space. I guess that could be as simple as having an opcode that is simply "skip me", so on the kernel side it'd be easy as it'd just drop it on the floor. You still need to app side to fill one, however, and then deal with "oops SQ ring is now full" too.

Quoted 4 lines

Sure, of course userspace would need to handle a misaligned big SQE at the end of the SQ analogously to mixed CQE sizes. I assume liburing should be able to do that mostly transparently, that logic could all be encapsulated by io_uring_get_sqe().

Yep I think so, we'd need a new helper to return the kind of SQE you want, and it'd just need to get a 64b one and mark it with the SKIP opcode first if being asked for a 128b one and we're one off from wrapping around.

--
Jens Axboe