Skip to contentSkip to conversation

add support for synchronous refill

PATCH Applied Pavel Begunkov 2 participants 8 messages Aug 17, 2025 – Aug 20, 2025

Subsystems (2)
lore
Share & tools
lore

Pavel Begunkov

author maintainer
#0 lore raw

cover add support for synchronous refill

Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.

For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next

Pavel Begunkov (2):
io_uring/zcrx: introduce io_parse_rqe()
io_uring/zcrx: allow synchronous buffer return

 include/uapi/linux/io_uring.h |  10 ++++
 io_uring/register.c           |   3 +
 io_uring/zcrx.c               | 100 +++++++++++++++++++++++++++++-----
 io_uring/zcrx.h               |   7 +++
 4 files changed, 107 insertions(+), 13 deletions(-)
Signature · 1 line
2.49.0
author maintainer
#1 lore raw

PATCH 1/2 io_uring/zcrx: introduce io_parse_rqe()

Add a helper for verifying a rqe and extracting a niov out of it. It'll be reused in following patches.

---
 io_uring/zcrx.c | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)
io_uring/zcrx.c +23 -13
index b3cfe0c04920..d510ebc3d382 100644
@@ -727,6 +727,28 @@ static struct io_uring_zcrx_rqe *io_zcrx_get_rqe(struct io_zcrx_ifq *ifq,
 	return &ifq->rqes[idx];
 }
 
+static inline bool io_parse_rqe(struct io_uring_zcrx_rqe *rqe,
+				struct io_zcrx_ifq *ifq,
+				struct net_iov **ret_niov)
+{
+	unsigned niov_idx, area_idx;
+	struct io_zcrx_area *area;
+
+	area_idx = rqe->off >> IORING_ZCRX_AREA_SHIFT;
+	niov_idx = (rqe->off & ~IORING_ZCRX_AREA_MASK) >> ifq->niov_shift;
+
+	if (unlikely(rqe->__pad || area_idx))
+		return false;
+	area = ifq->area;
+
+	if (unlikely(niov_idx >= area->nia.num_niovs))
+		return false;
+	niov_idx = array_index_nospec(niov_idx, area->nia.num_niovs);
+
+	*ret_niov = &area->nia.niovs[niov_idx];
+	return true;
+}
+
 static void io_zcrx_ring_refill(struct page_pool *pp,
 				struct io_zcrx_ifq *ifq)
 {
@@ -741,23 +763,11 @@ static void io_zcrx_ring_refill(struct page_pool *pp,
 
 	do {
 		struct io_uring_zcrx_rqe *rqe = io_zcrx_get_rqe(ifq, mask);
-		struct io_zcrx_area *area;
 		struct net_iov *niov;
-		unsigned niov_idx, area_idx;
 		netmem_ref netmem;
 
-		area_idx = rqe->off >> IORING_ZCRX_AREA_SHIFT;
-		niov_idx = (rqe->off & ~IORING_ZCRX_AREA_MASK) >> ifq->niov_shift;
-
-		if (unlikely(rqe->__pad || area_idx))
+		if (!io_parse_rqe(rqe, ifq, &niov))
 			continue;
-		area = ifq->area;
-
-		if (unlikely(niov_idx >= area->nia.num_niovs))
-			continue;
-		niov_idx = array_index_nospec(niov_idx, area->nia.num_niovs);
-
-		niov = &area->nia.niovs[niov_idx];
 		if (!io_zcrx_put_niov_uref(niov))
 			continue;
 
Signature · 1 line
2.49.0
author maintainer
#2 lore raw

PATCH 2/2 io_uring/zcrx: allow synchronous buffer return

In reply to Pavel Begunkov

Returning buffers via a ring is performant and convenient, but it becomes a problem when/if the user misconfigured the ring size and it becomes full. Add a synchronous way to return buffers back to the page pool via a new register opcode. It's supposed to be a reliable slow path for refilling.

---
 include/uapi/linux/io_uring.h | 10 ++++++
 io_uring/register.c           |  3 ++
 io_uring/zcrx.c               | 64 +++++++++++++++++++++++++++++++++++
 io_uring/zcrx.h               |  7 ++++
 4 files changed, 84 insertions(+)
include/uapi/linux/io_uring.h +10 -0
index 6957dc539d83..97b206df4cc1 100644
@@ -665,6 +665,9 @@ enum io_uring_register_op {
 
 	IORING_REGISTER_MEM_REGION		= 34,
 
+	/* return zcrx buffers back into circulation */
+	IORING_REGISTER_ZCRX_REFILL		= 35,
+
 	/* this goes last */
 	IORING_REGISTER_LAST,
 
@@ -1046,6 +1049,13 @@ struct io_uring_zcrx_ifq_reg {
 	__u64	__resv[3];
 };
 
+struct io_uring_zcrx_refill {
+	__u32		zcrx_id;
+	__u32		nr_entries;
+	__u64		rqes;
+	__u64		__resv[2];
+};
+
 #ifdef __cplusplus
 }
 #endif
io_uring/register.c +3 -0
index a59589249fce..5155ea627f65 100644
@@ -835,6 +835,9 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,
 			break;
 		ret = io_register_mem_region(ctx, arg);
 		break;
+	case IORING_REGISTER_ZCRX_REFILL:
+		ret = io_zcrx_return_bufs(ctx, arg, nr_args);
+		break;
 	default:
 		ret = -EINVAL;
 		break;
io_uring/zcrx.c +64 -0

70 lines not on this page · raw message

io_uring/zcrx.h +7 -0
index a48871b5adad..33ef61503092 100644
@@ -63,6 +63,8 @@ struct io_zcrx_ifq {
 };
 
 #if defined(CONFIG_IO_URING_ZCRX)
+int io_zcrx_return_bufs(struct io_ring_ctx *ctx,
+			void __user *arg, unsigned nr_arg);
 int io_register_zcrx_ifq(struct io_ring_ctx *ctx,
 			 struct io_uring_zcrx_ifq_reg __user *arg);
 void io_unregister_zcrx_ifqs(struct io_ring_ctx *ctx);
@@ -95,6 +97,11 @@ static inline struct io_mapped_region *io_zcrx_get_region(struct io_ring_ctx *ct
 {
 	return NULL;
 }
+static inline int io_zcrx_return_bufs(struct io_ring_ctx *ctx,
+				      void __user *arg, unsigned nr_arg)
+{
+	return -EOPNOTSUPP;
+}
 #endif
 
 int io_recvzc(struct io_kiocb *req, unsigned int issue_flags);
Signature · 1 line
2.49.0
maintainer Applied
#3 lore raw
In reply to Pavel Begunkov
Quoted 9 lines · Pavel Begunkov

Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.

For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next

[...]

Applied, thanks!

[1/2] io_uring/zcrx: introduce io_parse_rqe()

      commit: 55e5f6bb0ddfbb5912fec373ed99f9e02b19e3ea
[2/2] io_uring/zcrx: allow synchronous buffer return
      commit: ebbedae5b04f4f4a4d79d2d7329baab74b5a0564

Best regards,

Signature · 1 line
Jens Axboe
author maintainer
#4 lore raw
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Pavel Begunkov

Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.

Quoted 6 lines
Quoted 4 lines

For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next

[...]

Applied, thanks!

Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.

Signature · 1 line
Pavel Begunkov
maintainer
#5 lore raw
Quoted 11 lines · Pavel Begunkov
Quoted 10 lines · Jens Axboe
Quoted 8 lines · Pavel Begunkov

Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.

For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

[...]

Applied, thanks!

Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.

Alright, but then please make it clear when you are sending something out for merging.

Signature · 1 line
Jens Axboe
author maintainer
#6 lore raw
Quoted 16 lines · Jens Axboe
Quoted 15 lines · Pavel Begunkov
Quoted 13 lines · Jens Axboe
Quoted 9 lines · Pavel Begunkov

Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.

For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next

[...]

Applied, thanks!

Quoted 7 lines
Quoted 4 lines

Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.

Alright, but then please make it clear when you are sending something out for merging.

That would be a pull request, should be obvious enough. Perhaps it'd make sense to tag for review only patches. Any preference what tag it should be or otherwise?

Signature · 1 line
Pavel Begunkov
maintainer
#7 lore raw
Quoted 22 lines · Pavel Begunkov
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Pavel Begunkov
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Pavel Begunkov

Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.

Quoted 6 lines
Quoted 4 lines

For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next

[...]

Applied, thanks!

Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.

Quoted 6 lines
Quoted 2 lines

Alright, but then please make it clear when you are sending something out for merging.

That would be a pull request, should be obvious enough. Perhaps it'd make sense to tag for review only patches. Any preference what tag it should be or otherwise?

Please just send patches rather than a pull request. PRs get messed up by people all the time, and there's no picking and choosing there, it's all or nothing.

A review tag would be good in the subject, anything easily recognizable should do it. Like for-review or whatever, doesn't matter to me.

Signature · 1 line
Jens Axboe