add support for synchronous refill
Subsystems (2)
Pavel Begunkov
maintainercover add support for synchronous refill
Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via io_uring "register" syscall, which should serve as a slow fallback path.
For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next
Pavel Begunkov (2):
io_uring/zcrx: introduce io_parse_rqe()
io_uring/zcrx: allow synchronous buffer return
include/uapi/linux/io_uring.h | 10 ++++ io_uring/register.c | 3 + io_uring/zcrx.c | 100 +++++++++++++++++++++++++++++----- io_uring/zcrx.h | 7 +++ 4 files changed, 107 insertions(+), 13 deletions(-)
Signature · 1 line
2.49.0
Pavel Begunkov
maintainerPATCH 1/2 io_uring/zcrx: introduce io_parse_rqe()
Add a helper for verifying a rqe and extracting a niov out of it. It'll be reused in following patches.
--- io_uring/zcrx.c | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-)
io_uring/zcrx.c +23 -13
@@ -727,6 +727,28 @@ static struct io_uring_zcrx_rqe *io_zcrx_get_rqe(struct io_zcrx_ifq *ifq,return &ifq->rqes[idx]; } +static inline bool io_parse_rqe (struct io_uring_zcrx_rqe *rqe, +struct io_zcrx_ifq *ifq, +struct net_iov **ret_niov) +{ +unsigned niov_idx, area_idx; +struct io_zcrx_area *area; + + area_idx = rqe->off >>IORING_ZCRX_AREA_SHIFT ; + niov_idx = (rqe->off & ~IORING_ZCRX_AREA_MASK ) >> ifq->niov_shift; + +if (unlikely (rqe->__pad || area_idx)) +return false; + area = ifq->area; + +if (unlikely (niov_idx >= area->nia.num_niovs)) +return false; + niov_idx =array_index_nospec (niov_idx, area->nia.num_niovs); + + *ret_niov = &area->nia.niovs[niov_idx]; +return true; +} +static void io_zcrx_ring_refill (struct page_pool *pp,struct io_zcrx_ifq *ifq) {
@@ -741,23 +763,11 @@ static void io_zcrx_ring_refill(struct page_pool *pp, do { struct io_uring_zcrx_rqe *rqe = io_zcrx_get_rqe(ifq, mask); - struct io_zcrx_area *area; struct net_iov *niov; - unsigned niov_idx, area_idx; netmem_ref netmem; - area_idx = rqe->off >> IORING_ZCRX_AREA_SHIFT; - niov_idx = (rqe->off & ~IORING_ZCRX_AREA_MASK) >> ifq->niov_shift; - - if (unlikely(rqe->__pad || area_idx)) + if (!io_parse_rqe(rqe, ifq, &niov)) continue; - area = ifq->area; - - if (unlikely(niov_idx >= area->nia.num_niovs)) - continue; - niov_idx = array_index_nospec(niov_idx, area->nia.num_niovs); - - niov = &area->nia.niovs[niov_idx]; if (!io_zcrx_put_niov_uref(niov)) continue;
Signature · 1 line
2.49.0
Pavel Begunkov
maintainerPATCH 2/2 io_uring/zcrx: allow synchronous buffer return
In reply to Pavel BegunkovReturning buffers via a ring is performant and convenient, but it becomes a problem when/if the user misconfigured the ring size and it becomes full. Add a synchronous way to return buffers back to the page pool via a new register opcode. It's supposed to be a reliable slow path for refilling.
--- include/uapi/linux/io_uring.h | 10 ++++++ io_uring/register.c | 3 ++ io_uring/zcrx.c | 64 +++++++++++++++++++++++++++++++++++ io_uring/zcrx.h | 7 ++++ 4 files changed, 84 insertions(+)
include/uapi/linux/io_uring.h +10 -0
@@ -665,6 +665,9 @@ enum io_uring_register_op {IORING_REGISTER_MEM_REGION =34 , +/* return zcrx buffers back into circulation */ +IORING_REGISTER_ZCRX_REFILL =35 , +/* this goes last */ IORING_REGISTER_LAST,
@@ -1046,6 +1049,13 @@ struct io_uring_zcrx_ifq_reg {__u64 __resv[3 ]; }; +struct io_uring_zcrx_refill { +__u32 zcrx_id; +__u32 nr_entries; +__u64 rqes; +__u64 __resv[2 ]; +}; + #ifdef __cplusplus }#endif
io_uring/register.c +3 -0
@@ -835,6 +835,9 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_mem_region (ctx, arg);break ; +case IORING_REGISTER_ZCRX_REFILL : + ret =io_zcrx_return_bufs (ctx, arg, nr_args); +break ;default : ret = -EINVAL ;break ;
io_uring/zcrx.c +64 -0
70 lines not on this page · raw message
io_uring/zcrx.h +7 -0
@@ -63,6 +63,8 @@ struct io_zcrx_ifq { };#if defined(CONFIG_IO_URING_ZCRX ) +int io_zcrx_return_bufs (struct io_ring_ctx *ctx, +void __user *arg,unsigned nr_arg);int io_register_zcrx_ifq (struct io_ring_ctx *ctx,struct io_uring_zcrx_ifq_reg __user *arg);void io_unregister_zcrx_ifqs (struct io_ring_ctx *ctx);
@@ -95,6 +97,11 @@ static inline struct io_mapped_region *io_zcrx_get_region(struct io_ring_ctx *ct {return NULL ; } +static inline int io_zcrx_return_bufs (struct io_ring_ctx *ctx, +void __user *arg,unsigned nr_arg) +{ +return -EOPNOTSUPP ; +}#endif int io_recvzc (struct io_kiocb *req,unsigned int issue_flags);
Signature · 1 line
2.49.0
Quoted 9 lines · Pavel Begunkov
Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via
io_uring"register" syscall, which should serve as a slow fallback path.For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next[...]
Applied, thanks!
[1/2] io_uring/zcrx: introduce io_parse_rqe()
commit: 55e5f6bb0ddfbb5912fec373ed99f9e02b19e3ea
[2/2] io_uring/zcrx: allow synchronous buffer return
commit: ebbedae5b04f4f4a4d79d2d7329baab74b5a0564Best regards,
Signature · 1 line
Jens Axboe
Pavel Begunkov
maintainerQuoted 7 lines · Jens Axboe
Quoted 5 lines · Pavel Begunkov
Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via
io_uring"register" syscall, which should serve as a slow fallback path.
Quoted 6 lines
Quoted 4 lines
For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next[...]
Applied, thanks!
Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 11 lines · Pavel Begunkov
Quoted 10 lines · Jens Axboe
Quoted 8 lines · Pavel Begunkov
Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via
io_uring"register" syscall, which should serve as a slow fallback path.For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
[...]
Applied, thanks!
Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.
Alright, but then please make it clear when you are sending something out for merging.
Signature · 1 line
Jens Axboe
Pavel Begunkov
maintainerQuoted 16 lines · Jens Axboe
Quoted 15 lines · Pavel Begunkov
Quoted 13 lines · Jens Axboe
Quoted 9 lines · Pavel Begunkov
Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via
io_uring"register" syscall, which should serve as a slow fallback path.For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next[...]
Applied, thanks!
Quoted 7 lines
Quoted 4 lines
Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.
Alright, but then please make it clear when you are sending something out for merging.
That would be a pull request, should be obvious enough. Perhaps it'd make sense to tag for review only patches. Any preference what tag it should be or otherwise?
Signature · 1 line
Pavel Begunkov
Jens Axboe
maintainerQuoted 22 lines · Pavel Begunkov
Quoted 21 lines · Jens Axboe
Quoted 19 lines · Pavel Begunkov
Quoted 7 lines · Jens Axboe
Quoted 5 lines · Pavel Begunkov
Returning buffers via a ring is efficient but can cause problems when the ring doesn't have space. Add a way to return buffers synchronously via
io_uring"register" syscall, which should serve as a slow fallback path.Quoted 6 lines
Quoted 4 lines
For a full branch with all relevant dependencies see
https://github.com/isilence/linux.git zcrx/for-next[...]
Applied, thanks!
Leave these and other series out please. I sent them for review, but it'll be easier to keep in a branch and rebase it if necessary. I hoped tagging with zcrx-next would be good enough of a sign.
Quoted 6 lines
Quoted 2 lines
Alright, but then please make it clear when you are sending something out for merging.
That would be a pull request, should be obvious enough. Perhaps it'd make sense to tag for review only patches. Any preference what tag it should be or otherwise?
Please just send patches rather than a pull request. PRs get messed up by people all the time, and there's no picking and choosing there, it's all or nothing.
A review tag would be good in the subject, anything easily recognizable should do it. Like for-review or whatever, doesn't matter to me.
Signature · 1 line
Jens Axboe