trivial fixes and a cleanup of provided buffers
cover trivial fixes and a cleanup of provided buffers
Jens,
I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.
thanks,
Gabriel Krisman Bertazi (3):
io_uring: Fix check of BID wrapping in provided buffers
io_uring: Allow the full buffer id space for provided buffers
io_uring: Use slab for struct io_buffer objects
include/linux/io_uring_types.h | 2 -- io_uring/io_uring.c | 4 ++- io_uring/io_uring.h | 1 + io_uring/kbuf.c | 58 +++++++++++++++++++--------------- 4 files changed, 37 insertions(+), 28 deletions(-)
Signature · 1 line
2.42.0
PATCH 1/3 io_uring: Fix check of BID wrapping in provided buffers
Commit 3851d25c75ed0 ("io_uring: check for rollover of buffer ID when providing buffers") introduced a check to prevent wrapping the BID counter when sqe->off is provided, but it's off-by-one too restrictive, rejecting the last possible BID (65534).
i.e., the following fails with -EINVAL.
io_uring_prep_provide_buffers(sqe, addr, size, 0xFFFF, 0, 0);
--- io_uring/kbuf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/kbuf.c +1 -1
@@ -352,7 +352,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe tmp =READ_ONCE (sqe->off);if (tmp >USHRT_MAX )return -E2BIG ; -if (tmp + p->nbufs >=USHRT_MAX ) +if (tmp + p->nbufs >USHRT_MAX )return -EINVAL ; p->bid = tmp;return 0 ;
Signature · 1 line
2.42.0
PATCH 2/3 io_uring: Allow the full buffer id space for provided buffers
In reply to Gabriel Krisman Bertazinbufs tracks the number of buffers and not the last bgid. In 16-bit, we have 2^16 valid buffers, but the check mistakenly rejects the last bid. Let's fix it to make the interface consistent with the documentation.
--- io_uring/kbuf.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
io_uring/kbuf.c +7 -4
@@ -19,12 +19,15 @@#define BGID_ARRAY 64 +/* BIDs are addressed by a 16-bit field in a CQE */ +#define MAX_BIDS_PER_BGID (1 << 16) +struct io_provide_buf {struct file *file;__u64 addr;__u32 len;__u32 bgid; -__u16 nbufs; +__u32 nbufs;__u16 bid; };
@@ -289,7 +292,7 @@ int io_remove_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)return -EINVAL ; tmp =READ_ONCE (sqe->fd); -if (!tmp || tmp >USHRT_MAX ) +if (!tmp || tmp >MAX_BIDS_PER_BGID )return -EINVAL ;memset (p,0 ,sizeof (*p));
@@ -332,7 +335,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqereturn -EINVAL ; tmp =READ_ONCE (sqe->fd); -if (!tmp || tmp >USHRT_MAX ) +if (!tmp || tmp >MAX_BIDS_PER_BGID )return -E2BIG ; p->nbufs = tmp; p->addr =READ_ONCE (sqe->addr);
@@ -352,7 +355,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe tmp =READ_ONCE (sqe->off);if (tmp >USHRT_MAX )return -E2BIG ; -if (tmp + p->nbufs >USHRT_MAX ) +if (tmp + p->nbufs >MAX_BIDS_PER_BGID )return -EINVAL ; p->bid = tmp;return 0 ;
Signature · 1 line
2.42.0
PATCH 3/3 io_uring: Use slab for struct io_buffer objects
In reply to Gabriel Krisman BertaziThe allocation of struct io_buffer for metadata of provided buffers is done through a custom allocator that directly gets pages and fragments them. But, slab would do just fine, as this is not a hot path (in fact, it is a deprecated feature) and, by keeping a custom allocator implementation we lose benefits like tracking, poisoning, sanitizers. Finally, the custom code is more complex and requires keeping the list of pages in struct ctx for no good reason. This patch cleans this path up and just uses slab.
I microbenchmarked it by forcing the allocation of a large number of objects with the least number of io_uring commands possible (keeping nbufs=USHRT_MAX), with and without the patch. There is a slight increase in time spent in the allocation with slab, of course, but even when allocating to system resources exhaustion, which is not very realistic and happened around 1/2 billion provided buffers for me, it wasn't a significant hit in system time. Specially if we think of a real-world scenario, an application doing register/unregister of provided buffers will hit ctx->io_buffers_cache more often than actually going to slab.
---
changes from v1
- reduce batch size to limit stack usage. (Jeff)
- Don't check kmem_cache_alloc_bulk return for < 0 (Jeff)
--- include/linux/io_uring_types.h | 2 -- io_uring/io_uring.c | 4 ++- io_uring/io_uring.h | 1 + io_uring/kbuf.c | 47 +++++++++++++++++++--------------- 4 files changed, 30 insertions(+), 24 deletions(-)
include/linux/io_uring_types.h +0 -2
@@ -342,8 +342,6 @@ struct io_ring_ctx {struct wait_queue_head rsrc_quiesce_wq;unsigned rsrc_quiesce; -struct list_head io_buffers_pages; -#if defined(CONFIG_UNIX )struct socket *ring_sock;#endif
io_uring/io_uring.c +3 -1
@@ -338,7 +338,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)spin_lock_init (&ctx->completion_lock);spin_lock_init (&ctx->timeout_lock);INIT_WQ_LIST (&ctx->iopoll_list); -INIT_LIST_HEAD (&ctx->io_buffers_pages);INIT_LIST_HEAD (&ctx->io_buffers_comp);INIT_LIST_HEAD (&ctx->defer_list);INIT_LIST_HEAD (&ctx->timeout_list);
@@ -4681,6 +4680,9 @@ static int __init io_uring_init(void)SLAB_ACCOUNT |SLAB_TYPESAFE_BY_RCU , offsetof(struct io_kiocb, cmd.data), sizeof_field(struct io_kiocb, cmd.data), NULL); + io_buf_cachep =kmem_cache_create ("io_buffer" ,sizeof (struct io_buffer ),0 , +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , +NULL );#ifdef CONFIG_SYSCTL register_sysctl_init ("kernel" , kernel_io_uring_disabled_table);
io_uring/io_uring.h +1 -0
@@ -330,6 +330,7 @@ static inline bool io_req_cache_empty(struct io_ring_ctx *ctx) }extern struct kmem_cache *req_cachep; +extern struct kmem_cache *io_buf_cachep;static inline struct io_kiocb *io_extract_req (struct io_ring_ctx *ctx) {
io_uring/kbuf.c +26 -21
83 lines not on this page · raw message
Signature · 1 line
2.42.0
Jens Axboe
maintainerQuoted 7 lines · Gabriel Krisman Bertazi
Jens,
I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.
Thanks, this looks good. For patches 1-2, do you have test cases you could send for liburing?
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 7 lines · Gabriel Krisman Bertazi
Jens,
I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.
Thanks, this looks good. For patches 1-2, do you have test cases you could send for liburing?
Sure. I'll adapt them to liburing and submit shortly.
Signature · 1 line
Gabriel Krisman Bertazi
Quoted 9 lines · Gabriel Krisman Bertazi
Jens,
I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.
[...]
Applied, thanks!
[1/3] io_uring: Fix check of BID wrapping in provided buffers
commit: ab69838e7c75b0edb699c1a8f42752b30333c46f
[2/3] io_uring: Allow the full buffer id space for provided buffers
commit: f74c746e476b9dad51448b9a9421aae72b60e25f
[3/3] io_uring: Use slab for struct io_buffer objects
commit: b3a4dbc89d4021b3f90ff6a13537111a004f9d07Best regards,
Signature · 1 line
Jens Axboe