Skip to contentSkip to conversation

trivial fixes and a cleanup of provided buffers

PATCH Applied Gabriel Krisman Bertazi 2 participants 7 messages Oct 5, 2023 io_uring

lore
Share & tools
lore
#0 lore raw

cover trivial fixes and a cleanup of provided buffers

Jens,

I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.

thanks,

Gabriel Krisman Bertazi (3):
io_uring: Fix check of BID wrapping in provided buffers
io_uring: Allow the full buffer id space for provided buffers
io_uring: Use slab for struct io_buffer objects

 include/linux/io_uring_types.h |  2 --
 io_uring/io_uring.c            |  4 ++-
 io_uring/io_uring.h            |  1 +
 io_uring/kbuf.c                | 58 +++++++++++++++++++---------------
 4 files changed, 37 insertions(+), 28 deletions(-)
Signature · 1 line
2.42.0
#1 lore raw

PATCH 1/3 io_uring: Fix check of BID wrapping in provided buffers

Commit 3851d25c75ed0 ("io_uring: check for rollover of buffer ID when providing buffers") introduced a check to prevent wrapping the BID counter when sqe->off is provided, but it's off-by-one too restrictive, rejecting the last possible BID (65534).

i.e., the following fails with -EINVAL.

     io_uring_prep_provide_buffers(sqe, addr, size, 0xFFFF, 0, 0);
Fixes 3851d25c75ed ("io_uring: check for rollover of buffer ID when providing buffers")
---
 io_uring/kbuf.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/kbuf.c +1 -1
index 556f4df25b0f..52dba81c3f50 100644
@@ -352,7 +352,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe
 	tmp = READ_ONCE(sqe->off);
 	if (tmp > USHRT_MAX)
 		return -E2BIG;
-	if (tmp + p->nbufs >= USHRT_MAX)
+	if (tmp + p->nbufs > USHRT_MAX)
 		return -EINVAL;
 	p->bid = tmp;
 	return 0;
Signature · 1 line
2.42.0
#2 lore raw

PATCH 2/3 io_uring: Allow the full buffer id space for provided buffers

In reply to Gabriel Krisman Bertazi

nbufs tracks the number of buffers and not the last bgid. In 16-bit, we have 2^16 valid buffers, but the check mistakenly rejects the last bid. Let's fix it to make the interface consistent with the documentation.

Fixes ddf0322db79c ("io_uring: add IORING_OP_PROVIDE_BUFFERS")
---
 io_uring/kbuf.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
io_uring/kbuf.c +7 -4
index 52dba81c3f50..12a357348733 100644
@@ -19,12 +19,15 @@
 
 #define BGID_ARRAY	64
 
+/* BIDs are addressed by a 16-bit field in a CQE */
+#define MAX_BIDS_PER_BGID (1 << 16)
+
 struct io_provide_buf {
 	struct file			*file;
 	__u64				addr;
 	__u32				len;
 	__u32				bgid;
-	__u16				nbufs;
+	__u32				nbufs;
 	__u16				bid;
 };
 
@@ -289,7 +292,7 @@ int io_remove_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 		return -EINVAL;
 
 	tmp = READ_ONCE(sqe->fd);
-	if (!tmp || tmp > USHRT_MAX)
+	if (!tmp || tmp > MAX_BIDS_PER_BGID)
 		return -EINVAL;
 
 	memset(p, 0, sizeof(*p));
@@ -332,7 +335,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe
 		return -EINVAL;
 
 	tmp = READ_ONCE(sqe->fd);
-	if (!tmp || tmp > USHRT_MAX)
+	if (!tmp || tmp > MAX_BIDS_PER_BGID)
 		return -E2BIG;
 	p->nbufs = tmp;
 	p->addr = READ_ONCE(sqe->addr);
@@ -352,7 +355,7 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe
 	tmp = READ_ONCE(sqe->off);
 	if (tmp > USHRT_MAX)
 		return -E2BIG;
-	if (tmp + p->nbufs > USHRT_MAX)
+	if (tmp + p->nbufs > MAX_BIDS_PER_BGID)
 		return -EINVAL;
 	p->bid = tmp;
 	return 0;
Signature · 1 line
2.42.0
#3 lore raw

PATCH 3/3 io_uring: Use slab for struct io_buffer objects

In reply to Gabriel Krisman Bertazi

The allocation of struct io_buffer for metadata of provided buffers is done through a custom allocator that directly gets pages and fragments them. But, slab would do just fine, as this is not a hot path (in fact, it is a deprecated feature) and, by keeping a custom allocator implementation we lose benefits like tracking, poisoning, sanitizers. Finally, the custom code is more complex and requires keeping the list of pages in struct ctx for no good reason. This patch cleans this path up and just uses slab.

I microbenchmarked it by forcing the allocation of a large number of objects with the least number of io_uring commands possible (keeping nbufs=USHRT_MAX), with and without the patch. There is a slight increase in time spent in the allocation with slab, of course, but even when allocating to system resources exhaustion, which is not very realistic and happened around 1/2 billion provided buffers for me, it wasn't a significant hit in system time. Specially if we think of a real-world scenario, an application doing register/unregister of provided buffers will hit ctx->io_buffers_cache more often than actually going to slab.

---
changes from v1
- reduce batch size to limit stack usage. (Jeff)
- Don't check kmem_cache_alloc_bulk return for < 0 (Jeff)

---
 include/linux/io_uring_types.h |  2 --
 io_uring/io_uring.c            |  4 ++-
 io_uring/io_uring.h            |  1 +
 io_uring/kbuf.c                | 47 +++++++++++++++++++---------------
 4 files changed, 30 insertions(+), 24 deletions(-)
include/linux/io_uring_types.h +0 -2
index 13d19b9be9f4..9dca731b1ca8 100644
@@ -342,8 +342,6 @@ struct io_ring_ctx {
 	struct wait_queue_head		rsrc_quiesce_wq;
 	unsigned			rsrc_quiesce;
 
-	struct list_head		io_buffers_pages;
-
 	#if defined(CONFIG_UNIX)
 		struct socket		*ring_sock;
 	#endif
io_uring/io_uring.c +3 -1
index 783ed0fff71b..548c32fc1e28 100644
@@ -338,7 +338,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
 	spin_lock_init(&ctx->completion_lock);
 	spin_lock_init(&ctx->timeout_lock);
 	INIT_WQ_LIST(&ctx->iopoll_list);
-	INIT_LIST_HEAD(&ctx->io_buffers_pages);
 	INIT_LIST_HEAD(&ctx->io_buffers_comp);
 	INIT_LIST_HEAD(&ctx->defer_list);
 	INIT_LIST_HEAD(&ctx->timeout_list);
@@ -4681,6 +4680,9 @@ static int __init io_uring_init(void)
 				SLAB_ACCOUNT | SLAB_TYPESAFE_BY_RCU,
 				offsetof(struct io_kiocb, cmd.data),
 				sizeof_field(struct io_kiocb, cmd.data), NULL);
+	io_buf_cachep = kmem_cache_create("io_buffer", sizeof(struct io_buffer), 0,
+					  SLAB_HWCACHE_ALIGN | SLAB_PANIC | SLAB_ACCOUNT,
+					  NULL);
 
 #ifdef CONFIG_SYSCTL
 	register_sysctl_init("kernel", kernel_io_uring_disabled_table);
io_uring/io_uring.h +1 -0
index 547c30582fb8..2ff719ae1b57 100644
@@ -330,6 +330,7 @@ static inline bool io_req_cache_empty(struct io_ring_ctx *ctx)
 }
 
 extern struct kmem_cache *req_cachep;
+extern struct kmem_cache *io_buf_cachep;
 
 static inline struct io_kiocb *io_extract_req(struct io_ring_ctx *ctx)
 {
io_uring/kbuf.c +26 -21

83 lines not on this page · raw message

Signature · 1 line
2.42.0
maintainer
#4 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 7 lines · Gabriel Krisman Bertazi

Jens,

I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.

Thanks, this looks good. For patches 1-2, do you have test cases you could send for liburing?

Signature · 1 line
Jens Axboe
#5 lore raw
Quoted 11 lines · Jens Axboe
Quoted 7 lines · Gabriel Krisman Bertazi

Jens,

I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.

Thanks, this looks good. For patches 1-2, do you have test cases you could send for liburing?

Sure. I'll adapt them to liburing and submit shortly.

Signature · 1 line
Gabriel Krisman Bertazi
maintainer Applied
#6 lore raw
In reply to Gabriel Krisman Bertazi
Quoted 9 lines · Gabriel Krisman Bertazi

Jens,

I'm resubmitting the slab conversion for provided buffers with the suggestions from Jeff (thanks!) for your consideration, and appended 2 minor fixes related to kbuf in the patchset. Since the patchset grew from 1 to 3 patches, i pretended it is not a v2 and restart the counting from 1.

[...]

Applied, thanks!

[1/3] io_uring: Fix check of BID wrapping in provided buffers

      commit: ab69838e7c75b0edb699c1a8f42752b30333c46f
[2/3] io_uring: Allow the full buffer id space for provided buffers
      commit: f74c746e476b9dad51448b9a9421aae72b60e25f
[3/3] io_uring: Use slab for struct io_buffer objects
      commit: b3a4dbc89d4021b3f90ff6a13537111a004f9d07

Best regards,

Signature · 1 line
Jens Axboe