io_uring: Use slab for struct io_buffer objects
PATCH 1/1 io_uring: Use slab for struct io_buffer objects
The allocation of struct io_buffer for metadata of provided buffers is done through a custom allocator that directly gets pages and fragments them. But, slab would do just fine, as this is not a hot path (in fact, it is a deprecated feature) and, by keeping a custom allocator implementation we lose benefits like tracking, poisoning, sanitizers. Finally, the custom code is more complex and requires keeping the list of pages in struct ctx for no good reason. This patch cleans this path up and just uses slab.
I microbenchmarked it by forcing the allocation of a large number of objects with the least number of io_uring commands possible (keeping nbufs=USHRT_MAX), with and without the patch. There is a slight increase in time spent in the allocation with slab, of course, but even when allocating to system resources exhaustion, which is not very realistic and happened around 1/2 billion provided buffers for me, it wasn't a significant hit in system time. Specially if we think of a real-world scenario, an application doing register/unregister of provided buffers will hit ctx->io_buffers_cache more often than actually going to slab.
--- include/linux/io_uring_types.h | 2 -- io_uring/io_uring.c | 5 +++- io_uring/io_uring.h | 1 + io_uring/kbuf.c | 47 +++++++++++++++++++--------------- 4 files changed, 31 insertions(+), 24 deletions(-)
include/linux/io_uring_types.h +0 -2
@@ -346,8 +346,6 @@ struct io_ring_ctx {struct wait_queue_head rsrc_quiesce_wq;unsigned rsrc_quiesce; -struct list_head io_buffers_pages; -#if defined(CONFIG_UNIX )struct socket *ring_sock;#endif
io_uring/io_uring.c +4 -1
@@ -314,7 +314,6 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)spin_lock_init (&ctx->completion_lock);spin_lock_init (&ctx->timeout_lock);INIT_WQ_LIST (&ctx->iopoll_list); -INIT_LIST_HEAD (&ctx->io_buffers_pages);INIT_LIST_HEAD (&ctx->io_buffers_comp);INIT_LIST_HEAD (&ctx->defer_list);INIT_LIST_HEAD (&ctx->timeout_list);
@@ -4622,6 +4621,10 @@ static int __init io_uring_init(void) offsetof(struct io_kiocb, cmd.data), sizeof_field(struct io_kiocb, cmd.data), NULL); + io_buf_cachep =kmem_cache_create ("io_buffer" ,sizeof (struct io_buffer ),0 , +SLAB_HWCACHE_ALIGN |SLAB_PANIC |SLAB_ACCOUNT , +NULL ); +return 0 ; };__initcall (io_uring_init);
io_uring/io_uring.h +1 -0
@@ -338,6 +338,7 @@ static inline bool io_req_cache_empty(struct io_ring_ctx *ctx) }extern struct kmem_cache *req_cachep; +extern struct kmem_cache *io_buf_cachep;static inline struct io_kiocb *io_extract_req (struct io_ring_ctx *ctx) {
io_uring/kbuf.c +26 -21
83 lines not on this page · raw message
Signature · 1 line
2.41.0
Quoted 20 lines · Gabriel Krisman Bertazi
The allocation of struct
io_bufferfor metadata of provided buffers is done through a custom allocator that directly gets pages and fragments them. But, slab would do just fine, as this is not a hot path (in fact, it is a deprecated feature) and, by keeping a custom allocator implementation we lose benefits like tracking, poisoning, sanitizers. Finally, the custom code is more complex and requires keeping the list of pages in struct ctx for no good reason. This patch cleans this path up and just uses slab.I microbenchmarked it by forcing the allocation of a large number of objects with the least number of
io_uringcommands possible (keeping nbufs=USHRT_MAX), with and without the patch. There is a slight increase in time spent in the allocation with slab, of course, but even when allocating to system resources exhaustion, which is not very realistic and happened around 1/2 billion provided buffers for me, it wasn't a significant hit in system time. Specially if we think of a real-world scenario, an application doing register/unregister of provided buffers will hitctx->io_buffers_cachemore often than actually going to slab.
Quoted 1 line
Signed-off-by Gabriel Krisman Bertazi <[email protected]>
Hi Jens,
Any feedback on this?
Signature · 1 line
Gabriel Krisman Bertazi
Hi, Gabriel,
I just have a couple of comments. I don't have an opinion on whether it makes sense to replace the existing allocator.
-Jeff
io_uring/kbuf.c int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe · line 370
Quoted 12 lines
io_uring/kbuf.c +3
-3@@ -362,11 +363,12 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqereturn 0 ; } +#define IO_BUFFER_ALLOC_BATCH (PAGE_SIZE/sizeof(struct io_buffer)) +static int io_refill_buffer_cache (struct io_ring_ctx *ctx) { -struct io_buffer *buf; -struct page *page; -int bufs_in_page; +struct io_buffer *bufs[IO_BUFFER_ALLOC_BATCH ];
That's a pretty large on-stack allocation.
io_uring/kbuf.c static int io_refill_buffer_cache(struct io_ring_ctx *ctx) · line 396
Quoted 3 lines
io_uring/kbuf.c +3
-0+ allocated =kmem_cache_alloc_bulk (io_buf_cachep,GFP_KERNEL_ACCOUNT , +ARRAY_SIZE (bufs), (void **) bufs); +if (unlikely (allocated <=0 )) {
Can't be less than 0.
Quoted 11 lines · Jeff Moyer
Hi, Gabriel,
I just have a couple of comments. I don't have an opinion on whether it makes sense to replace the existing allocator.
-Jeff
Quoted 4 lines
io_uring/kbuf.c +0
-0@@ -362,11 +363,12 @@ int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqereturn 0 ; }
Quoted 10 lines
io_uring/kbuf.c int io_provide_buffers_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe · line 370
Quoted 8 lines
io_uring/kbuf.c +3
-3+#define IO_BUFFER_ALLOC_BATCH (PAGE_SIZE/sizeof(struct io_buffer)) +static int io_refill_buffer_cache (struct io_ring_ctx *ctx) { -struct io_buffer *buf; -struct page *page; -int bufs_in_page; +struct io_buffer *bufs[IO_BUFFER_ALLOC_BATCH ];That's a pretty large on-stack allocation.
Indeed, that is definitely too large. Thanks for pointing it out.
Also, I just noticed the define above should actually read:
#define IO_BUFFER_ALLOC_BATCH (PAGE_SIZE/sizeof(struct io_buffer *))
I'll follow up with a v2, after I retest the impact a smaller allocation batch would have.
Quoted 6 lines
io_uring/kbuf.c static int io_refill_buffer_cache(struct io_ring_ctx *ctx) · line 396
Quoted 4 lines
io_uring/kbuf.c +3
-0+ allocated =kmem_cache_alloc_bulk (io_buf_cachep,GFP_KERNEL_ACCOUNT , +ARRAY_SIZE (bufs), (void **) bufs); +if (unlikely (allocated <=0 )) {Can't be less than 0.
Thanks, will fix.
Quoted 1 line
Signature · 1 line
Gabriel Krisman Bertazi