Clean up alloc_cache allocations
v2 cover Clean up alloc_cache allocations
Hi, Jens, Pavel.
I sent this v2 originally during US thanksgiving week, so I'm resending now under a more suitable time, rebased and retested on top of Jen's for-6.14/io_uring branch. It keeps the changes requested in v1:
renaming the allocation helper and introducing a callback instead of zeroing the entire object, as suggested by Jens.
This was tested against liburing testsuite, with lockdep and KASAN enabled.
For v1, please see:
https://lore.kernel.org/io-uring/[email protected]/T/#t
Thanks,
Gabriel Krisman Bertazi (9):
io_uring: Fold allocation into alloc_cache helper
io_uring: Add generic helper to allocate async data io_uring/futex: Allocate ifd with generic alloc_cache helper io_uring/poll: Allocate apoll with generic alloc_cache helper io_uring/uring_cmd: Allocate async data through generic helper io_uring/net: Allocate msghdr async data through helper io_uring/rw: Allocate async data through helper
io_uring: Move old async data allocation helper to header io_uring/msg_ring: Drop custom destructor
io_uring/alloc_cache.h | 13 +++++++++++++ io_uring/futex.c | 13 +------------ io_uring/io_uring.c | 17 ++--------------- io_uring/io_uring.h | 23 +++++++++++++++++++++++ io_uring/msg_ring.c | 7 ------- io_uring/msg_ring.h | 1 - io_uring/net.c | 35 ++++++++++++++++++----------------- io_uring/poll.c | 13 +++++-------- io_uring/rw.c | 36 ++++++++++++++++-------------------- io_uring/timeout.c | 5 ++--- io_uring/uring_cmd.c | 20 ++------------------ io_uring/waitid.c | 4 ++-- 12 files changed, 84 insertions(+), 103 deletions(-)
Signature · 1 line
2.47.0
PATCH RESEND v2 1/9 io_uring: Fold allocation into alloc_cache helper
The allocation paths that use alloc_cache duplicate the same code pattern, sometimes in a quite convoluted way. Fold the allocation into the cache code itself, making it just an allocator function, and keeping the cache policy invisible to callers. Another justification for doing this, beyond code simplicity, is that it makes it trivial to test the impact of disabling the cache and using slab directly, which I've used for slab improvement experiments.
One relevant detail is that we provide a callback to optionally initialize memory only when we actually reach slab. This allows us to avoid blindly executing the allocation with GFP_ZERO and only clean fields when they matter.
---
since v1:
- add a callback to initialize objects coming from slab
- rename io_alloc_cache_alloc -> io_cache_alloc
--- io_uring/alloc_cache.h | 13 +++++++++++++ 1 file changed, 13 insertions(+)
io_uring/alloc_cache.h +13 -0
@@ -30,6 +30,19 @@ static inline void *io_alloc_cache_get(struct io_alloc_cache *cache)return NULL ; } +static inline void *io_cache_alloc (struct io_alloc_cache *cache,gfp_t gfp, +void (*init_once)(void *obj)) +{ +if (unlikely (!cache->nr_cached)) { +void *obj =kmalloc (cache->elem_size, gfp); + +if (obj && init_once) +init_once (obj); +return obj; + } +return io_alloc_cache_get (cache); +} +/* returns false if the cache was initialized properly */ static inline bool io_alloc_cache_init (struct io_alloc_cache *cache,unsigned max_nr,size_t size)
Signature · 1 line
2.47.0
PATCH RESEND v2 2/9 io_uring: Add generic helper to allocate async data
In reply to Gabriel Krisman BertaziThis helper replaces io_alloc_async_data by using the folded allocation. Do it in a header to allow the compiler to decide whether to inline.
--- io_uring/io_uring.h | 10 ++++++++++ 1 file changed, 10 insertions(+)
io_uring/io_uring.h +10 -0
@@ -222,6 +222,16 @@ static inline void io_req_set_res(struct io_kiocb *req, s32 res, u32 cflags) req->cqe.flags = cflags; } +static inline void *io_uring_alloc_async_data (struct io_alloc_cache *cache, +struct io_kiocb *req, +void (*init_once)(void *obj)) +{ + req->async_data =io_cache_alloc (cache,GFP_KERNEL , init_once); +if (req->async_data) + req->flags |=REQ_F_ASYNC_DATA ; +return req->async_data; +} +static inline bool req_has_async_data (struct io_kiocb *req) {return req->flags &REQ_F_ASYNC_DATA ;
Signature · 1 line
2.47.0
PATCH RESEND v2 3/9 io_uring/futex: Allocate ifd with generic alloc_cache helper
In reply to Gabriel Krisman BertaziInstead of open-coding the allocation, use the generic alloc_cache helper.
--- io_uring/futex.c | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-)
io_uring/futex.c +1 -12
@@ -251,17 +251,6 @@ static void io_futex_wake_fn(struct wake_q_head *wake_q, struct futex_q *q)io_req_task_work_add (req); } -static struct io_futex_data *io_alloc_ifd (struct io_ring_ctx *ctx) -{ -struct io_futex_data *ifd; - - ifd =io_alloc_cache_get (&ctx->futex_cache); -if (ifd) -return ifd; - -return kmalloc (sizeof (struct io_futex_data ),GFP_NOWAIT ); -} -int io_futexv_wait (struct io_kiocb *req,unsigned int issue_flags) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex);
@@ -331,7 +320,7 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) }io_ring_submit_lock (ctx, issue_flags); - ifd =io_alloc_ifd (ctx); + ifd =io_cache_alloc (&ctx->futex_cache,GFP_NOWAIT ,NULL );if (!ifd) { ret = -ENOMEM ; goto done_unlock;
Signature · 1 line
2.47.0
PATCH RESEND v2 4/9 io_uring/poll: Allocate apoll with generic alloc_cache helper
In reply to Gabriel Krisman BertaziThis abstracts away the cache details to simplify the code.
--- io_uring/poll.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-)
io_uring/poll.c +5 -8
@@ -648,15 +648,12 @@ static struct async_poll *io_req_alloc_apoll(struct io_kiocb *req,if (req->flags &REQ_F_POLLED ) { apoll = req->apoll;kfree (apoll->double_poll); - }else if (!(issue_flags &IO_URING_F_UNLOCKED )) { - apoll =io_alloc_cache_get (&ctx->apoll_cache); -if (!apoll) - goto alloc_apoll; - apoll->poll.retries =APOLL_MAX_RETRY ; }else { -alloc_apoll: - apoll =kmalloc (sizeof (*apoll),GFP_ATOMIC ); -if (unlikely (!apoll)) +if (!(issue_flags &IO_URING_F_UNLOCKED )) + apoll =io_cache_alloc (&ctx->apoll_cache,GFP_ATOMIC ,NULL ); +else + apoll =kmalloc (sizeof (*apoll),GFP_ATOMIC ); +if (!apoll)return NULL ; apoll->poll.retries =APOLL_MAX_RETRY ; }
Signature · 1 line
2.47.0
PATCH RESEND v2 5/9 io_uring/uring_cmd: Allocate async data through generic helper
In reply to Gabriel Krisman BertaziThis abstracts away the cache details and simplify the code.
--- io_uring/io_uring.h | 1 + io_uring/uring_cmd.c | 20 ++------------------ 2 files changed, 3 insertions(+), 18 deletions(-)
io_uring/io_uring.h +1 -0
@@ -8,6 +8,7 @@#include <linux/poll.h> #include <linux/io_uring_types.h> #include <uapi/linux/eventpoll.h> +#include "alloc_cache.h" #include "io-wq.h" #include "slist.h" #include "filetable.h"
io_uring/uring_cmd.c +2 -18
@@ -16,22 +16,6 @@#include "rsrc.h" #include "uring_cmd.h" -static struct uring_cache *io_uring_async_get (struct io_kiocb *req) -{ -struct io_ring_ctx *ctx = req->ctx; -struct uring_cache *cache; - - cache =io_alloc_cache_get (&ctx->uring_cache); -if (cache) { - req->flags |=REQ_F_ASYNC_DATA ; - req->async_data = cache; -return cache; - } -if (!io_alloc_async_data (req)) -return req->async_data; -return NULL ; -} -static void io_req_uring_cleanup (struct io_kiocb *req,unsigned int issue_flags) { struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd);
@@ -185,8 +169,8 @@ static int io_uring_cmd_prep_setup(struct io_kiocb *req, struct io_uring_cmd *ioucmd = io_kiocb_to_cmd(req, struct io_uring_cmd);struct uring_cache *cache; - cache =io_uring_async_get (req); -if (unlikely (!cache)) + cache =io_uring_alloc_async_data (&req->ctx->uring_cache, req,NULL ); +if (!cache)return -ENOMEM ;if (!(req->flags &REQ_F_FORCE_ASYNC )) {
Signature · 1 line
2.47.0
PATCH RESEND v2 6/9 io_uring/net: Allocate msghdr async data through helper
In reply to Gabriel Krisman BertaziThis abstracts away the cache details.
--- io_uring/net.c | 35 ++++++++++++++++++----------------- 1 file changed, 18 insertions(+), 17 deletions(-)
io_uring/net.c +18 -17
@@ -155,30 +155,31 @@ static void io_netmsg_recycle(struct io_kiocb *req, unsigned int issue_flags) } } +static void io_msg_async_data_init (void *obj) +{ +struct io_async_msghdr *hdr = (struct io_async_msghdr *)obj; + + hdr->free_iov =NULL ; + hdr->free_iov_nr =0 ; +} +static struct io_async_msghdr *io_msg_alloc_async (struct io_kiocb *req) {struct io_ring_ctx *ctx = req->ctx;struct io_async_msghdr *hdr; - hdr =io_alloc_cache_get (&ctx->netmsg_cache); -if (hdr) { -if (hdr->free_iov) { -kasan_mempool_unpoison_object (hdr->free_iov, - hdr->free_iov_nr *sizeof (struct iovec )); - req->flags |=REQ_F_NEED_CLEANUP ; - } - req->flags |=REQ_F_ASYNC_DATA ; - req->async_data = hdr; -return hdr; - } + hdr =io_uring_alloc_async_data (&ctx->netmsg_cache, req, + io_msg_async_data_init); +if (!hdr) +return NULL ; -if (!io_alloc_async_data (req)) { - hdr = req->async_data; - hdr->free_iov_nr =0 ; - hdr->free_iov =NULL ; -return hdr; +/* If the async data was cached, we might have an iov cached inside. */ +if (hdr->free_iov) { +kasan_mempool_unpoison_object (hdr->free_iov, + hdr->free_iov_nr *sizeof (struct iovec )); + req->flags |=REQ_F_NEED_CLEANUP ; } -return NULL ; +return hdr; }/* assign new iovec to kmsg, if we need to */
Signature · 1 line
2.47.0
PATCH RESEND v2 7/9 io_uring/rw: Allocate async data through helper
In reply to Gabriel Krisman BertaziThis abstract away the cache details.
--- io_uring/rw.c | 36 ++++++++++++++++-------------------- 1 file changed, 16 insertions(+), 20 deletions(-)
io_uring/rw.c +16 -20
@@ -208,33 +208,29 @@ static void io_req_rw_cleanup(struct io_kiocb *req, unsigned int issue_flags) } } +static void io_rw_async_data_init (void *obj) +{ +struct io_async_rw *rw = (struct io_async_rw *)obj; + + rw->free_iovec =0 ; + rw->bytes_done =0 ; +} +static int io_rw_alloc_async (struct io_kiocb *req) {struct io_ring_ctx *ctx = req->ctx;struct io_async_rw *rw; - rw =io_alloc_cache_get (&ctx->rw_cache); -if (rw) { -if (rw->free_iovec) { -kasan_mempool_unpoison_object (rw->free_iovec, - rw->free_iov_nr *sizeof (struct iovec )); - req->flags |=REQ_F_NEED_CLEANUP ; - } - req->flags |=REQ_F_ASYNC_DATA ; - req->async_data = rw; - goto done; - } - -if (!io_alloc_async_data (req)) { - rw = req->async_data; - rw->free_iovec =NULL ; - rw->free_iov_nr =0 ; -done: + rw =io_uring_alloc_async_data (&ctx->rw_cache, req, io_rw_async_data_init); +if (!rw) +return -ENOMEM ; +if (rw->free_iovec) { +kasan_mempool_unpoison_object (rw->free_iovec, + rw->free_iov_nr *sizeof (struct iovec )); + req->flags |=REQ_F_NEED_CLEANUP ; rw->bytes_done =0 ; -return 0 ; } - -return -ENOMEM ; +return 0 ; }static int io_prep_rw_setup (struct io_kiocb *req,int ddir,bool do_import)
Signature · 1 line
2.47.0
PATCH RESEND v2 8/9 io_uring: Move old async data allocation helper to header
In reply to Gabriel Krisman BertaziThere are two remaining uses of the old async data allocator that do not rely on the alloc cache. I don't want to make them use the new allocator helper because that would require a if(cache) check, which will result in dead code for the cached case (for callers passing a cache, gcc can't prove the cache isn't NULL, and will therefore preserve the check. Since this is an inline function and just a few lines long, keep a second helper to deal with cases where we don't have an async data cache.
No functional change intended here. This is just moving the helper around and making it inline.
--- io_uring/io_uring.c | 13 ------------- io_uring/io_uring.h | 12 ++++++++++++ io_uring/timeout.c | 5 ++--- io_uring/waitid.c | 4 ++-- 4 files changed, 16 insertions(+), 18 deletions(-)
io_uring/io_uring.c +0 -13
@@ -1639,19 +1639,6 @@ io_req_flags_t io_file_get_flags(struct file *file)return res; } -bool io_alloc_async_data (struct io_kiocb *req) -{ -const struct io_issue_def *def = &io_issue_defs[req->opcode]; - -WARN_ON_ONCE (!def->async_size); - req->async_data =kmalloc (def->async_size,GFP_KERNEL ); -if (req->async_data) { - req->flags |=REQ_F_ASYNC_DATA ; -return false; - } -return true; -} -static u32 io_get_sequence (struct io_kiocb *req) {u32 seq = req->ctx->cached_sq_head;
io_uring/io_uring.h +12 -0
@@ -12,6 +12,7 @@#include "io-wq.h" #include "slist.h" #include "filetable.h" +#include "opdef.h" #ifndef CREATE_TRACE_POINTS #include <trace/events/io_uring.h>
@@ -233,6 +234,17 @@ static inline void *io_uring_alloc_async_data(struct io_alloc_cache *cache,return req->async_data; } +static inline void *io_uring_alloc_async_data_nocache (struct io_kiocb *req) +{ +const struct io_issue_def *def = &io_issue_defs[req->opcode]; + +WARN_ON_ONCE (!def->async_size); + req->async_data =kmalloc (def->async_size,GFP_KERNEL ); +if (req->async_data) + req->flags |=REQ_F_ASYNC_DATA ; +return req->async_data; +} +static inline bool req_has_async_data (struct io_kiocb *req) {return req->flags &REQ_F_ASYNC_DATA ;
io_uring/timeout.c +2 -3
@@ -525,10 +525,9 @@ static int __io_timeout_prep(struct io_kiocb *req,if (WARN_ON_ONCE (req_has_async_data (req)))return -EFAULT ; -if (io_alloc_async_data (req)) + data =io_uring_alloc_async_data_nocache (req); +if (!data)return -ENOMEM ; - - data = req->async_data; data->req = req; data->flags = flags;
io_uring/waitid.c +2 -2
@@ -303,10 +303,10 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags)struct io_waitid_async *iwa;int ret; -if (io_alloc_async_data (req)) + iwa =io_uring_alloc_async_data_nocache (req); +if (!iwa)return -ENOMEM ; - iwa = req->async_data; iwa->req = req; ret = kernel_waitid_prepare(&iwa->wo, iw->which, iw->upid, &iw->info,
Signature · 1 line
2.47.0
PATCH RESEND v2 9/9 io_uring/msg_ring: Drop custom destructor
In reply to Gabriel Krisman Bertazikfree can handle slab objects nowadays. Drop the extra callback and just use kfree.
--- io_uring/io_uring.c | 4 ++-- io_uring/msg_ring.c | 7 ------- io_uring/msg_ring.h | 1 - 3 files changed, 2 insertions(+), 10 deletions(-)
io_uring/io_uring.c +2 -2
@@ -361,7 +361,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)io_alloc_cache_free (&ctx->netmsg_cache, io_netmsg_cache_free);io_alloc_cache_free (&ctx->rw_cache, io_rw_cache_free);io_alloc_cache_free (&ctx->uring_cache, kfree); -io_alloc_cache_free (&ctx->msg_cache, io_msg_cache_free); +io_alloc_cache_free (&ctx->msg_cache, kfree);io_futex_cache_free (ctx);kvfree (ctx->cancel_table.hbs);xa_destroy (&ctx->io_bl_xa);
@@ -2692,7 +2692,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)io_alloc_cache_free (&ctx->netmsg_cache, io_netmsg_cache_free);io_alloc_cache_free (&ctx->rw_cache, io_rw_cache_free);io_alloc_cache_free (&ctx->uring_cache, kfree); -io_alloc_cache_free (&ctx->msg_cache, io_msg_cache_free); +io_alloc_cache_free (&ctx->msg_cache, kfree);io_futex_cache_free (ctx);io_destroy_buffers (ctx);io_free_region (ctx, &ctx->param_region);
io_uring/msg_ring.c +0 -7
@@ -354,10 +354,3 @@ int io_uring_sync_msg_ring(struct io_uring_sqe *sqe)return __io_msg_ring_data (fd_file (f)->private_data, &io_msg,IO_URING_F_UNLOCKED ); } - -void io_msg_cache_free (const void *entry) -{ -struct io_kiocb *req = (struct io_kiocb *) entry; - -kmem_cache_free (req_cachep, req); -}
io_uring/msg_ring.h +0 -1
@@ -4,4 +4,3 @@ int io_uring_sync_msg_ring(struct io_uring_sqe *sqe);int io_msg_ring_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe);int io_msg_ring (struct io_kiocb *req,unsigned int issue_flags);void io_msg_ring_cleanup (struct io_kiocb *req); -void io_msg_cache_free (const void *entry);
Signature · 1 line
2.47.0
Quoted 10 lines · Gabriel Krisman Bertazi
I sent this v2 originally during US thanksgiving week, so I'm resending now under a more suitable time, rebased and retested on top of Jen's for-6.14/io_uring branch. It keeps the changes requested in v1:
renaming the allocation helper and introducing a callback instead of zeroing the entire object, as suggested by Jens.This was tested against liburing testsuite, with lockdep and KASAN enabled.
[...]
Applied, thanks!
[1/9] io_uring: Fold allocation into alloc_cache helper
commit: b1031968b14f83f4bb96ecc4de4f6350ae0f6dad
[2/9] io_uring: Add generic helper to allocate async data
commit: 694022b01368387cc1ed8485e279dfe27939ee43
[3/9] io_uring/futex: Allocate ifd with generic alloc_cache helper
commit: b42176e5055a628217ff1536111a9e2df23db835
[4/9] io_uring/poll: Allocate apoll with generic alloc_cache helper
commit: 4cc6fd392489cd76c7aa138eddace19dbcea366e
[5/9] io_uring/uring_cmd: Allocate async data through generic helper
commit: 02b3c515d0be7e77dd19920e30cf637e9c7a167d
[6/9] io_uring/net: Allocate msghdr async data through helper
commit: 23d91035cafa30d186242ebdf583aa1b55f1c59e
[7/9] io_uring/rw: Allocate async data through helper
commit: 8cf0c459993ee2911f4f01fba21b1987b102c887
[8/9] io_uring: Move old async data allocation helper to header
commit: 6cd2993dcdc17cac5cf6b11034abc6014caab71b
[9/9] io_uring/msg_ring: Drop custom destructor
commit: cd7f9fee711c9ca4b909ffaadcac0302358db841Best regards,
Signature · 1 line
Jens Axboe