Provide more efficient buffer registration
Jens Axboe
maintainercover Provide more efficient buffer registration
Hi,
Pretty much what the subject line says, it's about 25k to 40k times faster to provide a way to duplicate an existing rings buffer registration than it is manually map/pin/register the buffers again with a new ring.
Patch 1 is just a prep patch, patch 2 adds refs to struct io_mapped_ubuf, and patch 3 finally adds the register opcode to allow a ring to duplicate the registered mappings from one ring to another.
This came about from discussing overhead from the varnish cache project for cases with more dynamic ring/thread creation.
include/uapi/linux/io_uring.h | 8 ++++ io_uring/register.c | 6 +++ io_uring/rsrc.c | 89 ++++++++++++++++++++++++++++++++++- io_uring/rsrc.h | 2 + 4 files changed, 104 insertions(+), 1 deletion(-)
Signature · 1 line
Jens Axboe
Jens Axboe
maintainerPATCH 1/3 io_uring/rsrc: clear 'slot' entry upfront
No functional changes in this patch, but clearing the slot pointer earlier will be required by a later change.
--- io_uring/rsrc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/rsrc.c +1 -1
@@ -114,6 +114,7 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slostruct io_mapped_ubuf *imu = *slot;unsigned int i; + *slot =NULL ;if (imu != &dummy_ubuf) {for (i =0 ; i < imu->nr_bvecs; i++)unpin_user_page (imu->bvec[i].bv_page);
@@ -121,7 +122,6 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **sloio_unaccount_mem (ctx, imu->acct_pages);kvfree (imu); } - *slot =NULL ; }static void io_rsrc_put_work (struct io_rsrc_node *node)
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 2/3 io_uring/rsrc: add reference count to struct io_mapped_ubuf
In reply to Jens AxboeCurrently there's a single ring owner of a mapped buffer, and hence the reference count will always be 1 when it's torn down and freed. However, in preparation for being able to link io_mapped_ubuf to different spots, add a reference count to manage the lifetime of it.
--- io_uring/rsrc.c | 3 +++ io_uring/rsrc.h | 1 + 2 files changed, 4 insertions(+)
io_uring/rsrc.c +3 -0
@@ -116,6 +116,8 @@ static void io_buffer_unmap(struct io_ring_ctx *ctx, struct io_mapped_ubuf **slo *slot =NULL ;if (imu != &dummy_ubuf) { +if (!refcount_dec_and_test (&imu->refs)) +return ;for (i =0 ; i < imu->nr_bvecs; i++)unpin_user_page (imu->bvec[i].bv_page);if (imu->acct_pages)
@@ -990,6 +992,7 @@ static int io_sqe_buffer_register(struct io_ring_ctx *ctx, struct iovec *iov, imu->folio_shift = data.folio_shift; imu->folio_mask = ~((1UL << data.folio_shift) -1 ); } +refcount_set (&imu->refs,1 ); off = (unsigned long ) iov->iov_base & ~imu->folio_mask; *pimu = imu; ret =0 ;
io_uring/rsrc.h +1 -0
@@ -47,6 +47,7 @@ struct io_mapped_ubuf {unsigned int folio_shift;unsigned long acct_pages;unsigned long folio_mask; +refcount_t refs; struct bio_vec bvec[] __counted_by(nr_bvecs); };
Signature · 1 line
2.45.2
Jens Axboe
maintainerPATCH 3/3 io_uring: add IORING_REGISTER_COPY_BUFFERS method
In reply to Jens AxboeBuffers can get registered with io_uring, which allows to skip the repeated pin_pages, unpin/unref pages for each O_DIRECT operation. This reduces the overhead of O_DIRECT IO.
However, registrering buffers can take some time. Normally this isn't an issue as it's done at initialization time (and hence less critical), but for cases where rings can be created and destroyed as part of an IO thread pool, registering the same buffers for multiple rings become a more time sensitive proposition. As an example, let's say an application has an IO memory pool of 500G. Initial registration takes:
Got 500 huge pages (each 1024MB)
Registered 500 pages in 409 msec
or about 0.4 seconds. If we go higher to 900 1GB huge pages being registered:
Registered 900 pages in 738 msec
which is, as expected, a fully linear scaling.
Rather than have each ring pin/map/register the same buffer pool, provide an io_uring_register(2) opcode to simply duplicate the buffers that are registered with another ring. Adding the same 900GB of registered buffers to the target ring can then be accomplished in:
Copied 900 pages in 17 usec
While timing differs a bit, this provides around a 25,000-40,000x speedup for this use case.
--- include/uapi/linux/io_uring.h | 8 ++++ io_uring/register.c | 6 +++ io_uring/rsrc.c | 84 +++++++++++++++++++++++++++++++++++ io_uring/rsrc.h | 1 + 4 files changed, 99 insertions(+)
include/uapi/linux/io_uring.h +8 -0
@@ -609,6 +609,9 @@ enum io_uring_register_op {IORING_REGISTER_CLOCK =29 , +/* copy registered buffers from source ring to current ring */ +IORING_REGISTER_COPY_BUFFERS =30 , +/* this goes last */ IORING_REGISTER_LAST,
@@ -694,6 +697,11 @@ struct io_uring_clock_register {__u32 __resv[3 ]; }; +struct io_uring_copy_buffers { +__u32 src_fd; +__u32 pad[7 ]; +}; +struct io_uring_buf {__u64 addr;__u32 len;
io_uring/register.c +6 -0
@@ -542,6 +542,12 @@ static int __io_uring_register(struct io_ring_ctx *ctx, unsigned opcode,break ; ret =io_register_clock (ctx, arg);break ; +case IORING_REGISTER_COPY_BUFFERS : + ret = -EINVAL ; +if (!arg || nr_args !=1 ) +break ; + ret =io_register_copy_buffers (ctx, arg); +break ;default : ret = -EINVAL ;break ;
io_uring/rsrc.c +84 -0
87 lines not on this page · raw message
io_uring/rsrc.h +1 -0
@@ -68,6 +68,7 @@ int io_import_fixed(int ddir, struct iov_iter *iter,struct io_mapped_ubuf *imu, u64 buf_addr, size_t len); +int io_register_copy_buffers (struct io_ring_ctx *ctx,void __user *arg);void __io_sqe_buffers_unregister (struct io_ring_ctx *ctx);int io_sqe_buffers_unregister (struct io_ring_ctx *ctx); int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
Signature · 1 line
2.45.2
Jens Axboe
maintainerQuoted 7 lines · Jens Axboe
Hi,
Pretty much what the subject line says, it's about 25k to 40k times faster to provide a way to duplicate an existing rings buffer registration than it is manually map/pin/register the buffers again with a new ring.
Quoted 6 lines
Patch 1 is just a prep patch, patch 2 adds refs to struct
io_mapped_ubuf, and patch 3 finally adds the register opcode to allow a ring to duplicate the registered mappings from one ring to another.This came about from discussing overhead from the varnish cache project for cases with more dynamic ring/thread creation.
Ignore this one, it was an old version... I'll send a v2 in a bit.
Signature · 1 line
Jens Axboe