io_uring: add IORING_OP_BPF for extending io_uring
v2 cover io_uring: add IORING_OP_BPF for extending io_uring
Hello,
Add IORING_OP_BPF for extending io_uring operations, follows typical cases:
- buffer registered zero copy [1]
Also there are some RAID like ublk servers which needs to generate data parity in case of ublk zero copy
- extend io_uring operations from application
Easy to add one new syscall with IORING_OP_BPF
- extend 64 byte SQE
bpf map can store IO data conveniently
- communicate in IO chain
IORING_OP_BPF can be used for communicate among IOs seamlessly without requiring extra syscall
- pretty handy to inject error for test purpose
Any comments & feedback are welcome!
[1] lpc2024: ublk based zero copy I/O - use case in Android
https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf
V2:
- per-ring struct ops (Stefan Metzmacher, Caleb Sander Mateos)
- refactor io_import_fixed()/io_prep_reg_iovec()/io_import_reg_vec() for allowing to handle multiple buffers for single request
- kernel selftests
- all kinds of comments from Caleb Sander Mateos
- support vectored and registered vector buffer
Ming Lei (12):
io_uring: make io_import_fixed() global
io_uring: refactor io_prep_reg_iovec() for BPF kfunc use
io_uring: refactor io_import_reg_vec() for BPF kfunc use
io_uring: prepare for extending io_uring with bpf
io_uring: bpf: extend io_uring with bpf struct_ops
io_uring: bpf: implement struct_ops registration
io_uring: bpf: add BPF buffer descriptor for IORING_OP_BPF
io_uring: bpf: add uring_bpf_memcpy() kfunc selftests/io_uring: add BPF struct_ops and kfunc tests selftests/io_uring: add bpf_memcpy selftest for uring_bpf_memcpy()
kfunc
selftests/io_uring: add copy_user_to_fixed() and copy_fixed_to_user()
bpf_memcpy tests
selftests/io_uring: add copy_user_to_reg_vec() and
copy_reg_vec_to_user() bpf_memcpy tests
Pavel Begunkov (1):
selftests/io_uring: update mini liburing
include/linux/io_uring_types.h | 5 + include/uapi/linux/io_uring.h | 40 ++ init/Kconfig | 7 + io_uring/Makefile | 1 + io_uring/bpf_op.c | 669 ++++++++++++++++++ io_uring/bpf_op.h | 61 ++ io_uring/io_uring.c | 5 + io_uring/io_uring.h | 6 +- io_uring/opdef.c | 16 + io_uring/rsrc.c | 46 +- io_uring/rsrc.h | 68 +- tools/include/io_uring/mini_liburing.h | 67 +- tools/testing/selftests/Makefile | 3 +- tools/testing/selftests/io_uring/.gitignore | 2 + tools/testing/selftests/io_uring/Makefile | 173 +++++ .../selftests/io_uring/basic_bpf_ops.bpf.c | 94 +++ .../selftests/io_uring/basic_bpf_ops.c | 215 ++++++ .../selftests/io_uring/bpf_memcpy.bpf.c | 98 +++ tools/testing/selftests/io_uring/bpf_memcpy.c | 517 ++++++++++++++ .../selftests/io_uring/include/iou_test.h | 98 +++ tools/testing/selftests/io_uring/runner.c | 206 ++++++ 21 files changed, 2339 insertions(+), 58 deletions(-) create mode 100644 io_uring/bpf_op.c create mode 100644 io_uring/bpf_op.h create mode 100644 tools/testing/selftests/io_uring/.gitignore create mode 100644 tools/testing/selftests/io_uring/Makefile create mode 100644 tools/testing/selftests/io_uring/basic_bpf_ops.bpf.c create mode 100644 tools/testing/selftests/io_uring/basic_bpf_ops.c create mode 100644 tools/testing/selftests/io_uring/bpf_memcpy.bpf.c create mode 100644 tools/testing/selftests/io_uring/bpf_memcpy.c create mode 100644 tools/testing/selftests/io_uring/include/iou_test.h create mode 100644 tools/testing/selftests/io_uring/runner.c
Signature · 1 line
2.47.0
PATCH v2 1/13 io_uring: make io_import_fixed() global
Refactor buffer import functions:
- Make io_import_fixed() global so BPF kfuncs can use it directly
- Make io_import_reg_buf() static inline in rsrc.h
This allows BPF kfuncs to import buffers without associating them with a request, useful when one request has multiple buffers.
--- io_uring/rsrc.c | 17 +++-------------- io_uring/rsrc.h | 18 +++++++++++++++--- 2 files changed, 18 insertions(+), 17 deletions(-)
io_uring/rsrc.c +3 -14
@@ -1069,9 +1069,9 @@ static int io_import_kbuf(int ddir, struct iov_iter *iter,return 0 ; } -static int io_import_fixed (int ddir,struct iov_iter *iter, -struct io_mapped_ubuf *imu, -u64 buf_addr,size_t len) +int io_import_fixed (int ddir,struct iov_iter *iter, +struct io_mapped_ubuf *imu, +u64 buf_addr,size_t len) {const struct bio_vec *bvec;size_t folio_mask;
@@ -1140,17 +1140,6 @@ inline struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,return NULL ; } -int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter, -u64 buf_addr,size_t len,int ddir, -unsigned issue_flags) -{ -struct io_rsrc_node *node; - - node =io_find_buf_node (req, issue_flags); -if (!node) -return -EFAULT ; -return io_import_fixed (ddir, iter, node->buf, buf_addr, len); -}/* Lock two rings at once. The rings must be different! */ static void lock_two_rings (struct io_ring_ctx *ctx1,struct io_ring_ctx *ctx2)
io_uring/rsrc.h +15 -3
@@ -61,9 +61,21 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr);struct io_rsrc_node *io_find_buf_node (struct io_kiocb *req,unsigned issue_flags); -int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter, -u64 buf_addr,size_t len,int ddir, -unsigned issue_flags); +int io_import_fixed (int ddir,struct iov_iter *iter, +struct io_mapped_ubuf *imu, +u64 buf_addr,size_t len); + +static inline int io_import_reg_buf (struct io_kiocb *req,struct iov_iter *iter, +u64 buf_addr,size_t len,int ddir, +unsigned issue_flags) +{ +struct io_rsrc_node *node; + + node =io_find_buf_node (req, issue_flags); +if (!node) +return -EFAULT ; +return io_import_fixed (ddir, iter, node->buf, buf_addr, len); +}int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags);
Signature · 1 line
2.47.0
PATCH v2 2/13 io_uring: refactor io_prep_reg_iovec() for BPF kfunc use
In reply to Ming LeiSplit io_prep_reg_iovec() into:
- __io_prep_reg_iovec(): core logic without request association
- io_prep_reg_iovec(): inline wrapper handling request flags
The core function takes explicit 'compat' and 'need_clean' parameters instead of accessing req directly. This allows BPF kfuncs to prepare vectored buffers without request association, enabling support for multiple buffers per request.
--- io_uring/rsrc.c | 11 +++++------ io_uring/rsrc.h | 21 +++++++++++++++++++-- 2 files changed, 24 insertions(+), 8 deletions(-)
io_uring/rsrc.c +5 -6
@@ -1540,8 +1540,8 @@ int io_import_reg_vec(int ddir, struct iov_iter *iter,return io_vec_fill_bvec (ddir, iter, imu, iov, nr_iovs, vec); } -int io_prep_reg_iovec (struct io_kiocb *req,struct iou_vec *iv, -const struct iovec __user *uvec,size_t uvec_segs) +int __io_prep_reg_iovec (struct iou_vec *iv,const struct iovec __user *uvec, +size_t uvec_segs,bool compat,bool *need_clean) {struct iovec *iov;int iovec_off, ret;
@@ -1551,17 +1551,16 @@ int io_prep_reg_iovec(struct io_kiocb *req, struct iou_vec *iv, ret =io_vec_realloc (iv, uvec_segs);if (ret)return ret; - req->flags |=REQ_F_NEED_CLEANUP ; +if (need_clean) + *need_clean = true; }/* pad iovec to the right */ iovec_off = iv->nr - uvec_segs; iov = iv->iovec + iovec_off; - res =iovec_from_user (uvec, uvec_segs, uvec_segs, iov, -io_is_compat (req->ctx)); + res =iovec_from_user (uvec, uvec_segs, uvec_segs, iov, compat);if (IS_ERR (res))return PTR_ERR (res); - req->flags |=REQ_F_IMPORT_BUFFER ;return 0 ; }
io_uring/rsrc.h +19 -2
@@ -4,6 +4,7 @@#include <linux/io_uring_types.h> #include <linux/lockdep.h> +#include "io_uring.h" #define IO_VEC_CACHE_SOFT_CAP 256
@@ -79,8 +80,24 @@ static inline int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,int io_import_reg_vec (int ddir,struct iov_iter *iter,struct io_kiocb *req,struct iou_vec *vec,unsigned nr_iovs,unsigned issue_flags); -int io_prep_reg_iovec (struct io_kiocb *req,struct iou_vec *iv, -const struct iovec __user *uvec,size_t uvec_segs); +int __io_prep_reg_iovec (struct iou_vec *iv,const struct iovec __user *uvec, +size_t uvec_segs,bool compat,bool *need_clean); + +static inline int io_prep_reg_iovec (struct io_kiocb *req,struct iou_vec *iv, +const struct iovec __user *uvec, +size_t uvec_segs) +{ +bool need_clean = false; +int ret; + + ret =__io_prep_reg_iovec (iv, uvec, uvec_segs, +io_is_compat (req->ctx), &need_clean); +if (need_clean) + req->flags |=REQ_F_NEED_CLEANUP ; +if (ret >=0 ) + req->flags |=REQ_F_IMPORT_BUFFER ; +return ret; +}int io_register_clone_buffers (struct io_ring_ctx *ctx,void __user *arg);int io_sqe_buffers_unregister (struct io_ring_ctx *ctx);
Signature · 1 line
2.47.0
PATCH v2 3/13 io_uring: refactor io_import_reg_vec() for BPF kfunc use
In reply to Ming LeiSplit io_import_reg_vec() into:
- __io_import_reg_vec(): core logic taking io_mapped_ubuf directly
- io_import_reg_vec(): inline wrapper handling buffer lookup and request flags
The core function takes 'imu' and 'need_clean' parameters instead of accessing req directly. This allows BPF kfuncs to import vectored buffers without request association, enabling support for multiple buffers per request.
--- io_uring/rsrc.c | 18 +++++------------- io_uring/rsrc.h | 29 ++++++++++++++++++++++++++--- 2 files changed, 31 insertions(+), 16 deletions(-)
io_uring/rsrc.c +5 -13
@@ -1476,23 +1476,14 @@ static int io_kern_bvec_size(struct iovec *iov, unsigned nr_iovs,return 0 ; } -int io_import_reg_vec (int ddir,struct iov_iter *iter, -struct io_kiocb *req,struct iou_vec *vec, -unsigned nr_iovs,unsigned issue_flags) +int __io_import_reg_vec (int ddir,struct iov_iter *iter, +struct io_mapped_ubuf *imu,struct iou_vec *vec, +unsigned nr_iovs,bool *need_clean) { -struct io_rsrc_node *node; -struct io_mapped_ubuf *imu;unsigned iovec_off;struct iovec *iov;unsigned nr_segs; - node =io_find_buf_node (req, issue_flags); -if (!node) -return -EFAULT ; - imu = node->buf; -if (!(imu->dir & (1 << ddir))) -return -EFAULT ; - iovec_off = vec->nr - nr_iovs; iov = vec->iovec + iovec_off;
@@ -1531,7 +1522,8 @@ int io_import_reg_vec(int ddir, struct iov_iter *iter, *vec = tmp_vec; iov = vec->iovec + iovec_off; - req->flags |=REQ_F_NEED_CLEANUP ; +if (need_clean) + *need_clean = true; }if (imu->is_kbuf)
io_uring/rsrc.h +26 -3
@@ -77,9 +77,32 @@ static inline int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,return -EFAULT ;return io_import_fixed (ddir, iter, node->buf, buf_addr, len); } -int io_import_reg_vec (int ddir,struct iov_iter *iter, -struct io_kiocb *req,struct iou_vec *vec, -unsigned nr_iovs,unsigned issue_flags); +int __io_import_reg_vec (int ddir,struct iov_iter *iter, +struct io_mapped_ubuf *imu,struct iou_vec *vec, +unsigned nr_iovs,bool *need_clean); + +static inline int io_import_reg_vec (int ddir,struct iov_iter *iter, +struct io_kiocb *req,struct iou_vec *vec, +unsigned nr_iovs,unsigned issue_flags) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +bool need_clean = false; +int ret; + + node =io_find_buf_node (req, issue_flags); +if (!node) +return -EFAULT ; + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return -EFAULT ; + + ret =__io_import_reg_vec (ddir, iter, imu, vec, nr_iovs, &need_clean); +if (need_clean) + req->flags |=REQ_F_NEED_CLEANUP ; +return ret; +} +int __io_prep_reg_iovec (struct iou_vec *iv,const struct iovec __user *uvec,size_t uvec_segs,bool compat,bool *need_clean);
Signature · 1 line
2.47.0
PATCH v2 4/13 io_uring: prepare for extending io_uring with bpf
In reply to Ming LeiAdd one bpf operation & related framework and prepare for extending io_uring with bpf.
--- include/uapi/linux/io_uring.h | 1 + init/Kconfig | 7 +++++++ io_uring/Makefile | 1 + io_uring/bpf_op.c | 26 ++++++++++++++++++++++++++ io_uring/bpf_op.h | 15 +++++++++++++++ io_uring/opdef.c | 16 ++++++++++++++++ 6 files changed, 66 insertions(+) create mode 100644 io_uring/bpf_op.c create mode 100644 io_uring/bpf_op.h
include/uapi/linux/io_uring.h +1 -0
@@ -303,6 +303,7 @@ enum io_uring_op {IORING_OP_PIPE ,IORING_OP_NOP128 ,IORING_OP_URING_CMD128 , +IORING_OP_BPF ,/* this goes last, obviously */ IORING_OP_LAST,
init/Kconfig +7 -0
@@ -1868,6 +1868,13 @@ config IO_URING applications to submit and complete IO through submission and completion rings that are shared between the kernel and application. +config IO_URING_BPF_OP + bool "Enable IO uring bpf extension" if EXPERT + depends on IO_URING && BPF + help + This option enables bpf extension for the io_uring interface, so + application can define its own io_uring operation by bpf program. + config GCOV_PROFILE_URING bool "Enable GCOV profiling on the io_uring subsystem" depends on IO_URING && GCOV_KERNEL
io_uring/Makefile +1 -0
@@ -22,3 +22,4 @@ obj-$(CONFIG_NET_RX_BUSY_POLL) += napi.o obj-$(CONFIG_NET) += net.o cmd_net.o obj-$(CONFIG_PROC_FS) += fdinfo.o obj-$(CONFIG_IO_URING_MOCK_FILE) += mock_file.o +obj-$(CONFIG_IO_URING_BPF_OP) += bpf_op.o
io_uring/bpf_op.c new file +26 -0
@@ -0,0 +1,26 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2025 Red Hat */ + +#include <linux/kernel.h> +#include <linux/errno.h> +#include <uapi/linux/io_uring.h> +#include "io_uring.h" +#include "bpf_op.h" + +int io_uring_bpf_issue (struct io_kiocb *req,unsigned int issue_flags) +{ +return -EOPNOTSUPP ; +} + +int io_uring_bpf_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe) +{ +return -EOPNOTSUPP ; +} + +void io_uring_bpf_fail (struct io_kiocb *req) +{ +} + +void io_uring_bpf_cleanup (struct io_kiocb *req) +{ +}
io_uring/bpf_op.h new file +15 -0
@@ -0,0 +1,15 @@ +// SPDX-License-Identifier: GPL-2.0 +#ifndef IOU_BPF_OP_H +#define IOU_BPF_OP_H + +struct io_kiocb ; +struct io_uring_sqe ; + +#ifdef CONFIG_IO_URING_BPF_OP +int io_uring_bpf_issue (struct io_kiocb *req,unsigned int issue_flags); +int io_uring_bpf_prep (struct io_kiocb *req,const struct io_uring_sqe *sqe); +void io_uring_bpf_fail (struct io_kiocb *req); +void io_uring_bpf_cleanup (struct io_kiocb *req); +#endif + +#endif
io_uring/opdef.c +16 -0
@@ -38,6 +38,7 @@#include "futex.h" #include "truncate.h" #include "zcrx.h" +#include "bpf_op.h" static int io_no_issue (struct io_kiocb *req,unsigned int issue_flags) {
@@ -593,6 +594,14 @@ const struct io_issue_def io_issue_defs[] = { .prep = io_uring_cmd_prep, .issue = io_uring_cmd, }, + [IORING_OP_BPF] = { +#if defined(CONFIG_IO_URING_BPF_OP ) + .prep = io_uring_bpf_prep, + .issue = io_uring_bpf_issue, +#else + .prep = io_eopnotsupp_prep, +#endif + }, }; const struct io_cold_def io_cold_defs[] = {
@@ -851,6 +860,13 @@ const struct io_cold_def io_cold_defs[] = { .sqe_copy = io_uring_cmd_sqe_copy, .cleanup = io_uring_cmd_cleanup, }, + [IORING_OP_BPF] = { + .name = "BPF", +#if defined(CONFIG_IO_URING_BPF_OP ) + .cleanup = io_uring_bpf_cleanup, + .fail = io_uring_bpf_fail, +#endif + }, };const char *io_uring_get_opcode (u8 opcode)
Signature · 1 line
2.47.0
PATCH v2 5/13 io_uring: bpf: extend io_uring with bpf struct_ops
In reply to Ming Leiio_uring can be extended with bpf struct_ops in the following ways:
1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.
[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf
2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently
3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer
4) pretty handy to inject error for test purpose
bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.
--- include/linux/io_uring_types.h | 3 + include/uapi/linux/io_uring.h | 12 ++ io_uring/bpf_op.c | 231 ++++++++++++++++++++++++++++++++- io_uring/bpf_op.h | 42 ++++++ io_uring/io_uring.c | 5 + io_uring/io_uring.h | 6 +- 6 files changed, 295 insertions(+), 4 deletions(-)
include/linux/io_uring_types.h +3 -0
@@ -8,6 +8,8 @@#include <linux/llist.h> #include <uapi/linux/io_uring.h> +struct uring_bpf_ops_kern ; +enum {/* * A hint to not wake right away but delay until there are enough of
@@ -276,6 +278,7 @@ struct io_ring_ctx {struct io_rings *rings;struct percpu_ref refs; +struct uring_bpf_ops_kern *bpf_ops;clockid_t clockid;enum tk_offsets clock_offset;
include/uapi/linux/io_uring.h +12 -0
@@ -74,6 +74,7 @@ struct io_uring_sqe {__u32 install_fd_flags;__u32 nop_flags;__u32 pipe_flags; +__u32 bpf_op_flags; };__u64 user_data;/* data to be passed back at completion time */ /* pack this to avoid bogus arm OABI complaints */
@@ -237,6 +238,9 @@ enum io_uring_sqe_flags_bit {*/ #define IORING_SETUP_SQE_MIXED (1U << 19) +/* Allow userspace to define io_uring operation by BPF prog */ +#define IORING_SETUP_BPF_OP (1U << 20) +enum io_uring_op {IORING_OP_NOP ,IORING_OP_READV ,
@@ -422,6 +426,13 @@ enum io_uring_op {#define IORING_RECVSEND_BUNDLE (1U << 4)#define IORING_SEND_VECTORIZED (1U << 5) +/* +* sqe->bpf_op_flags top 8bits is for storing bpf prog sub op +* The other 24bits are used for bpf prog +*/ +#define IORING_BPF_OP_BITS 8 +#define IORING_BPF_OP_SHIFT 24 +/* * cqe.res for IORING_CQE_F_NOTIF if * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -626,6 +637,7 @@ struct io_uring_params {#define IORING_FEAT_MIN_TIMEOUT (1U << 15)#define IORING_FEAT_RW_ATTR (1U << 16)#define IORING_FEAT_NO_IOWAIT (1U << 17) +#define IORING_FEAT_BPF (1U << 18)/* * io_uring_register(2) opcodes and arguments
io_uring/bpf_op.c +229 -2
253 lines not on this page · raw message
io_uring/bpf_op.h +42 -0
54 lines not on this page · raw message
io_uring/io_uring.c +5 -0
@@ -105,6 +105,7 @@#include "rw.h" #include "alloc_cache.h" #include "eventfd.h" +#include "bpf_op.h" #define SQE_COMMON_FLAGS (IOSQE_FIXED_FILE | IOSQE_IO_LINK | \ IOSQE_IO_HARDLINK | IOSQE_ASYNC)
@@ -353,6 +354,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)io_napi_init (ctx);mutex_init (&ctx->mmap_lock); +if (io_bpf_alloc (ctx)) + goto free_ref; +return ctx; free_ref:
@@ -2867,6 +2871,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)if (ctx->hash_map)io_wq_put_hash (ctx->hash_map);io_napi_free (ctx); +io_bpf_free (ctx);kvfree (ctx->cancel_table.hbs);xa_destroy (&ctx->io_bl_xa);kfree (ctx);
io_uring/io_uring.h +4 -2
@@ -48,7 +48,8 @@ struct io_ctx_config { IORING_FEAT_RECVSEND_BUNDLE |\ IORING_FEAT_MIN_TIMEOUT |\ IORING_FEAT_RW_ATTR |\ - IORING_FEAT_NO_IOWAIT) + IORING_FEAT_NO_IOWAIT |\ + IORING_FEAT_BPF)#define IORING_SETUP_FLAGS (IORING_SETUP_IOPOLL |\ IORING_SETUP_SQPOLL |\
@@ -69,7 +70,8 @@ struct io_ctx_config { IORING_SETUP_NO_SQARRAY |\ IORING_SETUP_HYBRID_IOPOLL |\ IORING_SETUP_CQE_MIXED |\ - IORING_SETUP_SQE_MIXED) + IORING_SETUP_SQE_MIXED |\ + IORING_SETUP_BPF_OP)#define IORING_ENTER_FLAGS (IORING_ENTER_GETEVENTS |\ IORING_ENTER_SQ_WAKEUP |\
Signature · 1 line
2.47.0
PATCH v2 6/13 io_uring: bpf: implement struct_ops registration
In reply to Ming LeiComplete the BPF struct_ops registration mechanism by implementing refcount-based lifecycle management:
- Add refcount field to struct uring_bpf_ops_kern for tracking active requests
- Add wait_queue_head_t bpf_wq to struct io_ring_ctx for synchronizing unregistration with in-flight requests
- Implement io_bpf_reg_unreg() to handle registration (refcount=1) and unregistration (wait for in-flight requests to complete)
- Update io_uring_bpf_prep() to increment refcount on success and reject new requests when refcount is zero (unregistration in progress)
- Update io_uring_bpf_cleanup() to decrement refcount and wake up waiters when it reaches zero
--- include/linux/io_uring_types.h | 2 + io_uring/bpf_op.c | 104 ++++++++++++++++++++++++++++++++- io_uring/bpf_op.h | 3 + 3 files changed, 106 insertions(+), 3 deletions(-)
include/linux/io_uring_types.h +2 -0
@@ -474,6 +474,8 @@ struct io_ring_ctx {struct io_mapped_region ring_region;/* used for optimised request parameter and wait argument passing */ struct io_mapped_region param_region; + +wait_queue_head_t bpf_wq; };/*
io_uring/bpf_op.c +101 -3
162 lines not on this page · raw message
io_uring/bpf_op.h +3 -0
@@ -27,14 +27,17 @@ typedef void (*uring_bpf_cleanup_t)(struct uring_bpf_data *data);struct uring_bpf_ops {unsigned short id; +int ring_fd;uring_bpf_prep_t prep_fn;uring_bpf_issue_t issue_fn;uring_bpf_fail_t fail_fn;uring_bpf_cleanup_t cleanup_fn; }; +/* TODO: manage it via `io_rsrc_node` */ struct uring_bpf_ops_kern {const struct uring_bpf_ops *ops; +int refcount; };#ifdef CONFIG_IO_URING_BPF_OP
Signature · 1 line
2.47.0
PATCH v2 7/13 io_uring: bpf: add BPF buffer descriptor for IORING_OP_BPF
In reply to Ming LeiAdd io_bpf_buf_desc struct and io_bpf_buf_type enum to describe buffer parameters for IORING_OP_BPF kfuncs. Supports plain userspace, registered, vectored, and registered vectored buffer types.
Registered buffers (FIXED, KFIXED, REG_VEC) refer to buffers pre-registered with io_uring and can be either userspace or kernel buffers depending on how they were registered.
--- include/uapi/linux/io_uring.h | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+)
include/uapi/linux/io_uring.h +27 -0
@@ -433,6 +433,33 @@ enum io_uring_op {#define IORING_BPF_OP_BITS 8#define IORING_BPF_OP_SHIFT 24 +/* +* BPF buffer descriptor types. +* +* Registered buffers (FIXED, KFIXED, REG_VEC) refer to buffers pre-registered +* with io_uring. These can be either userspace or kernel buffers depending on +* how they were registered. +* +* For KFIXED, addr is an offset from the registered buffer start. +* For REG_VEC with kernel buffers, each iov.iov_base is offset-based. +*/ +enum io_bpf_buf_type { +IO_BPF_BUF_USER =0 ,/* plain userspace buffer */ +IO_BPF_BUF_FIXED =1 ,/* registered buffer (absolute address) */ +IO_BPF_BUF_VEC =2 ,/* vectored buffer (iovec array) */ +IO_BPF_BUF_KFIXED =3 ,/* registered buffer (offset-based) */ +IO_BPF_BUF_REG_VEC =4 ,/* registered vectored buffer */ +}; + +/* BPF buffer descriptor for IORING_OP_BPF */ +struct io_bpf_buf_desc { +__u8 type;/* IO_BPF_BUF_* */ +__u8 reserved; +__u16 buf_index;/* registered buffer index (FIXED/KFIXED/REG_VEC) */ +__u32 len;/* length (non-vec) or nr_vecs (vec types) */ +__u64 addr;/* userspace address, iovec ptr, or offset (KFIXED) */ +}; +/* * cqe.res for IORING_CQE_F_NOTIF if * IORING_SEND_ZC_REPORT_USAGE was requested
Signature · 1 line
2.47.0
PATCH v2 8/13 io_uring: bpf: add uring_bpf_memcpy() kfunc
In reply to Ming LeiAdd uring_bpf_memcpy() kfunc that copies data between io_uring BPF buffers. This kfunc supports all 5 buffer types defined in io_bpf_buf_desc:
- IO_BPF_BUF_USER: plain userspace buffer
- IO_BPF_BUF_FIXED: fixed buffer (absolute address within buffer)
- IO_BPF_BUF_VEC: vectored userspace buffer (iovec array)
- IO_BPF_BUF_KFIXED: kernel fixed buffer (offset-based addressing)
- IO_BPF_BUF_KVEC: kernel vectored buffer
Add helper functions for buffer import:
- io_bpf_import_fixed_buf(): handles FIXED/KFIXED types with proper node reference counting
- io_bpf_import_kvec_buf(): handles KVEC using __io_prep_reg_iovec() and __io_import_reg_vec()
- io_bpf_import_buffer(): unified dispatcher for all buffer types
- io_bpf_copy_iters(): page-based copy between iov_iters
The kfunc properly manages buffer node references and submit lock for registered buffer access.
--- io_uring/bpf_op.c | 320 +++++++++++++++++++++++++++++++++++++++++++++- io_uring/bpf_op.h | 3 +- 2 files changed, 321 insertions(+), 2 deletions(-)
io_uring/bpf_op.c +319 -1
@@ -10,9 +10,11 @@#include <linux/btf.h> #include <linux/btf_ids.h> #include <linux/filter.h> +#include <linux/uio.h> #include <uapi/linux/io_uring.h> #include "io_uring.h" #include "register.h" +#include "rsrc.h" #include "bpf_op.h" static inline unsigned char uring_bpf_get_op (u32 op_flags)
@@ -47,6 +49,7 @@ int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) data->opf = opf; data->ops = ops; + data->issue_flags =0 ; ret = ops->prep_fn (data, sqe);if (!ret) {/* Only increment refcount on success (uring_lock already held) */
@@ -74,7 +77,13 @@ static int __io_uring_bpf_issue(struct io_kiocb *req)int io_uring_bpf_issue (struct io_kiocb *req,unsigned int issue_flags) { -return __io_uring_bpf_issue (req); + struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data); +int ret; + + data->issue_flags = issue_flags; + ret =__io_uring_bpf_issue (req); + data->issue_flags =0 ; +return ret; }void io_uring_bpf_fail (struct io_kiocb *req)
@@ -291,6 +300,235 @@ static struct bpf_struct_ops bpf_uring_bpf_ops = { .owner = THIS_MODULE, }; +/* +* Helper to copy data between two iov_iters using page extraction. +* Extracts pages from source iterator and copies them to destination. +* Returns number of bytes copied or negative error code. +*/ +static ssize_t io_bpf_copy_iters (struct iov_iter *src,struct iov_iter *dst, +size_t len) +{ +#define MAX_PAGES_PER_LOOP 32 +struct page *pages[MAX_PAGES_PER_LOOP ]; +size_t total_copied =0 ; +bool need_unpin; + + need_unpin =iov_iter_extract_will_pin (src); + +while (len >0 ) { +struct page **page_array = pages; +size_t offset, copied =0 ; +ssize_t extracted; +unsigned int nr_pages; +size_t chunk_len; +int i; + + chunk_len =min_t (size_t, len,MAX_PAGES_PER_LOOP *PAGE_SIZE ); + extracted =iov_iter_extract_pages (src, &page_array, chunk_len, +MAX_PAGES_PER_LOOP ,0 , &offset); +if (extracted <=0 ) { +if (total_copied >0 ) +break ; +return extracted <0 ? extracted : -EFAULT ; + } + + nr_pages =DIV_ROUND_UP (offset + extracted,PAGE_SIZE ); + +for (i =0 ; i < nr_pages && copied < extracted; i++) { +size_t page_offset = (i ==0 ) ? offset :0 ; +size_t page_len =min_t (size_t, extracted - copied, +PAGE_SIZE - page_offset); +size_t n; + + n =copy_page_to_iter (page_array[i], page_offset, page_len, dst); + copied += n; +if (n < page_len) +break ; + } + +if (need_unpin) +unpin_user_pages (page_array, nr_pages); + + total_copied += copied; + len -= copied; + +if (copied < extracted) +break ; + } + +return total_copied; +#undef MAX_PAGES_PER_LOOP +} + +/* +* Helper to import fixed buffer (FIXED or KFIXED). +* Must be called with submit lock held. +* +* FIXED: addr is absolute userspace address within buffer +* KFIXED: addr is offset from buffer start +* +* Returns node with incremented refcount on success, ERR_PTR on failure. +*/ +static struct io_rsrc_node *io_bpf_import_fixed_buf (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +int ret; + + node =io_rsrc_node_lookup (&ctx->buf_table, desc->buf_index); +if (!node) +return ERR_PTR (-EFAULT ); + + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return ERR_PTR (-EFAULT ); + + node->refs++; + + ret =io_import_fixed (ddir, iter, imu, desc->addr, desc->len); +if (ret) { + node->refs--; +return ERR_PTR (ret); + } + +return node; +} + +/* +* Helper to import registered vectored buffer (KVEC). +* Must be called with submit lock held. +* +* addr: userspace iovec pointer +* len: number of iovecs +* buf_index: registered buffer index +* +* Returns node with incremented refcount on success, ERR_PTR on failure. +* Caller must call io_vec_free(vec) after use. +*/ +static struct io_rsrc_node *io_bpf_import_reg_vec (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +int ret; + + node =io_rsrc_node_lookup (&ctx->buf_table, desc->buf_index); +if (!node) +return ERR_PTR (-EFAULT ); + + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return ERR_PTR (-EFAULT ); + + node->refs++; + +/* Prepare iovec from userspace */ + ret =__io_prep_reg_iovec (vec,u64_to_user_ptr (desc->addr), + desc->len, ctx->compat,NULL ); +if (ret) + goto err; + +/* Import vectored buffer from registered buffer */ + ret =__io_import_reg_vec (ddir, iter, imu, vec, desc->len,NULL ); +if (ret) + goto err; + +return node; +err: + node->refs--; +return ERR_PTR (ret); +} + +/* +* Helper to import a vectored user buffer (VEC) into iou_vec. +* Allocates space in vec and copies iovec from userspace. +* +* Returns 0 on success, negative error code on failure. +* Caller must call io_vec_free(vec) after use. +*/ +static int io_bpf_import_vec_buf (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +unsigned nr_vecs = desc->len; +struct iovec *iov; +size_t total_len =0 ; +void *res; +int ret, i; + +if (nr_vecs > vec->nr) { + ret =io_vec_realloc (vec, nr_vecs); +if (ret) +return ret; + } + + iov = vec->iovec; + res =iovec_from_user (u64_to_user_ptr (desc->addr), nr_vecs, + nr_vecs, iov, ctx->compat); +if (IS_ERR (res)) +return PTR_ERR (res); + +for (i =0 ; i < nr_vecs; i++) + total_len += iov[i].iov_len; + +iov_iter_init (iter, ddir, iov, nr_vecs, total_len); +return 0 ; +} + +/* +* Helper to import a buffer into an iov_iter based on io_bpf_buf_desc. +* Supports all 5 buffer types: USER, FIXED, VEC, KFIXED, KVEC. +* Must be called with submit lock held for FIXED/KFIXED/KVEC types. +* +* @ctx: ring context +* @iter: output iterator +* @desc: buffer descriptor +* @ddir: direction (ITER_SOURCE for source, ITER_DEST for destination) +* @vec: iou_vec for VEC/KVEC types (caller must call io_vec_free after use) +* +* Returns node pointer (may be NULL for USER/VEC), or ERR_PTR on failure. +* Caller must drop node reference when done if non-NULL. +*/ +static struct io_rsrc_node *io_bpf_import_buffer (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +int ret; + +switch (desc->type) { +case IO_BPF_BUF_USER : +/* Plain user buffer */ + ret =import_ubuf (ddir,u64_to_user_ptr (desc->addr), + desc->len, iter); +return ret ?ERR_PTR (ret) :NULL ; + +case IO_BPF_BUF_FIXED : +case IO_BPF_BUF_KFIXED : +/* FIXED: addr is absolute address within buffer */ +/* KFIXED: addr is offset from buffer start */ +return io_bpf_import_fixed_buf (ctx, iter, desc, ddir); + +case IO_BPF_BUF_VEC : +/* Vectored user buffer - addr is iovec ptr, len is nr_vecs */ + ret =io_bpf_import_vec_buf (ctx, iter, desc, ddir, vec); +return ret ?ERR_PTR (ret) :NULL ; + +case IO_BPF_BUF_REG_VEC : +/* Registered vectored buffer */ +return io_bpf_import_reg_vec (ctx, iter, desc, ddir, vec); + +default : +return ERR_PTR (-EINVAL ); + } +} +__bpf_kfunc_start_defs (); __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res) {
@@ -300,10 +538,90 @@ __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)req_set_fail (req);io_req_set_res (req, res,0 ); } + +/** +* uring_bpf_memcpy - Copy data between io_uring BPF buffers +* @data: BPF request data containing request context +* @dst: Destination buffer descriptor +* @src: Source buffer descriptor +* +* Copies data from source buffer to destination buffer. +* Supports all 5 buffer types: USER, FIXED, VEC, KFIXED, REG_VEC. +* The copy length is min of actual buffer sizes (for VEC types, +* total bytes across all vectors, not nr_vecs). +* +* Returns: Number of bytes copied on success, negative error code on failure +*/ +__bpf_kfunc ssize_t uring_bpf_memcpy(const struct uring_bpf_data *data, +struct io_bpf_buf_desc *dst, +struct io_bpf_buf_desc *src) +{ +struct io_kiocb *req =cmd_to_io_kiocb ((void *)data); +struct io_ring_ctx *ctx = req->ctx; +unsigned int issue_flags = data->issue_flags; +struct io_rsrc_node *src_node, *dst_node; +struct iov_iter src_iter, dst_iter; +struct iou_vec src_vec = {}; +struct iou_vec dst_vec = {}; +ssize_t ret; +size_t len; + +/* Validate buffer types */ +if (src->type >IO_BPF_BUF_REG_VEC || dst->type >IO_BPF_BUF_REG_VEC ) +return -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + +/* Import source buffer */ + src_node =io_bpf_import_buffer (ctx, &src_iter, src,ITER_SOURCE , + &src_vec); +if (IS_ERR (src_node)) { + ret =PTR_ERR (src_node); + goto unlock; + } + +/* Import destination buffer */ + dst_node =io_bpf_import_buffer (ctx, &dst_iter, dst,ITER_DEST , + &dst_vec); +if (IS_ERR (dst_node)) { + ret =PTR_ERR (dst_node); + goto put_src; + } + +/* +* Calculate copy length from actual iterator sizes. +* For VEC types, desc->len is nr_vecs, not total bytes. +*/ + len =min (iov_iter_count (&src_iter),iov_iter_count (&dst_iter)); +if (!len) { + ret =0 ; + goto put_dst; + } +if (len >MAX_RW_COUNT ) { + ret = -EINVAL ; + goto put_dst; + } + +/* Copy data between iterators */ + ret =io_bpf_copy_iters (&src_iter, &dst_iter, len); + +put_dst: +io_vec_free (&dst_vec); +if (dst_node) +io_put_rsrc_node (ctx, dst_node); +put_src: +io_vec_free (&src_vec); +if (src_node) +io_put_rsrc_node (ctx, src_node); +unlock: +io_ring_submit_unlock (ctx, issue_flags); +return ret; +}__bpf_kfunc_end_defs ();BTF_KFUNCS_START (uring_bpf_kfuncs)BTF_ID_FLAGS (func ,uring_bpf_set_result ) +BTF_ID_FLAGS (func, uring_bpf_memcpy)BTF_KFUNCS_END (uring_bpf_kfuncs) static const struct btf_kfunc_id_set uring_kfunc_set = {
io_uring/bpf_op.h +2 -1
@@ -13,10 +13,11 @@ struct uring_bpf_data {void *req_data;/* not for bpf prog */ const struct uring_bpf_ops *ops;u32 opf; +u32 issue_flags;/* io_uring issue flags */ /* writeable for bpf prog */ u8 pdu[64 -sizeof (void *) - -sizeof (struct uring_bpf_ops *) -sizeof (u32)]; +sizeof (struct uring_bpf_ops *) -2 *sizeof (u32)]; }; typedef int (*uring_bpf_prep_t)(struct uring_bpf_data *data,
Signature · 1 line
2.47.0
Re: [PATCH V2 08/13] io_uring: bpf: add uring_bpf_memcpy() kfunc
Quoted 19 lines · Ming Lei
Add
uring_bpf_memcpy()kfunc that copies data betweenio_uringBPF buffers. This kfunc supports all 5 buffer types defined inio_bpf_buf_desc:-
IO_BPF_BUF_USER: plain userspace buffer
-IO_BPF_BUF_FIXED: fixed buffer (absolute address within buffer)
-IO_BPF_BUF_VEC: vectored userspace buffer (iovec array)
-IO_BPF_BUF_KFIXED: kernel fixed buffer (offset-based addressing)
-IO_BPF_BUF_KVEC: kernel vectored bufferAdd helper functions for buffer import:
-io_bpf_import_fixed_buf(): handles FIXED/KFIXED types with proper node reference counting
-io_bpf_import_kvec_buf(): handles KVEC using__io_prep_reg_iovec()and__io_import_reg_vec()
-io_bpf_import_buffer(): unified dispatcher for all buffer types
-io_bpf_copy_iters(): page-based copy betweeniov_iters
Quoted 2 lines
The kfunc properly manages buffer node references and submit lock for registered buffer access.
Ming,
Thank you for working on this patch series and sorry I didn't have a chance to take a look at this revision earlier.
My main reservation with this approach is that a dedicated kfunc is needed for each data manipulation operation an io_uring BPF program conceivably wants to perform. Even for the initial RAID parity calculation use case, I imagine we'd need a different kfunc for each RAID stripe data width and parity width combination. Other data operations added in the future would also need new kfuncs to support them.
struct io_bpf_buf_desc is certainly more flexible than the approach in v1, but I worry that fully consuming the iterator in a single kfunc call prevents composing data operations during a single pass over the buffer. For example, if the input data needs to be encrypted and then parity needs to be generated over the encrypted data, calling an encryption kfunc and a parity generation kfunc in sequence would perform two passes over the data. To do it in a single pass, you'd need something like an "encrypt_and_gen_parity()" kfunc, which seems very niche.
The hope was that io_uring BPF programs would provide a generic approach to registered buffer data manipulation, but the dependency on a large set of kfuncs seems a significant hurdle. I suspect a fully generic approach would require essentially reproducing the struct iov_iter abstraction in kfunc form.
Here's an alternate idea I've been mulling over: could ublk devices implement the proposed user dma-buf abstraction
(https://lore.kernel.org/io-uring/[email protected]/)?
A client process could then pre-register its data pages as a dma-buf against a ublk device, which would provide a file handle to the ublk server for the dma-buf memory that it could mmap() into its address space. A ublk I/O using a dma-buf would be dispatched to the ublk server with the virtual address of the data buffer in the ublk server's address space. Then the ublk server could directly access the data buffer in userspace.
From what I understand, this is similar to the original concept for ublk zero-copy, but with an additional pre-registration step. I think that may address the concern about mapping pages into the ublk server's address space in the I/O path. It would be an opt-in optimization for client applications, so it can require that the pages that will be shared with the ublk server are MAP_SHARED.
Quoted 125 lines
Signed-off-by Ming Lei <[email protected]>--- io_uring/bpf_op.c | 320 +++++++++++++++++++++++++++++++++++++++++++++- io_uring/bpf_op.h | 3 +- 2 files changed, 321 insertions(+), 2 deletions(-)io_uring/bpf_op.c +85
-1@@ -10,9 +10,11 @@#include <linux/btf.h> #include <linux/btf_ids.h> #include <linux/filter.h> +#include <linux/uio.h> #include <uapi/linux/io_uring.h> #include "io_uring.h" #include "register.h" +#include "rsrc.h" #include "bpf_op.h" static inline unsigned char uring_bpf_get_op (u32 op_flags)@@ -47,6 +49,7 @@ int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) data->opf = opf; data->ops = ops; + data->issue_flags =0 ; ret = ops->prep_fn (data, sqe);if (!ret) {/* Only increment refcount on success (uring_lock already held) */ @@ -74,7 +77,13 @@ static int __io_uring_bpf_issue(struct io_kiocb *req)int io_uring_bpf_issue (struct io_kiocb *req,unsigned int issue_flags) { -return __io_uring_bpf_issue (req); + struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data); +int ret; + + data->issue_flags = issue_flags; + ret =__io_uring_bpf_issue (req); + data->issue_flags =0 ; +return ret; }void io_uring_bpf_fail (struct io_kiocb *req)@@ -291,6 +300,235 @@ static struct bpf_struct_ops bpf_uring_bpf_ops = { .owner = THIS_MODULE, }; +/* +* Helper to copy data between two iov_iters using page extraction. +* Extracts pages from source iterator and copies them to destination. +* Returns number of bytes copied or negative error code. +*/ +static ssize_t io_bpf_copy_iters (struct iov_iter *src,struct iov_iter *dst, +size_t len) +{ +#define MAX_PAGES_PER_LOOP 32 +struct page *pages[MAX_PAGES_PER_LOOP ]; +size_t total_copied =0 ; +bool need_unpin; + + need_unpin =iov_iter_extract_will_pin (src); + +while (len >0 ) { +struct page **page_array = pages; +size_t offset, copied =0 ; +ssize_t extracted; +unsigned int nr_pages; +size_t chunk_len; +int i; + + chunk_len =min_t (size_t, len,MAX_PAGES_PER_LOOP *PAGE_SIZE ); + extracted =iov_iter_extract_pages (src, &page_array, chunk_len, +MAX_PAGES_PER_LOOP ,0 , &offset); +if (extracted <=0 ) { +if (total_copied >0 ) +break ; +return extracted <0 ? extracted : -EFAULT ; + } + + nr_pages =DIV_ROUND_UP (offset + extracted,PAGE_SIZE ); + +for (i =0 ; i < nr_pages && copied < extracted; i++) { +size_t page_offset = (i ==0 ) ? offset :0 ; +size_t page_len =min_t (size_t, extracted - copied, +PAGE_SIZE - page_offset); +size_t n; + + n =copy_page_to_iter (page_array[i], page_offset, page_len, dst); + copied += n; +if (n < page_len) +break ; + } + +if (need_unpin) +unpin_user_pages (page_array, nr_pages); + + total_copied += copied; + len -= copied; + +if (copied < extracted) +break ; + } + +return total_copied; +#undef MAX_PAGES_PER_LOOP +} + +/* +* Helper to import fixed buffer (FIXED or KFIXED). +* Must be called with submit lock held. +* +* FIXED: addr is absolute userspace address within buffer +* KFIXED: addr is offset from buffer start +* +* Returns node with incremented refcount on success, ERR_PTR on failure. +*/ +static struct io_rsrc_node *io_bpf_import_fixed_buf (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir) +{ +struct io_rsrc_node *node;
io_uring/bpf_op.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 387
Quoted 10 lines
io_uring/bpf_op.c +10
-0+struct io_mapped_ubuf *imu; +int ret; + + node =io_rsrc_node_lookup (&ctx->buf_table, desc->buf_index); +if (!node) +return ERR_PTR (-EFAULT ); + + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return ERR_PTR (-EFAULT );
This is already checked in io_import_fixed()? Same comment for io_bpf_import_reg_vec().
io_uring/bpf_op.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 393
Quoted 6 lines
io_uring/bpf_op.c +6
-0+ + node->refs++; + + ret =io_import_fixed (ddir, iter, imu, desc->addr, desc->len); +if (ret) { + node->refs--;
Just wait to increment node->refs until io_import_fixed() succeeds? Same comment for io_bpf_import_reg_vec().
Quoted 110 lines
io_uring/bpf_op.c +110
-0+return ERR_PTR (ret); + } + +return node; +} + +/* +* Helper to import registered vectored buffer (KVEC). +* Must be called with submit lock held. +* +* addr: userspace iovec pointer +* len: number of iovecs +* buf_index: registered buffer index +* +* Returns node with incremented refcount on success, ERR_PTR on failure. +* Caller must call io_vec_free(vec) after use. +*/ +static struct io_rsrc_node *io_bpf_import_reg_vec (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +int ret; + + node =io_rsrc_node_lookup (&ctx->buf_table, desc->buf_index); +if (!node) +return ERR_PTR (-EFAULT ); + + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return ERR_PTR (-EFAULT ); + + node->refs++; + +/* Prepare iovec from userspace */ + ret =__io_prep_reg_iovec (vec,u64_to_user_ptr (desc->addr), + desc->len, ctx->compat,NULL ); +if (ret) + goto err; + +/* Import vectored buffer from registered buffer */ + ret =__io_import_reg_vec (ddir, iter, imu, vec, desc->len,NULL ); +if (ret) + goto err; + +return node; +err: + node->refs--; +return ERR_PTR (ret); +} + +/* +* Helper to import a vectored user buffer (VEC) into iou_vec. +* Allocates space in vec and copies iovec from userspace. +* +* Returns 0 on success, negative error code on failure. +* Caller must call io_vec_free(vec) after use. +*/ +static int io_bpf_import_vec_buf (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +unsigned nr_vecs = desc->len; +struct iovec *iov; +size_t total_len =0 ; +void *res; +int ret, i; + +if (nr_vecs > vec->nr) { + ret =io_vec_realloc (vec, nr_vecs); +if (ret) +return ret; + } + + iov = vec->iovec; + res =iovec_from_user (u64_to_user_ptr (desc->addr), nr_vecs, + nr_vecs, iov, ctx->compat); +if (IS_ERR (res)) +return PTR_ERR (res); + +for (i =0 ; i < nr_vecs; i++) + total_len += iov[i].iov_len; + +iov_iter_init (iter, ddir, iov, nr_vecs, total_len); +return 0 ; +} + +/* +* Helper to import a buffer into an iov_iter based on io_bpf_buf_desc. +* Supports all 5 buffer types: USER, FIXED, VEC, KFIXED, KVEC. +* Must be called with submit lock held for FIXED/KFIXED/KVEC types. +* +* @ctx: ring context +* @iter: output iterator +* @desc: buffer descriptor +* @ddir: direction (ITER_SOURCE for source, ITER_DEST for destination) +* @vec: iou_vec for VEC/KVEC types (caller must call io_vec_free after use) +* +* Returns node pointer (may be NULL for USER/VEC), or ERR_PTR on failure. +* Caller must drop node reference when done if non-NULL. +*/ +static struct io_rsrc_node *io_bpf_import_buffer (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +int ret;
io_uring/bpf_op.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 513
Quoted 10 lines
io_uring/bpf_op.c +10
-0+ +switch (desc->type) { +case IO_BPF_BUF_USER : +/* Plain user buffer */ + ret =import_ubuf (ddir,u64_to_user_ptr (desc->addr), + desc->len, iter); +return ret ?ERR_PTR (ret) :NULL ; + +case IO_BPF_BUF_FIXED : +case IO_BPF_BUF_KFIXED :
Seems like these are handled identically, are both needed?
Quoted 34 lines
io_uring/bpf_op.c +27
-0+/* FIXED: addr is absolute address within buffer */ +/* KFIXED: addr is offset from buffer start */ +return io_bpf_import_fixed_buf (ctx, iter, desc, ddir); + +case IO_BPF_BUF_VEC : +/* Vectored user buffer - addr is iovec ptr, len is nr_vecs */ + ret =io_bpf_import_vec_buf (ctx, iter, desc, ddir, vec); +return ret ?ERR_PTR (ret) :NULL ; + +case IO_BPF_BUF_REG_VEC : +/* Registered vectored buffer */ +return io_bpf_import_reg_vec (ctx, iter, desc, ddir, vec); + +default : +return ERR_PTR (-EINVAL ); + } +} +__bpf_kfunc_start_defs (); __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res) {@@ -300,10 +538,90 @@ __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)req_set_fail (req);io_req_set_res (req, res,0 ); } + +/** +* uring_bpf_memcpy - Copy data between io_uring BPF buffers +* @data: BPF request data containing request context +* @dst: Destination buffer descriptor +* @src: Source buffer descriptor +* +* Copies data from source buffer to destination buffer. +* Supports all 5 buffer types: USER, FIXED, VEC, KFIXED, REG_VEC.
io_uring/bpf_op.c __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res) · line 559
Quoted 10 lines
io_uring/bpf_op.c +10
-0+* The copy length is min of actual buffer sizes (for VEC types, +* total bytes across all vectors, not nr_vecs). +* +* Returns: Number of bytes copied on success, negative error code on failure +*/ +__bpf_kfunc ssize_t uring_bpf_memcpy(const struct uring_bpf_data *data, +struct io_bpf_buf_desc *dst, +struct io_bpf_buf_desc *src) +{ +struct io_kiocb *req =cmd_to_io_kiocb ((void *)data);
Could omit the explicit void * cast
Best,
Caleb
Quoted 88 lines
io_uring/bpf_op.c +61
-0+struct io_ring_ctx *ctx = req->ctx; +unsigned int issue_flags = data->issue_flags; +struct io_rsrc_node *src_node, *dst_node; +struct iov_iter src_iter, dst_iter; +struct iou_vec src_vec = {}; +struct iou_vec dst_vec = {}; +ssize_t ret; +size_t len; + +/* Validate buffer types */ +if (src->type >IO_BPF_BUF_REG_VEC || dst->type >IO_BPF_BUF_REG_VEC ) +return -EINVAL ; + +io_ring_submit_lock (ctx, issue_flags); + +/* Import source buffer */ + src_node =io_bpf_import_buffer (ctx, &src_iter, src,ITER_SOURCE , + &src_vec); +if (IS_ERR (src_node)) { + ret =PTR_ERR (src_node); + goto unlock; + } + +/* Import destination buffer */ + dst_node =io_bpf_import_buffer (ctx, &dst_iter, dst,ITER_DEST , + &dst_vec); +if (IS_ERR (dst_node)) { + ret =PTR_ERR (dst_node); + goto put_src; + } + +/* +* Calculate copy length from actual iterator sizes. +* For VEC types, desc->len is nr_vecs, not total bytes. +*/ + len =min (iov_iter_count (&src_iter),iov_iter_count (&dst_iter)); +if (!len) { + ret =0 ; + goto put_dst; + } +if (len >MAX_RW_COUNT ) { + ret = -EINVAL ; + goto put_dst; + } + +/* Copy data between iterators */ + ret =io_bpf_copy_iters (&src_iter, &dst_iter, len); + +put_dst: +io_vec_free (&dst_vec); +if (dst_node) +io_put_rsrc_node (ctx, dst_node); +put_src: +io_vec_free (&src_vec); +if (src_node) +io_put_rsrc_node (ctx, src_node); +unlock: +io_ring_submit_unlock (ctx, issue_flags); +return ret; +}__bpf_kfunc_end_defs ();BTF_KFUNCS_START (uring_bpf_kfuncs)BTF_ID_FLAGS (func ,uring_bpf_set_result ) +BTF_ID_FLAGS (func, uring_bpf_memcpy)BTF_KFUNCS_END (uring_bpf_kfuncs) static const struct btf_kfunc_id_set uring_kfunc_set = {io_uring/bpf_op.h +2
-2@@ -13,10 +13,11 @@ struct uring_bpf_data {void *req_data;/* not for bpf prog */ const struct uring_bpf_ops *ops;u32 opf; +u32 issue_flags;/* io_uring issue flags */ /* writeable for bpf prog */ u8 pdu[64 -sizeof (void *) - -sizeof (struct uring_bpf_ops *) -sizeof (u32)]; +sizeof (struct uring_bpf_ops *) -2 *sizeof (u32)]; }; typedef int (*uring_bpf_prep_t)(struct uring_bpf_data *data, --2.47.0
Re: [PATCH V2 08/13] io_uring: bpf: add uring_bpf_memcpy() kfunc
Quoted 41 lines · Caleb Sander Mateos
Quoted 19 lines · Ming Lei
Add
uring_bpf_memcpy()kfunc that copies data betweenio_uringBPF buffers. This kfunc supports all 5 buffer types defined inio_bpf_buf_desc:-
IO_BPF_BUF_USER: plain userspace buffer
-IO_BPF_BUF_FIXED: fixed buffer (absolute address within buffer)
-IO_BPF_BUF_VEC: vectored userspace buffer (iovec array)
-IO_BPF_BUF_KFIXED: kernel fixed buffer (offset-based addressing)
-IO_BPF_BUF_KVEC: kernel vectored bufferAdd helper functions for buffer import:
-io_bpf_import_fixed_buf(): handles FIXED/KFIXED types with proper node reference counting
-io_bpf_import_kvec_buf(): handles KVEC using__io_prep_reg_iovec()and__io_import_reg_vec()
-io_bpf_import_buffer(): unified dispatcher for all buffer types
-io_bpf_copy_iters(): page-based copy betweeniov_itersQuoted 2 lines
The kfunc properly manages buffer node references and submit lock for registered buffer access.
Ming,
Thank you for working on this patch series and sorry I didn't have a chance to take a look at this revision earlier.
My main reservation with this approach is that a dedicated kfunc is needed for each data manipulation operation an
io_uringBPF program conceivably wants to perform. Even for the initial RAID parity calculation use case, I imagine we'd need a different kfunc for each RAID stripe data width and parity width combination. Other data operations added in the future would also need new kfuncs to support them.
structio_bpf_buf_descis certainly more flexible than the approach in v1, but I worry that fully consuming the iterator in a single kfunc call prevents composing data operations during a single pass over the buffer. For example, if the input data needs to be encrypted and then parity needs to be generated over the encrypted data, calling an encryption kfunc and a parity generation kfunc in sequence would
Quoted 8 lines
perform two passes over the data. To do it in a single pass, you'd need something like an "
encrypt_and_gen_parity()" kfunc, which seems very niche.
The hope was thatio_uringBPF programs would provide a generic approach to registered buffer data manipulation, but the dependency on a large set of kfuncs seems a significant hurdle. I suspect a fully generic approach would require essentially reproducing the structiov_iterabstraction in kfunc form.
Agree.
XOR/encrypt and copy should not be too hard to support if the kfunc interface is well defined, and there is actually compression requirement in Android's use case, which looks more complicated.
Quoted 17 lines
Here's an alternate idea I've been mulling over: could ublk devices implement the proposed user dma-buf abstraction
(https://lore.kernel.org/io-uring/[email protected]/)?
A client process could then pre-register its data pages as a dma-buf against a ublk device, which would provide a file handle to the ublk server for the dma-buf memory that it couldmmap()into its address space. A ublk I/O using a dma-buf would be dispatched to the ublk server with the virtual address of the data buffer in the ublk server's address space. Then the ublk server could directly access the data buffer in userspace.
From what I understand, this is similar to the original concept for ublk zero-copy, but with an additional pre-registration step. I think that may address the concern about mapping pages into the ublk server's address space in the I/O path. It would be an opt-in optimization for client applications, so it can require that the pages that will be shared with the ublk server areMAP_SHARED.
Yeah, I did look into Pavel's dma buffer based zc.
dma-buf is very handy to share everywhere, also it can avoid runtime dma mapping. The only drawback is that client has to take dma-buf based io-uring interface, so many application may not benefit from it.
Let's see how dma-buf related framework evolves.
Especially, for ublk, the focus is how to wire client dma buffer with request buffer in ublk server side.
Quoted 131 lines
Quoted 131 lines
Signed-off-by Ming Lei <[email protected]>--- io_uring/bpf_op.c | 320 +++++++++++++++++++++++++++++++++++++++++++++- io_uring/bpf_op.h | 3 +- 2 files changed, 321 insertions(+), 2 deletions(-)io_uring/bpf_op.c +90
-1@@ -10,9 +10,11 @@#include <linux/btf.h> #include <linux/btf_ids.h> #include <linux/filter.h> +#include <linux/uio.h> #include <uapi/linux/io_uring.h> #include "io_uring.h" #include "register.h" +#include "rsrc.h" #include "bpf_op.h" static inline unsigned char uring_bpf_get_op (u32 op_flags)@@ -47,6 +49,7 @@ int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) data->opf = opf; data->ops = ops; + data->issue_flags =0 ; ret = ops->prep_fn (data, sqe);if (!ret) {/* Only increment refcount on success (uring_lock already held) */ @@ -74,7 +77,13 @@ static int __io_uring_bpf_issue(struct io_kiocb *req)int io_uring_bpf_issue (struct io_kiocb *req,unsigned int issue_flags) { -return __io_uring_bpf_issue (req); + struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data); +int ret; + + data->issue_flags = issue_flags; + ret =__io_uring_bpf_issue (req); + data->issue_flags =0 ; +return ret; }void io_uring_bpf_fail (struct io_kiocb *req)@@ -291,6 +300,235 @@ static struct bpf_struct_ops bpf_uring_bpf_ops = { .owner = THIS_MODULE, }; +/* +* Helper to copy data between two iov_iters using page extraction. +* Extracts pages from source iterator and copies them to destination. +* Returns number of bytes copied or negative error code. +*/ +static ssize_t io_bpf_copy_iters (struct iov_iter *src,struct iov_iter *dst, +size_t len) +{ +#define MAX_PAGES_PER_LOOP 32 +struct page *pages[MAX_PAGES_PER_LOOP ]; +size_t total_copied =0 ; +bool need_unpin; + + need_unpin =iov_iter_extract_will_pin (src); + +while (len >0 ) { +struct page **page_array = pages; +size_t offset, copied =0 ; +ssize_t extracted; +unsigned int nr_pages; +size_t chunk_len; +int i; + + chunk_len =min_t (size_t, len,MAX_PAGES_PER_LOOP *PAGE_SIZE ); + extracted =iov_iter_extract_pages (src, &page_array, chunk_len, +MAX_PAGES_PER_LOOP ,0 , &offset); +if (extracted <=0 ) { +if (total_copied >0 ) +break ; +return extracted <0 ? extracted : -EFAULT ; + } + + nr_pages =DIV_ROUND_UP (offset + extracted,PAGE_SIZE ); + +for (i =0 ; i < nr_pages && copied < extracted; i++) { +size_t page_offset = (i ==0 ) ? offset :0 ; +size_t page_len =min_t (size_t, extracted - copied, +PAGE_SIZE - page_offset); +size_t n; + + n =copy_page_to_iter (page_array[i], page_offset, page_len, dst); + copied += n; +if (n < page_len) +break ; + } + +if (need_unpin) +unpin_user_pages (page_array, nr_pages); + + total_copied += copied; + len -= copied; + +if (copied < extracted) +break ; + } + +return total_copied; +#undef MAX_PAGES_PER_LOOP +} + +/* +* Helper to import fixed buffer (FIXED or KFIXED). +* Must be called with submit lock held. +* +* FIXED: addr is absolute userspace address within buffer +* KFIXED: addr is offset from buffer start +* +* Returns node with incremented refcount on success, ERR_PTR on failure. +*/ +static struct io_rsrc_node *io_bpf_import_fixed_buf (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +int ret; + + node =io_rsrc_node_lookup (&ctx->buf_table, desc->buf_index); +if (!node)
Quoted 8 lines
io_uring/bpf_op.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 387
Quoted 5 lines
io_uring/bpf_op.c +5
-0+return ERR_PTR (-EFAULT ); + + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return ERR_PTR (-EFAULT );This is already checked in
io_import_fixed()? Same comment forio_bpf_import_reg_vec().
OK.
Quoted 10 lines
io_uring/bpf_op.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 393
Quoted 7 lines
io_uring/bpf_op.c +6
-0+ + node->refs++; + + ret =io_import_fixed (ddir, iter, imu, desc->addr, desc->len); +if (ret) { + node->refs--;Just wait to increment
node->refsuntilio_import_fixed()succeeds? Same comment forio_bpf_import_reg_vec().
Fine.
Quoted 115 lines
Quoted 115 lines
io_uring/bpf_op.c +114
-0+return ERR_PTR (ret); + } + +return node; +} + +/* +* Helper to import registered vectored buffer (KVEC). +* Must be called with submit lock held. +* +* addr: userspace iovec pointer +* len: number of iovecs +* buf_index: registered buffer index +* +* Returns node with incremented refcount on success, ERR_PTR on failure. +* Caller must call io_vec_free(vec) after use. +*/ +static struct io_rsrc_node *io_bpf_import_reg_vec (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +struct io_rsrc_node *node; +struct io_mapped_ubuf *imu; +int ret; + + node =io_rsrc_node_lookup (&ctx->buf_table, desc->buf_index); +if (!node) +return ERR_PTR (-EFAULT ); + + imu = node->buf; +if (!(imu->dir & (1 << ddir))) +return ERR_PTR (-EFAULT ); + + node->refs++; + +/* Prepare iovec from userspace */ + ret =__io_prep_reg_iovec (vec,u64_to_user_ptr (desc->addr), + desc->len, ctx->compat,NULL ); +if (ret) + goto err; + +/* Import vectored buffer from registered buffer */ + ret =__io_import_reg_vec (ddir, iter, imu, vec, desc->len,NULL ); +if (ret) + goto err; + +return node; +err: + node->refs--; +return ERR_PTR (ret); +} + +/* +* Helper to import a vectored user buffer (VEC) into iou_vec. +* Allocates space in vec and copies iovec from userspace. +* +* Returns 0 on success, negative error code on failure. +* Caller must call io_vec_free(vec) after use. +*/ +static int io_bpf_import_vec_buf (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +unsigned nr_vecs = desc->len; +struct iovec *iov; +size_t total_len =0 ; +void *res; +int ret, i; + +if (nr_vecs > vec->nr) { + ret =io_vec_realloc (vec, nr_vecs); +if (ret) +return ret; + } + + iov = vec->iovec; + res =iovec_from_user (u64_to_user_ptr (desc->addr), nr_vecs, + nr_vecs, iov, ctx->compat); +if (IS_ERR (res)) +return PTR_ERR (res); + +for (i =0 ; i < nr_vecs; i++) + total_len += iov[i].iov_len; + +iov_iter_init (iter, ddir, iov, nr_vecs, total_len); +return 0 ; +} + +/* +* Helper to import a buffer into an iov_iter based on io_bpf_buf_desc. +* Supports all 5 buffer types: USER, FIXED, VEC, KFIXED, KVEC. +* Must be called with submit lock held for FIXED/KFIXED/KVEC types. +* +* @ctx: ring context +* @iter: output iterator +* @desc: buffer descriptor +* @ddir: direction (ITER_SOURCE for source, ITER_DEST for destination) +* @vec: iou_vec for VEC/KVEC types (caller must call io_vec_free after use) +* +* Returns node pointer (may be NULL for USER/VEC), or ERR_PTR on failure. +* Caller must drop node reference when done if non-NULL. +*/ +static struct io_rsrc_node *io_bpf_import_buffer (struct io_ring_ctx *ctx, +struct iov_iter *iter, +const struct io_bpf_buf_desc *desc, +int ddir,struct iou_vec *vec) +{ +int ret; + +switch (desc->type) { +case IO_BPF_BUF_USER : +/* Plain user buffer */
Quoted 8 lines
io_uring/bpf_op.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 513
Quoted 6 lines
io_uring/bpf_op.c +6
-0+ ret =import_ubuf (ddir,u64_to_user_ptr (desc->addr), + desc->len, iter); +return ret ?ERR_PTR (ret) :NULL ; + +case IO_BPF_BUF_FIXED : +case IO_BPF_BUF_KFIXED :Seems like these are handled identically, are both needed?
Yeah, but adding two could be more flexible.
Quoted 39 lines
Quoted 39 lines
io_uring/bpf_op.c +31
-0+/* FIXED: addr is absolute address within buffer */ +/* KFIXED: addr is offset from buffer start */ +return io_bpf_import_fixed_buf (ctx, iter, desc, ddir); + +case IO_BPF_BUF_VEC : +/* Vectored user buffer - addr is iovec ptr, len is nr_vecs */ + ret =io_bpf_import_vec_buf (ctx, iter, desc, ddir, vec); +return ret ?ERR_PTR (ret) :NULL ; + +case IO_BPF_BUF_REG_VEC : +/* Registered vectored buffer */ +return io_bpf_import_reg_vec (ctx, iter, desc, ddir, vec); + +default : +return ERR_PTR (-EINVAL ); + } +} +__bpf_kfunc_start_defs (); __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res) {@@ -300,10 +538,90 @@ __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)req_set_fail (req);io_req_set_res (req, res,0 ); } + +/** +* uring_bpf_memcpy - Copy data between io_uring BPF buffers +* @data: BPF request data containing request context +* @dst: Destination buffer descriptor +* @src: Source buffer descriptor +* +* Copies data from source buffer to destination buffer. +* Supports all 5 buffer types: USER, FIXED, VEC, KFIXED, REG_VEC. +* The copy length is min of actual buffer sizes (for VEC types, +* total bytes across all vectors, not nr_vecs). +* +* Returns: Number of bytes copied on success, negative error code on failure
Quoted 8 lines
io_uring/bpf_op.c __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res) · line 559
Quoted 6 lines
io_uring/bpf_op.c +6
-0+*/ +__bpf_kfunc ssize_t uring_bpf_memcpy(const struct uring_bpf_data *data, +struct io_bpf_buf_desc *dst, +struct io_bpf_buf_desc *src) +{ +struct io_kiocb *req =cmd_to_io_kiocb ((void *)data);Could omit the explicit void * cast
OK.
Thanks,
Ming
Re: [PATCH V2 08/13] io_uring: bpf: add uring_bpf_memcpy() kfunc
In reply to Caleb Sander MateosQuoted 28 lines · Caleb Sander Mateos
Quoted 19 lines · Ming Lei
Add
uring_bpf_memcpy()kfunc that copies data betweenio_uringBPF buffers. This kfunc supports all 5 buffer types defined inio_bpf_buf_desc:-
IO_BPF_BUF_USER: plain userspace buffer
-IO_BPF_BUF_FIXED: fixed buffer (absolute address within buffer)
-IO_BPF_BUF_VEC: vectored userspace buffer (iovec array)
-IO_BPF_BUF_KFIXED: kernel fixed buffer (offset-based addressing)
-IO_BPF_BUF_KVEC: kernel vectored bufferAdd helper functions for buffer import:
-io_bpf_import_fixed_buf(): handles FIXED/KFIXED types with proper node reference counting
-io_bpf_import_kvec_buf(): handles KVEC using__io_prep_reg_iovec()and__io_import_reg_vec()
-io_bpf_import_buffer(): unified dispatcher for all buffer types
-io_bpf_copy_iters(): page-based copy betweeniov_itersQuoted 2 lines
The kfunc properly manages buffer node references and submit lock for registered buffer access.
Ming,
Thank you for working on this patch series and sorry I didn't have a chance to take a look at this revision earlier.
Quoted 16 lines
My main reservation with this approach is that a dedicated kfunc is needed for each data manipulation operation an
io_uringBPF program conceivably wants to perform. Even for the initial RAID parity calculation use case, I imagine we'd need a different kfunc for each RAID stripe data width and parity width combination. Other data operations added in the future would also need new kfuncs to support them.
structio_bpf_buf_descis certainly more flexible than the approach in v1, but I worry that fully consuming the iterator in a single kfunc call prevents composing data operations during a single pass over the buffer. For example, if the input data needs to be encrypted and then parity needs to be generated over the encrypted data, calling an encryption kfunc and a parity generation kfunc in sequence would perform two passes over the data. To do it in a single pass, you'd need something like an "encrypt_and_gen_parity()" kfunc, which seems very niche.
parity buffer should be owned by ublk server.
So here you want to calculate parity over encrypted registered buffer in one pass.
Technically it should be doable, however it depends if the in-kernel encryption lib supports it, or it may be too slow.
Quoted 5 lines
The hope was that
io_uringBPF programs would provide a generic approach to registered buffer data manipulation, but the dependency on a large set of kfuncs seems a significant hurdle. I suspect a fully generic approach would require essentially reproducing the structiov_iterabstraction in kfunc form.
bpf iterator might be the solution, but looks one limit is that it is hard or not possible to produce one writeable buffer from bpf_iter_next() kfunc, so one encrypt kfunc is still needed. If the encrypt lib supports to generate parity data in one pass, the kfunc may support it, however, I feel it might be hard.
Also I guess one extra pass for calculating parity shouldn't be too bad, and here one extra decrypt kfunc is enough, and calculating & storing parity can be done directly because the parity buffer is not registered.
Thanks,
Ming
PATCH v2 9/13 selftests/io_uring: update mini liburing
In reply to Ming LeiFrom: Pavel Begunkov <[email protected]>
Add a helper for creating a ring from parameters and add support for IORING_SETUP_NO_SQARRAY.
Add io_uring_unregister_buffers().
--- tools/include/io_uring/mini_liburing.h | 67 ++++++++++++++++++++------ 1 file changed, 53 insertions(+), 14 deletions(-)
tools/include/io_uring/mini_liburing.h +53 -14
@@ -1,11 +1,13 @@/* SPDX-License-Identifier: MIT */ #include <linux/io_uring.h> +#include <signal.h> #include <sys/mman.h> #include <sys/syscall.h> #include <stdio.h> #include <string.h> #include <unistd.h> +#include <sys/uio.h> struct io_sq_ring {unsigned int *head;
@@ -55,6 +57,7 @@ struct io_uring {struct io_uring_sq sq;struct io_uring_cq cq;int ring_fd; +unsigned flags; };#if defined(__x86_64) || defined(__i386__)
@@ -72,7 +75,14 @@ static inline int io_uring_mmap(int fd, struct io_uring_params *p,void *ptr;int ret; - sq->ring_sz = p->sq_off.array + p->sq_entries *sizeof (unsigned int ); +if (p->flags &IORING_SETUP_NO_SQARRAY ) { + sq->ring_sz = p->cq_off.cqes; + sq->ring_sz += p->cq_entries *sizeof (struct io_uring_cqe ); + }else { + sq->ring_sz = p->sq_off.array; + sq->ring_sz += p->sq_entries *sizeof (unsigned int ); + } + ptr =mmap (0 , sq->ring_sz,PROT_READ |PROT_WRITE ,MAP_SHARED |MAP_POPULATE , fd,IORING_OFF_SQ_RING );if (ptr ==MAP_FAILED )
@@ -83,7 +93,8 @@ static inline int io_uring_mmap(int fd, struct io_uring_params *p, sq->kring_entries = ptr + p->sq_off.ring_entries; sq->kflags = ptr + p->sq_off.flags; sq->kdropped = ptr + p->sq_off.dropped; - sq->array = ptr + p->sq_off.array; +if (!(p->flags &IORING_SETUP_NO_SQARRAY )) + sq->array = ptr + p->sq_off.array; size = p->sq_entries *sizeof (struct io_uring_sqe ); sq->sqes = mmap(0, size, PROT_READ | PROT_WRITE,
@@ -126,28 +137,39 @@ static inline int io_uring_enter(int fd, unsigned int to_submit, flags, sig, _NSIG / 8); } -static inline int io_uring_queue_init (unsigned int entries, +static inline int io_uring_queue_init_params (unsigned int entries,struct io_uring *ring, -unsigned int flags) +struct io_uring_params *p) { -struct io_uring_params p;int fd, ret;memset (ring,0 ,sizeof (*ring)); -memset (&p,0 ,sizeof (p)); - p.flags = flags; - fd =io_uring_setup (entries, &p); + fd =io_uring_setup (entries, p);if (fd <0 )return fd; - ret =io_uring_mmap (fd, &p, &ring->sq, &ring->cq); -if (!ret) + ret =io_uring_mmap (fd, p, &ring->sq, &ring->cq); +if (!ret) { ring->ring_fd = fd; -else + ring->flags = p->flags; + }else {close (fd); + }return ret; } +static inline int io_uring_queue_init (unsigned int entries, +struct io_uring *ring, +unsigned int flags) +{ +struct io_uring_params p; + +memset (&p,0 ,sizeof (p)); + p.flags = flags; + +return io_uring_queue_init_params (entries, ring, &p); +} +/* Get a sqe */ static inline struct io_uring_sqe *io_uring_get_sqe (struct io_uring *ring) {
@@ -199,10 +221,18 @@ static inline int io_uring_submit(struct io_uring *ring) ktail = *sq->ktail; to_submit = sq->sqe_tail - sq->sqe_head; -for (submitted =0 ; submitted < to_submit; submitted++) { -read_barrier (); - sq->array[ktail++ & mask] = sq->sqe_head++ & mask; + +if (!(ring->flags &IORING_SETUP_NO_SQARRAY )) { +for (submitted =0 ; submitted < to_submit; submitted++) { +read_barrier (); + sq->array[ktail++ & mask] = sq->sqe_head++ & mask; + } + }else { + ktail += to_submit; + sq->sqe_head += to_submit; + submitted = to_submit; } +if (!submitted)return 0 ;
@@ -255,6 +285,15 @@ static inline int io_uring_register_buffers(struct io_uring *ring,return (ret <0 ) ? -errno : ret; } +static inline int io_uring_unregister_buffers (struct io_uring *ring) +{ +int ret; + + ret =syscall (__NR_io_uring_register, ring->ring_fd, +IORING_UNREGISTER_BUFFERS ,NULL ,0 ); +return (ret <0 ) ? -errno : ret; +} +static inline void io_uring_prep_send (struct io_uring_sqe *sqe,int sockfd,const void *buf,size_t len,int flags) {
Signature · 1 line
2.47.0
PATCH v2 10/13 selftests/io_uring: add BPF struct_ops and kfunc tests
In reply to Ming LeiAdd selftests for io_uring BPF struct_ops and kfunc functionality:
- basic_bpf_ops: Tests IORING_OP_BPF struct_ops registration and execution with multiple struct_ops support
The test framework includes:
- runner.c: Main test runner with auto-discovery
- iou_test.h: Common test infrastructure
- Makefile: Build system with BPF skeleton generation
--- tools/testing/selftests/Makefile | 3 +- tools/testing/selftests/io_uring/.gitignore | 2 + tools/testing/selftests/io_uring/Makefile | 172 ++++++++++++++ .../selftests/io_uring/basic_bpf_ops.bpf.c | 94 ++++++++ .../selftests/io_uring/basic_bpf_ops.c | 215 ++++++++++++++++++ .../selftests/io_uring/include/iou_test.h | 98 ++++++++ tools/testing/selftests/io_uring/runner.c | 206 +++++++++++++++++ 7 files changed, 789 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/io_uring/.gitignore create mode 100644 tools/testing/selftests/io_uring/Makefile create mode 100644 tools/testing/selftests/io_uring/basic_bpf_ops.bpf.c create mode 100644 tools/testing/selftests/io_uring/basic_bpf_ops.c create mode 100644 tools/testing/selftests/io_uring/include/iou_test.h create mode 100644 tools/testing/selftests/io_uring/runner.c
tools/testing/selftests/Makefile +2 -1
@@ -44,6 +44,7 @@ TARGETS += futex TARGETS += gpio TARGETS += hid TARGETS += intel_pstate +TARGETS += io_uring TARGETS += iommu TARGETS += ipc TARGETS += ir
@@ -147,7 +148,7 @@ endif # User can optionally provide a TARGETS skiplist. By default we skip # targets using BPF since it has cutting edge build time dependencies # which require more effort to install. -SKIP_TARGETS ?= bpf sched_ext +SKIP_TARGETS ?= bpf io_uring sched_ext ifneq ($(SKIP_TARGETS),) TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS)) override TARGETS := $(TMP)
tools/testing/selftests/io_uring/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +/build/ +/runner
tools/testing/selftests/io_uring/Makefile new file +172 -0
172 lines not on this page · raw message
tools/testing/selftests/io_uring/basic_bpf_ops.bpf.c new file +94 -0
94 lines not on this page · raw message
tools/testing/selftests/io_uring/basic_bpf_ops.c new file +215 -0
215 lines not on this page · raw message
tools/testing/selftests/io_uring/include/iou_test.h new file +98 -0
98 lines not on this page · raw message
tools/testing/selftests/io_uring/runner.c new file +206 -0
206 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH v2 11/13 selftests/io_uring: add bpf_memcpy selftest for uring_bpf_memcpy() kfunc
In reply to Ming LeiAdd a selftest to verify the uring_bpf_memcpy() kfunc works correctly with different buffer types. The test uses BPF struct_ops to implement prep/issue/fail/cleanup operations for the IORING_OP_BPF opcode.
Three test cases are included:
- copy_user_to_user(): Tests IO_BPF_BUF_USER buffer type (flat userspace buffer) for both source and destination
- copy_vec_to_vec(): Tests IO_BPF_BUF_VEC buffer type (iovec array split into multiple chunks) for both source and destination
- copy_user_to_vec(): Tests mixed buffer types with USER source and VEC destination
All tests allocate source/destination buffers, fill the source with a pattern, invoke uring_bpf_memcpy() via io_uring submission, and verify the copy succeeded by checking CQE result and destination buffer contents.
--- tools/testing/selftests/io_uring/Makefile | 1 + .../selftests/io_uring/bpf_memcpy.bpf.c | 98 +++++ tools/testing/selftests/io_uring/bpf_memcpy.c | 374 ++++++++++++++++++ 3 files changed, 473 insertions(+) create mode 100644 tools/testing/selftests/io_uring/bpf_memcpy.bpf.c create mode 100644 tools/testing/selftests/io_uring/bpf_memcpy.c
tools/testing/selftests/io_uring/Makefile +1 -0
@@ -150,6 +150,7 @@ all_test_bpfprogs := $(foreach prog,$(wildcard *.bpf.c),$(INCLUDE_DIR)/$(patsubs auto-test-targets := \ basic_bpf_ops \ + bpf_memcpy \ testcase-targets := $(addsuffix .o,$(addprefix $(IOUOBJ_DIR)/,$(auto-test-targets)))
tools/testing/selftests/io_uring/bpf_memcpy.bpf.c new file +98 -0
98 lines not on this page · raw message
tools/testing/selftests/io_uring/bpf_memcpy.c new file +374 -0
374 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH v2 12/13 selftests/io_uring: add copy_user_to_fixed() and copy_fixed_to_user() bpf_memcpy tests
In reply to Ming LeiAdd tests for IO_BPF_BUF_FIXED buffer type to verify uring_bpf_memcpy() kfunc works correctly with registered fixed buffers.
Changes:
- Add io_uring_unregister_buffers() to mini_liburing.h
- Add fixed buffer index tracking (src_buf_index/dst_buf_index) to test_ctx
- Add register_fixed_bufs()/unregister_fixed_bufs() helpers to manage buffer registration with the io_uring ring
- Update allocate_buf()/free_buf() to handle IO_BPF_BUF_FIXED type
- Add copy_user_to_fixed(): Tests USER source to FIXED destination
- Add copy_fixed_to_user(): Tests FIXED source to USER destination
--- tools/testing/selftests/io_uring/bpf_memcpy.c | 98 ++++++++++++++++++- 1 file changed, 97 insertions(+), 1 deletion(-)
tools/testing/selftests/io_uring/bpf_memcpy.c +97 -1
@@ -37,6 +37,10 @@ struct test_ctx {struct iovec dst_vec[MAX_VECS ];int src_nr_vec;int dst_nr_vec; + +/* Fixed buffer support */ +__u16 src_buf_index; +__u16 dst_buf_index; };static enum iou_test_status bpf_setup (struct test_ctx *ctx)
@@ -119,6 +123,7 @@ static int allocate_buf(char **buf, size_t size, __u8 buf_type,switch (buf_type) {case IO_BPF_BUF_USER : +case IO_BPF_BUF_FIXED : p =aligned_alloc (4096 , size);if (!p)return -ENOMEM ;
@@ -150,6 +155,7 @@ static void free_buf(char *buf, __u8 buf_type)switch (buf_type) {case IO_BPF_BUF_USER :case IO_BPF_BUF_VEC : +case IO_BPF_BUF_FIXED :free (buf);break ;default :
@@ -157,8 +163,48 @@ static void free_buf(char *buf, __u8 buf_type) } } +static enum iou_test_status register_fixed_bufs (struct test_ctx *ctx) +{ +struct iovec iovecs[2 ]; +int nr_iovecs =0 ; +int ret; + +if (ctx->src_type ==IO_BPF_BUF_FIXED ) { + ctx->src_buf_index = nr_iovecs; + iovecs[nr_iovecs].iov_base = ctx->src_buf; + iovecs[nr_iovecs].iov_len = ctx->src_buf_size; + nr_iovecs++; + } + +if (ctx->dst_type ==IO_BPF_BUF_FIXED ) { + ctx->dst_buf_index = nr_iovecs; + iovecs[nr_iovecs].iov_base = ctx->dst_buf; + iovecs[nr_iovecs].iov_len = ctx->dst_buf_size; + nr_iovecs++; + } + +if (nr_iovecs ==0 ) +return IOU_TEST_PASS ; + + ret =io_uring_register_buffers (&ctx->ring, iovecs, nr_iovecs); +if (ret) { +IOU_ERR ("Failed to register buffers: %d" , ret); +return IOU_TEST_FAIL ; + } + +return IOU_TEST_PASS ; +} + +static void unregister_fixed_bufs (struct test_ctx *ctx) +{ +if (ctx->src_type ==IO_BPF_BUF_FIXED || + ctx->dst_type ==IO_BPF_BUF_FIXED ) +io_uring_unregister_buffers (&ctx->ring); +} +static enum iou_test_status allocate_bufs (struct test_ctx *ctx) { +enum iou_test_status status;int ret; ret = allocate_buf(&ctx->src_buf, ctx->src_buf_size, ctx->src_type,
@@ -181,6 +227,16 @@ static enum iou_test_status allocate_bufs(struct test_ctx *ctx)memset (ctx->src_buf,TEST_PATTERN , ctx->src_buf_size);memset (ctx->dst_buf,0 , ctx->dst_buf_size); +/* Register fixed buffers if needed */ + status =register_fixed_bufs (ctx); +if (status !=IOU_TEST_PASS ) { +free_buf (ctx->dst_buf, ctx->dst_type); + ctx->dst_buf =NULL ; +free_buf (ctx->src_buf, ctx->src_type); + ctx->src_buf =NULL ; +return status; + } +/* Build buffer descriptors */ memset (ctx->descs,0 ,sizeof (ctx->descs)); ctx->descs[0 ].type = ctx->src_type;
@@ -189,6 +245,10 @@ static enum iou_test_status allocate_bufs(struct test_ctx *ctx)if (ctx->src_type ==IO_BPF_BUF_VEC ) { ctx->descs[0 ].addr = (__u64 )(uintptr_t )ctx->src_vec; ctx->descs[0 ].len = ctx->src_nr_vec; + }else if (ctx->src_type ==IO_BPF_BUF_FIXED ) { + ctx->descs[0 ].addr = (__u64 )(uintptr_t )ctx->src_buf; + ctx->descs[0 ].len = ctx->src_buf_size; + ctx->descs[0 ].buf_index = ctx->src_buf_index; }else { ctx->descs[0 ].addr = (__u64 )(uintptr_t )ctx->src_buf; ctx->descs[0 ].len = ctx->src_buf_size;
@@ -197,6 +257,10 @@ static enum iou_test_status allocate_bufs(struct test_ctx *ctx)if (ctx->dst_type ==IO_BPF_BUF_VEC ) { ctx->descs[1 ].addr = (__u64 )(uintptr_t )ctx->dst_vec; ctx->descs[1 ].len = ctx->dst_nr_vec; + }else if (ctx->dst_type ==IO_BPF_BUF_FIXED ) { + ctx->descs[1 ].addr = (__u64 )(uintptr_t )ctx->dst_buf; + ctx->descs[1 ].len = ctx->dst_buf_size; + ctx->descs[1 ].buf_index = ctx->dst_buf_index; }else { ctx->descs[1 ].addr = (__u64 )(uintptr_t )ctx->dst_buf; ctx->descs[1 ].len = ctx->dst_buf_size;
@@ -207,6 +271,8 @@ static enum iou_test_status allocate_bufs(struct test_ctx *ctx)static void free_bufs (struct test_ctx *ctx) { +unregister_fixed_bufs (ctx); +if (ctx->src_buf) {free_buf (ctx->src_buf, ctx->src_type); ctx->src_buf =NULL ;
@@ -332,6 +398,28 @@ static enum iou_test_status copy_user_to_vec(struct test_ctx *ctx)return test_copy (ctx); } +static enum iou_test_status copy_user_to_fixed (struct test_ctx *ctx) +{ + ctx->src_type =IO_BPF_BUF_USER ; + ctx->dst_type =IO_BPF_BUF_FIXED ; + ctx->src_buf_size =TEST_BUF_SIZE ; + ctx->dst_buf_size =TEST_BUF_SIZE ; + ctx->desc ="USER -> FIXED" ; + +return test_copy (ctx); +} + +static enum iou_test_status copy_fixed_to_user (struct test_ctx *ctx) +{ + ctx->src_type =IO_BPF_BUF_FIXED ; + ctx->dst_type =IO_BPF_BUF_USER ; + ctx->src_buf_size =TEST_BUF_SIZE ; + ctx->dst_buf_size =TEST_BUF_SIZE ; + ctx->desc ="FIXED -> USER" ; + +return test_copy (ctx); +} +static enum iou_test_status run (void *ctx_ptr) {struct test_ctx *ctx = ctx_ptr;
@@ -349,6 +437,14 @@ static enum iou_test_status run(void *ctx_ptr)if (status !=IOU_TEST_PASS )return status; + status =copy_user_to_fixed (ctx); +if (status !=IOU_TEST_PASS ) +return status; + + status =copy_fixed_to_user (ctx); +if (status !=IOU_TEST_PASS ) +return status; +return IOU_TEST_PASS ; }
@@ -366,7 +462,7 @@ static void cleanup(void *ctx_ptr)struct iou_test bpf_memcpy_test = { .name ="bpf_memcpy" , - .description ="Test uring_bpf_memcpy() kfunc with USER, VEC, and mixed buffer types" , + .description ="Test uring_bpf_memcpy() kfunc with USER, VEC, FIXED buffer types" , .setup = setup, .run = run, .cleanup = cleanup,
Signature · 1 line
2.47.0
PATCH v2 13/13 selftests/io_uring: add copy_user_to_reg_vec() and copy_reg_vec_to_user() bpf_memcpy tests
In reply to Ming LeiAdd tests for IO_BPF_BUF_REG_VEC buffer type to verify uring_bpf_memcpy() kfunc works correctly with registered vectored buffers.
IO_BPF_BUF_REG_VEC combines the vectored buffer layout (iovec array) with registered buffer optimization - the underlying buffer is registered with io_uring while the descriptor points to an iovec array for flexible scatter-gather access.
Changes:
- Add is_registered_buf() helper to check if buffer type needs registration
- Update allocate_buf()/free_buf() to handle IO_BPF_BUF_REG_VEC type
- Update register_fixed_bufs()/unregister_fixed_bufs() to handle REG_VEC
- Update allocate_bufs() to set up REG_VEC descriptors with buf_index
- Add copy_user_to_reg_vec(): Tests USER source to REG_VEC destination
- Add copy_reg_vec_to_user(): Tests REG_VEC source to USER destination
--- tools/testing/selftests/io_uring/bpf_memcpy.c | 57 +++++++++++++++++-- 1 file changed, 52 insertions(+), 5 deletions(-)
tools/testing/selftests/io_uring/bpf_memcpy.c +52 -5
@@ -130,6 +130,7 @@ static int allocate_buf(char **buf, size_t size, __u8 buf_type, *buf = p;return 0 ;case IO_BPF_BUF_VEC : +case IO_BPF_BUF_REG_VEC :if (nr_vec <=0 || nr_vec >MAX_VECS )return -EINVAL ; p =aligned_alloc (4096 , size);
@@ -156,6 +157,7 @@ static void free_buf(char *buf, __u8 buf_type)case IO_BPF_BUF_USER :case IO_BPF_BUF_VEC :case IO_BPF_BUF_FIXED : +case IO_BPF_BUF_REG_VEC :free (buf);break ;default :
@@ -163,20 +165,25 @@ static void free_buf(char *buf, __u8 buf_type) } } +static inline bool is_registered_buf (__u8 type) +{ +return type ==IO_BPF_BUF_FIXED || type ==IO_BPF_BUF_REG_VEC ; +} +static enum iou_test_status register_fixed_bufs (struct test_ctx *ctx) {struct iovec iovecs[2 ];int nr_iovecs =0 ;int ret; -if (ctx->src_type ==IO_BPF_BUF_FIXED ) { +if (is_registered_buf (ctx->src_type)) { ctx->src_buf_index = nr_iovecs; iovecs[nr_iovecs].iov_base = ctx->src_buf; iovecs[nr_iovecs].iov_len = ctx->src_buf_size; nr_iovecs++; } -if (ctx->dst_type ==IO_BPF_BUF_FIXED ) { +if (is_registered_buf (ctx->dst_type)) { ctx->dst_buf_index = nr_iovecs; iovecs[nr_iovecs].iov_base = ctx->dst_buf; iovecs[nr_iovecs].iov_len = ctx->dst_buf_size;
@@ -197,8 +204,8 @@ static enum iou_test_status register_fixed_bufs(struct test_ctx *ctx)static void unregister_fixed_bufs (struct test_ctx *ctx) { -if (ctx->src_type ==IO_BPF_BUF_FIXED || - ctx->dst_type ==IO_BPF_BUF_FIXED ) +if (is_registered_buf (ctx->src_type) || +is_registered_buf (ctx->dst_type))io_uring_unregister_buffers (&ctx->ring); }
@@ -249,6 +256,10 @@ static enum iou_test_status allocate_bufs(struct test_ctx *ctx) ctx->descs[0 ].addr = (__u64 )(uintptr_t )ctx->src_buf; ctx->descs[0 ].len = ctx->src_buf_size; ctx->descs[0 ].buf_index = ctx->src_buf_index; + }else if (ctx->src_type ==IO_BPF_BUF_REG_VEC ) { + ctx->descs[0 ].addr = (__u64 )(uintptr_t )ctx->src_vec; + ctx->descs[0 ].len = ctx->src_nr_vec; + ctx->descs[0 ].buf_index = ctx->src_buf_index; }else { ctx->descs[0 ].addr = (__u64 )(uintptr_t )ctx->src_buf; ctx->descs[0 ].len = ctx->src_buf_size;
@@ -261,6 +272,10 @@ static enum iou_test_status allocate_bufs(struct test_ctx *ctx) ctx->descs[1 ].addr = (__u64 )(uintptr_t )ctx->dst_buf; ctx->descs[1 ].len = ctx->dst_buf_size; ctx->descs[1 ].buf_index = ctx->dst_buf_index; + }else if (ctx->dst_type ==IO_BPF_BUF_REG_VEC ) { + ctx->descs[1 ].addr = (__u64 )(uintptr_t )ctx->dst_vec; + ctx->descs[1 ].len = ctx->dst_nr_vec; + ctx->descs[1 ].buf_index = ctx->dst_buf_index; }else { ctx->descs[1 ].addr = (__u64 )(uintptr_t )ctx->dst_buf; ctx->descs[1 ].len = ctx->dst_buf_size;
@@ -420,6 +435,30 @@ static enum iou_test_status copy_fixed_to_user(struct test_ctx *ctx)return test_copy (ctx); } +static enum iou_test_status copy_user_to_reg_vec (struct test_ctx *ctx) +{ + ctx->src_type =IO_BPF_BUF_USER ; + ctx->dst_type =IO_BPF_BUF_REG_VEC ; + ctx->src_buf_size =TEST_BUF_SIZE ; + ctx->dst_buf_size =TEST_BUF_SIZE ; + ctx->dst_nr_vec =4 ; + ctx->desc ="USER -> REG_VEC" ; + +return test_copy (ctx); +} + +static enum iou_test_status copy_reg_vec_to_user (struct test_ctx *ctx) +{ + ctx->src_type =IO_BPF_BUF_REG_VEC ; + ctx->dst_type =IO_BPF_BUF_USER ; + ctx->src_buf_size =TEST_BUF_SIZE ; + ctx->dst_buf_size =TEST_BUF_SIZE ; + ctx->src_nr_vec =4 ; + ctx->desc ="REG_VEC -> USER" ; + +return test_copy (ctx); +} +static enum iou_test_status run (void *ctx_ptr) {struct test_ctx *ctx = ctx_ptr;
@@ -445,6 +484,14 @@ static enum iou_test_status run(void *ctx_ptr)if (status !=IOU_TEST_PASS )return status; + status =copy_user_to_reg_vec (ctx); +if (status !=IOU_TEST_PASS ) +return status; + + status =copy_reg_vec_to_user (ctx); +if (status !=IOU_TEST_PASS ) +return status; +return IOU_TEST_PASS ; }
@@ -462,7 +509,7 @@ static void cleanup(void *ctx_ptr)struct iou_test bpf_memcpy_test = { .name ="bpf_memcpy" , - .description ="Test uring_bpf_memcpy() kfunc with USER, VEC, FIXED buffer types" , + .description ="Test uring_bpf_memcpy() kfunc with USER, VEC, FIXED, REG_VEC buffer types" , .setup = setup, .run = run, .cleanup = cleanup,
Signature · 1 line
2.47.0
Jens Axboe
maintainerQuoted 17 lines · Ming Lei
Hello,
Add
IORING_OP_BPFfor extendingio_uringoperations, follows typical cases:- buffer registered zero copy [1]
Also there are some RAID like ublk servers which needs to generate data parity in case of ublk zero copy
- extend
io_uringoperations from applicationEasy to add one new syscall with
IORING_OP_BPF- extend 64 byte SQE
bpf map can store IO data conveniently
Quoted 8 lines
- communicate in IO chain
IORING_OP_BPFcan be used for communicate among IOs seamlessly without requiring extra syscall- pretty handy to inject error for test purpose
Any comments & feedback are welcome!
Ming, can you respin your series against the current tree?
Signature · 1 line
Jens Axboe
Quoted 20 lines · Jens Axboe
Quoted 19 lines · Ming Lei
Hello,
Add
IORING_OP_BPFfor extendingio_uringoperations, follows typical cases:- buffer registered zero copy [1]
Also there are some RAID like ublk servers which needs to generate data parity in case of ublk zero copy
- extend
io_uringoperations from applicationEasy to add one new syscall with
IORING_OP_BPF- extend 64 byte SQE
bpf map can store IO data conveniently
- communicate in IO chain
Quoted 8 lines
Quoted 6 lines
IORING_OP_BPFcan be used for communicate among IOs seamlessly without requiring extra syscall- pretty handy to inject error for test purpose
Any comments & feedback are welcome!
Ming, can you respin your series against the current tree?
OK, I will post V3 for review.
Thanks,
Ming