BPF controlled io_uring

This series introduces a way to override the standard io_uring_enter syscall execution with an extendible event loop, which can be controlled by BPF via new io_uring struct_ops or from within the kernel.

There are multiple use cases I want to cover with this:

- Syscall avoidance. Instead of returning to the userspace for CQE processing, a part of the logic can be moved into BPF to avoid excessive number of syscalls.

- Access to in-kernel io_uring resources. For example, there are registered buffers that can't be directly accessed by the userspace, however we can give BPF the ability to peek at them. It can be used to take a look at in-buffer app level headers to decide what to do with data next and issuing IO using it.

- Smarter request ordering and linking. Request links are pretty limited and inflexible as they can't pass information from one request to another. With BPF we can peek at CQEs and memory and compile a subsequent request.

- Feature semi-deprecation. It can be used to simplify handling of deprecated features by moving it into the callback out core io_uring. For example, it should be trivial to simulate IOSQE_IO_DRAIN. Another target could be request linking logic.

- It can serve as a base for custom algorithms and fine tuning. Often, it'd be impractical to introduce a generic feature because it's either niche or requires a lot of configuration. For example, there is support min-wait, however BPF can help to further fine tune it by doing it in multiple steps with different number of CQEs / timeouts. Another feature people were asking about is allowing to over queue SQEs but make the kernel to maintain a given QD.

- Smarter polling. Napi polling is performed only once per syscall and then it switches to waiting. We can do smarter and intermix polling with waiting using the hook.

It might need more specialised kfuncs in the future, but the core functionality is implemented with just two simple functions. One returns region memory, which gives BPF access to CQ/SQ/etc. And the second is for submitting requests. It's also given a structure as an argument, which is used to pass waiting parameters.

It showed good numbers in a test that sequentially executes N nop requests, where BPF was more than twice as fast than a 2-nop request link implementation.

v8: - Remove an check that is "always true" to silence smatch
- Kill unused variables from selftests

v7: - Fix CQ overflow flushing deadlock and add a selftest

v6: - Fix inversed check on ejection leaving function pointer and
add a selftest checking that.
- Add spdx headers
- Remove sqe reassignment in selftests

v5: - Selftests are now using vmlinux.h
- Checking for unexpected loop return codes
- Remove KF_TRUSTED_ARGS (default)
- Squashed one of the patches, it's more sensible this way

v4: - Separated the event loop from the normal waiting path.
- Improved the selftest.

v3: - Removed most of utility kfuncs and replaced it with a single
helper returning the ring memory.
- Added KF_TRUSTED_ARGS to kfuncs
- Fix ifdef guarding
- Added a selftest
- Adjusted the waiting loop
- Reused the bpf lock section for task_work execution

Pavel Begunkov (5):
io_uring: introduce callback driven main loop io_uring/bpf-ops: implement loop_step with BPF struct_ops io_uring/bpf-ops: add kfunc helpers io_uring/bpf-ops: implement bpf ops registration selftests/io_uring: add a bpf io_uring selftest

 include/linux/io_uring_types.h                |  10 +
 io_uring/Kconfig                              |   5 +
 io_uring/Makefile                             |   3 +-
 io_uring/bpf-ops.c                            | 271 ++++++++++++++++++
 io_uring/bpf-ops.h                            |  28 ++
 io_uring/io_uring.c                           |  13 +
 io_uring/loop.c                               |  97 +++++++
 io_uring/loop.h                               |  27 ++
 io_uring/wait.h                               |   1 +
 tools/testing/selftests/Makefile              |   3 +-
 tools/testing/selftests/io_uring/Makefile     | 162 +++++++++++
 tools/testing/selftests/io_uring/common.h     |   7 +
 .../selftests/io_uring/nops_loop.bpf.c        | 131 +++++++++
 tools/testing/selftests/io_uring/nops_loop.c  | 110 +++++++
 .../testing/selftests/io_uring/overflow.bpf.c |  51 ++++
 tools/testing/selftests/io_uring/overflow.c   |  82 ++++++
 tools/testing/selftests/io_uring/unreg.bpf.c  |  27 ++
 tools/testing/selftests/io_uring/unreg.c      | 113 ++++++++
 18 files changed, 1139 insertions(+), 2 deletions(-)
 create mode 100644 io_uring/bpf-ops.c
 create mode 100644 io_uring/bpf-ops.h
 create mode 100644 io_uring/loop.c
 create mode 100644 io_uring/loop.h
 create mode 100644 tools/testing/selftests/io_uring/Makefile
 create mode 100644 tools/testing/selftests/io_uring/common.h
 create mode 100644 tools/testing/selftests/io_uring/nops_loop.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/nops_loop.c
 create mode 100644 tools/testing/selftests/io_uring/overflow.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/overflow.c
 create mode 100644 tools/testing/selftests/io_uring/unreg.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/unreg.c

2.52.0

The io_uring_enter() has a fixed order of execution: it submits requests, waits for completions, and returns to the user. Allow to optionally replace it with a custom loop driven by a callback called loop_step. The basic requirements to the callback is that it should be able to submit requests, wait for completions, parse them and repeat. Most of the communication including parameter passing can be implemented via shared memory.

The callback should return IOU_LOOP_CONTINUE to continue execution or IOU_LOOP_STOP to return to the user space. Note that the kernel may decide to prematurely terminate it as well, e.g. in case the process was signalled or killed.

The hook takes a structure with parameters. It can be used to ask the kernel to wait for CQEs by setting cq_wait_idx to the CQE index it wants to wait for. Spurious wake ups are possible and even likely, the callback is expected to handle it. There will be more parameters in the future like timeout.

It can be used with kernel callbacks, for example, as a slow path deprecation mechanism overwiting SQEs and emulating the wanted behaviour, however it's more useful together with BPF programs implemented in following patches.

Note that keeping it separately from the normal io_uring wait loop makes things much simpler and cleaner. It keeps it in one place instead of spreading a bunch of checks in different places including disabling the submission path. It holds the lock by default, which is a better fit for BPF synchronisation and the loop execution model. It nicely avoids existing quirks like forced wake ups on timeout request completion. And it should be easier to implement new features.

---
 include/linux/io_uring_types.h |  5 ++
 io_uring/Makefile              |  2 +-
 io_uring/io_uring.c            | 11 ++++
 io_uring/loop.c                | 97 ++++++++++++++++++++++++++++++++++
 io_uring/loop.h                | 27 ++++++++++
 io_uring/wait.h                |  1 +
 6 files changed, 142 insertions(+), 1 deletion(-)
 create mode 100644 io_uring/loop.c
 create mode 100644 io_uring/loop.h

index 3e4a82a6f817..cceac329fcfd 100644
@@ -41,6 +41,8 @@ enum io_uring_cmd_flags {
 	IO_URING_F_COMPAT		= (1 << 12),
 };
 
+struct iou_loop_params;
+
 struct io_wq_work_node {
 	struct io_wq_work_node *next;
 };

@@ -355,6 +357,9 @@ struct io_ring_ctx {
 		struct io_alloc_cache	rw_cache;
 		struct io_alloc_cache	cmd_cache;
 
+		int (*loop_step)(struct io_ring_ctx *ctx,
+				 struct iou_loop_params *);
+
 		/*
 		 * Any cancelable uring_cmd is added to this list in
 		 * ->uring_cmd() by io_uring_cmd_insert_cancelable()

index 931f9156132a..1c1f47de32a4 100644
@@ -14,7 +14,7 @@ obj-$(CONFIG_IO_URING)		+= io_uring.o opdef.o kbuf.o rsrc.o notif.o \
 					advise.o openclose.o statx.o timeout.o \
 					cancel.o waitid.o register.o \
 					truncate.o memmap.o alloc_cache.o \
-					query.o
+					query.o loop.o
 
 obj-$(CONFIG_IO_URING_ZCRX)	+= zcrx.o
 obj-$(CONFIG_IO_WQ)		+= io-wq.o

index 1e627b7a2f3a..0c8bb4e8480a 100644
@@ -95,6 +95,7 @@
 #include "eventfd.h"
 #include "wait.h"
 #include "bpf_filter.h"
+#include "loop.h"
 
 #define SQE_COMMON_FLAGS (IOSQE_FIXED_FILE | IOSQE_IO_LINK | \
 			  IOSQE_IO_HARDLINK | IOSQE_ASYNC)

@@ -589,6 +590,11 @@ void io_cqring_do_overflow_flush(struct io_ring_ctx *ctx)
 	mutex_unlock(&ctx->uring_lock);
 }
 
+void io_cqring_overflow_flush_locked(struct io_ring_ctx *ctx)
+{
+	__io_cqring_overflow_flush(ctx, false);
+}
+
 /* must to be called somewhat shortly after putting a request */
 static inline void io_put_task(struct io_kiocb *req)
 {

@@ -2582,6 +2588,11 @@ SYSCALL_DEFINE6(io_uring_enter, unsigned int, fd, u32, to_submit,
 	if (unlikely(smp_load_acquire(&ctx->flags) & IORING_SETUP_R_DISABLED))
 		goto out;
 
+	if (io_has_loop_ops(ctx)) {
+		ret = io_run_loop(ctx);
+		goto out;
+	}
+
 	/*
 	 * For SQ polling, the thread will do all submissions and completions.
 	 * Just return the requested submit count, and wake the thread if

new file mode 100644
index 000000000000..d7718b9ce61e
@@ -0,0 +1,27 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef IOU_LOOP_H
+#define IOU_LOOP_H
+
+#include <linux/io_uring_types.h>
+
+struct iou_loop_params {
+	/*
+	 * The CQE index to wait for. Only serves as a hint and can still be
+	 * woken up earlier.
+	 */
+	__u32			cq_wait_idx;
+};
+
+enum {
+	IOU_LOOP_CONTINUE = 0,
+	IOU_LOOP_STOP,
+};
+
+static inline bool io_has_loop_ops(struct io_ring_ctx *ctx)
+{
+	return data_race(ctx->loop_step);
+}
+
+int io_run_loop(struct io_ring_ctx *ctx);
+
+#endif

index 5e236f74e1af..037e512dd80c 100644
@@ -25,6 +25,7 @@ int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags,
 		   struct ext_arg *ext_arg);
 int io_run_task_work_sig(struct io_ring_ctx *ctx);
 void io_cqring_do_overflow_flush(struct io_ring_ctx *ctx);
+void io_cqring_overflow_flush_locked(struct io_ring_ctx *ctx);
 
 static inline unsigned int __io_cqring_events(struct io_ring_ctx *ctx)
 {

2.52.0

Introduce io_uring BPF struct ops implementing the loop_step callback, which will allow BPF to overwrite the default io_uring event loop logic.

The callback takes an io_uring context, the main role of which is to be passed to io_uring kfuncs. The other argument is a struct iou_loop_params, which BPF can use to request CQ waiting and communicate other parameters. See the event loop description in the previous patch for more details.

---
 io_uring/Kconfig    |   5 ++
 io_uring/Makefile   |   1 +
 io_uring/bpf-ops.c  | 127 ++++++++++++++++++++++++++++++++++++++++++++
 io_uring/bpf-ops.h  |  14 +++++
 io_uring/io_uring.c |   1 +
 5 files changed, 148 insertions(+)
 create mode 100644 io_uring/bpf-ops.c
 create mode 100644 io_uring/bpf-ops.h

index a7ae23cf1035..a283d9e53787 100644
@@ -14,3 +14,8 @@ config IO_URING_BPF
 	def_bool y
 	depends on BPF
 	depends on NET
+
+config IO_URING_BPF_OPS
+	def_bool y
+	depends on IO_URING
+	depends on BPF_SYSCALL && BPF_JIT && DEBUG_INFO_BTF

index 1c1f47de32a4..c54e328d1410 100644
@@ -25,3 +25,4 @@ obj-$(CONFIG_NET) += net.o cmd_net.o
 obj-$(CONFIG_PROC_FS) += fdinfo.o
 obj-$(CONFIG_IO_URING_MOCK_FILE) += mock_file.o
 obj-$(CONFIG_IO_URING_BPF) += bpf_filter.o
+obj-$(CONFIG_IO_URING_BPF_OPS) += bpf-ops.o

new file mode 100644
index 000000000000..e8a08ae2df0a
@@ -0,0 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef IOU_BPF_OPS_H
+#define IOU_BPF_OPS_H
+
+#include <linux/io_uring_types.h>
+
+struct io_uring_bpf_ops {
+	int (*loop_step)(struct io_ring_ctx *ctx, struct iou_loop_params *lp);
+
+	__u32 ring_fd;
+	void *priv;
+};
+
+#endif /* IOU_BPF_OPS_H */

index 0c8bb4e8480a..548ea5a080a0 100644
@@ -87,6 +87,7 @@
 #include "msg_ring.h"
 #include "memmap.h"
 #include "zcrx.h"
+#include "bpf-ops.h"
 
 #include "timeout.h"
 #include "poll.h"

2.52.0

Add two kfuncs that should cover most of the needs:

1. bpf_io_uring_submit_sqes(), which allows to submit io_uring requests. It mirrors the normal user space submission path and follows all related io_uring_enter(2) rules. i.e. SQEs are taken from the SQ according to head/tail values. In case of IORING_SETUP_SQ_REWIND, it'll submit first N entries.

2. bpf_io_uring_get_region() returns a pointer to the specified region, where io_uring regions are kernel-userspace shared chunks of memory. It takes the size as an argument, which should be a load time constant. There are 3 types of regions:
- IOU_REGION_SQ returns the submission queue.
- IOU_REGION_CQ stores the CQ, SQ/CQ headers and the sqarray. In other words, it gives same memory that would normally be mmap'ed with IORING_FEAT_SINGLE_MMAP enabled IORING_OFF_SQ_RING.
- IOU_REGION_MEM represents the memory / parameter region. It can be used to store request indirect parameters and for kernel - user communication.

It intentionally provides a thin but flexible API and expects BPF programs to implement CQ/SQ header parsing, CQ walking, etc. That mirrors how the normal user space works with rings and should help to minimise kernel / kfunc helpers changes while introducing new generic io_uring features.

---
 io_uring/bpf-ops.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++
 io_uring/bpf-ops.h |  6 +++++
 2 files changed, 61 insertions(+)

index 975db5a78188..17518f4ecca9 100644
@@ -5,11 +5,58 @@
 
 #include "io_uring.h"
 #include "register.h"
+#include "memmap.h"
 #include "bpf-ops.h"
 #include "loop.h"
 
 static const struct btf_type *loop_params_type;
 
+__bpf_kfunc_start_defs();
+
+__bpf_kfunc int bpf_io_uring_submit_sqes(struct io_ring_ctx *ctx, u32 nr)
+{
+	return io_submit_sqes(ctx, nr);
+}
+
+__bpf_kfunc
+__u8 *bpf_io_uring_get_region(struct io_ring_ctx *ctx, __u32 region_id,
+			      const size_t rdwr_buf_size)
+{
+	struct io_mapped_region *r;
+
+	lockdep_assert_held(&ctx->uring_lock);
+
+	switch (region_id) {
+	case IOU_REGION_MEM:
+		r = &ctx->param_region;
+		break;
+	case IOU_REGION_CQ:
+		r = &ctx->ring_region;
+		break;
+	case IOU_REGION_SQ:
+		r = &ctx->sq_region;
+		break;
+	default:
+		return NULL;
+	}
+
+	if (unlikely(rdwr_buf_size > io_region_size(r)))
+		return NULL;
+	return io_region_get_ptr(r);
+}
+
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(io_uring_kfunc_set)
+BTF_ID_FLAGS(func, bpf_io_uring_submit_sqes, KF_SLEEPABLE);
+BTF_ID_FLAGS(func, bpf_io_uring_get_region, KF_RET_NULL);
+BTF_KFUNCS_END(io_uring_kfunc_set)
+
+static const struct btf_kfunc_id_set bpf_io_uring_kfunc_set = {
+	.owner = THIS_MODULE,
+	.set = &io_uring_kfunc_set,
+};
+
 static int io_bpf_ops__loop_step(struct io_ring_ctx *ctx,
 				 struct iou_loop_params *lp)
 {

@@ -68,12 +115,20 @@ io_lookup_struct_type(struct btf *btf, const char *name)
 
 static int bpf_io_init(struct btf *btf)
 {
+	int ret;
+
 	loop_params_type = io_lookup_struct_type(btf, "iou_loop_params");
 	if (!loop_params_type) {
 		pr_err("io_uring: Failed to locate iou_loop_params\n");
 		return -EINVAL;
 	}
 
+	ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+					&bpf_io_uring_kfunc_set);
+	if (ret) {
+		pr_err("io_uring: Failed to register kfuncs (%d)\n", ret);
+		return ret;
+	}
 	return 0;
 }
 

index e8a08ae2df0a..b9e589ad519a 100644
@@ -4,6 +4,12 @@
 
 #include <linux/io_uring_types.h>
 
+enum {
+	IOU_REGION_MEM,
+	IOU_REGION_CQ,
+	IOU_REGION_SQ,
+};
+
 struct io_uring_bpf_ops {
 	int (*loop_step)(struct io_ring_ctx *ctx, struct iou_loop_params *lp);
 

2.52.0

Implement BPF struct ops registration. It's registered off the BPF path, and can be removed by BPF as well as io_uring. To protect it, introduce a global lock synchronising registration. ctx->uring_lock can be nested under it. ctx->bpf_ops is write protected by both locks and so it's safe to read it under either of them.

---
 include/linux/io_uring_types.h |  5 ++
 io_uring/bpf-ops.c             | 91 +++++++++++++++++++++++++++++++++-
 io_uring/bpf-ops.h             |  8 +++
 io_uring/io_uring.c            |  1 +
 4 files changed, 104 insertions(+), 1 deletion(-)

index cceac329fcfd..976d85f82f86 100644
@@ -8,6 +8,9 @@
 #include <linux/llist.h>
 #include <uapi/linux/io_uring.h>
 
+struct iou_loop_params;
+struct io_uring_bpf_ops;
+
 enum {
 	/*
 	 * A hint to not wake right away but delay until there are enough of

@@ -481,6 +484,8 @@ struct io_ring_ctx {
 	DECLARE_HASHTABLE(napi_ht, 4);
 #endif
 
+	struct io_uring_bpf_ops		*bpf_ops;
+
 	/*
 	 * Protection for resize vs mmap races - both the mmap and resize
 	 * side will need to grab this lock, to prevent either side from

index b9e589ad519a..b39b3fd3acda 100644
@@ -17,4 +17,12 @@ struct io_uring_bpf_ops {
 	void *priv;
 };
 
+#ifdef CONFIG_IO_URING_BPF_OPS
+void io_unregister_bpf_ops(struct io_ring_ctx *ctx);
+#else
+static inline void io_unregister_bpf_ops(struct io_ring_ctx *ctx)
+{
+}
+#endif
+
 #endif /* IOU_BPF_OPS_H */

index 548ea5a080a0..b154dac396d3 100644
@@ -2154,6 +2154,7 @@ static __cold void io_req_caches_free(struct io_ring_ctx *ctx)
 
 static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)
 {
+	io_unregister_bpf_ops(ctx);
 	io_sq_thread_finish(ctx);
 
 	mutex_lock(&ctx->uring_lock);

2.52.0

Add simple io_uring BPF selftests. nops_loop emulates a typical event loop but with NOP requests. It maintains a given QD, submits requests and reaps completions until it processes a pre-determined number of requests. Other tests check CQE overflows and that unregistration of BPF from io_uring works well.

---
 tools/testing/selftests/Makefile              |   3 +-
 tools/testing/selftests/io_uring/Makefile     | 162 ++++++++++++++++++
 tools/testing/selftests/io_uring/common.h     |   7 +
 .../selftests/io_uring/nops_loop.bpf.c        | 131 ++++++++++++++
 tools/testing/selftests/io_uring/nops_loop.c  | 110 ++++++++++++
 .../testing/selftests/io_uring/overflow.bpf.c |  51 ++++++
 tools/testing/selftests/io_uring/overflow.c   |  82 +++++++++
 tools/testing/selftests/io_uring/unreg.bpf.c  |  27 +++
 tools/testing/selftests/io_uring/unreg.c      | 113 ++++++++++++
 9 files changed, 685 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/selftests/io_uring/Makefile
 create mode 100644 tools/testing/selftests/io_uring/common.h
 create mode 100644 tools/testing/selftests/io_uring/nops_loop.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/nops_loop.c
 create mode 100644 tools/testing/selftests/io_uring/overflow.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/overflow.c
 create mode 100644 tools/testing/selftests/io_uring/unreg.bpf.c
 create mode 100644 tools/testing/selftests/io_uring/unreg.c

index 450f13ba4cca..f618efaaf684 100644
@@ -131,6 +131,7 @@ TARGETS += vfio
 TARGETS += x86
 TARGETS += x86/bugs
 TARGETS += zram
+TARGETS += io_uring
 #Please keep the TARGETS list alphabetically sorted
 # Run "make quicktest=1 run_tests" or
 # "make quicktest=1 kselftest" from top level Makefile

@@ -148,7 +149,7 @@ endif
 # User can optionally provide a TARGETS skiplist. By default we skip
 # targets using BPF since it has cutting edge build time dependencies
 # which require more effort to install.
-SKIP_TARGETS ?= bpf sched_ext
+SKIP_TARGETS ?= bpf sched_ext io_uring
 ifneq ($(SKIP_TARGETS),)
 	TMP := $(filter-out $(SKIP_TARGETS), $(TARGETS))
 	override TARGETS := $(TMP)

new file mode 100644
index 000000000000..9e726ca72d0c
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#define CQ_ENTRIES 8
+#define SQ_ENTRIES 8
+
+#define SLOT_RES	0
+#define SLOT_NR_CQES	1
+#define SLOT_NR_SQES	2

new file mode 100644
index 000000000000..fef7f9df59b5
@@ -0,0 +1,27 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#include <linux/types.h>
+#include <linux/stddef.h>
+#include <bpf/bpf_helpers.h>
+#include <bpf/bpf_tracing.h>
+#include "vmlinux.h"
+#include "common.h"
+
+char LICENSE[] SEC("license") = "Dual BSD/GPL";
+
+SEC("struct_ops.s/unreg_loop_step")
+int BPF_PROG(unreg_loop_step, struct io_ring_ctx *ring,
+			      struct iou_loop_params *ls)
+{
+	struct io_uring_sqe *sqes;
+
+	sqes = (void *)bpf_io_uring_get_region(ring, IOU_REGION_SQ,
+						sizeof(struct io_uring_sqe));
+	if (sqes)
+		sqes->user_data++;
+	return IOU_LOOP_STOP;
+}
+
+SEC(".struct_ops.link")
+struct io_uring_bpf_ops unreg_ops = {
+	.loop_step = (void *)unreg_loop_step,
+};

2.52.0

ok

...

Depends on the workload. If you're keeping queue depth 8 with a storage workload, it's all you need, and those don't go too high. For net recv workloads, it's batched well if you get 8-16 CQEs per wait, but unfortunately too often it's close to 1. You'd still bump the CQ size in cases there are bursts in traffic.

We can come up with cases when it's higher, but what's important is that you don't need to process all CQEs in a single BPF call. I bound the loop at CQ_ENTRIES for convenience, but the user can set the CQ size to whatever is needed, process some capped number every time, and finish the rest if any next time it's called. E.g.

CQ_ENTRIES = 2^14
const max_cqes_batch = 32;

cqes_to_process = cq_hdr->tail - cq_hdr->head; cqes_to_process = min(cqes_to_process, max_cqes_batch); process_cqes_nr(cqes_to_process, ...);

if (cqes_left)

	to_wait = 0;

I'll make the CQ size a load time constant and clean up the program, should make it a better example.

Sure, but let's do that as a follow up when it's needed?

The other two programs are not toys but rather tests. Nop requests in this one do nothing useful, but that'll be a good queue processing starting template. Replace nops with reads and it's doing actual IO. Add writes requests with some offset logic and it copies data b/w files. I had such examples, but selftests is not the best place for that. It can use abstractions, and I want to make them reusable instead of people copy-pasting from selftests.

Pavel Begunkov

Sure, but please still post them as extra patches so it's easier to see what's the end result.

Also please reply to that thread:
https://lore.kernel.org/bpf/CALTww28QMg=YXqKWpWLZrLO+xiqOe3LGyput8dx68-dnQsxg=g@mail.gmail.com/

It's not clear to me whether your io_uring+bpf setup will work for Xiao's use case.
I don't think we need 2 ways of doing it.
In networking bpf is hooked at xdp, tc, socket levels, but those are different abstractions.

We discussed this with Ming on the list before, that's one of the use cases I target as well, there is no reason why it shouldn't work. The difference is that this approach gives a flexible framework for extensibility and covers a good bunch of other needs, which is exactly the reason I moved from a BPF opcode approach, while Ming's proposal is more specific but argued to be a way easier to plug into ublk servers. If you ask me, we need a solution that covers a broader spectrum of use cases, but I guess it all can be argued in either way.

But it'd be interesting to discuss how a BPF abstraction around registered buffers should look like. E.g. passing an index to all kfuncs like that, or making it a KF_ACQUIRE/RELEASE object, or maybe something else. I'll reply later.

Pavel Begunkov

One big drawback of Pavel's approach is that it needs a totally different userspace
implementation for using BPF, which is just hard to use in existing io_uring applications.

But BPF opcode approach is seamless with existing io_uring applications, because it simply uses existing io_uring interface.

Thanks,
Ming Lei

Not really, it depends on how you write it. If you need to rewrite CQ handling in BPF, e.g. parsing CQEs and stopping at interesting completions to calculate checksums / etc. with BPF, then I get it, it's more involved. It could be useful for other reasons as well, but that's beside the point. But you can do that without ever touching CQ/SQ in BPF, for example, execute all your BPF work at the beginning of a syscall and indicate about this work via user-BPF shared memory.

The downside is breaking links in two if you use those, but perhaps you remember my opinion on io_uring links, with the amount of uAPI side effects and locking complexity it adds to io_uring while not being too flexible, I'd rather replace them with something more generic, like this BPF struct_ops.

In either case, I need it for a broader set of issues. It should work for things like checksumming / other manipulations of internally installed registered buffers, but even without it, there are enough use cases to justify that.

Pavel Begunkov

I meant SQE has to be allocated & built in bpf prog, then it is inevitable to move application logic into bpf prog.

If the logic is complicated, it is one big thing.

raid5 is a great example with complicated fast io code path, I'd suggest someone or Xiao or Caleb, try it. Compare your approach to the bpf opcode and draw some useful conclusions. Code should be more convincing.

For bpf opcode approach I believe Xiao has already built an example.

Thanks,
Ming

But that's what I'm saying, you don't need building sqes in bpf to augment it with BPF checksumming / etc. I can elaborate when I get some time, but to give a short example:

# bpf-prog.c:

unsigned csum_jobs;
unsigned regbuf_idx_to_csum;

loop() {
	if (csum_jobs) {
		kfunc_regbuf_do_csum(regbuf_idx_to_csum);
		csum_jobs = 0;
		...
	}

	// proceed to the default submit / lopp implementation
	// you'd expect from normal io_uring_submit_and_wait()
}

# user-prog.c

ring = create_ring();
load_my_bpf();

sqe = get_and_prep_sqe();
// normal submit+wait
io_uring_submit_and_wait();

cqe = get_cqes();

if (cqe->type == ...) {
	skel->bss->csum_jobs++;
	skel->bss->regbuf_idx_to_csum = idx_from_cqe(cqe);
	// use bpf arenas, io_uring regions for the real tning
}

// queue more sqes if needed

// this will do csum'ing and followed by submit+wait io_uring_submit_and_wait();

The implementation above would be simple enough. And on top, someone could try to do BPF submissions to cover cases like you had with group request ordering (forgot the actual name).

It might be entertaining, but I wanted to stress that it's not a blocker for my work. It can handle the use case with registered buffers, but I aim at a more wider range of applications, which wouldn't be covered by the opcode approach. And that's the reason why I moved away from opcodes, it wasn't flexible enough.

Pavel Begunkov

As a follow up to this discussion I sent out more realistic examples together with v9. It also has a very much toy program working with registered buffers as you mentioned ublk wanting to checksum them.

https://lore.kernel.org/io-uring/[email protected]

Pavel Begunkov