Skip to contentSkip to conversation

io_uring: add IORING_OP_BPF for extending io_uring

PATCH Superseded Ming Lei 6 participants 41 messages Nov 4, 2025 – Dec 31, 2025

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

cover io_uring: add IORING_OP_BPF for extending io_uring

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

- buffer registered zero copy [1]

Also there are some RAID like ublk servers which needs to generate data parity in case of ublk zero copy

- extend io_uring operations from application

Easy to add one new syscall with IORING_OP_BPF

- extend 64 byte SQE

bpf map can store IO data conveniently

- communicate in IO chain

IORING_OP_BPF can be used for communicate among IOs seamlessly without requiring extra syscall

- pretty handy to inject error for test purpose

The 1st 3 patches adds IORING_OP_BPF framework & struct_ops.

The 4th patch adds both fixed and plain user buffer support, in future vector and fixed vector buffer can be supported.

The last patch exports io_uring_bpf_req_memcpy() for copying data among different request buffers, so far fixed and plain buffers are supported, this is also one demo for showing how to solve above requirements with IORING_OP_BPF.

Follows liburing support and tests:

https://github.com/ming1/liburing/commits/uring_bpf/

Any comments & feedback are welcome!

[1] lpc2024: ublk based zero copy I/O - use case in Android

https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

Ming Lei (5):
io_uring: prepare for extending io_uring with bpf
io_uring: bpf: add io_uring_ctx setup for BPF into one list
io_uring: bpf: extend io_uring with bpf struct_ops
io_uring: bpf: add buffer support for IORING_OP_BPF
io_uring: bpf: add io_uring_bpf_req_memcpy() kfunc

 include/linux/io_uring_types.h |   5 +
 include/uapi/linux/io_uring.h  |  56 ++++
 init/Kconfig                   |   7 +
 io_uring/Makefile              |   1 +
 io_uring/bpf.c                 | 556 +++++++++++++++++++++++++++++++++
 io_uring/io_uring.c            |   8 +
 io_uring/io_uring.h            |   6 +-
 io_uring/opdef.c               |  10 +
 io_uring/uring_bpf.h           |  86 +++++
 9 files changed, 733 insertions(+), 2 deletions(-)
 create mode 100644 io_uring/bpf.c
 create mode 100644 io_uring/uring_bpf.h
Signature · 1 line
2.47.0
author
#1 lore raw

PATCH 1/5 io_uring: prepare for extending io_uring with bpf

Add one bpf operation & related framework and prepare for extending io_uring with bpf.

---
 include/uapi/linux/io_uring.h |  1 +
 init/Kconfig                  |  7 +++++++
 io_uring/Makefile             |  1 +
 io_uring/bpf.c                | 26 ++++++++++++++++++++++++++
 io_uring/opdef.c              | 10 ++++++++++
 io_uring/uring_bpf.h          | 26 ++++++++++++++++++++++++++
 6 files changed, 71 insertions(+)
 create mode 100644 io_uring/bpf.c
 create mode 100644 io_uring/uring_bpf.h
include/uapi/linux/io_uring.h +1 -0
index 04797a9b76bc..b167c1d4ce6e 100644
@@ -303,6 +303,7 @@ enum io_uring_op {
 	IORING_OP_PIPE,
 	IORING_OP_NOP128,
 	IORING_OP_URING_CMD128,
+	IORING_OP_BPF,
 
 	/* this goes last, obviously */
 	IORING_OP_LAST,
init/Kconfig +7 -0
index cab3ad28ca49..14d566516643 100644
@@ -1843,6 +1843,13 @@ config IO_URING
 	  applications to submit and complete IO through submission and
 	  completion rings that are shared between the kernel and application.
 
+config IO_URING_BPF
+	bool "Enable IO uring bpf extension" if EXPERT
+	depends on IO_URING && BPF
+	help
+	  This option enables bpf extension for the io_uring interface, so
+	  application can define its own io_uring operation by bpf program.
+
 config GCOV_PROFILE_URING
 	bool "Enable GCOV profiling on the io_uring subsystem"
 	depends on IO_URING && GCOV_KERNEL
io_uring/Makefile +1 -0
index bc4e4a3fa0a5..35eeeaf64489 100644
@@ -22,3 +22,4 @@ obj-$(CONFIG_NET_RX_BUSY_POLL)	+= napi.o
 obj-$(CONFIG_NET) += net.o cmd_net.o
 obj-$(CONFIG_PROC_FS) += fdinfo.o
 obj-$(CONFIG_IO_URING_MOCK_FILE) += mock_file.o
+obj-$(CONFIG_IO_URING_BPF)	+= bpf.o
io_uring/bpf.c new file +26 -0
new file mode 100644
index 000000000000..8c47df13c7b5
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Red Hat */
+
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <uapi/linux/io_uring.h>
+#include "io_uring.h"
+#include "uring_bpf.h"
+
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+	return -ECANCELED;
+}
+
+int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
+{
+	return -EOPNOTSUPP;
+}
+
+void io_uring_bpf_fail(struct io_kiocb *req)
+{
+}
+
+void io_uring_bpf_cleanup(struct io_kiocb *req)
+{
+}
io_uring/opdef.c +10 -0
index df52d760240e..d93ee3d577d4 100644
@@ -38,6 +38,7 @@
 #include "futex.h"
 #include "truncate.h"
 #include "zcrx.h"
+#include "uring_bpf.h"
 
 static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -593,6 +594,10 @@ const struct io_issue_def io_issue_defs[] = {
 		.prep			= io_uring_cmd_prep,
 		.issue			= io_uring_cmd,
 	},
+	[IORING_OP_BPF] = {
+		.prep			= io_uring_bpf_prep,
+		.issue			= io_uring_bpf_issue,
+	},
 };
 
 const struct io_cold_def io_cold_defs[] = {
@@ -851,6 +856,11 @@ const struct io_cold_def io_cold_defs[] = {
 		.sqe_copy		= io_uring_cmd_sqe_copy,
 		.cleanup		= io_uring_cmd_cleanup,
 	},
+	[IORING_OP_BPF] = {
+		.name			= "BPF",
+		.cleanup		= io_uring_bpf_cleanup,
+		.fail			= io_uring_bpf_fail,
+	},
 };
 
 const char *io_uring_get_opcode(u8 opcode)
io_uring/uring_bpf.h new file +26 -0
new file mode 100644
index 000000000000..bde774ce6ac0
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef IOU_BPF_H
+#define IOU_BPF_H
+
+#ifdef CONFIG_IO_URING_BPF
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags);
+int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
+void io_uring_bpf_fail(struct io_kiocb *req);
+void io_uring_bpf_cleanup(struct io_kiocb *req);
+#else
+static inline int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+	return -ECANCELED;
+}
+static inline int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
+{
+	return -EOPNOTSUPP;
+}
+static inline void io_uring_bpf_fail(struct io_kiocb *req)
+{
+}
+static inline void io_uring_bpf_cleanup(struct io_kiocb *req)
+{
+}
+#endif
+#endif
Signature · 1 line
2.47.0
#2 lore raw

Re: [PATCH 1/5] io_uring: prepare for extending io_uring with bpf

Quoted 100 lines · Ming Lei

Add one bpf operation & related framework and prepare for extending io_uring with bpf.

---
 include/uapi/linux/io_uring.h |  1 +
 init/Kconfig                  |  7 +++++++
 io_uring/Makefile             |  1 +
 io_uring/bpf.c                | 26 ++++++++++++++++++++++++++
 io_uring/opdef.c              | 10 ++++++++++
 io_uring/uring_bpf.h          | 26 ++++++++++++++++++++++++++
 6 files changed, 71 insertions(+)
 create mode 100644 io_uring/bpf.c
 create mode 100644 io_uring/uring_bpf.h
include/uapi/linux/io_uring.h +1 -0
index 04797a9b76bc..b167c1d4ce6e 100644
@@ -303,6 +303,7 @@ enum io_uring_op {
        IORING_OP_PIPE,
        IORING_OP_NOP128,
        IORING_OP_URING_CMD128,
+       IORING_OP_BPF,
 
        /* this goes last, obviously */
        IORING_OP_LAST,
init/Kconfig +7 -0
index cab3ad28ca49..14d566516643 100644
@@ -1843,6 +1843,13 @@ config IO_URING
          applications to submit and complete IO through submission and
          completion rings that are shared between the kernel and application.
 
+config IO_URING_BPF
+       bool "Enable IO uring bpf extension" if EXPERT
+       depends on IO_URING && BPF
+       help
+         This option enables bpf extension for the io_uring interface, so
+         application can define its own io_uring operation by bpf program.
+
 config GCOV_PROFILE_URING
        bool "Enable GCOV profiling on the io_uring subsystem"
        depends on IO_URING && GCOV_KERNEL
io_uring/Makefile +1 -0
index bc4e4a3fa0a5..35eeeaf64489 100644
@@ -22,3 +22,4 @@ obj-$(CONFIG_NET_RX_BUSY_POLL)        += napi.o
 obj-$(CONFIG_NET) += net.o cmd_net.o
 obj-$(CONFIG_PROC_FS) += fdinfo.o
 obj-$(CONFIG_IO_URING_MOCK_FILE) += mock_file.o
+obj-$(CONFIG_IO_URING_BPF)     += bpf.o
io_uring/bpf.c new file +26 -0
new file mode 100644
index 000000000000..8c47df13c7b5
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Red Hat */
+
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <uapi/linux/io_uring.h>
+#include "io_uring.h"
+#include "uring_bpf.h"
+
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+       return -ECANCELED;
+}
+
+int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
+{
+       return -EOPNOTSUPP;
+}
+
+void io_uring_bpf_fail(struct io_kiocb *req)
+{
+}
+
+void io_uring_bpf_cleanup(struct io_kiocb *req)
+{
+}
io_uring/opdef.c +1 -0
index df52d760240e..d93ee3d577d4 100644
@@ -38,6 +38,7 @@
 #include "futex.h"
 #include "truncate.h"
 #include "zcrx.h"
+#include "uring_bpf.h"
 
 static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags)
 {

io_uring/opdef.c const struct io_issue_def io_issue_defs[] = { · line 599

Quoted 6 lines
io_uring/opdef.c +3 -0
@@ -593,6 +594,10 @@ const struct io_issue_def io_issue_defs[] = {
                .prep                   = io_uring_cmd_prep,
                .issue                  = io_uring_cmd,
        },
+       [IORING_OP_BPF] = {
+               .prep                   = io_uring_bpf_prep,
+               .issue                  = io_uring_bpf_issue,

Should this set .prep = io_eopnotsupp_prep for !CONFIG_IO_URING_BPF and remove the stub implementations? That would be more consistent with the other opcodes when they are configured out, and ensure io_uring_op_supported(IORING_OP_BPF) returns false for !CONFIG_IO_URING_BPF.

Best,
Caleb

Quoted 51 lines
io_uring/opdef.c +6 -0
+       },
 };
 
 const struct io_cold_def io_cold_defs[] = {
@@ -851,6 +856,11 @@ const struct io_cold_def io_cold_defs[] = {
                .sqe_copy               = io_uring_cmd_sqe_copy,
                .cleanup                = io_uring_cmd_cleanup,
        },
+       [IORING_OP_BPF] = {
+               .name                   = "BPF",
+               .cleanup                = io_uring_bpf_cleanup,
+               .fail                   = io_uring_bpf_fail,
+       },
 };
 
 const char *io_uring_get_opcode(u8 opcode)
io_uring/uring_bpf.h new file +26 -1
new file mode 100644
index 000000000000..bde774ce6ac0
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef IOU_BPF_H
+#define IOU_BPF_H
+
+#ifdef CONFIG_IO_URING_BPF
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags);
+int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
+void io_uring_bpf_fail(struct io_kiocb *req);
+void io_uring_bpf_cleanup(struct io_kiocb *req);
+#else
+static inline int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+       return -ECANCELED;
+}
+static inline int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
+{
+       return -EOPNOTSUPP;
+}
+static inline void io_uring_bpf_fail(struct io_kiocb *req)
+{
+}
+static inline void io_uring_bpf_cleanup(struct io_kiocb *req)
+{
+}
+#endif
+#endif
--

2.47.0

author
#3 lore raw

Re: [PATCH 1/5] io_uring: prepare for extending io_uring with bpf

Quoted 98 lines · Caleb Sander Mateos
Quoted 97 lines · Ming Lei

Add one bpf operation & related framework and prepare for extending io_uring with bpf.

---
 include/uapi/linux/io_uring.h |  1 +
 init/Kconfig                  |  7 +++++++
 io_uring/Makefile             |  1 +
 io_uring/bpf.c                | 26 ++++++++++++++++++++++++++
 io_uring/opdef.c              | 10 ++++++++++
 io_uring/uring_bpf.h          | 26 ++++++++++++++++++++++++++
 6 files changed, 71 insertions(+)
 create mode 100644 io_uring/bpf.c
 create mode 100644 io_uring/uring_bpf.h
include/uapi/linux/io_uring.h +1 -0
index 04797a9b76bc..b167c1d4ce6e 100644
@@ -303,6 +303,7 @@ enum io_uring_op {
        IORING_OP_PIPE,
        IORING_OP_NOP128,
        IORING_OP_URING_CMD128,
+       IORING_OP_BPF,
 
        /* this goes last, obviously */
        IORING_OP_LAST,
init/Kconfig +7 -0
index cab3ad28ca49..14d566516643 100644
@@ -1843,6 +1843,13 @@ config IO_URING
          applications to submit and complete IO through submission and
          completion rings that are shared between the kernel and application.
 
+config IO_URING_BPF
+       bool "Enable IO uring bpf extension" if EXPERT
+       depends on IO_URING && BPF
+       help
+         This option enables bpf extension for the io_uring interface, so
+         application can define its own io_uring operation by bpf program.
+
 config GCOV_PROFILE_URING
        bool "Enable GCOV profiling on the io_uring subsystem"
        depends on IO_URING && GCOV_KERNEL
io_uring/Makefile +1 -0
index bc4e4a3fa0a5..35eeeaf64489 100644
@@ -22,3 +22,4 @@ obj-$(CONFIG_NET_RX_BUSY_POLL)        += napi.o
 obj-$(CONFIG_NET) += net.o cmd_net.o
 obj-$(CONFIG_PROC_FS) += fdinfo.o
 obj-$(CONFIG_IO_URING_MOCK_FILE) += mock_file.o
+obj-$(CONFIG_IO_URING_BPF)     += bpf.o
io_uring/bpf.c new file +26 -0
new file mode 100644
index 000000000000..8c47df13c7b5
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/* Copyright (c) 2024 Red Hat */
+
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <uapi/linux/io_uring.h>
+#include "io_uring.h"
+#include "uring_bpf.h"
+
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+       return -ECANCELED;
+}
+
+int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
+{
+       return -EOPNOTSUPP;
+}
+
+void io_uring_bpf_fail(struct io_kiocb *req)
+{
+}
+
+void io_uring_bpf_cleanup(struct io_kiocb *req)
+{
+}
io_uring/opdef.c +1 -0
index df52d760240e..d93ee3d577d4 100644
@@ -38,6 +38,7 @@
 #include "futex.h"
 #include "truncate.h"
 #include "zcrx.h"
+#include "uring_bpf.h"
 
Quoted 15 lines

io_uring/opdef.c const struct io_issue_def io_issue_defs[] = { · line 599

Quoted 9 lines
io_uring/opdef.c +3 -0
 static int io_no_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -593,6 +594,10 @@ const struct io_issue_def io_issue_defs[] = {
                .prep                   = io_uring_cmd_prep,
                .issue                  = io_uring_cmd,
        },
+       [IORING_OP_BPF] = {
+               .prep                   = io_uring_bpf_prep,
+               .issue                  = io_uring_bpf_issue,

Should this set .prep = io_eopnotsupp_prep for !CONFIG_IO_URING_BPF and remove the stub implementations? That would be more consistent with the other opcodes when they are configured out, and ensure io_uring_op_supported(IORING_OP_BPF) returns false for !CONFIG_IO_URING_BPF.

Yeah, good point!

Thanks,
Ming

author
#4 lore raw

PATCH 2/5 io_uring: bpf: add io_uring_ctx setup for BPF into one list

In reply to Ming Lei

Add io_uring_ctx setup for BPF into one list, and prepare for syncing bpf struct_ops register and un-register.

---
 include/linux/io_uring_types.h |  5 +++++
 include/uapi/linux/io_uring.h  |  5 +++++
 io_uring/bpf.c                 | 15 +++++++++++++++
 io_uring/io_uring.c            |  7 +++++++
 io_uring/io_uring.h            |  3 ++-
 io_uring/uring_bpf.h           | 11 +++++++++++
 6 files changed, 45 insertions(+), 1 deletion(-)
include/linux/io_uring_types.h +5 -0
index 92780764d5fa..d2e098c3fd2c 100644
@@ -465,6 +465,11 @@ struct io_ring_ctx {
 	struct io_mapped_region		ring_region;
 	/* used for optimised request parameter and wait argument passing  */
 	struct io_mapped_region		param_region;
+
+#ifdef CONFIG_IO_URING_BPF
+	/* added to uring_bpf_ctx_list */
+	struct list_head		bpf_node;
+#endif
 };
 
 /*
include/uapi/linux/io_uring.h +5 -0
index b167c1d4ce6e..b8c49813b4e5 100644
@@ -237,6 +237,11 @@ enum io_uring_sqe_flags_bit {
  */
 #define IORING_SETUP_SQE_MIXED		(1U << 19)
 
+/*
+ * Allow to submit bpf IO
+ */
+#define IORING_SETUP_BPF		(1U << 20)
+
 enum io_uring_op {
 	IORING_OP_NOP,
 	IORING_OP_READV,
io_uring/bpf.c +15 -0
index 8c47df13c7b5..bb1e37d1e804 100644
@@ -7,6 +7,9 @@
 #include "io_uring.h"
 #include "uring_bpf.h"
 
+static DEFINE_MUTEX(uring_bpf_ctx_lock);
+static LIST_HEAD(uring_bpf_ctx_list);
+
 int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
 	return -ECANCELED;
@@ -24,3 +27,15 @@ void io_uring_bpf_fail(struct io_kiocb *req)
 void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
 }
+
+void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
+{
+	guard(mutex)(&uring_bpf_ctx_lock);
+	list_add(&ctx->bpf_node, &uring_bpf_ctx_list);
+}
+
+void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
+{
+	guard(mutex)(&uring_bpf_ctx_lock);
+	list_del(&ctx->bpf_node);
+}
io_uring/io_uring.c +7 -0
index 3f0489261d11..38f03f6c28cb 100644
@@ -105,6 +105,7 @@
 #include "rw.h"
 #include "alloc_cache.h"
 #include "eventfd.h"
+#include "uring_bpf.h"
 
 #define SQE_COMMON_FLAGS (IOSQE_FIXED_FILE | IOSQE_IO_LINK | \
 			  IOSQE_IO_HARDLINK | IOSQE_ASYNC)
@@ -352,6 +353,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
 	io_napi_init(ctx);
 	mutex_init(&ctx->mmap_lock);
 
+	if (ctx->flags & IORING_SETUP_BPF)
+		uring_bpf_add_ctx(ctx);
+
 	return ctx;
 
 free_ref:
@@ -2855,6 +2859,9 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)
 	if (!(ctx->flags & IORING_SETUP_NO_SQARRAY))
 		static_branch_dec(&io_key_has_sqarray);
 
+	if (ctx->flags & IORING_SETUP_BPF)
+		uring_bpf_del_ctx(ctx);
+
 	percpu_ref_exit(&ctx->refs);
 	free_uid(ctx->user);
 	io_req_caches_free(ctx);
io_uring/io_uring.h +2 -1
index 23c268ab1c8f..4baf21a9e1ee 100644
@@ -55,7 +55,8 @@
 			IORING_SETUP_NO_SQARRAY |\
 			IORING_SETUP_HYBRID_IOPOLL |\
 			IORING_SETUP_CQE_MIXED |\
-			IORING_SETUP_SQE_MIXED)
+			IORING_SETUP_SQE_MIXED |\
+			IORING_SETUP_BPF)
 
 #define IORING_ENTER_FLAGS (IORING_ENTER_GETEVENTS |\
 			IORING_ENTER_SQ_WAKEUP |\
io_uring/uring_bpf.h +11 -0
index bde774ce6ac0..b6cda6df99b1 100644
@@ -7,6 +7,10 @@ int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags);
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
 void io_uring_bpf_fail(struct io_kiocb *req);
 void io_uring_bpf_cleanup(struct io_kiocb *req);
+
+void uring_bpf_add_ctx(struct io_ring_ctx *ctx);
+void uring_bpf_del_ctx(struct io_ring_ctx *ctx);
+
 #else
 static inline int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -22,5 +26,12 @@ static inline void io_uring_bpf_fail(struct io_kiocb *req)
 static inline void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
 }
+
+static inline void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
+{
+}
+static inline void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
+{
+}
 #endif
 #endif
Signature · 1 line
2.47.0
#5 lore raw

Re: [PATCH 2/5] io_uring: bpf: add io_uring_ctx setup for BPF into one list

Quoted 35 lines · Ming Lei

Add io_uring_ctx setup for BPF into one list, and prepare for syncing bpf struct_ops register and un-register.

---
 include/linux/io_uring_types.h |  5 +++++
 include/uapi/linux/io_uring.h  |  5 +++++
 io_uring/bpf.c                 | 15 +++++++++++++++
 io_uring/io_uring.c            |  7 +++++++
 io_uring/io_uring.h            |  3 ++-
 io_uring/uring_bpf.h           | 11 +++++++++++
 6 files changed, 45 insertions(+), 1 deletion(-)
include/linux/io_uring_types.h +5 -0
index 92780764d5fa..d2e098c3fd2c 100644
@@ -465,6 +465,11 @@ struct io_ring_ctx {
        struct io_mapped_region         ring_region;
        /* used for optimised request parameter and wait argument passing  */
        struct io_mapped_region         param_region;
+
+#ifdef CONFIG_IO_URING_BPF
+       /* added to uring_bpf_ctx_list */
+       struct list_head                bpf_node;
+#endif
 };
 
 /*
include/uapi/linux/io_uring.h +0 -0
index b167c1d4ce6e..b8c49813b4e5 100644

include/uapi/linux/io_uring.h enum io_uring_sqe_flags_bit { · line 243

Quoted 7 lines
include/uapi/linux/io_uring.h +4 -0
@@ -237,6 +237,11 @@ enum io_uring_sqe_flags_bit {
  */
 #define IORING_SETUP_SQE_MIXED         (1U << 19)
 
+/*
+ * Allow to submit bpf IO
+ */
+#define IORING_SETUP_BPF               (1U << 20)

Is the setup flag really necessary? It doesn't look like there's much overhead to allowing BPF programs to be used on any io_ring_ctx, so I would be inclined to avoid needing to set an additional flag to use it.

Best,
Caleb

Quoted 111 lines
include/uapi/linux/io_uring.h +1 -0
+
 enum io_uring_op {
        IORING_OP_NOP,
        IORING_OP_READV,
io_uring/bpf.c +15 -0
index 8c47df13c7b5..bb1e37d1e804 100644
@@ -7,6 +7,9 @@
 #include "io_uring.h"
 #include "uring_bpf.h"
 
+static DEFINE_MUTEX(uring_bpf_ctx_lock);
+static LIST_HEAD(uring_bpf_ctx_list);
+
 int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
        return -ECANCELED;
@@ -24,3 +27,15 @@ void io_uring_bpf_fail(struct io_kiocb *req)
 void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
 }
+
+void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
+{
+       guard(mutex)(&uring_bpf_ctx_lock);
+       list_add(&ctx->bpf_node, &uring_bpf_ctx_list);
+}
+
+void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
+{
+       guard(mutex)(&uring_bpf_ctx_lock);
+       list_del(&ctx->bpf_node);
+}
io_uring/io_uring.c +7 -0
index 3f0489261d11..38f03f6c28cb 100644
@@ -105,6 +105,7 @@
 #include "rw.h"
 #include "alloc_cache.h"
 #include "eventfd.h"
+#include "uring_bpf.h"
 
 #define SQE_COMMON_FLAGS (IOSQE_FIXED_FILE | IOSQE_IO_LINK | \
                          IOSQE_IO_HARDLINK | IOSQE_ASYNC)
@@ -352,6 +353,9 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p)
        io_napi_init(ctx);
        mutex_init(&ctx->mmap_lock);
 
+       if (ctx->flags & IORING_SETUP_BPF)
+               uring_bpf_add_ctx(ctx);
+
        return ctx;
 
 free_ref:
@@ -2855,6 +2859,9 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx)
        if (!(ctx->flags & IORING_SETUP_NO_SQARRAY))
                static_branch_dec(&io_key_has_sqarray);
 
+       if (ctx->flags & IORING_SETUP_BPF)
+               uring_bpf_del_ctx(ctx);
+
        percpu_ref_exit(&ctx->refs);
        free_uid(ctx->user);
        io_req_caches_free(ctx);
io_uring/io_uring.h +2 -1
index 23c268ab1c8f..4baf21a9e1ee 100644
@@ -55,7 +55,8 @@
                        IORING_SETUP_NO_SQARRAY |\
                        IORING_SETUP_HYBRID_IOPOLL |\
                        IORING_SETUP_CQE_MIXED |\
-                       IORING_SETUP_SQE_MIXED)
+                       IORING_SETUP_SQE_MIXED |\
+                       IORING_SETUP_BPF)
 
 #define IORING_ENTER_FLAGS (IORING_ENTER_GETEVENTS |\
                        IORING_ENTER_SQ_WAKEUP |\
io_uring/uring_bpf.h +11 -1
index bde774ce6ac0..b6cda6df99b1 100644
@@ -7,6 +7,10 @@ int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags);
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
 void io_uring_bpf_fail(struct io_kiocb *req);
 void io_uring_bpf_cleanup(struct io_kiocb *req);
+
+void uring_bpf_add_ctx(struct io_ring_ctx *ctx);
+void uring_bpf_del_ctx(struct io_ring_ctx *ctx);
+
 #else
 static inline int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -22,5 +26,12 @@ static inline void io_uring_bpf_fail(struct io_kiocb *req)
 static inline void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
 }
+
+static inline void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
+{
+}
+static inline void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
+{
+}
 #endif
 #endif
--

2.47.0

author
#6 lore raw

Re: [PATCH 2/5] io_uring: bpf: add io_uring_ctx setup for BPF into one list

Quoted 39 lines · Caleb Sander Mateos
Quoted 38 lines · Ming Lei

Add io_uring_ctx setup for BPF into one list, and prepare for syncing bpf struct_ops register and un-register.

---
 include/linux/io_uring_types.h |  5 +++++
 include/uapi/linux/io_uring.h  |  5 +++++
 io_uring/bpf.c                 | 15 +++++++++++++++
 io_uring/io_uring.c            |  7 +++++++
 io_uring/io_uring.h            |  3 ++-
 io_uring/uring_bpf.h           | 11 +++++++++++
 6 files changed, 45 insertions(+), 1 deletion(-)
include/linux/io_uring_types.h +5 -0
index 92780764d5fa..d2e098c3fd2c 100644
@@ -465,6 +465,11 @@ struct io_ring_ctx {
        struct io_mapped_region         ring_region;
        /* used for optimised request parameter and wait argument passing  */
        struct io_mapped_region         param_region;
+
+#ifdef CONFIG_IO_URING_BPF
+       /* added to uring_bpf_ctx_list */
+       struct list_head                bpf_node;
+#endif
 };
 
 /*
include/uapi/linux/io_uring.h +0 -0
index b167c1d4ce6e..b8c49813b4e5 100644
@@ -237,6 +237,11 @@ enum io_uring_sqe_flags_bit {
  */
 #define IORING_SETUP_SQE_MIXED         (1U << 19)
 
Quoted 9 lines

include/uapi/linux/io_uring.h enum io_uring_sqe_flags_bit { · line 243

Quoted 4 lines
include/uapi/linux/io_uring.h +4 -0
+/*
+ * Allow to submit bpf IO
+ */
+#define IORING_SETUP_BPF               (1U << 20)

Is the setup flag really necessary? It doesn't look like there's much overhead to allowing BPF programs to be used on any io_ring_ctx, so I would be inclined to avoid needing to set an additional flag to use it.

It is used for dealing with unregistering & registering bpf prog vs. handling IO.

Thanks,
Ming

#7 lore raw

Re: [PATCH 2/5] io_uring: bpf: add io_uring_ctx setup for BPF into one list

Quoted 51 lines · Ming Lei
Quoted 39 lines · Caleb Sander Mateos
Quoted 38 lines · Ming Lei

Add io_uring_ctx setup for BPF into one list, and prepare for syncing bpf struct_ops register and un-register.

---
 include/linux/io_uring_types.h |  5 +++++
 include/uapi/linux/io_uring.h  |  5 +++++
 io_uring/bpf.c                 | 15 +++++++++++++++
 io_uring/io_uring.c            |  7 +++++++
 io_uring/io_uring.h            |  3 ++-
 io_uring/uring_bpf.h           | 11 +++++++++++
 6 files changed, 45 insertions(+), 1 deletion(-)
include/linux/io_uring_types.h +5 -0
index 92780764d5fa..d2e098c3fd2c 100644
@@ -465,6 +465,11 @@ struct io_ring_ctx {
        struct io_mapped_region         ring_region;
        /* used for optimised request parameter and wait argument passing  */
        struct io_mapped_region         param_region;
+
+#ifdef CONFIG_IO_URING_BPF
+       /* added to uring_bpf_ctx_list */
+       struct list_head                bpf_node;
+#endif
 };
 
 /*
include/uapi/linux/io_uring.h +0 -0
index b167c1d4ce6e..b8c49813b4e5 100644
@@ -237,6 +237,11 @@ enum io_uring_sqe_flags_bit {
  */
 #define IORING_SETUP_SQE_MIXED         (1U << 19)
 
Quoted 9 lines

include/uapi/linux/io_uring.h enum io_uring_sqe_flags_bit { · line 243

Quoted 4 lines
include/uapi/linux/io_uring.h +4 -0
+/*
+ * Allow to submit bpf IO
+ */
+#define IORING_SETUP_BPF               (1U << 20)

Is the setup flag really necessary? It doesn't look like there's much overhead to allowing BPF programs to be used on any io_ring_ctx, so I would be inclined to avoid needing to set an additional flag to use it.

Quoted 1 line

It is used for dealing with unregistering & registering bpf prog vs. handling IO.

It looks like it's only necessary because the io_uring BPF program lookup uses uring_lock for protection instead of srcu in contexts where uring_lock is already held. So then registering and unregistering an io_uring BPF program requires acquiring uring_lock for all io_ring_ctxs that support BPF programs. I think this mechanism could be avoided entirely if the BPF program were specified by bpf_fd instead. Then it would be protected by the existing RCU + refcount of a file descriptor lookup. The bpf_fd could also be registered as an io_uring fixed file to avoid the reference count overhead, in which case the lookup would be protected by uring_lock.

Best,
Caleb

author
#8 lore raw

PATCH 3/5 io_uring: bpf: extend io_uring with bpf struct_ops

In reply to Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

---
 include/uapi/linux/io_uring.h |   9 ++
 io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
 io_uring/io_uring.c           |   1 +
 io_uring/io_uring.h           |   3 +-
 io_uring/uring_bpf.h          |  30 ++++
 5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
 		__u32		install_fd_flags;
 		__u32		nop_flags;
 		__u32		pipe_flags;
+		__u32		bpf_op_flags;
 	};
 	__u64	user_data;	/* data to be passed back at completion time */
 	/* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
 #define IORING_RECVSEND_BUNDLE		(1U << 4)
 #define IORING_SEND_VECTORIZED		(1U << 5)
 
+/*
+ * sqe->bpf_op_flags		top 8bits is for storing bpf op
+ *				The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS	(8)
+#define IORING_BPF_OP_SHIFT	(24)
+
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
  * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
 #define IORING_FEAT_MIN_TIMEOUT		(1U << 15)
 #define IORING_FEAT_RW_ATTR		(1U << 16)
 #define IORING_FEAT_NO_IOWAIT		(1U << 17)
+#define IORING_FEAT_BPF			(1U << 18)
 
 /*
  * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +269 -2

300 lines not on this page · raw message

io_uring/io_uring.c +1 -0
index 38f03f6c28cb..d2517e09407a 100644
@@ -3851,6 +3851,7 @@ static int __init io_uring_init(void)
 	register_sysctl_init("kernel", kernel_io_uring_disabled_table);
 #endif
 
+	io_bpf_init();
 	return 0;
 };
 __initcall(io_uring_init);
io_uring/io_uring.h +2 -1
index 4baf21a9e1ee..3f19bb079bcc 100644
@@ -34,7 +34,8 @@
 			IORING_FEAT_RECVSEND_BUNDLE |\
 			IORING_FEAT_MIN_TIMEOUT |\
 			IORING_FEAT_RW_ATTR |\
-			IORING_FEAT_NO_IOWAIT)
+			IORING_FEAT_NO_IOWAIT |\
+			IORING_FEAT_BPF);
 
 #define IORING_SETUP_FLAGS (IORING_SETUP_IOPOLL |\
 			IORING_SETUP_SQPOLL |\
io_uring/uring_bpf.h +30 -0
index b6cda6df99b1..c76eba887d22 100644
@@ -2,6 +2,29 @@
 #ifndef IOU_BPF_H
 #define IOU_BPF_H
 
+struct uring_bpf_data {
+	/* readonly for bpf prog */
+	struct file     *file;
+	u32		opf;
+
+	/* writeable for bpf prog */
+	u8              pdu[64 - sizeof(struct file *) - sizeof(u32)];
+};
+
+typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
+			       const struct io_uring_sqe *sqe);
+typedef int (*uring_io_issue_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_fail_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_cleanup_t)(struct uring_bpf_data *data);
+
+struct uring_bpf_ops {
+	unsigned short		id;
+	uring_io_prep_t		prep_fn;
+	uring_io_issue_t	issue_fn;
+	uring_io_fail_t		fail_fn;
+	uring_io_cleanup_t	cleanup_fn;
+};
+
 #ifdef CONFIG_IO_URING_BPF
 int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags);
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
@@ -11,6 +34,8 @@ void io_uring_bpf_cleanup(struct io_kiocb *req);
 void uring_bpf_add_ctx(struct io_ring_ctx *ctx);
 void uring_bpf_del_ctx(struct io_ring_ctx *ctx);
 
+int __init io_bpf_init(void);
+
 #else
 static inline int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -33,5 +58,10 @@ static inline void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
 static inline void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
 {
 }
+
+static inline int __init io_bpf_init(void)
+{
+	return 0;
+}
 #endif
 #endif
Signature · 1 line
2.47.0
#9 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

kernel test robot noticed the following build errors:

Hi Ming,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20251104]
[cannot apply to bpf-next/net bpf-next/master bpf/master linus/master v6.18-rc4 v6.18-rc3 v6.18-rc2 v6.18-rc4] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Ming-Lei/io_uring-prepare-for-extending-io_uring-with-bpf/20251105-002757
base:   next-20251104
patch link:    https://lore.kernel.org/r/20251104162123.1086035-4-ming.lei%40redhat.com
patch subject: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops
config: parisc-randconfig-r121-20251107 (https://download.01.org/0day-ci/archive/20251108/[email protected]/config)
compiler: hppa-linux-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251108/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

io_uring/bpf.c: In function 'uring_bpf_ops_is_valid_access':

>> io_uring/bpf.c:117:9: error: implicit declaration of function 'bpf_tracing_btf_ctx_access'; did you mean 'bpf_sock_convert_ctx_access'? [-Werror=implicit-function-declaration]
     return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
            ^~~~~~~~~~~~~~~~~~~~~~~~~~
            bpf_sock_convert_ctx_access
   In file included from include/linux/bpf_verifier.h:7,
                    from io_uring/bpf.c:9:
   io_uring/bpf.c: In function 'io_bpf_init':
   include/linux/bpf.h:2044:50: warning: statement with no effect [-Wunused-value]
    #define register_bpf_struct_ops(st_ops, type) ({ (void *)(st_ops); 0; })
                                                     ^~~~~~~~~~~~~~~~
   io_uring/bpf.c:303:8: note: in expansion of macro 'register_bpf_struct_ops'
     err = register_bpf_struct_ops(&bpf_uring_bpf_ops, uring_bpf_ops);
           ^~~~~~~~~~~~~~~~~~~~~~~
   In file included from <command-line>:
   In function 'io_kiocb_cmd_sz_check',
       inlined from 'io_uring_bpf_prep' at io_uring/bpf.c:41:32:
   include/linux/compiler_types.h:603:38: error: call to '__compiletime_assert_513' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
                                         ^
   include/linux/compiler_types.h:584:4: note: in definition of macro '__compiletime_assert'
       prefix ## suffix();    \
       ^~~~~~
   include/linux/compiler_types.h:603:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
     ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:2: note: in expansion of macro 'BUILD_BUG_ON'
     BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
     ^~~~~~~~~~~~
   In function 'io_kiocb_cmd_sz_check',
       inlined from '__io_uring_bpf_issue.isra.4' at io_uring/bpf.c:58:32,
       inlined from 'io_uring_bpf_issue' at io_uring/bpf.c:72:9:
   include/linux/compiler_types.h:603:38: error: call to '__compiletime_assert_513' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
                                         ^
   include/linux/compiler_types.h:584:4: note: in definition of macro '__compiletime_assert'
       prefix ## suffix();    \
       ^~~~~~
   include/linux/compiler_types.h:603:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
     ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:2: note: in expansion of macro 'BUILD_BUG_ON'
     BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
     ^~~~~~~~~~~~
   In function 'io_kiocb_cmd_sz_check',
       inlined from '__io_uring_bpf_issue.isra.4' at io_uring/bpf.c:58:32,
       inlined from 'io_uring_bpf_issue' at io_uring/bpf.c:77:9:
   include/linux/compiler_types.h:603:38: error: call to '__compiletime_assert_513' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
                                         ^
   include/linux/compiler_types.h:584:4: note: in definition of macro '__compiletime_assert'
       prefix ## suffix();    \
       ^~~~~~
   include/linux/compiler_types.h:603:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
     ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:2: note: in expansion of macro 'BUILD_BUG_ON'
     BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
     ^~~~~~~~~~~~
   In function 'io_kiocb_cmd_sz_check',
       inlined from 'io_uring_bpf_fail' at io_uring/bpf.c:82:32:
   include/linux/compiler_types.h:603:38: error: call to '__compiletime_assert_513' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
                                         ^
   include/linux/compiler_types.h:584:4: note: in definition of macro '__compiletime_assert'
       prefix ## suffix();    \
       ^~~~~~
   include/linux/compiler_types.h:603:2: note: in expansion of macro '_compiletime_assert'
     _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
     ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
    #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
                                        ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:2: note: in expansion of macro 'BUILD_BUG_ON_MSG'
     BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
     ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:2: note: in expansion of macro 'BUILD_BUG_ON'
     BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
     ^~~~~~~~~~~~

include/linux/io_uring_types.h: In function 'io_uring_bpf_cleanup':
include/linux/compiler_types.h:603:38: error: call to '__compiletime_assert_513' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
_compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
^
include/linux/compiler_types.h:584:4: note: in definition of macro '__compiletime_assert'

vim +117 io_uring/bpf.c

111
112 static bool uring_bpf_ops_is_valid_access(int off, int size,

   113					       enum bpf_access_type type,
   114					       const struct bpf_prog *prog,
   115					       struct bpf_insn_access_aux *info)
   116	{
 > 117		return bpf_tracing_btf_ctx_access(off, size, type, prog, info);

118 }
119

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#10 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

In reply to Ming Lei
kernel test robot noticed the following build errors:

Hi Ming,

kernel test robot noticed the following build errors:

[auto build test ERROR on next-20251104]
[cannot apply to bpf-next/net bpf-next/master bpf/master linus/master v6.18-rc4 v6.18-rc3 v6.18-rc2 v6.18-rc4] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Ming-Lei/io_uring-prepare-for-extending-io_uring-with-bpf/20251105-002757
base:   next-20251104
patch link:    https://lore.kernel.org/r/20251104162123.1086035-4-ming.lei%40redhat.com
patch subject: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

:::::: branch date: 3 days ago
:::::: commit date: 3 days ago
config: hexagon-allmodconfig (https://download.01.org/0day-ci/archive/20251108/[email protected]/config)
compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251108/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/r/[email protected]/

All errors (new ones prefixed by >>):

io_uring/bpf.c:29:28: warning: unused function 'uring_bpf_get_flags' [-Wunused-function]
29 | static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)
| ^~~~~~~~~~~~~~~~~~~
In file included from io_uring/bpf.c:14:
In file included from io_uring/io_uring.h:9:

>> include/linux/io_uring_types.h:655:2: error: call to '__compiletime_assert_586' declared with 'error' attribute: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     655 |         BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
         |         ^
   include/linux/build_bug.h:50:2: note: expanded from macro 'BUILD_BUG_ON'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^
   include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^
   include/linux/compiler_types.h:603:2: note: expanded from macro 'compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^
   include/linux/compiler_types.h:591:2: note: expanded from macro '_compiletime_assert'
     591 |         __compiletime_assert(condition, msg, prefix, suffix)
         |         ^
   include/linux/compiler_types.h:584:4: note: expanded from macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^
   <scratch space>:174:1: note: expanded from here
     174 | __compiletime_assert_586
         | ^
   In file included from io_uring/bpf.c:14:
   In file included from io_uring/io_uring.h:9:
>> include/linux/io_uring_types.h:655:2: error: call to '__compiletime_assert_586' declared with 'error' attribute: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
   include/linux/build_bug.h:50:2: note: expanded from macro 'BUILD_BUG_ON'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^
   include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^
   include/linux/compiler_types.h:603:2: note: expanded from macro 'compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^
   include/linux/compiler_types.h:591:2: note: expanded from macro '_compiletime_assert'
     591 |         __compiletime_assert(condition, msg, prefix, suffix)
         |         ^
   include/linux/compiler_types.h:584:4: note: expanded from macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^
   <scratch space>:174:1: note: expanded from here
     174 | __compiletime_assert_586
         | ^
   In file included from io_uring/bpf.c:14:
   In file included from io_uring/io_uring.h:9:
>> include/linux/io_uring_types.h:655:2: error: call to '__compiletime_assert_586' declared with 'error' attribute: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
   include/linux/build_bug.h:50:2: note: expanded from macro 'BUILD_BUG_ON'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^
   include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^
   include/linux/compiler_types.h:603:2: note: expanded from macro 'compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^
   include/linux/compiler_types.h:591:2: note: expanded from macro '_compiletime_assert'
     591 |         __compiletime_assert(condition, msg, prefix, suffix)
         |         ^
   include/linux/compiler_types.h:584:4: note: expanded from macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^
   <scratch space>:174:1: note: expanded from here
     174 | __compiletime_assert_586
         | ^
   In file included from io_uring/bpf.c:14:
   In file included from io_uring/io_uring.h:9:
>> include/linux/io_uring_types.h:655:2: error: call to '__compiletime_assert_586' declared with 'error' attribute: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
   include/linux/build_bug.h:50:2: note: expanded from macro 'BUILD_BUG_ON'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^
   include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^
   include/linux/compiler_types.h:603:2: note: expanded from macro 'compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^
   include/linux/compiler_types.h:591:2: note: expanded from macro '_compiletime_assert'
     591 |         __compiletime_assert(condition, msg, prefix, suffix)
         |         ^
   include/linux/compiler_types.h:584:4: note: expanded from macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^
   <scratch space>:174:1: note: expanded from here
     174 | __compiletime_assert_586
         | ^
   In file included from io_uring/bpf.c:14:
   In file included from io_uring/io_uring.h:9:
>> include/linux/io_uring_types.h:655:2: error: call to '__compiletime_assert_586' declared with 'error' attribute: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
   include/linux/build_bug.h:50:2: note: expanded from macro 'BUILD_BUG_ON'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^
   include/linux/build_bug.h:39:37: note: expanded from macro 'BUILD_BUG_ON_MSG'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^
   include/linux/compiler_types.h:603:2: note: expanded from macro 'compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^
   include/linux/compiler_types.h:591:2: note: expanded from macro '_compiletime_assert'
     591 |         __compiletime_assert(condition, msg, prefix, suffix)
         |         ^
   include/linux/compiler_types.h:584:4: note: expanded from macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^
   <scratch space>:174:1: note: expanded from here
     174 | __compiletime_assert_586
         | ^

1 warning and 5 errors generated.

vim +655 include/linux/io_uring_types.h

e27f928ee1cb06 io_uring/io_uring_types.h      Jens Axboe          2022-05-24  652  
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  653  static inline void io_kiocb_cmd_sz_check(size_t cmd_sz)
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  654  {
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11 @655  	BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  656  }
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  657  #define io_kiocb_to_cmd(req, cmd_type) ( \
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  658  	io_kiocb_cmd_sz_check(sizeof(cmd_type)) , \
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  659  	((cmd_type *)&(req)->cmd) \
f2ccb5aed7bce1 include/linux/io_uring_types.h Stefan Metzmacher   2022-08-11  660  )

09fdd35162c289 include/linux/io_uring_types.h Caleb Sander Mateos 2025-02-28 661

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#11 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

In reply to Ming Lei

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 57 lines
---
  include/uapi/linux/io_uring.h |   9 ++
  io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
  io_uring/io_uring.c           |   1 +
  io_uring/io_uring.h           |   3 +-
  io_uring/uring_bpf.h          |  30 ++++
  5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
  		__u32		install_fd_flags;
  		__u32		nop_flags;
  		__u32		pipe_flags;
+		__u32		bpf_op_flags;
  	};
  	__u64	user_data;	/* data to be passed back at completion time */
  	/* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
  #define IORING_RECVSEND_BUNDLE		(1U << 4)
  #define IORING_SEND_VECTORIZED		(1U << 5)
 
+/*
+ * sqe->bpf_op_flags		top 8bits is for storing bpf op
+ *				The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS	(8)
+#define IORING_BPF_OP_SHIFT	(24)
+
  /*
   * cqe.res for IORING_CQE_F_NOTIF if
   * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
  #define IORING_FEAT_MIN_TIMEOUT		(1U << 15)
  #define IORING_FEAT_RW_ATTR		(1U << 16)
  #define IORING_FEAT_NO_IOWAIT		(1U << 17)
+#define IORING_FEAT_BPF			(1U << 18)
 
  /*
   * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +6 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
  #include <linux/kernel.h>
  #include <linux/errno.h>
  #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>

io_uring/bpf.c line 22

Quoted 10 lines
io_uring/bpf.c +5 -0
+#include <linux/filter.h>
  #include "io_uring.h"
  #include "uring_bpf.h"
 
+#define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
+
  static DEFINE_MUTEX(uring_bpf_ctx_lock);
  static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

metze

author
#12 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 89 lines · Stefan Metzmacher

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 59 lines
---
  include/uapi/linux/io_uring.h |   9 ++
  io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
  io_uring/io_uring.c           |   1 +
  io_uring/io_uring.h           |   3 +-
  io_uring/uring_bpf.h          |  30 ++++
  5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
  		__u32		install_fd_flags;
  		__u32		nop_flags;
  		__u32		pipe_flags;
+		__u32		bpf_op_flags;
  	};
  	__u64	user_data;	/* data to be passed back at completion time */
  	/* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
  #define IORING_RECVSEND_BUNDLE		(1U << 4)
  #define IORING_SEND_VECTORIZED		(1U << 5)
+/*
+ * sqe->bpf_op_flags		top 8bits is for storing bpf op
+ *				The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS	(8)
+#define IORING_BPF_OP_SHIFT	(24)
+
  /*
   * cqe.res for IORING_CQE_F_NOTIF if
   * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
  #define IORING_FEAT_MIN_TIMEOUT		(1U << 15)
  #define IORING_FEAT_RW_ATTR		(1U << 16)
  #define IORING_FEAT_NO_IOWAIT		(1U << 17)
+#define IORING_FEAT_BPF			(1U << 18)
  /*
   * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +8 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
  #include <linux/kernel.h>
  #include <linux/errno.h>
  #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
  #include "io_uring.h"
  #include "uring_bpf.h"
+#define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
Quoted 8 lines

io_uring/bpf.c line 22

Quoted 5 lines
io_uring/bpf.c +3 -0
+
  static DEFINE_MUTEX(uring_bpf_ctx_lock);
  static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

So far there is only 62 in-tree io_uring operation defined, I feel 256 should be enough.

Quoted 4 lines

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

per-ctx requirement looks reasonable, and it shouldn't be hard to support.

Quoted 3 lines

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

I considered per-ctx prog before, one drawback is the prog can't be shared among io_ring_ctx, which could waste memory. In my ublk case, there can be lots of devices sharing same bpf prog.

thanks,
Ming

#13 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Am 13.11.25 um 11:59 schrieb Ming Lei:

Quoted 110 lines · Stefan Metzmacher
Quoted 89 lines

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 59 lines
---
   include/uapi/linux/io_uring.h |   9 ++
   io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
   io_uring/io_uring.c           |   1 +
   io_uring/io_uring.h           |   3 +-
   io_uring/uring_bpf.h          |  30 ++++
   5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
   		__u32		install_fd_flags;
   		__u32		nop_flags;
   		__u32		pipe_flags;
+		__u32		bpf_op_flags;
   	};
   	__u64	user_data;	/* data to be passed back at completion time */
   	/* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
   #define IORING_RECVSEND_BUNDLE		(1U << 4)
   #define IORING_SEND_VECTORIZED		(1U << 5)
+/*
+ * sqe->bpf_op_flags		top 8bits is for storing bpf op
+ *				The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS	(8)
+#define IORING_BPF_OP_SHIFT	(24)
+
   /*
    * cqe.res for IORING_CQE_F_NOTIF if
    * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
   #define IORING_FEAT_MIN_TIMEOUT		(1U << 15)
   #define IORING_FEAT_RW_ATTR		(1U << 16)
   #define IORING_FEAT_NO_IOWAIT		(1U << 17)
+#define IORING_FEAT_BPF			(1U << 18)
   /*
    * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +8 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
   #include <linux/kernel.h>
   #include <linux/errno.h>
   #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
   #include "io_uring.h"
   #include "uring_bpf.h"
+#define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
Quoted 8 lines

io_uring/bpf.c line 22

Quoted 5 lines
io_uring/bpf.c +3 -0
+
   static DEFINE_MUTEX(uring_bpf_ctx_lock);
   static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

So far there is only 62 in-tree io_uring operation defined, I feel 256 should be enough.

Quoted 3 lines

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

per-ctx requirement looks reasonable, and it shouldn't be hard to support.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

I considered per-ctx prog before, one drawback is the prog can't be shared among io_ring_ctx, which could waste memory. In my ublk case, there can be lots of devices sharing same bpf prog.

Can't the ublk instances coordinate and use the same bpf_fd? new instances could request it via a unix socket and SCM_RIGHTS from a long running loading process. On the other hand do they really want to share?

I don't know much about bpf in details, so I'm wondering in your example from
https://github.com/ming1/liburing/commit/625b69ddde15ad80e078c684ba166f49c1174fa4

Would memory_map be global in the whole system or would
each loaded instance of the program have it's own instance of memory_map?

Thanks!
metze

author
#14 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 114 lines · Stefan Metzmacher

Am 13.11.25 um 11:59 schrieb Ming Lei:

Quoted 113 lines · Stefan Metzmacher
Quoted 89 lines

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 59 lines
---
   include/uapi/linux/io_uring.h |   9 ++
   io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
   io_uring/io_uring.c           |   1 +
   io_uring/io_uring.h           |   3 +-
   io_uring/uring_bpf.h          |  30 ++++
   5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
   		__u32		install_fd_flags;
   		__u32		nop_flags;
   		__u32		pipe_flags;
+		__u32		bpf_op_flags;
   	};
   	__u64	user_data;	/* data to be passed back at completion time */
   	/* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
   #define IORING_RECVSEND_BUNDLE		(1U << 4)
   #define IORING_SEND_VECTORIZED		(1U << 5)
+/*
+ * sqe->bpf_op_flags		top 8bits is for storing bpf op
+ *				The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS	(8)
+#define IORING_BPF_OP_SHIFT	(24)
+
   /*
    * cqe.res for IORING_CQE_F_NOTIF if
    * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
   #define IORING_FEAT_MIN_TIMEOUT		(1U << 15)
   #define IORING_FEAT_RW_ATTR		(1U << 16)
   #define IORING_FEAT_NO_IOWAIT		(1U << 17)
+#define IORING_FEAT_BPF			(1U << 18)
   /*
    * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +8 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
   #include <linux/kernel.h>
   #include <linux/errno.h>
   #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
   #include "io_uring.h"
   #include "uring_bpf.h"
+#define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
Quoted 8 lines

io_uring/bpf.c line 22

Quoted 5 lines
io_uring/bpf.c +3 -0
+
   static DEFINE_MUTEX(uring_bpf_ctx_lock);
   static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

So far there is only 62 in-tree io_uring operation defined, I feel 256 should be enough.

Quoted 3 lines

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

per-ctx requirement looks reasonable, and it shouldn't be hard to support.

Quoted 3 lines

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

Quoted 8 lines
Quoted 3 lines

I considered per-ctx prog before, one drawback is the prog can't be shared among io_ring_ctx, which could waste memory. In my ublk case, there can be lots of devices sharing same bpf prog.

Can't the ublk instances coordinate and use the same bpf_fd? new instances could request it via a unix socket and SCM_RIGHTS from a long running loading process. On the other hand do they really want to share?

struct_ops is typically registered once, used everywhere, such as sched_ext and socket example.

This patch follows this usage, so every io_uring application can access it like the in-kernel operations.

I can understand the requirement for per-io-ring-ctx struct_ops, which won't cause conflict among different applications.

For example, ublk/raid5, there are 100 such devices, each device is created in dedicated process and uses its own io-uring, so 100 same struct_ops prog are registered in memory. Given struct_ops prog is registered as per-io-ring-ctx, it may not be shared by bpf_fd, IMO.

Quoted 7 lines

I don't know much about bpf in details, so I'm wondering in your example from
https://github.com/ming1/liburing/commit/625b69ddde15ad80e078c684ba166f49c1174fa4

Would memory_map be global in the whole system or would
each loaded instance of the program have it's own instance of memory_map?

bpf map is global.

At default, each loaded prog owns the map, but it may be exported for others by pinning the map.

It is easy to verify by writing test code in tools/testing/selftests/

But I am not an bpf expert...

Thanks,
Ming

#15 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 131 lines · Ming Lei
Quoted 114 lines · Stefan Metzmacher

Am 13.11.25 um 11:59 schrieb Ming Lei:

Quoted 113 lines · Stefan Metzmacher
Quoted 89 lines

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 59 lines
---
   include/uapi/linux/io_uring.h |   9 ++
   io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
   io_uring/io_uring.c           |   1 +
   io_uring/io_uring.h           |   3 +-
   io_uring/uring_bpf.h          |  30 ++++
   5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
                __u32           install_fd_flags;
                __u32           nop_flags;
                __u32           pipe_flags;
+               __u32           bpf_op_flags;
        };
        __u64   user_data;      /* data to be passed back at completion time */
        /* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
   #define IORING_RECVSEND_BUNDLE               (1U << 4)
   #define IORING_SEND_VECTORIZED               (1U << 5)
+/*
+ * sqe->bpf_op_flags           top 8bits is for storing bpf op
+ *                             The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS     (8)
+#define IORING_BPF_OP_SHIFT    (24)
+
   /*
    * cqe.res for IORING_CQE_F_NOTIF if
    * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
   #define IORING_FEAT_MIN_TIMEOUT              (1U << 15)
   #define IORING_FEAT_RW_ATTR          (1U << 16)
   #define IORING_FEAT_NO_IOWAIT                (1U << 17)
+#define IORING_FEAT_BPF                        (1U << 18)
   /*
    * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +8 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
   #include <linux/kernel.h>
   #include <linux/errno.h>
   #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
   #include "io_uring.h"
   #include "uring_bpf.h"
+#define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
Quoted 8 lines

io_uring/bpf.c line 22

Quoted 5 lines
io_uring/bpf.c +3 -0
+
   static DEFINE_MUTEX(uring_bpf_ctx_lock);
   static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

So far there is only 62 in-tree io_uring operation defined, I feel 256 should be enough.

Quoted 3 lines

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

per-ctx requirement looks reasonable, and it shouldn't be hard to support.

Quoted 3 lines

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

Quoted 8 lines
Quoted 3 lines

I considered per-ctx prog before, one drawback is the prog can't be shared among io_ring_ctx, which could waste memory. In my ublk case, there can be lots of devices sharing same bpf prog.

Can't the ublk instances coordinate and use the same bpf_fd? new instances could request it via a unix socket and SCM_RIGHTS from a long running loading process. On the other hand do they really want to share?

struct_ops is typically registered once, used everywhere, such as sched_ext and socket example.

This patch follows this usage, so every io_uring application can access it like the in-kernel operations.

Quoted 6 lines

I can understand the requirement for per-io-ring-ctx struct_ops, which won't cause conflict among different applications.

For example, ublk/raid5, there are 100 such devices, each device is created in dedicated process and uses its own io-uring, so 100 same struct_ops prog are registered in memory. Given struct_ops prog is registered as per-io-ring-ctx, it may not be shared by bpf_fd, IMO.

I agree with Stefan that a global IORING_OP_BPF op to BPF program mapping will be difficult to coordinate between processes. For example, consider two different ublk server programs that each want to use a different BPF program. Ideally, each should be an independent program and not need to know the op ids used by the other. On the other hand, a multithreaded process may have multiple io_ring_ctxs and want to use the same IORING_OP_BPF ops with all of them. So a process-level mapping seems to make the most sense. And that's exactly the mapping level that we would get from using the BPF program file descriptor to specify the IORING_OP_BPF op. Additionally, as Stefan points out, the IORING_OP_BPF program could be shared with another process by sending the file descriptor using SCM_RIGHTS. And the file descriptor lookup overhead could be avoided in the I/O path using io_uring's existing support for registered files.

Best,
Caleb

Quoted 21 lines
Quoted 8 lines

I don't know much about bpf in details, so I'm wondering in your example from
https://github.com/ming1/liburing/commit/625b69ddde15ad80e078c684ba166f49c1174fa4

Would memory_map be global in the whole system or would
each loaded instance of the program have it's own instance of memory_map?

bpf map is global.

At default, each loaded prog owns the map, but it may be exported for others by pinning the map.

It is easy to verify by writing test code in tools/testing/selftests/

But I am not an bpf expert...

Thanks,
Ming

author
#16 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 139 lines · Caleb Sander Mateos
Quoted 131 lines · Ming Lei
Quoted 114 lines · Stefan Metzmacher

Am 13.11.25 um 11:59 schrieb Ming Lei:

Quoted 113 lines · Stefan Metzmacher
Quoted 89 lines

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 59 lines
---
   include/uapi/linux/io_uring.h |   9 ++
   io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
   io_uring/io_uring.c           |   1 +
   io_uring/io_uring.h           |   3 +-
   io_uring/uring_bpf.h          |  30 ++++
   5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
                __u32           install_fd_flags;
                __u32           nop_flags;
                __u32           pipe_flags;
+               __u32           bpf_op_flags;
        };
        __u64   user_data;      /* data to be passed back at completion time */
        /* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
   #define IORING_RECVSEND_BUNDLE               (1U << 4)
   #define IORING_SEND_VECTORIZED               (1U << 5)
+/*
+ * sqe->bpf_op_flags           top 8bits is for storing bpf op
+ *                             The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS     (8)
+#define IORING_BPF_OP_SHIFT    (24)
+
   /*
    * cqe.res for IORING_CQE_F_NOTIF if
    * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
   #define IORING_FEAT_MIN_TIMEOUT              (1U << 15)
   #define IORING_FEAT_RW_ATTR          (1U << 16)
   #define IORING_FEAT_NO_IOWAIT                (1U << 17)
+#define IORING_FEAT_BPF                        (1U << 18)
   /*
    * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +8 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
   #include <linux/kernel.h>
   #include <linux/errno.h>
   #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
   #include "io_uring.h"
   #include "uring_bpf.h"
+#define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
Quoted 8 lines

io_uring/bpf.c line 22

Quoted 5 lines
io_uring/bpf.c +3 -0
+
   static DEFINE_MUTEX(uring_bpf_ctx_lock);
   static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

So far there is only 62 in-tree io_uring operation defined, I feel 256 should be enough.

Quoted 3 lines

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

per-ctx requirement looks reasonable, and it shouldn't be hard to support.

Quoted 3 lines

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

Quoted 8 lines
Quoted 3 lines

I considered per-ctx prog before, one drawback is the prog can't be shared among io_ring_ctx, which could waste memory. In my ublk case, there can be lots of devices sharing same bpf prog.

Can't the ublk instances coordinate and use the same bpf_fd? new instances could request it via a unix socket and SCM_RIGHTS from a long running loading process. On the other hand do they really want to share?

struct_ops is typically registered once, used everywhere, such as sched_ext and socket example.

This patch follows this usage, so every io_uring application can access it like the in-kernel operations.

Quoted 6 lines

I can understand the requirement for per-io-ring-ctx struct_ops, which won't cause conflict among different applications.

For example, ublk/raid5, there are 100 such devices, each device is created in dedicated process and uses its own io-uring, so 100 same struct_ops prog are registered in memory. Given struct_ops prog is registered as per-io-ring-ctx, it may not be shared by bpf_fd, IMO.

Quoted 5 lines

I agree with Stefan that a global IORING_OP_BPF op to BPF program mapping will be difficult to coordinate between processes. For example, consider two different ublk server programs that each want to use a different BPF program. Ideally, each should be an independent program and not need to know the op ids used by the other.

Each processes can query free slots by checking bpftool struc_ops.

Quoted 7 lines

On the other hand, a multithreaded process may have multiple io_ring_ctxs and want to use the same IORING_OP_BPF ops with all of them. So a process-level mapping seems to make the most sense. And that's exactly the mapping level that we would get from using the BPF program file descriptor to specify the IORING_OP_BPF op. Additionally, as Stefan points out, the IORING_OP_BPF program could be shared with another process by sending the file descriptor using SCM_RIGHTS. And

io_uring FD doesn't support SCM_RIGHTS.

If one privileged process sends bpf prog FD via SCM_RIGHTS, that means this privileged process may be allowed to register any IORING_OP_BPF program, sounds like CONFIG_BPF_UNPRIV_DEFAULT_OFF == n. Probably it is fine if we just expose 'struct uring_bpf_data' and not expose struct io_kiocb to bpf prog.

Another ways is to register global struct_ops prog in the following way:

- the 1st 256 progs are stored in plain array, which can be for really global/generic progs

- the others(256 ~ 65535) progs are stored in xarray, processes can lookup free slots and use them in dynamic allocation way.

I'd suggest to start with global register, which is easy to use, and extend to per-uring-ctx struct_ops in future.

Thanks,
Ming

#17 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 141 lines · Ming Lei
Quoted 139 lines · Caleb Sander Mateos
Quoted 137 lines · Ming Lei
Quoted 114 lines · Stefan Metzmacher

Am 13.11.25 um 11:59 schrieb Ming Lei:

Quoted 113 lines · Stefan Metzmacher
Quoted 89 lines

Hi Ming,

Quoted 19 lines

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

Quoted 6 lines

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

This sounds useful to me.

Quoted 59 lines
---
   include/uapi/linux/io_uring.h |   9 ++
   io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
   io_uring/io_uring.c           |   1 +
   io_uring/io_uring.h           |   3 +-
   io_uring/uring_bpf.h          |  30 ++++
   5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +9 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
                __u32           install_fd_flags;
                __u32           nop_flags;
                __u32           pipe_flags;
+               __u32           bpf_op_flags;
        };
        __u64   user_data;      /* data to be passed back at completion time */
        /* pack this to avoid bogus arm OABI complaints */
@@ -427,6 +428,13 @@ enum io_uring_op {
   #define IORING_RECVSEND_BUNDLE               (1U << 4)
   #define IORING_SEND_VECTORIZED               (1U << 5)
+/*
+ * sqe->bpf_op_flags           top 8bits is for storing bpf op
+ *                             The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS     (8)
+#define IORING_BPF_OP_SHIFT    (24)
+
   /*
    * cqe.res for IORING_CQE_F_NOTIF if
    * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
   #define IORING_FEAT_MIN_TIMEOUT              (1U << 15)
   #define IORING_FEAT_RW_ATTR          (1U << 16)
   #define IORING_FEAT_NO_IOWAIT                (1U << 17)
+#define IORING_FEAT_BPF                        (1U << 18)
   /*
    * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +8 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
   #include <linux/kernel.h>
   #include <linux/errno.h>
   #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
   #include "io_uring.h"
   #include "uring_bpf.h"
+#define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
Quoted 8 lines

io_uring/bpf.c line 21

Quoted 5 lines
io_uring/bpf.c +3 -0
+
   static DEFINE_MUTEX(uring_bpf_ctx_lock);
   static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

This indicates to me that the whole system with all applications in all namespaces need to coordinate in order to use these 256 ops?

So far there is only 62 in-tree io_uring operation defined, I feel 256 should be enough.

Quoted 3 lines

I think in order to have something useful, this should be per struct io_ring_ctx and each application should be able to load its own bpf programs.

per-ctx requirement looks reasonable, and it shouldn't be hard to support.

Quoted 3 lines

Something that uses bpf_prog_get_type() based on a bpf_fd like SIOCKCMATTACH in net/kcm/kcmsock.c.

Quoted 8 lines
Quoted 3 lines

I considered per-ctx prog before, one drawback is the prog can't be shared among io_ring_ctx, which could waste memory. In my ublk case, there can be lots of devices sharing same bpf prog.

Can't the ublk instances coordinate and use the same bpf_fd? new instances could request it via a unix socket and SCM_RIGHTS from a long running loading process. On the other hand do they really want to share?

struct_ops is typically registered once, used everywhere, such as sched_ext and socket example.

This patch follows this usage, so every io_uring application can access it like the in-kernel operations.

I can understand the requirement for per-io-ring-ctx struct_ops, which won't cause conflict among different applications.

For example, ublk/raid5, there are 100 such devices, each device is created in dedicated process and uses its own io-uring, so 100 same struct_ops prog are registered in memory. Given struct_ops prog is registered as per-io-ring-ctx, it may not be shared by bpf_fd, IMO.

Quoted 7 lines
Quoted 5 lines

I agree with Stefan that a global IORING_OP_BPF op to BPF program mapping will be difficult to coordinate between processes. For example, consider two different ublk server programs that each want to use a different BPF program. Ideally, each should be an independent program and not need to know the op ids used by the other.

Each processes can query free slots by checking bpftool struc_ops.

This seems like a time-of-check-to-time-of-use race condition, no? I worry that a malicious process could exploit this by intentionally trying to register IORING_OP_BPF programs concurrently with another process checking for free slots and registering its own programs. If the malicious process is able to register its program in the free slot before the other process, it could trick the other process into executing a malicious IORING_OP_BPF program instead of the intended one, no?

Quoted 10 lines
Quoted 8 lines

On the other hand, a multithreaded process may have multiple io_ring_ctxs and want to use the same IORING_OP_BPF ops with all of them. So a process-level mapping seems to make the most sense. And that's exactly the mapping level that we would get from using the BPF program file descriptor to specify the IORING_OP_BPF op. Additionally, as Stefan points out, the IORING_OP_BPF program could be shared with another process by sending the file descriptor using SCM_RIGHTS. And

io_uring FD doesn't support SCM_RIGHTS.

I (and I assume Stefan) were referring to the BPF program file descriptor, not the io_uring one.

Quoted 7 lines

If one privileged process sends bpf prog FD via SCM_RIGHTS, that means this privileged process may be allowed to register any IORING_OP_BPF program, sounds like CONFIG_BPF_UNPRIV_DEFAULT_OFF == n. Probably it is fine if we just expose 'struct uring_bpf_data' and not expose struct io_kiocb to bpf prog.

Quoted 7 lines

Another ways is to register global struct_ops prog in the following way:

- the 1st 256 progs are stored in plain array, which can be for really global/generic progs

- the others(256 ~ 65535) progs are stored in xarray, processes can lookup free slots and use them in dynamic allocation way.

Why do you feel a separate xarray for IORING_OP_BPF programs is needed? It seems to me like a process's file descriptor table already serves this purpose well.

Quoted 3 lines

I'd suggest to start with global register, which is easy to use, and extend to per-uring-ctx struct_ops in future.

But as soon as global IORING_OP_BPF program opcodes are supported, they become a part of the UAPI. That will make it difficult to remove them in the future. So I would prefer to pin down this core aspect of the UAPI when this feature is merged initially.

Thanks,
Caleb

#18 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

In reply to Ming Lei
Quoted 1 line · Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

So I have a probably dumb question I ran into as I tried to figure this stuff out. You define this maximum here...

Quoted 1 line

+#define MAX_BPF_OPS_COUNT (1 << IORING_BPF_OP_BITS)

...which sizes the bpf_ops array:

Quoted 1 line

+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

Later, you do registration here:

Quoted 15 lines
io_uring/bpf.c +15 -0
+static int io_bpf_reg_unreg(struct uring_bpf_ops *ops, bool reg)
+{
+	struct io_ring_ctx *ctx;
+	int ret = 0;
+
+	guard(mutex)(&uring_bpf_ctx_lock);
+	list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+		mutex_lock(&ctx->uring_lock);
+
+	if (reg) {
+		if (bpf_ops[ops->id].issue_fn)
+			ret = -EBUSY;
+		else
+			bpf_ops[ops->id] = *ops;
+	} else {

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 213

Quoted 10 lines
io_uring/bpf.c +10 -0
+		bpf_ops[ops->id] = (struct uring_bpf_ops) {0};
+	}
+
+	synchronize_srcu(&uring_bpf_srcu);
+
+	list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+		mutex_unlock(&ctx->uring_lock);
+
+	return ret;
+}

Nowhere do I find a test ensuring that ops->id is within range; MAX_BPF_OPS_COUNT never appears in a test. What am I missing?

Thanks,

jon

author
#19 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 35 lines · Jonathan Corbet
Quoted 1 line · Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

So I have a probably dumb question I ran into as I tried to figure this stuff out. You define this maximum here...

Quoted 1 line

+#define MAX_BPF_OPS_COUNT (1 << IORING_BPF_OP_BITS)

...which sizes the bpf_ops array:

Quoted 1 line

+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

Later, you do registration here:

Quoted 20 lines
io_uring/bpf.c +20 -0
+static int io_bpf_reg_unreg(struct uring_bpf_ops *ops, bool reg)
+{
+	struct io_ring_ctx *ctx;
+	int ret = 0;
+
+	guard(mutex)(&uring_bpf_ctx_lock);
+	list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+		mutex_lock(&ctx->uring_lock);
+
+	if (reg) {
+		if (bpf_ops[ops->id].issue_fn)
+			ret = -EBUSY;
+		else
+			bpf_ops[ops->id] = *ops;
+	} else {
+		bpf_ops[ops->id] = (struct uring_bpf_ops) {0};
+	}
+
+	synchronize_srcu(&uring_bpf_srcu);
+
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 213

Quoted 5 lines
io_uring/bpf.c +5 -0
+	list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+		mutex_unlock(&ctx->uring_lock);
+
+	return ret;
+}

Nowhere do I find a test ensuring that ops->id is within range; MAX_BPF_OPS_COUNT never appears in a test. What am I missing?

bits of ops->id is limited by IORING_BPF_OP_BITS and it is stored in top 8bits of ->bpf_flags, so ops->id is within the range naturally.

Thanks,
Ming

author
#20 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 46 lines · Ming Lei
Quoted 35 lines · Jonathan Corbet
Quoted 1 line · Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

So I have a probably dumb question I ran into as I tried to figure this stuff out. You define this maximum here...

Quoted 1 line

+#define MAX_BPF_OPS_COUNT (1 << IORING_BPF_OP_BITS)

...which sizes the bpf_ops array:

Quoted 1 line

+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];

Later, you do registration here:

Quoted 20 lines
io_uring/bpf.c +20 -0
+static int io_bpf_reg_unreg(struct uring_bpf_ops *ops, bool reg)
+{
+   struct io_ring_ctx *ctx;
+   int ret = 0;
+
+   guard(mutex)(&uring_bpf_ctx_lock);
+   list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+           mutex_lock(&ctx->uring_lock);
+
+   if (reg) {
+           if (bpf_ops[ops->id].issue_fn)
+                   ret = -EBUSY;
+           else
+                   bpf_ops[ops->id] = *ops;
+   } else {
+           bpf_ops[ops->id] = (struct uring_bpf_ops) {0};
+   }
+
+   synchronize_srcu(&uring_bpf_srcu);
+
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 213

Quoted 5 lines
io_uring/bpf.c +5 -0
+   list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+           mutex_unlock(&ctx->uring_lock);
+
+   return ret;
+}

Nowhere do I find a test ensuring that ops->id is within range; MAX_BPF_OPS_COUNT never appears in a test. What am I missing?

Quoted 2 lines

bits of ops->id is limited by IORING_BPF_OP_BITS and it is stored in top 8bits of ->bpf_flags, so ops->id is within the range naturally.

oops, misunderstand your point.

Yes, ios->id needs to be checked in io_bpf_reg_unreg(), sorry for the noise.

Thanks,

#21 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

In reply to Ming Lei
Quoted 49 lines · Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

---
 include/uapi/linux/io_uring.h |   9 ++
 io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
 io_uring/io_uring.c           |   1 +
 io_uring/io_uring.h           |   3 +-
 io_uring/uring_bpf.h          |  30 ++++
 5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +1 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
                __u32           install_fd_flags;
                __u32           nop_flags;
                __u32           pipe_flags;
+               __u32           bpf_op_flags;
        };
        __u64   user_data;      /* data to be passed back at completion time */
        /* pack this to avoid bogus arm OABI complaints */

include/uapi/linux/io_uring.h enum io_uring_op { · line 436

Quoted 9 lines
include/uapi/linux/io_uring.h +6 -0
@@ -427,6 +428,13 @@ enum io_uring_op {
 #define IORING_RECVSEND_BUNDLE         (1U << 4)
 #define IORING_SEND_VECTORIZED         (1U << 5)
 
+/*
+ * sqe->bpf_op_flags           top 8bits is for storing bpf op
+ *                             The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS     (8)
+#define IORING_BPF_OP_SHIFT    (24)

Could omit the parentheses here

Quoted 35 lines
include/uapi/linux/io_uring.h +2 -0
+
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
  * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
 #define IORING_FEAT_MIN_TIMEOUT                (1U << 15)
 #define IORING_FEAT_RW_ATTR            (1U << 16)
 #define IORING_FEAT_NO_IOWAIT          (1U << 17)
+#define IORING_FEAT_BPF                        (1U << 18)
 
 /*
  * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +10 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
 #include "io_uring.h"
 #include "uring_bpf.h"
 
+#define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
+
 static DEFINE_MUTEX(uring_bpf_ctx_lock);
 static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);

io_uring/bpf.c line 29

Quoted 10 lines
io_uring/bpf.c +6 -2
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];
 
-int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
 {
-       return -ECANCELED;
+       return (unsigned char)(op_flags >> IORING_BPF_OP_SHIFT);
+}
+
+static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)

u32?

io_uring/bpf.c line 42

Quoted 13 lines
io_uring/bpf.c +9 -0
+{
+       return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+}
+
+static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
+{
+       return &bpf_ops[uring_bpf_get_op(data->opf)];
 }
 
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);

u32?

Quoted 10 lines
io_uring/bpf.c +10 -0
+       struct uring_bpf_ops *ops;
+
+       if (!(req->ctx->flags & IORING_SETUP_BPF))
+               return -EACCES;
+
+       data->opf = op_flags;
+       ops = &bpf_ops[uring_bpf_get_op(data->opf)];
+
+       if (ops->prep_fn)
+               return ops->prep_fn(data, sqe);

io_uring/bpf.c line 62

Quoted 10 lines
io_uring/bpf.c +7 -0
        return -EOPNOTSUPP;
 }
 
+static int __io_uring_bpf_issue(struct io_kiocb *req)
+{
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->issue_fn)
+               return ops->issue_fn(data);

Doesn't this need to use rcu_dereference() to access ops->issue_fn since io_bpf_reg_unreg() may concurrently modify it?

Also, it doesn't look safe to propagate the BPF ->issue_fn() return value to the ->issue() return value. If the BPF program returns IOU_ISSUE_SKIP_COMPLETE = -EIOCBQUEUED, the io_uring request will never be completed. And it looks like ->issue() implementations are meant to return either IOU_COMPLETE, IOU_RETRY, or IOU_ISSUE_SKIP_COMPLETE. If the BPF program returns some other value, it would be nice to propagate it to the io_uring CQE result and return IOU_COMPLETE, similar to io_uring_cmd().

Quoted 40 lines
io_uring/bpf.c +27 -0
+       return -ECANCELED;
+}
+
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+       if (issue_flags & IO_URING_F_UNLOCKED) {
+               int idx, ret;
+
+               idx = srcu_read_lock(&uring_bpf_srcu);
+               ret = __io_uring_bpf_issue(req);
+               srcu_read_unlock(&uring_bpf_srcu, idx);
+
+               return ret;
+       }
+       return __io_uring_bpf_issue(req);
+}
+
 void io_uring_bpf_fail(struct io_kiocb *req)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->fail_fn)
+               ops->fail_fn(data);
 }
 
 void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->cleanup_fn)
+               ops->cleanup_fn(data);
 }
 
 void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
@@ -39,3 +106,203 @@ void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
        guard(mutex)(&uring_bpf_ctx_lock);
        list_del(&ctx->bpf_node);
 }

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 118

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+static const struct btf_type *uring_bpf_data_type;
+
+static bool uring_bpf_ops_is_valid_access(int off, int size,
+                                      enum bpf_access_type type,
+                                      const struct bpf_prog *prog,
+                                      struct bpf_insn_access_aux *info)
+{
+       return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}

Just use bpf_tracing_btf_ctx_access instead of defining another equivalent function?

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 128

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+static int uring_bpf_ops_btf_struct_access(struct bpf_verifier_log *log,
+                                       const struct bpf_reg_state *reg,
+                                       int off, int size)
+{
+       const struct btf_type *t;
+
+       t = btf_type_by_id(reg->btf, reg->btf_id);
+       if (t != uring_bpf_data_type) {
+               bpf_log(log, "only read is supported\n");

What does this log line mean?

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 133

Quoted 5 lines
io_uring/bpf.c +5 -0
+               return -EACCES;
+       }
+
+       if (off < offsetof(struct uring_bpf_data, pdu) ||
+                       off + size >= sizeof(struct uring_bpf_data))

Should be > instead of >=? Otherwise the last byte of pdu isn't usable.

Quoted 18 lines
io_uring/bpf.c +18 -0
+               return -EACCES;
+
+       return NOT_INIT;
+}
+
+static const struct bpf_verifier_ops io_bpf_verifier_ops = {
+       .get_func_proto = bpf_base_func_proto,
+       .is_valid_access = uring_bpf_ops_is_valid_access,
+       .btf_struct_access = uring_bpf_ops_btf_struct_access,
+};
+
+static int uring_bpf_ops_init(struct btf *btf)
+{
+       s32 type_id;
+
+       type_id = btf_find_by_name_kind(btf, "uring_bpf_data", BTF_KIND_STRUCT);
+       if (type_id < 0)
+               return -EINVAL;

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 161

Quoted 10 lines
io_uring/bpf.c +10 -0
+       uring_bpf_data_type = btf_type_by_id(btf, type_id);
+       return 0;
+}
+
+static int uring_bpf_ops_check_member(const struct btf_type *t,
+                                  const struct btf_member *member,
+                                  const struct bpf_prog *prog)
+{
+       return 0;
+}

It looks like struct bpf_struct_ops's .check_member can be omitted if it always succeeds

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 172

Quoted 11 lines
io_uring/bpf.c +11 -0
+
+static int uring_bpf_ops_init_member(const struct btf_type *t,
+                                const struct btf_member *member,
+                                void *kdata, const void *udata)
+{
+       const struct uring_bpf_ops *uuring_bpf_ops;
+       struct uring_bpf_ops *kuring_bpf_ops;
+       u32 moff;
+
+       uuring_bpf_ops = (const struct uring_bpf_ops *)udata;
+       kuring_bpf_ops = (struct uring_bpf_ops *)kdata;

Don't need to explicitly cast from (const) void *. That could allow these initializers to be combined with the variable declarations.

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 178

Quoted 6 lines
io_uring/bpf.c +6 -0
+
+       moff = __btf_member_bit_offset(t, member) / 8;
+
+       switch (moff) {
+       case offsetof(struct uring_bpf_ops, id):
+               /* For dev_id, this function has to copy it and return 1 to

What does "dev_id" refer to?

Quoted 8 lines
io_uring/bpf.c +8 -0
+                * indicate that the data has been handled by the struct_ops
+                * type, or the verifier will reject the map if the value of
+                * those fields is not zero.
+                */
+               kuring_bpf_ops->id = uuring_bpf_ops->id;
+               return 1;
+       }
+       return 0;

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 196

Quoted 10 lines
io_uring/bpf.c +10 -0
+}
+
+static int io_bpf_reg_unreg(struct uring_bpf_ops *ops, bool reg)
+{
+       struct io_ring_ctx *ctx;
+       int ret = 0;
+
+       guard(mutex)(&uring_bpf_ctx_lock);
+       list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+               mutex_lock(&ctx->uring_lock);

Locking multiple io_ring_ctx's uring_locks is deadlock prone. See lock_two_rings() for example, which takes care to acquire multiple uring_locks in a consistent order. Would it be possible to lock one io_ring_ctx at a time and set some flag to indicate that srcu_read_lock() needs to be used?

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 204

Quoted 8 lines
io_uring/bpf.c +8 -0
+
+       if (reg) {
+               if (bpf_ops[ops->id].issue_fn)
+                       ret = -EBUSY;
+               else
+                       bpf_ops[ops->id] = *ops;
+       } else {
+               bpf_ops[ops->id] = (struct uring_bpf_ops) {0};

Don't these need to use rcu_assign_pointer() to assign bpf_ops[ops->id].issue_fn since __io_uring_bpf_issue() may read it concurrently?

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 210

Quoted 6 lines
io_uring/bpf.c +6 -0
+       }
+
+       synchronize_srcu(&uring_bpf_srcu);
+
+       list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+               mutex_unlock(&ctx->uring_lock);

It might be preferable to call synchronize_srcu() after releasing the uring_locks (and maybe uring_bpf_ctx_lock). That would minimize the latency injected into io_uring requests in case synchronize_srcu() blocks for a long time.

Quoted 11 lines
io_uring/bpf.c +11 -0
+
+       return ret;
+}
+
+static int io_bpf_reg(void *kdata, struct bpf_link *link)
+{
+       struct uring_bpf_ops *ops = kdata;
+
+       return io_bpf_reg_unreg(ops, true);
+}
+

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 231

Quoted 10 lines
io_uring/bpf.c +10 -0
+static void io_bpf_unreg(void *kdata, struct bpf_link *link)
+{
+       struct uring_bpf_ops *ops = kdata;
+
+       io_bpf_reg_unreg(ops, false);
+}
+
+static int io_bpf_prep_io(struct uring_bpf_data *data, const struct io_uring_sqe *sqe)
+{
+       return -EOPNOTSUPP;

The return value for the stub functions doesn't matter, return 0 for simplicity?

Also, could the stub functions be renamed to more clearly indicate that they are only used for their signature and shouldn't be called directly?

Quoted 13 lines
io_uring/bpf.c +13 -0
+}
+
+static int io_bpf_issue_io(struct uring_bpf_data *data)
+{
+       return -ECANCELED;
+}
+
+static void io_bpf_fail_io(struct uring_bpf_data *data)
+{
+}
+
+static void io_bpf_cleanup_io(struct uring_bpf_data *data)
+{

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 254

Quoted 10 lines
io_uring/bpf.c +10 -0
+}
+
+static struct uring_bpf_ops __bpf_uring_bpf_ops = {
+       .prep_fn        = io_bpf_prep_io,
+       .issue_fn       = io_bpf_issue_io,
+       .fail_fn        = io_bpf_fail_io,
+       .cleanup_fn     = io_bpf_cleanup_io,
+};
+
+static struct bpf_struct_ops bpf_uring_bpf_ops = {

const?

Quoted 13 lines
io_uring/bpf.c +13 -0
+       .verifier_ops = &io_bpf_verifier_ops,
+       .init = uring_bpf_ops_init,
+       .check_member = uring_bpf_ops_check_member,
+       .init_member = uring_bpf_ops_init_member,
+       .reg = io_bpf_reg,
+       .unreg = io_bpf_unreg,
+       .name = "uring_bpf_ops",
+       . = &__bpf_uring_bpf_ops,
+       .owner = THIS_MODULE,
+};
+
+__bpf_kfunc_start_defs();
+__bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 277

Quoted 10 lines
io_uring/bpf.c +10 -0
+{
+       struct io_kiocb *req = cmd_to_io_kiocb(data);
+
+       if (res < 0)
+               req_set_fail(req);
+       io_req_set_res(req, res, 0);
+}
+
+/* io_kiocb layout might be changed */
+__bpf_kfunc struct io_kiocb *uring_bpf_data_to_req(struct uring_bpf_data *data)

How would the returned struct io_kiocb * be used in an io_uring BPF program?

Quoted 18 lines
io_uring/bpf.c +18 -0
+{
+       return cmd_to_io_kiocb(data);
+}
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(uring_bpf_kfuncs)
+BTF_ID_FLAGS(func, uring_bpf_set_result)
+BTF_ID_FLAGS(func, uring_bpf_data_to_req)
+BTF_KFUNCS_END(uring_bpf_kfuncs)
+
+static const struct btf_kfunc_id_set uring_kfunc_set = {
+       .owner = THIS_MODULE,
+       .set   = &uring_bpf_kfuncs,
+};
+
+int __init io_bpf_init(void)
+{
+       int err;

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 305

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+       err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &uring_kfunc_set);
+       if (err) {
+               pr_warn("error while setting UBLK BPF tracing kfuncs: %d", err);
+               return err;
+       }
+
+       err = register_bpf_struct_ops(&bpf_uring_bpf_ops, uring_bpf_ops);
+       if (err)
+               pr_warn("error while registering io_uring bpf struct ops: %d", err);

Is there a reason this error isn't fatal?

io_uring/io_uring.c static int __init io_uring_init(void) · line 3854

Quoted 12 lines
io_uring/bpf.c +3 -0
+
+       return 0;
+}
io_uring/io_uring.c +1 -0
index 38f03f6c28cb..d2517e09407a 100644
@@ -3851,6 +3851,7 @@ static int __init io_uring_init(void)
        register_sysctl_init("kernel", kernel_io_uring_disabled_table);
 #endif
 
+       io_bpf_init();

It doesn't look like there are any particular initialization ordering requirements with the rest of io_uring_init(). How about making a separate __initcall() in bpf.c so io_bpf_init() doesn't need to be visible outside that file?

Quoted 8 lines
io_uring/io_uring.c +0 -0
        return 0;
 };
 __initcall(io_uring_init);
io_uring/io_uring.h +0 -0
index 4baf21a9e1ee..3f19bb079bcc 100644

io_uring/io_uring.h line 38

Quoted 6 lines
io_uring/io_uring.h +2 -1
@@ -34,7 +34,8 @@
                        IORING_FEAT_RECVSEND_BUNDLE |\
                        IORING_FEAT_MIN_TIMEOUT |\
                        IORING_FEAT_RW_ATTR |\
-                       IORING_FEAT_NO_IOWAIT)
+                       IORING_FEAT_NO_IOWAIT |\
+                       IORING_FEAT_BPF);

Unintentional semicolon?

io_uring/uring_bpf.h line 6

Quoted 13 lines
io_uring/io_uring.h +0 -0
 #define IORING_SETUP_FLAGS (IORING_SETUP_IOPOLL |\
                        IORING_SETUP_SQPOLL |\
io_uring/uring_bpf.h +2 -0
index b6cda6df99b1..c76eba887d22 100644
@@ -2,6 +2,29 @@
 #ifndef IOU_BPF_H
 #define IOU_BPF_H
 
+struct uring_bpf_data {
+       /* readonly for bpf prog */

It doesn't look like uring_bpf_ops_btf_struct_access() actually allows these fields to be accessed?

io_uring/uring_bpf.h line 18

Quoted 12 lines
io_uring/uring_bpf.h +12 -0
+       struct file     *file;
+       u32             opf;
+
+       /* writeable for bpf prog */
+       u8              pdu[64 - sizeof(struct file *) - sizeof(u32)];
+};
+
+typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
+                              const struct io_uring_sqe *sqe);
+typedef int (*uring_io_issue_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_fail_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_cleanup_t)(struct uring_bpf_data *data);

"uring_io" seems like a strange name for function typedefs specific to io_uring BPF. How about renaming these to "uring_bpf_..." instead?

Best,
Caleb

Quoted 35 lines
io_uring/uring_bpf.h +16 -1
+
+struct uring_bpf_ops {
+       unsigned short          id;
+       uring_io_prep_t         prep_fn;
+       uring_io_issue_t        issue_fn;
+       uring_io_fail_t         fail_fn;
+       uring_io_cleanup_t      cleanup_fn;
+};
+
 #ifdef CONFIG_IO_URING_BPF
 int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags);
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
@@ -11,6 +34,8 @@ void io_uring_bpf_cleanup(struct io_kiocb *req);
 void uring_bpf_add_ctx(struct io_ring_ctx *ctx);
 void uring_bpf_del_ctx(struct io_ring_ctx *ctx);
 
+int __init io_bpf_init(void);
+
 #else
 static inline int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
@@ -33,5 +58,10 @@ static inline void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
 static inline void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
 {
 }
+
+static inline int __init io_bpf_init(void)
+{
+       return 0;
+}
 #endif
 #endif
--

2.47.0

author
#22 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 139 lines · Caleb Sander Mateos
Quoted 49 lines · Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

---
 include/uapi/linux/io_uring.h |   9 ++
 io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
 io_uring/io_uring.c           |   1 +
 io_uring/io_uring.h           |   3 +-
 io_uring/uring_bpf.h          |  30 ++++
 5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +1 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
                __u32           install_fd_flags;
                __u32           nop_flags;
                __u32           pipe_flags;
+               __u32           bpf_op_flags;
        };
        __u64   user_data;      /* data to be passed back at completion time */
        /* pack this to avoid bogus arm OABI complaints */

include/uapi/linux/io_uring.h enum io_uring_op { · line 436

Quoted 9 lines
include/uapi/linux/io_uring.h +6 -0
@@ -427,6 +428,13 @@ enum io_uring_op {
 #define IORING_RECVSEND_BUNDLE         (1U << 4)
 #define IORING_SEND_VECTORIZED         (1U << 5)
 
+/*
+ * sqe->bpf_op_flags           top 8bits is for storing bpf op
+ *                             The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS     (8)
+#define IORING_BPF_OP_SHIFT    (24)

Could omit the parentheses here

Quoted 35 lines
include/uapi/linux/io_uring.h +2 -0
+
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
  * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
 #define IORING_FEAT_MIN_TIMEOUT                (1U << 15)
 #define IORING_FEAT_RW_ATTR            (1U << 16)
 #define IORING_FEAT_NO_IOWAIT          (1U << 17)
+#define IORING_FEAT_BPF                        (1U << 18)
 
 /*
  * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +10 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
 #include "io_uring.h"
 #include "uring_bpf.h"
 
+#define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
+
 static DEFINE_MUTEX(uring_bpf_ctx_lock);
 static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);

io_uring/bpf.c line 29

Quoted 10 lines
io_uring/bpf.c +6 -2
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];
 
-int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
 {
-       return -ECANCELED;
+       return (unsigned char)(op_flags >> IORING_BPF_OP_SHIFT);
+}
+
+static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)

u32?

io_uring/bpf.c line 42

Quoted 13 lines
io_uring/bpf.c +9 -0
+{
+       return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+}
+
+static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
+{
+       return &bpf_ops[uring_bpf_get_op(data->opf)];
 }
 
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);

u32?

Quoted 13 lines
io_uring/bpf.c +10 -0
+       struct uring_bpf_ops *ops;
+
+       if (!(req->ctx->flags & IORING_SETUP_BPF))
+               return -EACCES;
+
+       data->opf = op_flags;
+       ops = &bpf_ops[uring_bpf_get_op(data->opf)];
+
+       if (ops->prep_fn)
+               return ops->prep_fn(data, sqe);
        return -EOPNOTSUPP;
 }
 
Quoted 10 lines

io_uring/bpf.c line 62

Quoted 7 lines
io_uring/bpf.c +7 -0
+static int __io_uring_bpf_issue(struct io_kiocb *req)
+{
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->issue_fn)
+               return ops->issue_fn(data);

Doesn't this need to use rcu_dereference() to access ops->issue_fn since io_bpf_reg_unreg() may concurrently modify it?

rcu isn't enough, io_bpf_reg_unreg() shouldn't be started unless one bpf IO is completed. Probably percpu-refcount may have to be used.

Quoted 5 lines

Also, it doesn't look safe to propagate the BPF ->issue_fn() return value to the ->issue() return value. If the BPF program returns IOU_ISSUE_SKIP_COMPLETE = -EIOCBQUEUED, the io_uring request will never be completed. And it looks like ->issue() implementations are

It depends on if bpf OP can support async IO, and it relies on bpf kfunc's capability actually.

But yes, it is better to start with sync bpf IO only.

Quoted 59 lines

meant to return either IOU_COMPLETE, IOU_RETRY, or IOU_ISSUE_SKIP_COMPLETE. If the BPF program returns some other value, it would be nice to propagate it to the io_uring CQE result and return IOU_COMPLETE, similar to io_uring_cmd().

Quoted 40 lines
io_uring/bpf.c +27 -0
+       return -ECANCELED;
+}
+
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+       if (issue_flags & IO_URING_F_UNLOCKED) {
+               int idx, ret;
+
+               idx = srcu_read_lock(&uring_bpf_srcu);
+               ret = __io_uring_bpf_issue(req);
+               srcu_read_unlock(&uring_bpf_srcu, idx);
+
+               return ret;
+       }
+       return __io_uring_bpf_issue(req);
+}
+
 void io_uring_bpf_fail(struct io_kiocb *req)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->fail_fn)
+               ops->fail_fn(data);
 }
 
 void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->cleanup_fn)
+               ops->cleanup_fn(data);
 }
 
 void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
@@ -39,3 +106,203 @@ void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
        guard(mutex)(&uring_bpf_ctx_lock);
        list_del(&ctx->bpf_node);
 }

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 118

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+static const struct btf_type *uring_bpf_data_type;
+
+static bool uring_bpf_ops_is_valid_access(int off, int size,
+                                      enum bpf_access_type type,
+                                      const struct bpf_prog *prog,
+                                      struct bpf_insn_access_aux *info)
+{
+       return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}

Just use bpf_tracing_btf_ctx_access instead of defining another equivalent function?

Quoted 12 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 128

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+static int uring_bpf_ops_btf_struct_access(struct bpf_verifier_log *log,
+                                       const struct bpf_reg_state *reg,
+                                       int off, int size)
+{
+       const struct btf_type *t;
+
+       t = btf_type_by_id(reg->btf, reg->btf_id);
+       if (t != uring_bpf_data_type) {
+               bpf_log(log, "only read is supported\n");

What does this log line mean?

We only support to write to uring_bpf_data.pdu, and readonly is for other type.

Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 133

Quoted 6 lines
io_uring/bpf.c +5 -0
+               return -EACCES;
+       }
+
+       if (off < offsetof(struct uring_bpf_data, pdu) ||
+                       off + size >= sizeof(struct uring_bpf_data))

Should be > instead of >=? Otherwise the last byte of pdu isn't usable.

Good catch!

Quoted 24 lines
Quoted 24 lines
io_uring/bpf.c +23 -0
+               return -EACCES;
+
+       return NOT_INIT;
+}
+
+static const struct bpf_verifier_ops io_bpf_verifier_ops = {
+       .get_func_proto = bpf_base_func_proto,
+       .is_valid_access = uring_bpf_ops_is_valid_access,
+       .btf_struct_access = uring_bpf_ops_btf_struct_access,
+};
+
+static int uring_bpf_ops_init(struct btf *btf)
+{
+       s32 type_id;
+
+       type_id = btf_find_by_name_kind(btf, "uring_bpf_data", BTF_KIND_STRUCT);
+       if (type_id < 0)
+               return -EINVAL;
+       uring_bpf_data_type = btf_type_by_id(btf, type_id);
+       return 0;
+}
+
+static int uring_bpf_ops_check_member(const struct btf_type *t,
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 161

Quoted 5 lines
io_uring/bpf.c +5 -0
+                                  const struct btf_member *member,
+                                  const struct bpf_prog *prog)
+{
+       return 0;
+}

It looks like struct bpf_struct_ops's .check_member can be omitted if it always succeeds

I think it is better to check each member of the struct_ops, also .sleepable need to be checked here.

Quoted 15 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 172

Quoted 12 lines
io_uring/bpf.c +11 -0
+
+static int uring_bpf_ops_init_member(const struct btf_type *t,
+                                const struct btf_member *member,
+                                void *kdata, const void *udata)
+{
+       const struct uring_bpf_ops *uuring_bpf_ops;
+       struct uring_bpf_ops *kuring_bpf_ops;
+       u32 moff;
+
+       uuring_bpf_ops = (const struct uring_bpf_ops *)udata;
+       kuring_bpf_ops = (struct uring_bpf_ops *)kdata;

Don't need to explicitly cast from (const) void *. That could allow these initializers to be combined with the variable declarations.

OK.

Quoted 9 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 178

Quoted 7 lines
io_uring/bpf.c +6 -0
+
+       moff = __btf_member_bit_offset(t, member) / 8;
+
+       switch (moff) {
+       case offsetof(struct uring_bpf_ops, id):
+               /* For dev_id, this function has to copy it and return 1 to

What does "dev_id" refer to?

Looks a typo.

Quoted 39 lines
Quoted 9 lines
io_uring/bpf.c +8 -0
+                * indicate that the data has been handled by the struct_ops
+                * type, or the verifier will reject the map if the value of
+                * those fields is not zero.
+                */
+               kuring_bpf_ops->id = uuring_bpf_ops->id;
+               return 1;
+       }
+       return 0;

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 196

Quoted 10 lines
io_uring/bpf.c +10 -0
+}
+
+static int io_bpf_reg_unreg(struct uring_bpf_ops *ops, bool reg)
+{
+       struct io_ring_ctx *ctx;
+       int ret = 0;
+
+       guard(mutex)(&uring_bpf_ctx_lock);
+       list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+               mutex_lock(&ctx->uring_lock);

Locking multiple io_ring_ctx's uring_locks is deadlock prone. See lock_two_rings() for example, which takes care to acquire multiple uring_locks in a consistent order. Would it be possible to lock one io_ring_ctx at a time and set some flag to indicate that srcu_read_lock() needs to be used?

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 204

Quoted 8 lines
io_uring/bpf.c +8 -0
+
+       if (reg) {
+               if (bpf_ops[ops->id].issue_fn)
+                       ret = -EBUSY;
+               else
+                       bpf_ops[ops->id] = *ops;
+       } else {
+               bpf_ops[ops->id] = (struct uring_bpf_ops) {0};

Don't these need to use rcu_assign_pointer() to assign bpf_ops[ops->id].issue_fn since __io_uring_bpf_issue() may read it concurrently?

Quoted 11 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 210

Quoted 6 lines
io_uring/bpf.c +6 -0
+       }
+
+       synchronize_srcu(&uring_bpf_srcu);
+
+       list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+               mutex_unlock(&ctx->uring_lock);

It might be preferable to call synchronize_srcu() after releasing the uring_locks (and maybe uring_bpf_ctx_lock). That would minimize the latency injected into io_uring requests in case synchronize_srcu() blocks for a long time.

I plan to switch to percpu-refcount, which is simple, and can avoid unreg when one such bpf io is inflight.

Quoted 25 lines
Quoted 12 lines
io_uring/bpf.c +11 -0
+
+       return ret;
+}
+
+static int io_bpf_reg(void *kdata, struct bpf_link *link)
+{
+       struct uring_bpf_ops *ops = kdata;
+
+       return io_bpf_reg_unreg(ops, true);
+}
+

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 231

Quoted 10 lines
io_uring/bpf.c +10 -0
+static void io_bpf_unreg(void *kdata, struct bpf_link *link)
+{
+       struct uring_bpf_ops *ops = kdata;
+
+       io_bpf_reg_unreg(ops, false);
+}
+
+static int io_bpf_prep_io(struct uring_bpf_data *data, const struct io_uring_sqe *sqe)
+{
+       return -EOPNOTSUPP;

The return value for the stub functions doesn't matter, return 0 for simplicity?

Quoted 3 lines

Also, could the stub functions be renamed to more clearly indicate that they are only used for their signature and shouldn't be called directly?

Looks fine.

Quoted 18 lines
Quoted 18 lines
io_uring/bpf.c +17 -0
+}
+
+static int io_bpf_issue_io(struct uring_bpf_data *data)
+{
+       return -ECANCELED;
+}
+
+static void io_bpf_fail_io(struct uring_bpf_data *data)
+{
+}
+
+static void io_bpf_cleanup_io(struct uring_bpf_data *data)
+{
+}
+
+static struct uring_bpf_ops __bpf_uring_bpf_ops = {
+       .prep_fn        = io_bpf_prep_io,
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 254

Quoted 6 lines
io_uring/bpf.c +6 -0
+       .issue_fn       = io_bpf_issue_io,
+       .fail_fn        = io_bpf_fail_io,
+       .cleanup_fn     = io_bpf_cleanup_io,
+};
+
+static struct bpf_struct_ops bpf_uring_bpf_ops = {

const?

int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)

Quoted 18 lines
Quoted 18 lines
io_uring/bpf.c +17 -0
+       .verifier_ops = &io_bpf_verifier_ops,
+       .init = uring_bpf_ops_init,
+       .check_member = uring_bpf_ops_check_member,
+       .init_member = uring_bpf_ops_init_member,
+       .reg = io_bpf_reg,
+       .unreg = io_bpf_unreg,
+       .name = "uring_bpf_ops",
+       . = &__bpf_uring_bpf_ops,
+       .owner = THIS_MODULE,
+};
+
+__bpf_kfunc_start_defs();
+__bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)
+{
+       struct io_kiocb *req = cmd_to_io_kiocb(data);
+
+       if (res < 0)
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 277

Quoted 6 lines
io_uring/bpf.c +6 -0
+               req_set_fail(req);
+       io_req_set_res(req, res, 0);
+}
+
+/* io_kiocb layout might be changed */
+__bpf_kfunc struct io_kiocb *uring_bpf_data_to_req(struct uring_bpf_data *data)

How would the returned struct io_kiocb * be used in an io_uring BPF program?

I plan to not expose io_kiocb to bpf prog in next version.

Quoted 23 lines
Quoted 23 lines
io_uring/bpf.c +22 -0
+{
+       return cmd_to_io_kiocb(data);
+}
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(uring_bpf_kfuncs)
+BTF_ID_FLAGS(func, uring_bpf_set_result)
+BTF_ID_FLAGS(func, uring_bpf_data_to_req)
+BTF_KFUNCS_END(uring_bpf_kfuncs)
+
+static const struct btf_kfunc_id_set uring_kfunc_set = {
+       .owner = THIS_MODULE,
+       .set   = &uring_bpf_kfuncs,
+};
+
+int __init io_bpf_init(void)
+{
+       int err;
+
+       err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &uring_kfunc_set);
+       if (err) {
+               pr_warn("error while setting UBLK BPF tracing kfuncs: %d", err);
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 305

Quoted 6 lines
io_uring/bpf.c +6 -0
+               return err;
+       }
+
+       err = register_bpf_struct_ops(&bpf_uring_bpf_ops, uring_bpf_ops);
+       if (err)
+               pr_warn("error while registering io_uring bpf struct ops: %d", err);

Is there a reason this error isn't fatal?

oops, the following line should be return err.

Quoted 12 lines
Quoted 12 lines
io_uring/bpf.c +3 -0
+
+       return 0;
+}
io_uring/io_uring.c +0 -0
index 38f03f6c28cb..d2517e09407a 100644
@@ -3851,6 +3851,7 @@ static int __init io_uring_init(void)
        register_sysctl_init("kernel", kernel_io_uring_disabled_table);
 #endif
 
Quoted 6 lines

io_uring/io_uring.c static int __init io_uring_init(void) · line 3854

Quoted 1 line
io_uring/io_uring.c +1 -0
+       io_bpf_init();

It doesn't look like there are any particular initialization ordering requirements with the rest of io_uring_init(). How about making a separate __initcall() in bpf.c so io_bpf_init() doesn't need to be visible outside that file?

Looks fine.

Quoted 17 lines
Quoted 9 lines
io_uring/io_uring.c +0 -0
        return 0;
 };
 __initcall(io_uring_init);
io_uring/io_uring.h +0 -0
index 4baf21a9e1ee..3f19bb079bcc 100644

io_uring/io_uring.h line 38

Quoted 6 lines
io_uring/io_uring.h +2 -1
@@ -34,7 +34,8 @@
                        IORING_FEAT_RECVSEND_BUNDLE |\
                        IORING_FEAT_MIN_TIMEOUT |\
                        IORING_FEAT_RW_ATTR |\
-                       IORING_FEAT_NO_IOWAIT)
+                       IORING_FEAT_NO_IOWAIT |\
+                       IORING_FEAT_BPF);

Unintentional semicolon?

Good catch!

Quoted 12 lines
Quoted 12 lines
io_uring/io_uring.h +0 -0
 #define IORING_SETUP_FLAGS (IORING_SETUP_IOPOLL |\
                        IORING_SETUP_SQPOLL |\
io_uring/uring_bpf.h +0 -0
index b6cda6df99b1..c76eba887d22 100644
@@ -2,6 +2,29 @@
 #ifndef IOU_BPF_H
 #define IOU_BPF_H
 
Quoted 5 lines

io_uring/uring_bpf.h line 6

Quoted 2 lines
io_uring/uring_bpf.h +2 -0
+struct uring_bpf_data {
+       /* readonly for bpf prog */

It doesn't look like uring_bpf_ops_btf_struct_access() actually allows these fields to be accessed?

pdu is writeable, the following two are readonly.

Quoted 16 lines

io_uring/uring_bpf.h line 18

Quoted 13 lines
io_uring/uring_bpf.h +12 -0
+       struct file     *file;
+       u32             opf;
+
+       /* writeable for bpf prog */
+       u8              pdu[64 - sizeof(struct file *) - sizeof(u32)];
+};
+
+typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
+                              const struct io_uring_sqe *sqe);
+typedef int (*uring_io_issue_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_fail_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_cleanup_t)(struct uring_bpf_data *data);

"uring_io" seems like a strange name for function typedefs specific to io_uring BPF. How about renaming these to "uring_bpf_..." instead?

Looks fine.

Thanks,
Ming

#23 lore raw

Re: [PATCH 3/5] io_uring: bpf: extend io_uring with bpf struct_ops

Quoted 161 lines · Ming Lei
Quoted 139 lines · Caleb Sander Mateos
Quoted 49 lines · Ming Lei

io_uring can be extended with bpf struct_ops in the following ways:

1) add new io_uring operation from application
- one typical use case is for operating device zero-copy buffer, which belongs to kernel, and not visible or too expensive to export to userspace, such as supporting copy data from this buffer to userspace, decompressing data to zero-copy buffer in Android case[1][2], or checksum/decrypting.

[1] https://lpc.events/event/18/contributions/1710/attachments/1440/3070/LPC2024_ublk_zero_copy.pdf

2) extend 64 byte SQE, since bpf map can be used to store IO data conveniently

3) communicate in IO chain, since bpf map can be shared among IOs, when one bpf IO is completed, data can be written to IO chain wide bpf map, then the following bpf IO can retrieve the data from this bpf map, this way is more flexible than io_uring built-in buffer

4) pretty handy to inject error for test purpose

bpf struct_ops is one very handy way to attach bpf prog with kernel, and this patch simply wires existed io_uring operation callbacks with added uring bpf struct_ops, so application can define its own uring bpf operations.

---
 include/uapi/linux/io_uring.h |   9 ++
 io_uring/bpf.c                | 271 +++++++++++++++++++++++++++++++++-
 io_uring/io_uring.c           |   1 +
 io_uring/io_uring.h           |   3 +-
 io_uring/uring_bpf.h          |  30 ++++
 5 files changed, 311 insertions(+), 3 deletions(-)
include/uapi/linux/io_uring.h +1 -0
index b8c49813b4e5..94d2050131ac 100644
@@ -74,6 +74,7 @@ struct io_uring_sqe {
                __u32           install_fd_flags;
                __u32           nop_flags;
                __u32           pipe_flags;
+               __u32           bpf_op_flags;
        };
        __u64   user_data;      /* data to be passed back at completion time */
        /* pack this to avoid bogus arm OABI complaints */

include/uapi/linux/io_uring.h enum io_uring_op { · line 436

Quoted 9 lines
include/uapi/linux/io_uring.h +6 -0
@@ -427,6 +428,13 @@ enum io_uring_op {
 #define IORING_RECVSEND_BUNDLE         (1U << 4)
 #define IORING_SEND_VECTORIZED         (1U << 5)
 
+/*
+ * sqe->bpf_op_flags           top 8bits is for storing bpf op
+ *                             The other 24bits are used for bpf prog
+ */
+#define IORING_BPF_OP_BITS     (8)
+#define IORING_BPF_OP_SHIFT    (24)

Could omit the parentheses here

Quoted 35 lines
include/uapi/linux/io_uring.h +2 -0
+
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
  * IORING_SEND_ZC_REPORT_USAGE was requested
@@ -631,6 +639,7 @@ struct io_uring_params {
 #define IORING_FEAT_MIN_TIMEOUT                (1U << 15)
 #define IORING_FEAT_RW_ATTR            (1U << 16)
 #define IORING_FEAT_NO_IOWAIT          (1U << 17)
+#define IORING_FEAT_BPF                        (1U << 18)
 
 /*
  * io_uring_register(2) opcodes and arguments
io_uring/bpf.c +10 -0
index bb1e37d1e804..8227be6d5a10 100644
@@ -4,28 +4,95 @@
 #include <linux/kernel.h>
 #include <linux/errno.h>
 #include <uapi/linux/io_uring.h>
+#include <linux/init.h>
+#include <linux/types.h>
+#include <linux/bpf_verifier.h>
+#include <linux/bpf.h>
+#include <linux/btf.h>
+#include <linux/btf_ids.h>
+#include <linux/filter.h>
 #include "io_uring.h"
 #include "uring_bpf.h"
 
+#define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
+
 static DEFINE_MUTEX(uring_bpf_ctx_lock);
 static LIST_HEAD(uring_bpf_ctx_list);
+DEFINE_STATIC_SRCU(uring_bpf_srcu);

io_uring/bpf.c line 29

Quoted 10 lines
io_uring/bpf.c +6 -2
+static struct uring_bpf_ops bpf_ops[MAX_BPF_OPS_COUNT];
 
-int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
 {
-       return -ECANCELED;
+       return (unsigned char)(op_flags >> IORING_BPF_OP_SHIFT);
+}
+
+static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)

u32?

io_uring/bpf.c line 42

Quoted 13 lines
io_uring/bpf.c +9 -0
+{
+       return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+}
+
+static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
+{
+       return &bpf_ops[uring_bpf_get_op(data->opf)];
 }
 
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);

u32?

Quoted 13 lines
io_uring/bpf.c +10 -0
+       struct uring_bpf_ops *ops;
+
+       if (!(req->ctx->flags & IORING_SETUP_BPF))
+               return -EACCES;
+
+       data->opf = op_flags;
+       ops = &bpf_ops[uring_bpf_get_op(data->opf)];
+
+       if (ops->prep_fn)
+               return ops->prep_fn(data, sqe);
        return -EOPNOTSUPP;
 }
 
Quoted 10 lines

io_uring/bpf.c line 62

Quoted 7 lines
io_uring/bpf.c +7 -0
+static int __io_uring_bpf_issue(struct io_kiocb *req)
+{
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->issue_fn)
+               return ops->issue_fn(data);

Doesn't this need to use rcu_dereference() to access ops->issue_fn since io_bpf_reg_unreg() may concurrently modify it?

rcu isn't enough, io_bpf_reg_unreg() shouldn't be started unless one bpf IO is completed. Probably percpu-refcount may have to be used.

Quoted 5 lines

Also, it doesn't look safe to propagate the BPF ->issue_fn() return value to the ->issue() return value. If the BPF program returns IOU_ISSUE_SKIP_COMPLETE = -EIOCBQUEUED, the io_uring request will never be completed. And it looks like ->issue() implementations are

Quoted 4 lines

It depends on if bpf OP can support async IO, and it relies on bpf kfunc's capability actually.

But yes, it is better to start with sync bpf IO only.

Yeah, the existing kfunc interface doesn't provide any way to complete the request asynchronously. I'm having a hard time imagining how the asynchronous completion would even be triggered. I guess the struct uring_bpf_data * would need to be stored somewhere and some other BPF program that runs later would call the kfunc to complete it? Seems like it would be difficult to prevent double completions. I agree supporting only synchronous BPF programs for now makes sense.

Quoted 73 lines
Quoted 60 lines

meant to return either IOU_COMPLETE, IOU_RETRY, or IOU_ISSUE_SKIP_COMPLETE. If the BPF program returns some other value, it would be nice to propagate it to the io_uring CQE result and return IOU_COMPLETE, similar to io_uring_cmd().

Quoted 40 lines
io_uring/bpf.c +27 -0
+       return -ECANCELED;
+}
+
+int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
+{
+       if (issue_flags & IO_URING_F_UNLOCKED) {
+               int idx, ret;
+
+               idx = srcu_read_lock(&uring_bpf_srcu);
+               ret = __io_uring_bpf_issue(req);
+               srcu_read_unlock(&uring_bpf_srcu, idx);
+
+               return ret;
+       }
+       return __io_uring_bpf_issue(req);
+}
+
 void io_uring_bpf_fail(struct io_kiocb *req)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->fail_fn)
+               ops->fail_fn(data);
 }
 
 void io_uring_bpf_cleanup(struct io_kiocb *req)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+       struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
+
+       if (ops->cleanup_fn)
+               ops->cleanup_fn(data);
 }
 
 void uring_bpf_add_ctx(struct io_ring_ctx *ctx)
@@ -39,3 +106,203 @@ void uring_bpf_del_ctx(struct io_ring_ctx *ctx)
        guard(mutex)(&uring_bpf_ctx_lock);
        list_del(&ctx->bpf_node);
 }

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 118

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+static const struct btf_type *uring_bpf_data_type;
+
+static bool uring_bpf_ops_is_valid_access(int off, int size,
+                                      enum bpf_access_type type,
+                                      const struct bpf_prog *prog,
+                                      struct bpf_insn_access_aux *info)
+{
+       return bpf_tracing_btf_ctx_access(off, size, type, prog, info);
+}

Just use bpf_tracing_btf_ctx_access instead of defining another equivalent function?

Quoted 12 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 128

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+static int uring_bpf_ops_btf_struct_access(struct bpf_verifier_log *log,
+                                       const struct bpf_reg_state *reg,
+                                       int off, int size)
+{
+       const struct btf_type *t;
+
+       t = btf_type_by_id(reg->btf, reg->btf_id);
+       if (t != uring_bpf_data_type) {
+               bpf_log(log, "only read is supported\n");

What does this log line mean?

Quoted 1 line

We only support to write to uring_bpf_data.pdu, and readonly is for other type.

I missed that ->btf_struct_access() is only called for write accesses. Makes sense.

Quoted 129 lines
Quoted 9 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 133

Quoted 7 lines
io_uring/bpf.c +5 -0
+               return -EACCES;
+       }
+
+       if (off < offsetof(struct uring_bpf_data, pdu) ||
+                       off + size >= sizeof(struct uring_bpf_data))

Should be > instead of >=? Otherwise the last byte of pdu isn't usable.

Good catch!

Quoted 24 lines
Quoted 24 lines
io_uring/bpf.c +23 -0
+               return -EACCES;
+
+       return NOT_INIT;
+}
+
+static const struct bpf_verifier_ops io_bpf_verifier_ops = {
+       .get_func_proto = bpf_base_func_proto,
+       .is_valid_access = uring_bpf_ops_is_valid_access,
+       .btf_struct_access = uring_bpf_ops_btf_struct_access,
+};
+
+static int uring_bpf_ops_init(struct btf *btf)
+{
+       s32 type_id;
+
+       type_id = btf_find_by_name_kind(btf, "uring_bpf_data", BTF_KIND_STRUCT);
+       if (type_id < 0)
+               return -EINVAL;
+       uring_bpf_data_type = btf_type_by_id(btf, type_id);
+       return 0;
+}
+
+static int uring_bpf_ops_check_member(const struct btf_type *t,
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 161

Quoted 5 lines
io_uring/bpf.c +5 -0
+                                  const struct btf_member *member,
+                                  const struct bpf_prog *prog)
+{
+       return 0;
+}

It looks like struct bpf_struct_ops's .check_member can be omitted if it always succeeds

I think it is better to check each member of the struct_ops, also .sleepable need to be checked here.

Quoted 15 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 172

Quoted 12 lines
io_uring/bpf.c +11 -0
+
+static int uring_bpf_ops_init_member(const struct btf_type *t,
+                                const struct btf_member *member,
+                                void *kdata, const void *udata)
+{
+       const struct uring_bpf_ops *uuring_bpf_ops;
+       struct uring_bpf_ops *kuring_bpf_ops;
+       u32 moff;
+
+       uuring_bpf_ops = (const struct uring_bpf_ops *)udata;
+       kuring_bpf_ops = (struct uring_bpf_ops *)kdata;

Don't need to explicitly cast from (const) void *. That could allow these initializers to be combined with the variable declarations.

OK.

Quoted 9 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 178

Quoted 7 lines
io_uring/bpf.c +6 -0
+
+       moff = __btf_member_bit_offset(t, member) / 8;
+
+       switch (moff) {
+       case offsetof(struct uring_bpf_ops, id):
+               /* For dev_id, this function has to copy it and return 1 to

What does "dev_id" refer to?

Looks a typo.

Quoted 39 lines
Quoted 9 lines
io_uring/bpf.c +8 -0
+                * indicate that the data has been handled by the struct_ops
+                * type, or the verifier will reject the map if the value of
+                * those fields is not zero.
+                */
+               kuring_bpf_ops->id = uuring_bpf_ops->id;
+               return 1;
+       }
+       return 0;

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 196

Quoted 10 lines
io_uring/bpf.c +10 -0
+}
+
+static int io_bpf_reg_unreg(struct uring_bpf_ops *ops, bool reg)
+{
+       struct io_ring_ctx *ctx;
+       int ret = 0;
+
+       guard(mutex)(&uring_bpf_ctx_lock);
+       list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+               mutex_lock(&ctx->uring_lock);

Locking multiple io_ring_ctx's uring_locks is deadlock prone. See lock_two_rings() for example, which takes care to acquire multiple uring_locks in a consistent order. Would it be possible to lock one io_ring_ctx at a time and set some flag to indicate that srcu_read_lock() needs to be used?

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 204

Quoted 8 lines
io_uring/bpf.c +8 -0
+
+       if (reg) {
+               if (bpf_ops[ops->id].issue_fn)
+                       ret = -EBUSY;
+               else
+                       bpf_ops[ops->id] = *ops;
+       } else {
+               bpf_ops[ops->id] = (struct uring_bpf_ops) {0};

Don't these need to use rcu_assign_pointer() to assign bpf_ops[ops->id].issue_fn since __io_uring_bpf_issue() may read it concurrently?

Quoted 11 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 210

Quoted 6 lines
io_uring/bpf.c +6 -0
+       }
+
+       synchronize_srcu(&uring_bpf_srcu);
+
+       list_for_each_entry(ctx, &uring_bpf_ctx_list, bpf_node)
+               mutex_unlock(&ctx->uring_lock);

It might be preferable to call synchronize_srcu() after releasing the uring_locks (and maybe uring_bpf_ctx_lock). That would minimize the latency injected into io_uring requests in case synchronize_srcu() blocks for a long time.

Quoted 2 lines

I plan to switch to percpu-refcount, which is simple, and can avoid unreg when one such bpf io is inflight.

Okay, I still think it would be simpler to specify the io_uring BPF program by bpf_fd. Then we could use the existing file descriptor/io_uring registered file lookup mechanisms instead of having to write a new one.

Quoted 59 lines
Quoted 26 lines
Quoted 13 lines
io_uring/bpf.c +11 -0
+
+       return ret;
+}
+
+static int io_bpf_reg(void *kdata, struct bpf_link *link)
+{
+       struct uring_bpf_ops *ops = kdata;
+
+       return io_bpf_reg_unreg(ops, true);
+}
+

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 231

Quoted 10 lines
io_uring/bpf.c +10 -0
+static void io_bpf_unreg(void *kdata, struct bpf_link *link)
+{
+       struct uring_bpf_ops *ops = kdata;
+
+       io_bpf_reg_unreg(ops, false);
+}
+
+static int io_bpf_prep_io(struct uring_bpf_data *data, const struct io_uring_sqe *sqe)
+{
+       return -EOPNOTSUPP;

The return value for the stub functions doesn't matter, return 0 for simplicity?

Quoted 3 lines

Also, could the stub functions be renamed to more clearly indicate that they are only used for their signature and shouldn't be called directly?

Looks fine.

Quoted 18 lines
Quoted 18 lines
io_uring/bpf.c +17 -0
+}
+
+static int io_bpf_issue_io(struct uring_bpf_data *data)
+{
+       return -ECANCELED;
+}
+
+static void io_bpf_fail_io(struct uring_bpf_data *data)
+{
+}
+
+static void io_bpf_cleanup_io(struct uring_bpf_data *data)
+{
+}
+
+static struct uring_bpf_ops __bpf_uring_bpf_ops = {
+       .prep_fn        = io_bpf_prep_io,
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 254

Quoted 6 lines
io_uring/bpf.c +6 -0
+       .issue_fn       = io_bpf_issue_io,
+       .fail_fn        = io_bpf_fail_io,
+       .cleanup_fn     = io_bpf_cleanup_io,
+};
+
+static struct bpf_struct_ops bpf_uring_bpf_ops = {

const?

Quoted 1 line

int __register_bpf_struct_ops(struct bpf_struct_ops *st_ops)

Good point. I was confused why a mutable pointer is needed, but looks like the BPF code internally assigns to the func_models field.

Quoted 117 lines
Quoted 19 lines
Quoted 19 lines
io_uring/bpf.c +17 -0
+       .verifier_ops = &io_bpf_verifier_ops,
+       .init = uring_bpf_ops_init,
+       .check_member = uring_bpf_ops_check_member,
+       .init_member = uring_bpf_ops_init_member,
+       .reg = io_bpf_reg,
+       .unreg = io_bpf_unreg,
+       .name = "uring_bpf_ops",
+       . = &__bpf_uring_bpf_ops,
+       .owner = THIS_MODULE,
+};
+
+__bpf_kfunc_start_defs();
+__bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)
+{
+       struct io_kiocb *req = cmd_to_io_kiocb(data);
+
+       if (res < 0)
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 277

Quoted 6 lines
io_uring/bpf.c +6 -0
+               req_set_fail(req);
+       io_req_set_res(req, res, 0);
+}
+
+/* io_kiocb layout might be changed */
+__bpf_kfunc struct io_kiocb *uring_bpf_data_to_req(struct uring_bpf_data *data)

How would the returned struct io_kiocb * be used in an io_uring BPF program?

I plan to not expose io_kiocb to bpf prog in next version.

Quoted 23 lines
Quoted 23 lines
io_uring/bpf.c +22 -0
+{
+       return cmd_to_io_kiocb(data);
+}
+__bpf_kfunc_end_defs();
+
+BTF_KFUNCS_START(uring_bpf_kfuncs)
+BTF_ID_FLAGS(func, uring_bpf_set_result)
+BTF_ID_FLAGS(func, uring_bpf_data_to_req)
+BTF_KFUNCS_END(uring_bpf_kfuncs)
+
+static const struct btf_kfunc_id_set uring_kfunc_set = {
+       .owner = THIS_MODULE,
+       .set   = &uring_bpf_kfuncs,
+};
+
+int __init io_bpf_init(void)
+{
+       int err;
+
+       err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &uring_kfunc_set);
+       if (err) {
+               pr_warn("error while setting UBLK BPF tracing kfuncs: %d", err);
Quoted 8 lines

io_uring/bpf.c void uring_bpf_del_ctx(struct io_ring_ctx *ctx) · line 305

Quoted 6 lines
io_uring/bpf.c +6 -0
+               return err;
+       }
+
+       err = register_bpf_struct_ops(&bpf_uring_bpf_ops, uring_bpf_ops);
+       if (err)
+               pr_warn("error while registering io_uring bpf struct ops: %d", err);

Is there a reason this error isn't fatal?

oops, the following line should be return err.

Quoted 12 lines
Quoted 12 lines
io_uring/bpf.c +3 -0
+
+       return 0;
+}
io_uring/io_uring.c +0 -0
index 38f03f6c28cb..d2517e09407a 100644
@@ -3851,6 +3851,7 @@ static int __init io_uring_init(void)
        register_sysctl_init("kernel", kernel_io_uring_disabled_table);
 #endif
 
Quoted 6 lines

io_uring/io_uring.c static int __init io_uring_init(void) · line 3854

Quoted 1 line
io_uring/io_uring.c +1 -0
+       io_bpf_init();

It doesn't look like there are any particular initialization ordering requirements with the rest of io_uring_init(). How about making a separate __initcall() in bpf.c so io_bpf_init() doesn't need to be visible outside that file?

Looks fine.

Quoted 17 lines
Quoted 9 lines
io_uring/io_uring.c +0 -0
        return 0;
 };
 __initcall(io_uring_init);
io_uring/io_uring.h +0 -0
index 4baf21a9e1ee..3f19bb079bcc 100644

io_uring/io_uring.h line 38

Quoted 6 lines
io_uring/io_uring.h +2 -1
@@ -34,7 +34,8 @@
                        IORING_FEAT_RECVSEND_BUNDLE |\
                        IORING_FEAT_MIN_TIMEOUT |\
                        IORING_FEAT_RW_ATTR |\
-                       IORING_FEAT_NO_IOWAIT)
+                       IORING_FEAT_NO_IOWAIT |\
+                       IORING_FEAT_BPF);

Unintentional semicolon?

Good catch!

Quoted 12 lines
Quoted 12 lines
io_uring/io_uring.h +0 -0
 #define IORING_SETUP_FLAGS (IORING_SETUP_IOPOLL |\
                        IORING_SETUP_SQPOLL |\
io_uring/uring_bpf.h +0 -0
index b6cda6df99b1..c76eba887d22 100644
@@ -2,6 +2,29 @@
 #ifndef IOU_BPF_H
 #define IOU_BPF_H
 
Quoted 7 lines
Quoted 5 lines

io_uring/uring_bpf.h line 6

Quoted 2 lines
io_uring/uring_bpf.h +2 -0
+struct uring_bpf_data {
+       /* readonly for bpf prog */

It doesn't look like uring_bpf_ops_btf_struct_access() actually allows these fields to be accessed?

pdu is writeable, the following two are readonly.

I'm new to BPF, what allows them to be read? Are BPF programs implicitly allowed to read within the bounds of any struct?

Best,
Caleb

Quoted 24 lines
Quoted 17 lines
Quoted 4 lines
io_uring/uring_bpf.h +2 -0
+       struct file     *file;
+       u32             opf;

io_uring/uring_bpf.h line 18

Quoted 10 lines
io_uring/uring_bpf.h +10 -0
+
+       /* writeable for bpf prog */
+       u8              pdu[64 - sizeof(struct file *) - sizeof(u32)];
+};
+
+typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
+                              const struct io_uring_sqe *sqe);
+typedef int (*uring_io_issue_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_fail_t)(struct uring_bpf_data *data);
+typedef void (*uring_io_cleanup_t)(struct uring_bpf_data *data);

"uring_io" seems like a strange name for function typedefs specific to io_uring BPF. How about renaming these to "uring_bpf_..." instead?

Looks fine.

Thanks,
Ming

author
#24 lore raw

PATCH 4/5 io_uring: bpf: add buffer support for IORING_OP_BPF

In reply to Ming Lei

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr, len=sqe->len)

Buffer 2 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr3, len=sqe->optlen)

The sqe->bpf_op_flags layout (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (3 bits)
Bits 20-18: Buffer 2 type (3 bits)
Bits 17-0: Custom BPF flags (18 bits)

Using 3-bit encoding for each buffer type allows for future expansion to 8 types (0-7). Currently types 0-2 are defined (none/plain/fixed) and 3-7 are reserved for future use.

Buffer 2 currently only supports none/plain types because the io_uring framework can only handle one fixed buffer per request (via req->buf_index). The 3-bit encoding provides room for future enhancements.

Buffer metadata (addresses, lengths) is stored in the extended uring_bpf_data structure and is accessible to BPF programs with readonly permissions. Buffer types can be extracted from the opf field using IORING_BPF_BUF1_TYPE() and IORING_BPF_BUF2_TYPE() macros.

Valid buffer combinations:
- 0 buffers
- 1 plain buffer
- 1 fixed buffer
- 2 plain buffers
- 1 fixed buffer + 1 plain buffer

---
 include/uapi/linux/io_uring.h | 45 +++++++++++++++++++++++--
 io_uring/bpf.c                | 63 ++++++++++++++++++++++++++++++++++-
 io_uring/uring_bpf.h          | 12 ++++++-
 3 files changed, 116 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +43 -2
index 94d2050131ac..950f4cfbbf86 100644
@@ -429,12 +429,53 @@ enum io_uring_op {
 #define IORING_SEND_VECTORIZED		(1U << 5)
 
 /*
- * sqe->bpf_op_flags		top 8bits is for storing bpf op
- *				The other 24bits are used for bpf prog
+ * sqe->bpf_op_flags layout (32 bits total):
+ *   Bits 31-24: BPF operation ID (8 bits, 256 possible operations)
+ *   Bits 23-21: Buffer 1 type (3 bits: none/plain/fixed/reserved)
+ *   Bits 20-18: Buffer 2 type (3 bits: none/plain/reserved)
+ *   Bits 17-0:  Custom BPF flags (18 bits, available for BPF programs)
+ *
+ * For IORING_OP_BPF, buffers are specified as follows:
+ *   Buffer 1 (plain):  addr=sqe->addr, len=sqe->len
+ *   Buffer 1 (fixed):  index=sqe->buf_index, offset=sqe->addr, len=sqe->len
+ *   Buffer 2 (plain):  addr=sqe->addr3, len=sqe->optlen
+ *
+ * Note: Buffer 1 can be none/plain/fixed. Buffer 2 can only be none/plain.
+ *       3-bit encoding for each buffer allows for future expansion to 8 types (0-7).
+ *       Currently only one fixed buffer per request is supported (buffer 1).
+ *       Valid combinations: 0 buffers, 1 plain, 1 fixed, 2 plain, 1 fixed + 1 plain.
  */
 #define IORING_BPF_OP_BITS	(8)
 #define IORING_BPF_OP_SHIFT	(24)
 
+/* Buffer type encoding in sqe->bpf_op_flags */
+#define IORING_BPF_BUF1_TYPE_SHIFT	(21)
+#define IORING_BPF_BUF2_TYPE_SHIFT	(18)
+#define IORING_BPF_BUF_TYPE_NONE	(0)	/* No buffer */
+#define IORING_BPF_BUF_TYPE_PLAIN	(1)	/* Plain user buffer */
+#define IORING_BPF_BUF_TYPE_FIXED	(2)	/* Fixed/registered buffer */
+#define IORING_BPF_BUF_TYPE_MASK	(7)	/* 3-bit mask */
+
+/* Helper macros to encode/decode buffer types */
+#define IORING_BPF_BUF1_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF1_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_BUF2_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF2_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_SET_BUF1_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF1_TYPE_SHIFT)
+#define IORING_BPF_SET_BUF2_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF2_TYPE_SHIFT)
+
+/* Custom BPF flags mask (18 bits available, bits 17-0) */
+#define IORING_BPF_CUSTOM_FLAGS_MASK	((1U << 18) - 1)
+
+/* Encode all components into sqe->bpf_op_flags */
+#define IORING_BPF_OP_FLAGS(op, buf1_type, buf2_type, flags) \
+	(((op) << IORING_BPF_OP_SHIFT) | \
+	 IORING_BPF_SET_BUF1_TYPE(buf1_type) | \
+	 IORING_BPF_SET_BUF2_TYPE(buf2_type) | \
+	 ((flags) & IORING_BPF_CUSTOM_FLAGS_MASK))
+
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
  * IORING_SEND_ZC_REPORT_USAGE was requested
io_uring/bpf.c +62 -1

95 lines not on this page · raw message

io_uring/uring_bpf.h +11 -1
index c76eba887d22..c919931cb4b0 100644
@@ -7,8 +7,18 @@ struct uring_bpf_data {
 	struct file     *file;
 	u32		opf;
 
+	/* Buffer 1 metadata - readable for bpf prog */
+	u32		buf1_len;		/* buffer 1 length, bytes 12-15 */
+	u64		buf1_addr;		/* buffer 1 address or offset, bytes 16-23 */
+
+	/* Buffer 2 metadata - readable for bpf prog (plain only) */
+	u64		buf2_addr;		/* buffer 2 address, bytes 24-31 */
+	u32		buf2_len;		/* buffer 2 length, bytes 32-35 */
+	u32		__pad;			/* padding, bytes 36-39 */
+
 	/* writeable for bpf prog */
-	u8              pdu[64 - sizeof(struct file *) - sizeof(u32)];
+	u8              pdu[64 - sizeof(struct file *) - 4 * sizeof(u32) -
+		2 * sizeof(u64)];
 };
 
 typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
Signature · 1 line
2.47.0
#25 lore raw

Re: [PATCH 4/5] io_uring: bpf: add buffer support for IORING_OP_BPF

Am 04.11.25 um 17:21 schrieb Ming Lei:

Quoted 217 lines

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr, len=sqe->len)

Buffer 2 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr3, len=sqe->optlen)

The sqe->bpf_op_flags layout (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (3 bits)
Bits 20-18: Buffer 2 type (3 bits)
Bits 17-0: Custom BPF flags (18 bits)

Using 3-bit encoding for each buffer type allows for future expansion to 8 types (0-7). Currently types 0-2 are defined (none/plain/fixed) and 3-7 are reserved for future use.

Buffer 2 currently only supports none/plain types because the io_uring framework can only handle one fixed buffer per request (via req->buf_index). The 3-bit encoding provides room for future enhancements.

Buffer metadata (addresses, lengths) is stored in the extended uring_bpf_data structure and is accessible to BPF programs with readonly permissions. Buffer types can be extracted from the opf field using IORING_BPF_BUF1_TYPE() and IORING_BPF_BUF2_TYPE() macros.

Valid buffer combinations:
- 0 buffers
- 1 plain buffer
- 1 fixed buffer
- 2 plain buffers
- 1 fixed buffer + 1 plain buffer

---
  include/uapi/linux/io_uring.h | 45 +++++++++++++++++++++++--
  io_uring/bpf.c                | 63 ++++++++++++++++++++++++++++++++++-
  io_uring/uring_bpf.h          | 12 ++++++-
  3 files changed, 116 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +43 -2
index 94d2050131ac..950f4cfbbf86 100644
@@ -429,12 +429,53 @@ enum io_uring_op {
  #define IORING_SEND_VECTORIZED		(1U << 5)
 
  /*
- * sqe->bpf_op_flags		top 8bits is for storing bpf op
- *				The other 24bits are used for bpf prog
+ * sqe->bpf_op_flags layout (32 bits total):
+ *   Bits 31-24: BPF operation ID (8 bits, 256 possible operations)
+ *   Bits 23-21: Buffer 1 type (3 bits: none/plain/fixed/reserved)
+ *   Bits 20-18: Buffer 2 type (3 bits: none/plain/reserved)
+ *   Bits 17-0:  Custom BPF flags (18 bits, available for BPF programs)
+ *
+ * For IORING_OP_BPF, buffers are specified as follows:
+ *   Buffer 1 (plain):  addr=sqe->addr, len=sqe->len
+ *   Buffer 1 (fixed):  index=sqe->buf_index, offset=sqe->addr, len=sqe->len
+ *   Buffer 2 (plain):  addr=sqe->addr3, len=sqe->optlen
+ *
+ * Note: Buffer 1 can be none/plain/fixed. Buffer 2 can only be none/plain.
+ *       3-bit encoding for each buffer allows for future expansion to 8 types (0-7).
+ *       Currently only one fixed buffer per request is supported (buffer 1).
+ *       Valid combinations: 0 buffers, 1 plain, 1 fixed, 2 plain, 1 fixed + 1 plain.
   */
  #define IORING_BPF_OP_BITS	(8)
  #define IORING_BPF_OP_SHIFT	(24)
 
+/* Buffer type encoding in sqe->bpf_op_flags */
+#define IORING_BPF_BUF1_TYPE_SHIFT	(21)
+#define IORING_BPF_BUF2_TYPE_SHIFT	(18)
+#define IORING_BPF_BUF_TYPE_NONE	(0)	/* No buffer */
+#define IORING_BPF_BUF_TYPE_PLAIN	(1)	/* Plain user buffer */
+#define IORING_BPF_BUF_TYPE_FIXED	(2)	/* Fixed/registered buffer */
+#define IORING_BPF_BUF_TYPE_MASK	(7)	/* 3-bit mask */
+
+/* Helper macros to encode/decode buffer types */
+#define IORING_BPF_BUF1_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF1_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_BUF2_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF2_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_SET_BUF1_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF1_TYPE_SHIFT)
+#define IORING_BPF_SET_BUF2_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF2_TYPE_SHIFT)
+
+/* Custom BPF flags mask (18 bits available, bits 17-0) */
+#define IORING_BPF_CUSTOM_FLAGS_MASK	((1U << 18) - 1)
+
+/* Encode all components into sqe->bpf_op_flags */
+#define IORING_BPF_OP_FLAGS(op, buf1_type, buf2_type, flags) \
+	(((op) << IORING_BPF_OP_SHIFT) | \
+	 IORING_BPF_SET_BUF1_TYPE(buf1_type) | \
+	 IORING_BPF_SET_BUF2_TYPE(buf2_type) | \
+	 ((flags) & IORING_BPF_CUSTOM_FLAGS_MASK))
+
  /*
   * cqe.res for IORING_CQE_F_NOTIF if
   * IORING_SEND_ZC_REPORT_USAGE was requested
io_uring/bpf.c +62 -1
index 8227be6d5a10..e837c3d57b96 100644
@@ -11,8 +11,10 @@
  #include <linux/btf.h>
  #include <linux/btf_ids.h>
  #include <linux/filter.h>
+#include <linux/uio.h>
  #include "io_uring.h"
  #include "uring_bpf.h"
+#include "rsrc.h"
 
  #define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
 
@@ -28,7 +30,7 @@ static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
 
  static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)
  {
-	return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+	return op_flags & IORING_BPF_CUSTOM_FLAGS_MASK;
  }
 
  static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
@@ -36,18 +38,77 @@ static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *dat
  	return &bpf_ops[uring_bpf_get_op(data->opf)];
  }
 
+static int io_bpf_prep_buffers(struct io_kiocb *req,
+			       const struct io_uring_sqe *sqe,
+			       struct uring_bpf_data *data,
+			       unsigned int op_flags)
+{
+	u8 buf1_type, buf2_type;
+
+	/* Extract buffer configuration from bpf_op_flags */
+	buf1_type = IORING_BPF_BUF1_TYPE(op_flags);
+	buf2_type = IORING_BPF_BUF2_TYPE(op_flags);
+
+	/* Prepare buffer 1 */
+	if (buf1_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer: addr=sqe->addr, len=sqe->len */
+		data->buf1_addr = READ_ONCE(sqe->addr);
+		data->buf1_len = READ_ONCE(sqe->len);
+	} else if (buf1_type == IORING_BPF_BUF_TYPE_FIXED) {
+		/* Fixed buffer: index=sqe->buf_index, offset=sqe->addr, len=sqe->len */
+		req->buf_index = READ_ONCE(sqe->buf_index);
+		data->buf1_addr = READ_ONCE(sqe->addr);  /* offset within fixed buffer */
+		data->buf1_len = READ_ONCE(sqe->len);
+
+		/* Validate buffer index */
+		if (unlikely(!req->ctx->buf_table.nr))
+			return -EFAULT;
+		if (unlikely(req->buf_index >= req->ctx->buf_table.nr))
+			return -EINVAL;
+	} else if (buf1_type == IORING_BPF_BUF_TYPE_NONE) {
+		data->buf1_addr = 0;
+		data->buf1_len = 0;
+	} else {
+		return -EINVAL;
+	}
+
+	/* Prepare buffer 2 (plain only - io_uring only supports one fixed buffer) */
+	if (buf2_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer: addr=sqe->addr3, len=sqe->optlen */
+		data->buf2_addr = READ_ONCE(sqe->addr3);
+		data->buf2_len = READ_ONCE(sqe->optlen);
+	} else if (buf2_type == IORING_BPF_BUF_TYPE_NONE) {
+		data->buf2_addr = 0;
+		data->buf2_len = 0;
+	} else {
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
  int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
  {
  	struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
  	unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);
  	struct uring_bpf_ops *ops;
+	int ret;
 
  	if (!(req->ctx->flags & IORING_SETUP_BPF))
  		return -EACCES;
 
+	if (uring_bpf_get_flags(op_flags))
+		return -EINVAL;
+
  	data->opf = op_flags;
  	ops = &bpf_ops[uring_bpf_get_op(data->opf)];
 
+	/* Prepare buffers based on buffer type flags */
+	ret = io_bpf_prep_buffers(req, sqe, data, op_flags);
+	if (ret)
+		return ret;
+
  	if (ops->prep_fn)
  		return ops->prep_fn(data, sqe);
  	return -EOPNOTSUPP;
io_uring/uring_bpf.h +0 -0
index c76eba887d22..c919931cb4b0 100644
@@ -7,8 +7,18 @@ struct uring_bpf_data {
  	struct file     *file;

io_uring/uring_bpf.h struct uring_bpf_data { · line 17

Quoted 10 lines
io_uring/uring_bpf.h +8 -0
  	u32		opf;
 
+	/* Buffer 1 metadata - readable for bpf prog */
+	u32		buf1_len;		/* buffer 1 length, bytes 12-15 */
+	u64		buf1_addr;		/* buffer 1 address or offset, bytes 16-23 */
+
+	/* Buffer 2 metadata - readable for bpf prog (plain only) */
+	u64		buf2_addr;		/* buffer 2 address, bytes 24-31 */
+	u32		buf2_len;		/* buffer 2 length, bytes 32-35 */
+	u32		__pad;			/* padding, bytes 36-39 */

I'm wondering if this should be in the generic struct uring_bpf_data at all. For io_uring_cmd we have helper functions like io_uring_cmd_import_fixed[_vec] which can be used by the implementation, but the layout on top of io_uring_cmd->pdu is up to the specific operation.

Thinking about it, I'm wondering if the bpf operations could be implemented on top of io_uring_cmd.

metze

author
#26 lore raw

Re: [PATCH 4/5] io_uring: bpf: add buffer support for IORING_OP_BPF

Quoted 215 lines · Stefan Metzmacher

Am 04.11.25 um 17:21 schrieb Ming Lei:

Quoted 214 lines

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr, len=sqe->len)

Buffer 2 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr3, len=sqe->optlen)

The sqe->bpf_op_flags layout (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (3 bits)
Bits 20-18: Buffer 2 type (3 bits)
Bits 17-0: Custom BPF flags (18 bits)

Using 3-bit encoding for each buffer type allows for future expansion to 8 types (0-7). Currently types 0-2 are defined (none/plain/fixed) and 3-7 are reserved for future use.

Buffer 2 currently only supports none/plain types because the io_uring framework can only handle one fixed buffer per request (via req->buf_index). The 3-bit encoding provides room for future enhancements.

Buffer metadata (addresses, lengths) is stored in the extended uring_bpf_data structure and is accessible to BPF programs with readonly permissions. Buffer types can be extracted from the opf field using IORING_BPF_BUF1_TYPE() and IORING_BPF_BUF2_TYPE() macros.

Valid buffer combinations:
- 0 buffers
- 1 plain buffer
- 1 fixed buffer
- 2 plain buffers
- 1 fixed buffer + 1 plain buffer

---
  include/uapi/linux/io_uring.h | 45 +++++++++++++++++++++++--
  io_uring/bpf.c                | 63 ++++++++++++++++++++++++++++++++++-
  io_uring/uring_bpf.h          | 12 ++++++-
  3 files changed, 116 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +43 -2
index 94d2050131ac..950f4cfbbf86 100644
@@ -429,12 +429,53 @@ enum io_uring_op {
  #define IORING_SEND_VECTORIZED		(1U << 5)
  /*
- * sqe->bpf_op_flags		top 8bits is for storing bpf op
- *				The other 24bits are used for bpf prog
+ * sqe->bpf_op_flags layout (32 bits total):
+ *   Bits 31-24: BPF operation ID (8 bits, 256 possible operations)
+ *   Bits 23-21: Buffer 1 type (3 bits: none/plain/fixed/reserved)
+ *   Bits 20-18: Buffer 2 type (3 bits: none/plain/reserved)
+ *   Bits 17-0:  Custom BPF flags (18 bits, available for BPF programs)
+ *
+ * For IORING_OP_BPF, buffers are specified as follows:
+ *   Buffer 1 (plain):  addr=sqe->addr, len=sqe->len
+ *   Buffer 1 (fixed):  index=sqe->buf_index, offset=sqe->addr, len=sqe->len
+ *   Buffer 2 (plain):  addr=sqe->addr3, len=sqe->optlen
+ *
+ * Note: Buffer 1 can be none/plain/fixed. Buffer 2 can only be none/plain.
+ *       3-bit encoding for each buffer allows for future expansion to 8 types (0-7).
+ *       Currently only one fixed buffer per request is supported (buffer 1).
+ *       Valid combinations: 0 buffers, 1 plain, 1 fixed, 2 plain, 1 fixed + 1 plain.
   */
  #define IORING_BPF_OP_BITS	(8)
  #define IORING_BPF_OP_SHIFT	(24)
+/* Buffer type encoding in sqe->bpf_op_flags */
+#define IORING_BPF_BUF1_TYPE_SHIFT	(21)
+#define IORING_BPF_BUF2_TYPE_SHIFT	(18)
+#define IORING_BPF_BUF_TYPE_NONE	(0)	/* No buffer */
+#define IORING_BPF_BUF_TYPE_PLAIN	(1)	/* Plain user buffer */
+#define IORING_BPF_BUF_TYPE_FIXED	(2)	/* Fixed/registered buffer */
+#define IORING_BPF_BUF_TYPE_MASK	(7)	/* 3-bit mask */
+
+/* Helper macros to encode/decode buffer types */
+#define IORING_BPF_BUF1_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF1_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_BUF2_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF2_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_SET_BUF1_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF1_TYPE_SHIFT)
+#define IORING_BPF_SET_BUF2_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF2_TYPE_SHIFT)
+
+/* Custom BPF flags mask (18 bits available, bits 17-0) */
+#define IORING_BPF_CUSTOM_FLAGS_MASK	((1U << 18) - 1)
+
+/* Encode all components into sqe->bpf_op_flags */
+#define IORING_BPF_OP_FLAGS(op, buf1_type, buf2_type, flags) \
+	(((op) << IORING_BPF_OP_SHIFT) | \
+	 IORING_BPF_SET_BUF1_TYPE(buf1_type) | \
+	 IORING_BPF_SET_BUF2_TYPE(buf2_type) | \
+	 ((flags) & IORING_BPF_CUSTOM_FLAGS_MASK))
+
  /*
   * cqe.res for IORING_CQE_F_NOTIF if
   * IORING_SEND_ZC_REPORT_USAGE was requested
io_uring/bpf.c +62 -1
index 8227be6d5a10..e837c3d57b96 100644
@@ -11,8 +11,10 @@
  #include <linux/btf.h>
  #include <linux/btf_ids.h>
  #include <linux/filter.h>
+#include <linux/uio.h>
  #include "io_uring.h"
  #include "uring_bpf.h"
+#include "rsrc.h"
  #define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
@@ -28,7 +30,7 @@ static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
  static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)
  {
-	return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+	return op_flags & IORING_BPF_CUSTOM_FLAGS_MASK;
  }
  static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
@@ -36,18 +38,77 @@ static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *dat
  	return &bpf_ops[uring_bpf_get_op(data->opf)];
  }
+static int io_bpf_prep_buffers(struct io_kiocb *req,
+			       const struct io_uring_sqe *sqe,
+			       struct uring_bpf_data *data,
+			       unsigned int op_flags)
+{
+	u8 buf1_type, buf2_type;
+
+	/* Extract buffer configuration from bpf_op_flags */
+	buf1_type = IORING_BPF_BUF1_TYPE(op_flags);
+	buf2_type = IORING_BPF_BUF2_TYPE(op_flags);
+
+	/* Prepare buffer 1 */
+	if (buf1_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer: addr=sqe->addr, len=sqe->len */
+		data->buf1_addr = READ_ONCE(sqe->addr);
+		data->buf1_len = READ_ONCE(sqe->len);
+	} else if (buf1_type == IORING_BPF_BUF_TYPE_FIXED) {
+		/* Fixed buffer: index=sqe->buf_index, offset=sqe->addr, len=sqe->len */
+		req->buf_index = READ_ONCE(sqe->buf_index);
+		data->buf1_addr = READ_ONCE(sqe->addr);  /* offset within fixed buffer */
+		data->buf1_len = READ_ONCE(sqe->len);
+
+		/* Validate buffer index */
+		if (unlikely(!req->ctx->buf_table.nr))
+			return -EFAULT;
+		if (unlikely(req->buf_index >= req->ctx->buf_table.nr))
+			return -EINVAL;
+	} else if (buf1_type == IORING_BPF_BUF_TYPE_NONE) {
+		data->buf1_addr = 0;
+		data->buf1_len = 0;
+	} else {
+		return -EINVAL;
+	}
+
+	/* Prepare buffer 2 (plain only - io_uring only supports one fixed buffer) */
+	if (buf2_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer: addr=sqe->addr3, len=sqe->optlen */
+		data->buf2_addr = READ_ONCE(sqe->addr3);
+		data->buf2_len = READ_ONCE(sqe->optlen);
+	} else if (buf2_type == IORING_BPF_BUF_TYPE_NONE) {
+		data->buf2_addr = 0;
+		data->buf2_len = 0;
+	} else {
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
  int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
  {
  	struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
  	unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);
  	struct uring_bpf_ops *ops;
+	int ret;
  	if (!(req->ctx->flags & IORING_SETUP_BPF))
  		return -EACCES;
+	if (uring_bpf_get_flags(op_flags))
+		return -EINVAL;
+
  	data->opf = op_flags;
  	ops = &bpf_ops[uring_bpf_get_op(data->opf)];
+	/* Prepare buffers based on buffer type flags */
+	ret = io_bpf_prep_buffers(req, sqe, data, op_flags);
+	if (ret)
+		return ret;
+
  	if (ops->prep_fn)
  		return ops->prep_fn(data, sqe);
  	return -EOPNOTSUPP;
io_uring/uring_bpf.h +6 -0
index c76eba887d22..c919931cb4b0 100644
@@ -7,8 +7,18 @@ struct uring_bpf_data {
  	struct file     *file;
  	u32		opf;
+	/* Buffer 1 metadata - readable for bpf prog */
+	u32		buf1_len;		/* buffer 1 length, bytes 12-15 */
+	u64		buf1_addr;		/* buffer 1 address or offset, bytes 16-23 */
+
+	/* Buffer 2 metadata - readable for bpf prog (plain only) */
+	u64		buf2_addr;		/* buffer 2 address, bytes 24-31 */
Quoted 8 lines

io_uring/uring_bpf.h struct uring_bpf_data { · line 17

Quoted 2 lines
io_uring/uring_bpf.h +2 -0
+	u32		buf2_len;		/* buffer 2 length, bytes 32-35 */
+	u32		__pad;			/* padding, bytes 36-39 */

I'm wondering if this should be in the generic struct uring_bpf_data at all. For io_uring_cmd we have helper functions like io_uring_cmd_import_fixed[_vec] which can be used by the implementation, but the layout on top of io_uring_cmd->pdu is up to the specific operation.

Actually all above are not necessary, I plan to remove all in V2 because data can be passed via arena map easily & efficiently, and all kinds of buffer(plain user, fixed, vectored, fixed vectored) handing can leave to kfunc.

Quoted 2 lines

Thinking about it, I'm wondering if the bpf operations could be implemented on top of io_uring_cmd.

It could be one mess for uring_cmd which should focus on in-tree kernel users, not see obvious benefit, IMO.

Thanks,
Ming

#27 lore raw

Re: [PATCH 4/5] io_uring: bpf: add buffer support for IORING_OP_BPF

Am 13.11.25 um 12:04 schrieb Ming Lei:

Quoted 225 lines · Stefan Metzmacher
Quoted 215 lines

Am 04.11.25 um 17:21 schrieb Ming Lei:

Quoted 214 lines

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr,

    len=sqe->len)

Buffer 2 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr3, len=sqe->optlen)

The sqe->bpf_op_flags layout (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (3 bits)
Bits 20-18: Buffer 2 type (3 bits)
Bits 17-0: Custom BPF flags (18 bits)

Using 3-bit encoding for each buffer type allows for future expansion to 8 types (0-7). Currently types 0-2 are defined (none/plain/fixed) and 3-7 are reserved for future use.

Buffer 2 currently only supports none/plain types because the io_uring framework can only handle one fixed buffer per request (via req->buf_index). The 3-bit encoding provides room for future enhancements.

Buffer metadata (addresses, lengths) is stored in the extended uring_bpf_data structure and is accessible to BPF programs with readonly permissions. Buffer types can be extracted from the opf field using IORING_BPF_BUF1_TYPE() and IORING_BPF_BUF2_TYPE() macros.

Valid buffer combinations:
- 0 buffers
- 1 plain buffer
- 1 fixed buffer
- 2 plain buffers
- 1 fixed buffer + 1 plain buffer

---
   include/uapi/linux/io_uring.h | 45 +++++++++++++++++++++++--
   io_uring/bpf.c                | 63 ++++++++++++++++++++++++++++++++++-
   io_uring/uring_bpf.h          | 12 ++++++-
   3 files changed, 116 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +43 -2
index 94d2050131ac..950f4cfbbf86 100644
@@ -429,12 +429,53 @@ enum io_uring_op {
   #define IORING_SEND_VECTORIZED		(1U << 5)
   /*
- * sqe->bpf_op_flags		top 8bits is for storing bpf op
- *				The other 24bits are used for bpf prog
+ * sqe->bpf_op_flags layout (32 bits total):
+ *   Bits 31-24: BPF operation ID (8 bits, 256 possible operations)
+ *   Bits 23-21: Buffer 1 type (3 bits: none/plain/fixed/reserved)
+ *   Bits 20-18: Buffer 2 type (3 bits: none/plain/reserved)
+ *   Bits 17-0:  Custom BPF flags (18 bits, available for BPF programs)
+ *
+ * For IORING_OP_BPF, buffers are specified as follows:
+ *   Buffer 1 (plain):  addr=sqe->addr, len=sqe->len
+ *   Buffer 1 (fixed):  index=sqe->buf_index, offset=sqe->addr, len=sqe->len
+ *   Buffer 2 (plain):  addr=sqe->addr3, len=sqe->optlen
+ *
+ * Note: Buffer 1 can be none/plain/fixed. Buffer 2 can only be none/plain.
+ *       3-bit encoding for each buffer allows for future expansion to 8 types (0-7).
+ *       Currently only one fixed buffer per request is supported (buffer 1).
+ *       Valid combinations: 0 buffers, 1 plain, 1 fixed, 2 plain, 1 fixed + 1 plain.
    */
   #define IORING_BPF_OP_BITS	(8)
   #define IORING_BPF_OP_SHIFT	(24)
+/* Buffer type encoding in sqe->bpf_op_flags */
+#define IORING_BPF_BUF1_TYPE_SHIFT	(21)
+#define IORING_BPF_BUF2_TYPE_SHIFT	(18)
+#define IORING_BPF_BUF_TYPE_NONE	(0)	/* No buffer */
+#define IORING_BPF_BUF_TYPE_PLAIN	(1)	/* Plain user buffer */
+#define IORING_BPF_BUF_TYPE_FIXED	(2)	/* Fixed/registered buffer */
+#define IORING_BPF_BUF_TYPE_MASK	(7)	/* 3-bit mask */
+
+/* Helper macros to encode/decode buffer types */
+#define IORING_BPF_BUF1_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF1_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_BUF2_TYPE(flags) \
+	(((flags) >> IORING_BPF_BUF2_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_SET_BUF1_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF1_TYPE_SHIFT)
+#define IORING_BPF_SET_BUF2_TYPE(type) \
+	(((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF2_TYPE_SHIFT)
+
+/* Custom BPF flags mask (18 bits available, bits 17-0) */
+#define IORING_BPF_CUSTOM_FLAGS_MASK	((1U << 18) - 1)
+
+/* Encode all components into sqe->bpf_op_flags */
+#define IORING_BPF_OP_FLAGS(op, buf1_type, buf2_type, flags) \
+	(((op) << IORING_BPF_OP_SHIFT) | \
+	 IORING_BPF_SET_BUF1_TYPE(buf1_type) | \
+	 IORING_BPF_SET_BUF2_TYPE(buf2_type) | \
+	 ((flags) & IORING_BPF_CUSTOM_FLAGS_MASK))
+
   /*
    * cqe.res for IORING_CQE_F_NOTIF if
    * IORING_SEND_ZC_REPORT_USAGE was requested
io_uring/bpf.c +62 -1
index 8227be6d5a10..e837c3d57b96 100644
@@ -11,8 +11,10 @@
   #include <linux/btf.h>
   #include <linux/btf_ids.h>
   #include <linux/filter.h>
+#include <linux/uio.h>
   #include "io_uring.h"
   #include "uring_bpf.h"
+#include "rsrc.h"
   #define MAX_BPF_OPS_COUNT	(1 << IORING_BPF_OP_BITS)
@@ -28,7 +30,7 @@ static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
   static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)
   {
-	return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+	return op_flags & IORING_BPF_CUSTOM_FLAGS_MASK;
   }
   static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
@@ -36,18 +38,77 @@ static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *dat
   	return &bpf_ops[uring_bpf_get_op(data->opf)];
   }
+static int io_bpf_prep_buffers(struct io_kiocb *req,
+			       const struct io_uring_sqe *sqe,
+			       struct uring_bpf_data *data,
+			       unsigned int op_flags)
+{
+	u8 buf1_type, buf2_type;
+
+	/* Extract buffer configuration from bpf_op_flags */
+	buf1_type = IORING_BPF_BUF1_TYPE(op_flags);
+	buf2_type = IORING_BPF_BUF2_TYPE(op_flags);
+
+	/* Prepare buffer 1 */
+	if (buf1_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer: addr=sqe->addr, len=sqe->len */
+		data->buf1_addr = READ_ONCE(sqe->addr);
+		data->buf1_len = READ_ONCE(sqe->len);
+	} else if (buf1_type == IORING_BPF_BUF_TYPE_FIXED) {
+		/* Fixed buffer: index=sqe->buf_index, offset=sqe->addr, len=sqe->len */
+		req->buf_index = READ_ONCE(sqe->buf_index);
+		data->buf1_addr = READ_ONCE(sqe->addr);  /* offset within fixed buffer */
+		data->buf1_len = READ_ONCE(sqe->len);
+
+		/* Validate buffer index */
+		if (unlikely(!req->ctx->buf_table.nr))
+			return -EFAULT;
+		if (unlikely(req->buf_index >= req->ctx->buf_table.nr))
+			return -EINVAL;
+	} else if (buf1_type == IORING_BPF_BUF_TYPE_NONE) {
+		data->buf1_addr = 0;
+		data->buf1_len = 0;
+	} else {
+		return -EINVAL;
+	}
+
+	/* Prepare buffer 2 (plain only - io_uring only supports one fixed buffer) */
+	if (buf2_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer: addr=sqe->addr3, len=sqe->optlen */
+		data->buf2_addr = READ_ONCE(sqe->addr3);
+		data->buf2_len = READ_ONCE(sqe->optlen);
+	} else if (buf2_type == IORING_BPF_BUF_TYPE_NONE) {
+		data->buf2_addr = 0;
+		data->buf2_len = 0;
+	} else {
+		return -EINVAL;
+	}
+
+	return 0;
+}
+
+
   int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
   {
   	struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
   	unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);
   	struct uring_bpf_ops *ops;
+	int ret;
   	if (!(req->ctx->flags & IORING_SETUP_BPF))
   		return -EACCES;
+	if (uring_bpf_get_flags(op_flags))
+		return -EINVAL;
+
   	data->opf = op_flags;
   	ops = &bpf_ops[uring_bpf_get_op(data->opf)];
+	/* Prepare buffers based on buffer type flags */
+	ret = io_bpf_prep_buffers(req, sqe, data, op_flags);
+	if (ret)
+		return ret;
+
   	if (ops->prep_fn)
   		return ops->prep_fn(data, sqe);
   	return -EOPNOTSUPP;
io_uring/uring_bpf.h +6 -0
index c76eba887d22..c919931cb4b0 100644
@@ -7,8 +7,18 @@ struct uring_bpf_data {
   	struct file     *file;
   	u32		opf;
+	/* Buffer 1 metadata - readable for bpf prog */
+	u32		buf1_len;		/* buffer 1 length, bytes 12-15 */
+	u64		buf1_addr;		/* buffer 1 address or offset, bytes 16-23 */
+
+	/* Buffer 2 metadata - readable for bpf prog (plain only) */
+	u64		buf2_addr;		/* buffer 2 address, bytes 24-31 */
Quoted 8 lines

io_uring/uring_bpf.h struct uring_bpf_data { · line 17

Quoted 2 lines
io_uring/uring_bpf.h +2 -0
+	u32		buf2_len;		/* buffer 2 length, bytes 32-35 */
+	u32		__pad;			/* padding, bytes 36-39 */

I'm wondering if this should be in the generic struct uring_bpf_data at all. For io_uring_cmd we have helper functions like io_uring_cmd_import_fixed[_vec] which can be used by the implementation, but the layout on top of io_uring_cmd->pdu is up to the specific operation.

Quoted 4 lines

Actually all above are not necessary, I plan to remove all in V2 because data can be passed via arena map easily & efficiently, and all kinds of buffer(plain user, fixed, vectored, fixed vectored) handing can leave to kfunc.

Ok, I guess I can learn a lot from the next version and its examples :-)

Quoted 5 lines
Quoted 2 lines

Thinking about it, I'm wondering if the bpf operations could be implemented on top of io_uring_cmd.

It could be one mess for uring_cmd which should focus on in-tree kernel users, not see obvious benefit, IMO.

Yes, I'm not saying we should do it, only that it would be possible...

metze

#28 lore raw

Re: [PATCH 4/5] io_uring: bpf: add buffer support for IORING_OP_BPF

In reply to Ming Lei
Quoted 4 lines · Ming Lei

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

I agree the 2 buffer limit seems a bit restrictive, and it would make more sense to expose kfuncs to import plain and fixed buffers. Then the BPF program could decide what buffers to import based on the SQE, BPF maps, etc. This would be analogous to how uring_cmd implementations import buffers.

Quoted 5 lines

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr,

Should this say "addr=" instead of "offset="? It's passed as buf_addr to io_bpf_import_buffer(), so it's an absolute userspace address. The offset into the fixed buffer depends on the starting address of the fixed buffer.

Quoted 46 lines

len=sqe->len)

Buffer 2 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr3, len=sqe->optlen)

The sqe->bpf_op_flags layout (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (3 bits)
Bits 20-18: Buffer 2 type (3 bits)
Bits 17-0: Custom BPF flags (18 bits)

Using 3-bit encoding for each buffer type allows for future expansion to 8 types (0-7). Currently types 0-2 are defined (none/plain/fixed) and 3-7 are reserved for future use.

Buffer 2 currently only supports none/plain types because the io_uring framework can only handle one fixed buffer per request (via req->buf_index). The 3-bit encoding provides room for future enhancements.

Buffer metadata (addresses, lengths) is stored in the extended uring_bpf_data structure and is accessible to BPF programs with readonly permissions. Buffer types can be extracted from the opf field using IORING_BPF_BUF1_TYPE() and IORING_BPF_BUF2_TYPE() macros.

Valid buffer combinations:
- 0 buffers
- 1 plain buffer
- 1 fixed buffer
- 2 plain buffers
- 1 fixed buffer + 1 plain buffer

---
 include/uapi/linux/io_uring.h | 45 +++++++++++++++++++++++--
 io_uring/bpf.c                | 63 ++++++++++++++++++++++++++++++++++-
 io_uring/uring_bpf.h          | 12 ++++++-
 3 files changed, 116 insertions(+), 4 deletions(-)
include/uapi/linux/io_uring.h +0 -0
index 94d2050131ac..950f4cfbbf86 100644

include/uapi/linux/io_uring.h enum io_uring_op { · line 436

Quoted 10 lines
include/uapi/linux/io_uring.h +5 -2
@@ -429,12 +429,53 @@ enum io_uring_op {
 #define IORING_SEND_VECTORIZED         (1U << 5)
 
 /*
- * sqe->bpf_op_flags           top 8bits is for storing bpf op
- *                             The other 24bits are used for bpf prog
+ * sqe->bpf_op_flags layout (32 bits total):
+ *   Bits 31-24: BPF operation ID (8 bits, 256 possible operations)
+ *   Bits 23-21: Buffer 1 type (3 bits: none/plain/fixed/reserved)
+ *   Bits 20-18: Buffer 2 type (3 bits: none/plain/reserved)
+ *   Bits 17-0:  Custom BPF flags (18 bits, available for BPF programs)

This doesn't seem accurate, as io_uring_bpf_prep() rejects any requests with these bits set?

Quoted 11 lines
include/uapi/linux/io_uring.h +10 -0
+ *
+ * For IORING_OP_BPF, buffers are specified as follows:
+ *   Buffer 1 (plain):  addr=sqe->addr, len=sqe->len
+ *   Buffer 1 (fixed):  index=sqe->buf_index, offset=sqe->addr, len=sqe->len
+ *   Buffer 2 (plain):  addr=sqe->addr3, len=sqe->optlen
+ *
+ * Note: Buffer 1 can be none/plain/fixed. Buffer 2 can only be none/plain.
+ *       3-bit encoding for each buffer allows for future expansion to 8 types (0-7).
+ *       Currently only one fixed buffer per request is supported (buffer 1).
+ *       Valid combinations: 0 buffers, 1 plain, 1 fixed, 2 plain, 1 fixed + 1 plain.
  */

include/uapi/linux/io_uring.h enum io_uring_op { · line 457

Quoted 10 lines
include/uapi/linux/io_uring.h +7 -0
 #define IORING_BPF_OP_BITS     (8)
 #define IORING_BPF_OP_SHIFT    (24)
 
+/* Buffer type encoding in sqe->bpf_op_flags */
+#define IORING_BPF_BUF1_TYPE_SHIFT     (21)
+#define IORING_BPF_BUF2_TYPE_SHIFT     (18)
+#define IORING_BPF_BUF_TYPE_NONE       (0)     /* No buffer */
+#define IORING_BPF_BUF_TYPE_PLAIN      (1)     /* Plain user buffer */
+#define IORING_BPF_BUF_TYPE_FIXED      (2)     /* Fixed/registered buffer */
+#define IORING_BPF_BUF_TYPE_MASK       (7)     /* 3-bit mask */

What do you think about replacing the bpf_op_flags field with a struct containing op, buffer_flags, custom_flags fields to reduce the number of bitwise operations needed to read and write these values?

include/uapi/linux/io_uring.h enum io_uring_op { · line 463

Quoted 6 lines
include/uapi/linux/io_uring.h +6 -0
+
+/* Helper macros to encode/decode buffer types */
+#define IORING_BPF_BUF1_TYPE(flags) \
+       (((flags) >> IORING_BPF_BUF1_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)
+#define IORING_BPF_BUF2_TYPE(flags) \
+       (((flags) >> IORING_BPF_BUF2_TYPE_SHIFT) & IORING_BPF_BUF_TYPE_MASK)

I'm not sure how userspace would use these helpers. Would it make sense to move them from the UAPI header to the kernel-internal code?

include/uapi/linux/io_uring.h enum io_uring_op { · line 470

Quoted 7 lines
include/uapi/linux/io_uring.h +7 -0
+#define IORING_BPF_SET_BUF1_TYPE(type) \
+       (((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF1_TYPE_SHIFT)
+#define IORING_BPF_SET_BUF2_TYPE(type) \
+       (((type) & IORING_BPF_BUF_TYPE_MASK) << IORING_BPF_BUF2_TYPE_SHIFT)
+
+/* Custom BPF flags mask (18 bits available, bits 17-0) */
+#define IORING_BPF_CUSTOM_FLAGS_MASK   ((1U << 18) - 1)

Use IORING_BPF_BUF2_TYPE_SHIFT instead of 18?

Quoted 50 lines
include/uapi/linux/io_uring.h +8 -0
+
+/* Encode all components into sqe->bpf_op_flags */
+#define IORING_BPF_OP_FLAGS(op, buf1_type, buf2_type, flags) \
+       (((op) << IORING_BPF_OP_SHIFT) | \
+        IORING_BPF_SET_BUF1_TYPE(buf1_type) | \
+        IORING_BPF_SET_BUF2_TYPE(buf2_type) | \
+        ((flags) & IORING_BPF_CUSTOM_FLAGS_MASK))
+
 /*
  * cqe.res for IORING_CQE_F_NOTIF if
  * IORING_SEND_ZC_REPORT_USAGE was requested
io_uring/bpf.c +14 -1
index 8227be6d5a10..e837c3d57b96 100644
@@ -11,8 +11,10 @@
 #include <linux/btf.h>
 #include <linux/btf_ids.h>
 #include <linux/filter.h>
+#include <linux/uio.h>
 #include "io_uring.h"
 #include "uring_bpf.h"
+#include "rsrc.h"
 
 #define MAX_BPF_OPS_COUNT      (1 << IORING_BPF_OP_BITS)
 
@@ -28,7 +30,7 @@ static inline unsigned char uring_bpf_get_op(unsigned int op_flags)
 
 static inline unsigned int uring_bpf_get_flags(unsigned int op_flags)
 {
-       return op_flags & ((1U << IORING_BPF_OP_SHIFT) - 1);
+       return op_flags & IORING_BPF_CUSTOM_FLAGS_MASK;
 }
 
 static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *data)
@@ -36,18 +38,77 @@ static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *dat
        return &bpf_ops[uring_bpf_get_op(data->opf)];
 }
 
+static int io_bpf_prep_buffers(struct io_kiocb *req,
+                              const struct io_uring_sqe *sqe,
+                              struct uring_bpf_data *data,
+                              unsigned int op_flags)
+{
+       u8 buf1_type, buf2_type;
+
+       /* Extract buffer configuration from bpf_op_flags */
+       buf1_type = IORING_BPF_BUF1_TYPE(op_flags);
+       buf2_type = IORING_BPF_BUF2_TYPE(op_flags);
+

io_uring/bpf.c static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *dat · line 61

Quoted 10 lines
io_uring/bpf.c +10 -0
+       /* Prepare buffer 1 */
+       if (buf1_type == IORING_BPF_BUF_TYPE_PLAIN) {
+               /* Plain user buffer: addr=sqe->addr, len=sqe->len */
+               data->buf1_addr = READ_ONCE(sqe->addr);
+               data->buf1_len = READ_ONCE(sqe->len);
+       } else if (buf1_type == IORING_BPF_BUF_TYPE_FIXED) {
+               /* Fixed buffer: index=sqe->buf_index, offset=sqe->addr, len=sqe->len */
+               req->buf_index = READ_ONCE(sqe->buf_index);
+               data->buf1_addr = READ_ONCE(sqe->addr);  /* offset within fixed buffer */
+               data->buf1_len = READ_ONCE(sqe->len);

Deduplicate these assignments with the ones in the IORING_BPF_BUF_TYPE_PLAIN case?

io_uring/bpf.c static inline struct uring_bpf_ops *uring_bpf_get_ops(struct uring_bpf_data *dat · line 67

Quoted 6 lines
io_uring/bpf.c +6 -0
+
+               /* Validate buffer index */
+               if (unlikely(!req->ctx->buf_table.nr))
+                       return -EFAULT;
+               if (unlikely(req->buf_index >= req->ctx->buf_table.nr))
+                       return -EINVAL;

Why validate these here? Won't io_import_reg_buf() check these anyways?

Best,
Caleb

Quoted 74 lines
io_uring/bpf.c +32 -0
+       } else if (buf1_type == IORING_BPF_BUF_TYPE_NONE) {
+               data->buf1_addr = 0;
+               data->buf1_len = 0;
+       } else {
+               return -EINVAL;
+       }
+
+       /* Prepare buffer 2 (plain only - io_uring only supports one fixed buffer) */
+       if (buf2_type == IORING_BPF_BUF_TYPE_PLAIN) {
+               /* Plain user buffer: addr=sqe->addr3, len=sqe->optlen */
+               data->buf2_addr = READ_ONCE(sqe->addr3);
+               data->buf2_len = READ_ONCE(sqe->optlen);
+       } else if (buf2_type == IORING_BPF_BUF_TYPE_NONE) {
+               data->buf2_addr = 0;
+               data->buf2_len = 0;
+       } else {
+               return -EINVAL;
+       }
+
+       return 0;
+}
+
+
 int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 {
        struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
        unsigned int op_flags = READ_ONCE(sqe->bpf_op_flags);
        struct uring_bpf_ops *ops;
+       int ret;
 
        if (!(req->ctx->flags & IORING_SETUP_BPF))
                return -EACCES;
 
+       if (uring_bpf_get_flags(op_flags))
+               return -EINVAL;
+
        data->opf = op_flags;
        ops = &bpf_ops[uring_bpf_get_op(data->opf)];
 
+       /* Prepare buffers based on buffer type flags */
+       ret = io_bpf_prep_buffers(req, sqe, data, op_flags);
+       if (ret)
+               return ret;
+
        if (ops->prep_fn)
                return ops->prep_fn(data, sqe);
        return -EOPNOTSUPP;
io_uring/uring_bpf.h +11 -2
index c76eba887d22..c919931cb4b0 100644
@@ -7,8 +7,18 @@ struct uring_bpf_data {
        struct file     *file;
        u32             opf;
 
+       /* Buffer 1 metadata - readable for bpf prog */
+       u32             buf1_len;               /* buffer 1 length, bytes 12-15 */
+       u64             buf1_addr;              /* buffer 1 address or offset, bytes 16-23 */
+
+       /* Buffer 2 metadata - readable for bpf prog (plain only) */
+       u64             buf2_addr;              /* buffer 2 address, bytes 24-31 */
+       u32             buf2_len;               /* buffer 2 length, bytes 32-35 */
+       u32             __pad;                  /* padding, bytes 36-39 */
+
        /* writeable for bpf prog */
-       u8              pdu[64 - sizeof(struct file *) - sizeof(u32)];
+       u8              pdu[64 - sizeof(struct file *) - 4 * sizeof(u32) -
+               2 * sizeof(u64)];
 };
 
 typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
--

2.47.0

author
#29 lore raw

Re: [PATCH 4/5] io_uring: bpf: add buffer support for IORING_OP_BPF

Quoted 11 lines · Caleb Sander Mateos
Quoted 4 lines · Ming Lei

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

I agree the 2 buffer limit seems a bit restrictive, and it would make more sense to expose kfuncs to import plain and fixed buffers. Then the BPF program could decide what buffers to import based on the SQE, BPF maps, etc. This would be analogous to how uring_cmd implementations import buffers.

Yes, this way is too restrictive.

I think there are at least two approaches:

- define public buffer descriptor, which can describe plain, fixed, vector, fixed vector buffer, ...

- user can pass this buffer descriptor array from sqe->addr & sqe->len (buf descriptor need to be defined in UAPI)

OR

- user can pass this buffer array from arena map (buf descriptor is just API between bpf prog and userspace)

The former could be better, because buf descriptor is part of UAPI, and user still can choose to use bpf map to pass buffer. But defining 'buf descriptor' may take a while...

The latter way could be easier to start...

Quoted 11 lines
Quoted 6 lines

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr,

Should this say "addr=" instead of "offset="? It's passed as buf_addr to io_bpf_import_buffer(), so it's an absolute userspace address. The offset into the fixed buffer depends on the starting address of the fixed buffer.

For user fixed buffer, offset is the buff addr.

For kernel fixed buffer, it is offset.

Thanks,
Ming

#30 lore raw

Re: [PATCH 4/5] io_uring: bpf: add buffer support for IORING_OP_BPF

Quoted 26 lines · Ming Lei
Quoted 11 lines · Caleb Sander Mateos
Quoted 4 lines · Ming Lei

Add support for passing 0-2 buffers to BPF operations through IORING_OP_BPF. Buffer types are encoded in sqe->bpf_op_flags using dedicated 3-bit fields for each buffer.

I agree the 2 buffer limit seems a bit restrictive, and it would make more sense to expose kfuncs to import plain and fixed buffers. Then the BPF program could decide what buffers to import based on the SQE, BPF maps, etc. This would be analogous to how uring_cmd implementations import buffers.

Yes, this way is too restrictive.

I think there are at least two approaches:

- define public buffer descriptor, which can describe plain, fixed, vector, fixed vector buffer, ...

- user can pass this buffer descriptor array from sqe->addr & sqe->len (buf descriptor need to be defined in UAPI)

OR

Quoted 8 lines

- user can pass this buffer array from arena map (buf descriptor is just API between bpf prog and userspace)

The former could be better, because buf descriptor is part of UAPI, and user still can choose to use bpf map to pass buffer. But defining 'buf descriptor' may take a while...

The latter way could be easier to start...

I think the latter approach (buffer interface specific to io_uring BPF program) probably makes more sense. Different programs may have different buffer needs (e.g. 1 buffer, 2 buffers, an arbitrary number of buffers). Programs that need an arbitrary number of buffers will probably have to specify the address and length of a struct iovec array in the SQE, or communicate via some BPF map. But this additional indirection would add unnecessary overhead for programs that only accept 1 or 2 buffers. Leaving it up to the io_uring BPF program how to interpret the SQE fields makes a lot of sense to me, as ->uring_cmd() implementations already have that responsibility.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Buffer 1 can be:
- None (no buffer)
- Plain user buffer (addr=sqe->addr, len=sqe->len)
- Fixed/registered buffer (index=sqe->buf_index, offset=sqe->addr,

Quoted 8 lines
Quoted 4 lines

Should this say "addr=" instead of "offset="? It's passed as buf_addr to io_bpf_import_buffer(), so it's an absolute userspace address. The offset into the fixed buffer depends on the starting address of the fixed buffer.

For user fixed buffer, offset is the buff addr.

For kernel fixed buffer, it is offset.

Yes, good point. I think fixed buffers are usually described in terms of user registered buffers (probably since those were the only kind of registered buffers until recently). It wouldn't hurt to be explicit that addr vs. offset depends on the type of registered buffer. I think it would also be fine to just say that this is an addr for both kinds of buffers, as you can think of kernel registered buffers as implicitly having a starting address of 0.

Best,
Caleb

author
#31 lore raw

PATCH 5/5 io_uring: bpf: add io_uring_bpf_req_memcpy() kfunc

In reply to Ming Lei

Add io_uring_bpf_req_memcpy() kfunc to enable BPF programs to copy data between buffers associated with IORING_OP_BPF requests.

The kfunc supports copying between:
- Plain user buffers (using import_ubuf())
- Fixed/registered buffers (using io_import_reg_buf())
- Mixed combinations (plain-to-fixed, fixed-to-plain)

This enables BPF programs to implement data transformation and processing operations directly within io_uring's request context, avoiding additional userspace copies.

Implementation details:

1. Add issue_flags tracking in struct uring_bpf_data:
- Replace __pad field with issue_flags (bytes 36-39)
- Initialized to 0 before ops->prep_fn()
- Saved from issue_flags parameter before ops->issue_fn()
- Required by io_import_reg_buf() for proper async handling

2. Add buffer preparation infrastructure:
- io_bpf_prep_buffers() extracts buffer metadata from SQE
- Buffer 1: plain (addr/len) or fixed (buf_index/addr/len)
- Buffer 2: plain only (addr3/optlen)
- Buffer types encoded in sqe->bpf_op_flags bits 23-18

3. io_uring_bpf_req_memcpy() implementation:
- Validates buffer IDs (1 or 2) and prevents same-buffer copies
- Extracts buffer metadata based on buffer ID
- Sets up iov_iters using import_ubuf() or io_import_reg_buf()
- Performs page-sized chunked copying via temporary buffer
- Returns bytes copied or negative error code

Buffer encoding in sqe->bpf_op_flags (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (0=none, 1=plain, 2=fixed) Bits 20-18: Buffer 2 type (0=none, 1=plain) Bits 17-0: Custom BPF flags (18 bits)

---
 io_uring/bpf.c       | 187 +++++++++++++++++++++++++++++++++++++++++++
 io_uring/uring_bpf.h |  11 ++-
 2 files changed, 197 insertions(+), 1 deletion(-)
io_uring/bpf.c +187 -0
index e837c3d57b96..ee4c617e3904 100644
@@ -109,6 +109,8 @@ int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
 	if (ret)
 		return ret;
 
+	/* ctx->uring_lock is held */
+	data->issue_flags = 0;
 	if (ops->prep_fn)
 		return ops->prep_fn(data, sqe);
 	return -EOPNOTSUPP;
@@ -126,6 +128,9 @@ static int __io_uring_bpf_issue(struct io_kiocb *req)
 
 int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
+	struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+
+	data->issue_flags = issue_flags;
 	if (issue_flags & IO_URING_F_UNLOCKED) {
 		int idx, ret;
 
@@ -143,6 +148,8 @@ void io_uring_bpf_fail(struct io_kiocb *req)
 	struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
 	struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
 
+	/* ctx->uring_lock is held */
+	data->issue_flags = 0;
 	if (ops->fail_fn)
 		ops->fail_fn(data);
 }
@@ -152,6 +159,8 @@ void io_uring_bpf_cleanup(struct io_kiocb *req)
 	struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
 	struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
 
+	/* ctx->uring_lock is held */
+	data->issue_flags = 0;
 	if (ops->cleanup_fn)
 		ops->cleanup_fn(data);
 }
@@ -324,6 +333,104 @@ static struct bpf_struct_ops bpf_uring_bpf_ops = {
 	.owner = THIS_MODULE,
 };
 
+/*
+ * Helper to copy data between two iov_iters using page extraction.
+ * Extracts pages from source iterator and copies them to destination.
+ * Returns number of bytes copied or negative error code.
+ */
+static ssize_t io_bpf_copy_iters(struct iov_iter *src, struct iov_iter *dst,
+				 size_t len)
+{
+#define MAX_PAGES_PER_LOOP 32
+	struct page *pages[MAX_PAGES_PER_LOOP];
+	size_t total_copied = 0;
+	bool need_unpin;
+
+	/* Determine if we'll need to unpin pages later */
+	need_unpin = user_backed_iter(src);
+
+	/* Process pages in chunks */
+	while (len > 0) {
+		struct page **page_array = pages;
+		size_t offset, copied = 0;
+		ssize_t extracted;
+		unsigned int nr_pages;
+		size_t chunk_len;
+		int i;
+
+		/* Extract up to MAX_PAGES_PER_LOOP pages */
+		chunk_len = min_t(size_t, len, MAX_PAGES_PER_LOOP * PAGE_SIZE);
+		extracted = iov_iter_extract_pages(src, &page_array, chunk_len,
+						   MAX_PAGES_PER_LOOP, 0, &offset);
+		if (extracted <= 0) {
+			if (total_copied > 0)
+				break;
+			return extracted < 0 ? extracted : -EFAULT;
+		}
+
+		nr_pages = DIV_ROUND_UP(offset + extracted, PAGE_SIZE);
+
+		/* Copy pages to destination iterator */
+		for (i = 0; i < nr_pages && copied < extracted; i++) {
+			size_t page_offset = (i == 0) ? offset : 0;
+			size_t page_len = min_t(size_t, extracted - copied,
+						PAGE_SIZE - page_offset);
+			size_t n;
+
+			n = copy_page_to_iter(pages[i], page_offset, page_len, dst);
+			copied += n;
+			if (n < page_len)
+				break;
+		}
+
+		/* Clean up extracted pages */
+		if (need_unpin)
+			unpin_user_pages(pages, nr_pages);
+
+		total_copied += copied;
+		len -= copied;
+
+		/* Stop if we didn't copy all extracted data */
+		if (copied < extracted)
+			break;
+	}
+
+	return total_copied;
+#undef MAX_PAGES_PER_LOOP
+}
+
+/*
+ * Helper to import a buffer into an iov_iter for BPF memcpy operations.
+ * Handles both plain user buffers and fixed/registered buffers.
+ *
+ * @req: io_kiocb request
+ * @iter: output iterator
+ * @buf_type: buffer type (plain or fixed)
+ * @addr: buffer address
+ * @offset: offset into buffer
+ * @len: length from offset
+ * @direction: ITER_SOURCE for source buffer, ITER_DEST for destination
+ * @issue_flags: io_uring issue flags
+ *
+ * Returns 0 on success, negative error code on failure.
+ */
+static int io_bpf_import_buffer(struct io_kiocb *req, struct iov_iter *iter,
+				u8 buf_type, u64 addr, unsigned int offset,
+				u32 len, int direction, unsigned int issue_flags)
+{
+	if (buf_type == IORING_BPF_BUF_TYPE_PLAIN) {
+		/* Plain user buffer */
+		return import_ubuf(direction, (void __user *)(addr + offset),
+				   len - offset, iter);
+	} else if (buf_type == IORING_BPF_BUF_TYPE_FIXED) {
+		/* Fixed buffer */
+		return io_import_reg_buf(req, iter, addr + offset,
+					 len - offset, direction, issue_flags);
+	}
+
+	return -EINVAL;
+}
+
 __bpf_kfunc_start_defs();
 __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)
 {
@@ -339,11 +446,91 @@ __bpf_kfunc struct io_kiocb *uring_bpf_data_to_req(struct uring_bpf_data *data)
 {
 	return cmd_to_io_kiocb(data);
 }
+
+/**
+ * io_uring_bpf_req_memcpy - Copy data between io_uring BPF request buffers
+ * @data: BPF request data containing buffer metadata
+ * @dest: Destination buffer descriptor (with buf_id and offset)
+ * @src: Source buffer descriptor (with buf_id and offset)
+ * @len: Number of bytes to copy
+ *
+ * Copies data between two different io_uring BPF request buffers (buf_id 1 and 2).
+ * Supports: plain-to-plain, fixed-to-plain, and plain-to-fixed.
+ * Does not support copying within the same buffer (src and dest must be different).
+ *
+ * Returns: Number of bytes copied on success, negative error code on failure
+ */
+__bpf_kfunc int io_uring_bpf_req_memcpy(struct uring_bpf_data *data,
+					struct bpf_req_mem_desc *dest,
+					struct bpf_req_mem_desc *src,
+					unsigned int len)
+{
+	struct io_kiocb *req = cmd_to_io_kiocb(data);
+	struct iov_iter dst_iter, src_iter;
+	u8 dst_type, src_type;
+	u64 dst_addr, src_addr;
+	u32 dst_len, src_len;
+	int ret;
+
+	/* Validate buffer IDs */
+	if (dest->buf_id < 1 || dest->buf_id > 2 ||
+	    src->buf_id < 1 || src->buf_id > 2)
+		return -EINVAL;
+
+	/* Don't allow copying within the same buffer */
+	if (src->buf_id == dest->buf_id)
+		return -EINVAL;
+
+	/* Extract source buffer metadata */
+	if (src->buf_id == 1) {
+		src_type = IORING_BPF_BUF1_TYPE(data->opf);
+		src_addr = data->buf1_addr;
+		src_len = data->buf1_len;
+	} else {
+		src_type = IORING_BPF_BUF2_TYPE(data->opf);
+		src_addr = data->buf2_addr;
+		src_len = data->buf2_len;
+	}
+
+	/* Extract destination buffer metadata */
+	if (dest->buf_id == 1) {
+		dst_type = IORING_BPF_BUF1_TYPE(data->opf);
+		dst_addr = data->buf1_addr;
+		dst_len = data->buf1_len;
+	} else {
+		dst_type = IORING_BPF_BUF2_TYPE(data->opf);
+		dst_addr = data->buf2_addr;
+		dst_len = data->buf2_len;
+	}
+
+	/* Validate offsets and lengths */
+	if (src->offset + len > src_len || dest->offset + len > dst_len)
+		return -EINVAL;
+
+	/* Initialize source iterator */
+	ret = io_bpf_import_buffer(req, &src_iter, src_type,
+				   src_addr, src->offset, src_len,
+				   ITER_SOURCE, data->issue_flags);
+	if (ret)
+		return ret;
+
+	/* Initialize destination iterator */
+	ret = io_bpf_import_buffer(req, &dst_iter, dst_type,
+				   dst_addr, dest->offset, dst_len,
+				   ITER_DEST, data->issue_flags);
+	if (ret)
+		return ret;
+
+	/* Extract pages from source iterator and copy to destination */
+	return io_bpf_copy_iters(&src_iter, &dst_iter, len);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(uring_bpf_kfuncs)
 BTF_ID_FLAGS(func, uring_bpf_set_result)
 BTF_ID_FLAGS(func, uring_bpf_data_to_req)
+BTF_ID_FLAGS(func, io_uring_bpf_req_memcpy)
 BTF_KFUNCS_END(uring_bpf_kfuncs)
 
 static const struct btf_kfunc_id_set uring_kfunc_set = {
io_uring/uring_bpf.h +10 -1
index c919931cb4b0..d6e0d6dff82e 100644
@@ -14,13 +14,22 @@ struct uring_bpf_data {
 	/* Buffer 2 metadata - readable for bpf prog (plain only) */
 	u64		buf2_addr;		/* buffer 2 address, bytes 24-31 */
 	u32		buf2_len;		/* buffer 2 length, bytes 32-35 */
-	u32		__pad;			/* padding, bytes 36-39 */
+	u32		issue_flags;		/* issue_flags from io_uring, bytes 36-39 */
 
 	/* writeable for bpf prog */
 	u8              pdu[64 - sizeof(struct file *) - 4 * sizeof(u32) -
 		2 * sizeof(u64)];
 };
 
+/*
+ * Descriptor for io_uring BPF request buffer.
+ * Used by io_uring_bpf_req_memcpy() to identify which buffer to copy from/to.
+ */
+struct bpf_req_mem_desc {
+	u8		buf_id;		/* Buffer ID: 1 or 2 */
+	unsigned int	offset;		/* Offset into buffer */
+};
+
 typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
 			       const struct io_uring_sqe *sqe);
 typedef int (*uring_io_issue_t)(struct uring_bpf_data *data);
Signature · 1 line
2.47.0
#32 lore raw

Re: [PATCH 5/5] io_uring: bpf: add io_uring_bpf_req_memcpy() kfunc

kernel test robot noticed the following build warnings:

Hi Ming,

kernel test robot noticed the following build warnings:

[auto build test WARNING on next-20251104]
[cannot apply to bpf-next/net bpf-next/master bpf/master linus/master v6.18-rc4 v6.18-rc3 v6.18-rc2 v6.18-rc4] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Ming-Lei/io_uring-prepare-for-extending-io_uring-with-bpf/20251105-002757
base:   next-20251104
patch link:    https://lore.kernel.org/r/20251104162123.1086035-6-ming.lei%40redhat.com
patch subject: [PATCH 5/5] io_uring: bpf: add io_uring_bpf_req_memcpy() kfunc
config: openrisc-allyesconfig (https://download.01.org/0day-ci/archive/20251108/[email protected]/config)
compiler: or1k-linux-gcc (GCC) 15.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20251108/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All warnings (new ones prefixed by >>):

io_uring/bpf.c: In function 'io_bpf_import_buffer':

>> io_uring/bpf.c:423:47: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     423 |                 return import_ubuf(direction, (void __user *)(addr + offset),
         |                                               ^
   In file included from include/linux/bpf_verifier.h:7,
                    from io_uring/bpf.c:9:
   io_uring/bpf.c: In function 'io_bpf_init':
   include/linux/bpf.h:2044:50: warning: statement with no effect [-Wunused-value]
    2044 | #define register_bpf_struct_ops(st_ops, type) ({ (void *)(st_ops); 0; })
         |                                                  ^~~~~~~~~~~~~~~~
   io_uring/bpf.c:551:15: note: in expansion of macro 'register_bpf_struct_ops'
     551 |         err = register_bpf_struct_ops(&bpf_uring_bpf_ops, uring_bpf_ops);
         |               ^~~~~~~~~~~~~~~~~~~~~~~
   In file included from <command-line>:
   In function 'io_kiocb_cmd_sz_check',
       inlined from 'io_uring_bpf_prep' at io_uring/bpf.c:93:32:
   include/linux/compiler_types.h:603:45: error: call to '__compiletime_assert_598' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |                                             ^
   include/linux/compiler_types.h:584:25: note: in definition of macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^~~~~~
   include/linux/compiler_types.h:603:9: note: in expansion of macro '_compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:9: note: in expansion of macro 'BUILD_BUG_ON_MSG'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:9: note: in expansion of macro 'BUILD_BUG_ON'
     655 |         BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
         |         ^~~~~~~~~~~~
   In function 'io_kiocb_cmd_sz_check',
       inlined from 'io_uring_bpf_issue' at io_uring/bpf.c:131:32:
   include/linux/compiler_types.h:603:45: error: call to '__compiletime_assert_598' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |                                             ^
   include/linux/compiler_types.h:584:25: note: in definition of macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^~~~~~
   include/linux/compiler_types.h:603:9: note: in expansion of macro '_compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:9: note: in expansion of macro 'BUILD_BUG_ON_MSG'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:9: note: in expansion of macro 'BUILD_BUG_ON'
     655 |         BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
         |         ^~~~~~~~~~~~
   In function 'io_kiocb_cmd_sz_check',
       inlined from 'io_uring_bpf_fail' at io_uring/bpf.c:148:32:
   include/linux/compiler_types.h:603:45: error: call to '__compiletime_assert_598' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |                                             ^
   include/linux/compiler_types.h:584:25: note: in definition of macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^~~~~~
   include/linux/compiler_types.h:603:9: note: in expansion of macro '_compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:9: note: in expansion of macro 'BUILD_BUG_ON_MSG'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:9: note: in expansion of macro 'BUILD_BUG_ON'
     655 |         BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
         |         ^~~~~~~~~~~~
   In function 'io_kiocb_cmd_sz_check',
       inlined from 'io_uring_bpf_cleanup' at io_uring/bpf.c:159:32:
   include/linux/compiler_types.h:603:45: error: call to '__compiletime_assert_598' declared with attribute error: BUILD_BUG_ON failed: cmd_sz > sizeof(struct io_cmd_data)
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |                                             ^
   include/linux/compiler_types.h:584:25: note: in definition of macro '__compiletime_assert'
     584 |                         prefix ## suffix();                             \
         |                         ^~~~~~
   include/linux/compiler_types.h:603:9: note: in expansion of macro '_compiletime_assert'
     603 |         _compiletime_assert(condition, msg, __compiletime_assert_, __COUNTER__)
         |         ^~~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:39:37: note: in expansion of macro 'compiletime_assert'
      39 | #define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
         |                                     ^~~~~~~~~~~~~~~~~~
   include/linux/build_bug.h:50:9: note: in expansion of macro 'BUILD_BUG_ON_MSG'
      50 |         BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
         |         ^~~~~~~~~~~~~~~~
   include/linux/io_uring_types.h:655:9: note: in expansion of macro 'BUILD_BUG_ON'
     655 |         BUILD_BUG_ON(cmd_sz > sizeof(struct io_cmd_data));
         |         ^~~~~~~~~~~~

vim +423 io_uring/bpf.c

401
402 /*
403 * Helper to import a buffer into an iov_iter for BPF memcpy operations.
404 * Handles both plain user buffers and fixed/registered buffers.
405 *
406 * @req: io_kiocb request
407 * @iter: output iterator
408 * @buf_type: buffer type (plain or fixed)
409 * @addr: buffer address
410 * @offset: offset into buffer
411 * @len: length from offset
412 * @direction: ITER_SOURCE for source buffer, ITER_DEST for destination
413 * @issue_flags: io_uring issue flags
414 *
415 * Returns 0 on success, negative error code on failure.
416 */
417 static int io_bpf_import_buffer(struct io_kiocb *req, struct iov_iter *iter,
418 u8 buf_type, u64 addr, unsigned int offset,
419 u32 len, int direction, unsigned int issue_flags)
420 {
421 if (buf_type == IORING_BPF_BUF_TYPE_PLAIN) {
422 /* Plain user buffer */

 > 423			return import_ubuf(direction, (void __user *)(addr + offset),

424 len - offset, iter);
425 } else if (buf_type == IORING_BPF_BUF_TYPE_FIXED) {
426 /* Fixed buffer */
427 return io_import_reg_buf(req, iter, addr + offset,
428 len - offset, direction, issue_flags);
429 }
430
431 return -EINVAL;
432 }
433

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#33 lore raw

Re: [PATCH 5/5] io_uring: bpf: add io_uring_bpf_req_memcpy() kfunc

In reply to Ming Lei
Quoted 96 lines · Ming Lei

Add io_uring_bpf_req_memcpy() kfunc to enable BPF programs to copy data between buffers associated with IORING_OP_BPF requests.

The kfunc supports copying between:
- Plain user buffers (using import_ubuf())
- Fixed/registered buffers (using io_import_reg_buf())
- Mixed combinations (plain-to-fixed, fixed-to-plain)

This enables BPF programs to implement data transformation and processing operations directly within io_uring's request context, avoiding additional userspace copies.

Implementation details:

1. Add issue_flags tracking in struct uring_bpf_data:
- Replace __pad field with issue_flags (bytes 36-39)
- Initialized to 0 before ops->prep_fn()
- Saved from issue_flags parameter before ops->issue_fn()
- Required by io_import_reg_buf() for proper async handling

2. Add buffer preparation infrastructure:
- io_bpf_prep_buffers() extracts buffer metadata from SQE
- Buffer 1: plain (addr/len) or fixed (buf_index/addr/len)
- Buffer 2: plain only (addr3/optlen)
- Buffer types encoded in sqe->bpf_op_flags bits 23-18

3. io_uring_bpf_req_memcpy() implementation:
- Validates buffer IDs (1 or 2) and prevents same-buffer copies
- Extracts buffer metadata based on buffer ID
- Sets up iov_iters using import_ubuf() or io_import_reg_buf()
- Performs page-sized chunked copying via temporary buffer
- Returns bytes copied or negative error code

Buffer encoding in sqe->bpf_op_flags (32 bits):
Bits 31-24: BPF operation ID (8 bits)
Bits 23-21: Buffer 1 type (0=none, 1=plain, 2=fixed) Bits 20-18: Buffer 2 type (0=none, 1=plain) Bits 17-0: Custom BPF flags (18 bits)

---
 io_uring/bpf.c       | 187 +++++++++++++++++++++++++++++++++++++++++++
 io_uring/uring_bpf.h |  11 ++-
 2 files changed, 197 insertions(+), 1 deletion(-)
io_uring/bpf.c +14 -0
index e837c3d57b96..ee4c617e3904 100644
@@ -109,6 +109,8 @@ int io_uring_bpf_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe)
        if (ret)
                return ret;
 
+       /* ctx->uring_lock is held */
+       data->issue_flags = 0;
        if (ops->prep_fn)
                return ops->prep_fn(data, sqe);
        return -EOPNOTSUPP;
@@ -126,6 +128,9 @@ static int __io_uring_bpf_issue(struct io_kiocb *req)
 
 int io_uring_bpf_issue(struct io_kiocb *req, unsigned int issue_flags)
 {
+       struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
+
+       data->issue_flags = issue_flags;
        if (issue_flags & IO_URING_F_UNLOCKED) {
                int idx, ret;
 
@@ -143,6 +148,8 @@ void io_uring_bpf_fail(struct io_kiocb *req)
        struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
        struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
 
+       /* ctx->uring_lock is held */
+       data->issue_flags = 0;
        if (ops->fail_fn)
                ops->fail_fn(data);
 }
@@ -152,6 +159,8 @@ void io_uring_bpf_cleanup(struct io_kiocb *req)
        struct uring_bpf_data *data = io_kiocb_to_cmd(req, struct uring_bpf_data);
        struct uring_bpf_ops *ops = uring_bpf_get_ops(data);
 
+       /* ctx->uring_lock is held */
+       data->issue_flags = 0;
        if (ops->cleanup_fn)
                ops->cleanup_fn(data);
 }
@@ -324,6 +333,104 @@ static struct bpf_struct_ops bpf_uring_bpf_ops = {
        .owner = THIS_MODULE,
 };
 
+/*
+ * Helper to copy data between two iov_iters using page extraction.
+ * Extracts pages from source iterator and copies them to destination.
+ * Returns number of bytes copied or negative error code.
+ */

io_uring/bpf.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 350

Quoted 10 lines
io_uring/bpf.c +10 -0
+static ssize_t io_bpf_copy_iters(struct iov_iter *src, struct iov_iter *dst,
+                                size_t len)
+{
+#define MAX_PAGES_PER_LOOP 32
+       struct page *pages[MAX_PAGES_PER_LOOP];
+       size_t total_copied = 0;
+       bool need_unpin;
+
+       /* Determine if we'll need to unpin pages later */
+       need_unpin = user_backed_iter(src);

Use iov_iter_extract_will_pin() for clarity?

Quoted 28 lines
io_uring/bpf.c +28 -0
+
+       /* Process pages in chunks */
+       while (len > 0) {
+               struct page **page_array = pages;
+               size_t offset, copied = 0;
+               ssize_t extracted;
+               unsigned int nr_pages;
+               size_t chunk_len;
+               int i;
+
+               /* Extract up to MAX_PAGES_PER_LOOP pages */
+               chunk_len = min_t(size_t, len, MAX_PAGES_PER_LOOP * PAGE_SIZE);
+               extracted = iov_iter_extract_pages(src, &page_array, chunk_len,
+                                                  MAX_PAGES_PER_LOOP, 0, &offset);
+               if (extracted <= 0) {
+                       if (total_copied > 0)
+                               break;
+                       return extracted < 0 ? extracted : -EFAULT;
+               }
+
+               nr_pages = DIV_ROUND_UP(offset + extracted, PAGE_SIZE);
+
+               /* Copy pages to destination iterator */
+               for (i = 0; i < nr_pages && copied < extracted; i++) {
+                       size_t page_offset = (i == 0) ? offset : 0;
+                       size_t page_len = min_t(size_t, extracted - copied,
+                                               PAGE_SIZE - page_offset);
+                       size_t n;

io_uring/bpf.c static struct bpf_struct_ops bpf_uring_bpf_ops = { · line 388

Quoted 10 lines
io_uring/bpf.c +10 -0
+
+                       n = copy_page_to_iter(pages[i], page_offset, page_len, dst);
+                       copied += n;
+                       if (n < page_len)
+                               break;
+               }
+
+               /* Clean up extracted pages */
+               if (need_unpin)
+                       unpin_user_pages(pages, nr_pages);

Could avoid the page pinning and unpinning cost when copying from a user iov_iter to a bvec iov_iter by using iov_iter_extract_pages(dst) and copy_page_from_iter(src). But this optimization could be implemented later.

Quoted 59 lines
io_uring/bpf.c +52 -0
+
+               total_copied += copied;
+               len -= copied;
+
+               /* Stop if we didn't copy all extracted data */
+               if (copied < extracted)
+                       break;
+       }
+
+       return total_copied;
+#undef MAX_PAGES_PER_LOOP
+}
+
+/*
+ * Helper to import a buffer into an iov_iter for BPF memcpy operations.
+ * Handles both plain user buffers and fixed/registered buffers.
+ *
+ * @req: io_kiocb request
+ * @iter: output iterator
+ * @buf_type: buffer type (plain or fixed)
+ * @addr: buffer address
+ * @offset: offset into buffer
+ * @len: length from offset
+ * @direction: ITER_SOURCE for source buffer, ITER_DEST for destination
+ * @issue_flags: io_uring issue flags
+ *
+ * Returns 0 on success, negative error code on failure.
+ */
+static int io_bpf_import_buffer(struct io_kiocb *req, struct iov_iter *iter,
+                               u8 buf_type, u64 addr, unsigned int offset,
+                               u32 len, int direction, unsigned int issue_flags)
+{
+       if (buf_type == IORING_BPF_BUF_TYPE_PLAIN) {
+               /* Plain user buffer */
+               return import_ubuf(direction, (void __user *)(addr + offset),
+                                  len - offset, iter);
+       } else if (buf_type == IORING_BPF_BUF_TYPE_FIXED) {
+               /* Fixed buffer */
+               return io_import_reg_buf(req, iter, addr + offset,
+                                        len - offset, direction, issue_flags);
+       }
+
+       return -EINVAL;
+}
+
 __bpf_kfunc_start_defs();
 __bpf_kfunc void uring_bpf_set_result(struct uring_bpf_data *data, int res)
 {
@@ -339,11 +446,91 @@ __bpf_kfunc struct io_kiocb *uring_bpf_data_to_req(struct uring_bpf_data *data)
 {
        return cmd_to_io_kiocb(data);
 }
+
+/**
+ * io_uring_bpf_req_memcpy - Copy data between io_uring BPF request buffers
+ * @data: BPF request data containing buffer metadata
+ * @dest: Destination buffer descriptor (with buf_id and offset)
+ * @src: Source buffer descriptor (with buf_id and offset)
+ * @len: Number of bytes to copy

io_uring/bpf.c __bpf_kfunc struct io_kiocb *uring_bpf_data_to_req(struct uring_bpf_data *data) · line 465

Quoted 10 lines
io_uring/bpf.c +10 -0
+ *
+ * Copies data between two different io_uring BPF request buffers (buf_id 1 and 2).
+ * Supports: plain-to-plain, fixed-to-plain, and plain-to-fixed.
+ * Does not support copying within the same buffer (src and dest must be different).
+ *
+ * Returns: Number of bytes copied on success, negative error code on failure
+ */
+__bpf_kfunc int io_uring_bpf_req_memcpy(struct uring_bpf_data *data,
+                                       struct bpf_req_mem_desc *dest,
+                                       struct bpf_req_mem_desc *src,

Curious, does struct bpf_req_mem_desc need to be registered anywhere for use as a kfunc argument? Or is any type automatically allowed to be used with a kfunc since BTF is generated for it?

Best,
Caleb

Quoted 102 lines
io_uring/bpf.c +63 -0
+                                       unsigned int len)
+{
+       struct io_kiocb *req = cmd_to_io_kiocb(data);
+       struct iov_iter dst_iter, src_iter;
+       u8 dst_type, src_type;
+       u64 dst_addr, src_addr;
+       u32 dst_len, src_len;
+       int ret;
+
+       /* Validate buffer IDs */
+       if (dest->buf_id < 1 || dest->buf_id > 2 ||
+           src->buf_id < 1 || src->buf_id > 2)
+               return -EINVAL;
+
+       /* Don't allow copying within the same buffer */
+       if (src->buf_id == dest->buf_id)
+               return -EINVAL;
+
+       /* Extract source buffer metadata */
+       if (src->buf_id == 1) {
+               src_type = IORING_BPF_BUF1_TYPE(data->opf);
+               src_addr = data->buf1_addr;
+               src_len = data->buf1_len;
+       } else {
+               src_type = IORING_BPF_BUF2_TYPE(data->opf);
+               src_addr = data->buf2_addr;
+               src_len = data->buf2_len;
+       }
+
+       /* Extract destination buffer metadata */
+       if (dest->buf_id == 1) {
+               dst_type = IORING_BPF_BUF1_TYPE(data->opf);
+               dst_addr = data->buf1_addr;
+               dst_len = data->buf1_len;
+       } else {
+               dst_type = IORING_BPF_BUF2_TYPE(data->opf);
+               dst_addr = data->buf2_addr;
+               dst_len = data->buf2_len;
+       }
+
+       /* Validate offsets and lengths */
+       if (src->offset + len > src_len || dest->offset + len > dst_len)
+               return -EINVAL;
+
+       /* Initialize source iterator */
+       ret = io_bpf_import_buffer(req, &src_iter, src_type,
+                                  src_addr, src->offset, src_len,
+                                  ITER_SOURCE, data->issue_flags);
+       if (ret)
+               return ret;
+
+       /* Initialize destination iterator */
+       ret = io_bpf_import_buffer(req, &dst_iter, dst_type,
+                                  dst_addr, dest->offset, dst_len,
+                                  ITER_DEST, data->issue_flags);
+       if (ret)
+               return ret;
+
+       /* Extract pages from source iterator and copy to destination */
+       return io_bpf_copy_iters(&src_iter, &dst_iter, len);
+}
+
 __bpf_kfunc_end_defs();
 
 BTF_KFUNCS_START(uring_bpf_kfuncs)
 BTF_ID_FLAGS(func, uring_bpf_set_result)
 BTF_ID_FLAGS(func, uring_bpf_data_to_req)
+BTF_ID_FLAGS(func, io_uring_bpf_req_memcpy)
 BTF_KFUNCS_END(uring_bpf_kfuncs)
 
 static const struct btf_kfunc_id_set uring_kfunc_set = {
io_uring/uring_bpf.h +10 -2
index c919931cb4b0..d6e0d6dff82e 100644
@@ -14,13 +14,22 @@ struct uring_bpf_data {
        /* Buffer 2 metadata - readable for bpf prog (plain only) */
        u64             buf2_addr;              /* buffer 2 address, bytes 24-31 */
        u32             buf2_len;               /* buffer 2 length, bytes 32-35 */
-       u32             __pad;                  /* padding, bytes 36-39 */
+       u32             issue_flags;            /* issue_flags from io_uring, bytes 36-39 */
 
        /* writeable for bpf prog */
        u8              pdu[64 - sizeof(struct file *) - 4 * sizeof(u32) -
                2 * sizeof(u64)];
 };
 
+/*
+ * Descriptor for io_uring BPF request buffer.
+ * Used by io_uring_bpf_req_memcpy() to identify which buffer to copy from/to.
+ */
+struct bpf_req_mem_desc {
+       u8              buf_id;         /* Buffer ID: 1 or 2 */
+       unsigned int    offset;         /* Offset into buffer */
+};
+
 typedef int (*uring_io_prep_t)(struct uring_bpf_data *data,
                               const struct io_uring_sqe *sqe);
 typedef int (*uring_io_issue_t)(struct uring_bpf_data *data);
--

2.47.0

#34 lore raw
In reply to Ming Lei
Quoted 3 lines · Ming Lei

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

BPF requests were tried long time ago and it wasn't great. Performance for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases were looking pretty ugly, internally, and for BPF writers as well.

I'm not so sure about your criteria, but my requirement was to at least being able to reuse all io_uring IO handling, i.e. submitting requests, and to wait/process completions, otherwise a lot of opportunities are wasted. My approach from a few months back [1] controlling requests from the outside was looking much better. At least it covered a bunch of needs without extra changes. I was just wiring up io_uring changes I wanted to make BPF writer lifes easier. Let me resend the bpf series with it.

It makes me wonder if they are complementary, but I'm not sure what your use cases are and what capabilities it might need.

[1] https://lore.kernel.org/io-uring/[email protected]/

Signature · 1 line
Pavel Begunkov
author
#35 lore raw
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Ming Lei

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

BPF requests were tried long time ago and it wasn't great. Performance

Care to share the link so I can learn from the lesson? Maybe things have changed now...

Quoted 2 lines

for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases

What is the overhead? In this patch, OP's prep() and issue() are defined in bpf prog, but in typical use case, the code size is pretty small, and bpf prog code is supposed to run in fast path.

Quoted 1 line

were looking pretty ugly, internally, and for BPF writers as well.

I am not sure what simple use cases you are talking about.

Quoted 5 lines

I'm not so sure about your criteria, but my requirement was to at least being able to reuse all io_uring IO handling, i.e. submitting requests, and to wait/process completions, otherwise a lot of opportunities are wasted. My approach from a few months back [1] controlling requests from

Please read the patchset.

This patchset defines new IORING_BPF_OP code, which's ->prep(), ->issue(), ..., are hooked with struct_ops prog, so all io_uring core code is used, just the exact IORING_BPF_OP behavior is defined by struct_ops prog.

Quoted 5 lines

the outside was looking much better. At least it covered a bunch of needs without extra changes. I was just wiring up io_uring changes I wanted to make BPF writer lifes easier. Let me resend the bpf series with it.

It makes me wonder if they are complementary, but I'm not sure what

I think the two are orthogonal in function, and they can co-exist.

Quoted 1 line

your use cases are and what capabilities it might need.

The main use cases are described in cover letter and the 3rd patch, please find the details there.

So far the main case is to access the registered (kernel)buffer from issue() callback of struct_ops, because the buffer doesn't have userspace mapping. The last two patches adds support to provide two buffers(fixed, plain) for IORING_BPF_OP, and in future vectored buffer will be added too, so IORING_BPF_OP can handle buffer flexibly, such as:

- use exported compress kfunc to compress data from kernel buffer into another buffer or inplace, then the following linked SQE can be submitted to write the built compressed data into storage

- in raid use case, calculate IO data parity from kernel buffer, and store the parity data to another plain user buffer, then the following linked SQE can be submitted to write the built parity data to storage

Even for userspace buffer, the BPF_OP can support similar handling for saving one extra io_uring_enter() syscall.

Quoted 2 lines

[1] https://lore.kernel.org/io-uring/[email protected]/

I looked at your patches, in which SQE is generated in bpf prog(kernel), and it can't be used in my case.

Thanks,
Ming

#36 lore raw
Quoted 10 lines · Ming Lei
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Ming Lei

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

BPF requests were tried long time ago and it wasn't great. Performance

Care to share the link so I can learn from the lesson? Maybe things have changed now...

https://lore.kernel.org/io-uring/[email protected]/T/#m31d0a2ac6e2213f912a200f5e8d88bd74f81406b

There were some extra features and testing from folks, but I don't think it was ever posted to the list.

Quoted 4 lines
Quoted 2 lines

for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases

What is the overhead? In this patch, OP's prep() and issue() are defined in

The overhead of creating, freeing and executing a request. If you use it with links, it's also overhead of that. That prototype could also optionally wait for completions, and it wasn't free either.

Quoted 5 lines

bpf prog, but in typical use case, the code size is pretty small, and bpf prog code is supposed to run in fast path.>

Quoted 1 line

were looking pretty ugly, internally, and for BPF writers as well.

I am not sure what simple use cases you are talking about.

As an example, creating a loop reading a file:
read N bytes; wait for completion; repeat

Quoted 10 lines
Quoted 4 lines

I'm not so sure about your criteria, but my requirement was to at least being able to reuse all io_uring IO handling, i.e. submitting requests, and to wait/process completions, otherwise a lot of opportunities are wasted. My approach from a few months back [1] controlling requests from

Please read the patchset.

This patchset defines new IORING_BPF_OP code, which's ->prep(), ->issue(), ..., are hooked with struct_ops prog, so all io_uring core code is used, just the exact IORING_BPF_OP behavior is defined by struct_ops prog.

Right, but I'm talking about what the io_uring BPF program is capable of doing.

Quoted 23 lines
Quoted 5 lines

the outside was looking much better. At least it covered a bunch of needs without extra changes. I was just wiring up io_uring changes I wanted to make BPF writer lifes easier. Let me resend the bpf series with it.

It makes me wonder if they are complementary, but I'm not sure what

I think the two are orthogonal in function, and they can co-exist.

Quoted 1 line

your use cases are and what capabilities it might need.

The main use cases are described in cover letter and the 3rd patch, please find the details there.

So far the main case is to access the registered (kernel)buffer from issue() callback of struct_ops, because the buffer doesn't have userspace mapping. The last two patches adds support to provide two buffers(fixed, plain) for IORING_BPF_OP, and in future vectored buffer will be added too, so IORING_BPF_OP can handle buffer flexibly, such as:

- use exported compress kfunc to compress data from kernel buffer into another buffer or inplace, then the following linked SQE can be submitted to write the built compressed data into storage

Quoted 6 lines

- in raid use case, calculate IO data parity from kernel buffer, and store the parity data to another plain user buffer, then the following linked SQE can be submitted to write the built parity data to storage

Even for userspace buffer, the BPF_OP can support similar handling for saving one extra io_uring_enter() syscall.

Sure, registered buffer handling was one of the use cases for that recent re-itarations as well, and David Wei had some thoughts for it as well. Though, it was not exactly about copying.

Quoted 3 lines
Quoted 1 line

[1] https://lore.kernel.org/io-uring/[email protected]/

I looked at your patches, in which SQE is generated in bpf prog(kernel),

Quick note: userspace and BPF are both allowed to submit requests / generate SQEs.

Quoted 1 line

and it can't be used in my case.

Hmm, how so? Let's say ublk registers a buffer and posts a completion. Then BPF runs, it sees the completion and does the necessary processing, probably using some kfuncs like the ones you introduced. After it can optionally queue up requests writing it to the storage or anything else.

The reason I'm asking is because it's supposed to be able to do anything the userspace can already achieve (and more). So, if it can't be used for this use cases, there should be some problem in my design.

Signature · 1 line
Pavel Begunkov
author
#37 lore raw
Quoted 9 lines · Pavel Begunkov
Quoted 8 lines · Ming Lei
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Ming Lei

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

BPF requests were tried long time ago and it wasn't great. Performance

Quoted 7 lines
Quoted 2 lines

Care to share the link so I can learn from the lesson? Maybe things have changed now...

https://lore.kernel.org/io-uring/[email protected]/T/#m31d0a2ac6e2213f912a200f5e8d88bd74f81406b

There were some extra features and testing from folks, but I don't think it was ever posted to the list.

Thanks for sharing the link:

```
The main problem solved is feeding completion information of other
requests in a form of CQEs back into BPF. I decided to wire up support
for multiple completion queues (aka CQs) and give BPF programs access to
them, so leaving userspace in control over synchronisation that should
be much more flexible that the link-based approach.
```

Looks it is totally different with my patch in motivation and policy.

I do _not_ want to move application logic into kernel by building SQE from kernel prog. With IORING_OP_BPF, the whole io_uring application is built & maintained completely in userspace, so I needn't to do cumbersome kernel/user communication just for setting up one SQE in prog, not mention maintaining SQE's relation with userspace side's.

Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases

What is the overhead? In this patch, OP's prep() and issue() are defined in

The overhead of creating, freeing and executing a request. If you use it with links, it's also overhead of that. That prototype could also optionally wait for completions, and it wasn't free either.

IORING_OP_BPF is same with existing normal io_uring request and link, wrt all above you mentioned.

IORING_OP_BPF's motivation is for being io_uring's supplementary or extention in function, not for improving performance.

Quoted 9 lines
Quoted 6 lines

bpf prog, but in typical use case, the code size is pretty small, and bpf prog code is supposed to run in fast path.>

Quoted 1 line

were looking pretty ugly, internally, and for BPF writers as well.

I am not sure what simple use cases you are talking about.

As an example, creating a loop reading a file:
read N bytes; wait for completion; repeat

IORING_OP_BPF isn't supposed to implement FS operation in bpf prog.

It doesn't mean IORING_OP_BPF can't support async issuing:

- issue_wait() can be added for offload in io-wq context

OR

- for typical FS AIO, in theory it can be supported too, just the struct_ops need to define one completion callback, and the callback can be called from ->ki_complete().

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

I'm not so sure about your criteria, but my requirement was to at least being able to reuse all io_uring IO handling, i.e. submitting requests, and to wait/process completions, otherwise a lot of opportunities are wasted. My approach from a few months back [1] controlling requests from

Quoted 8 lines
Quoted 5 lines

Please read the patchset.

This patchset defines new IORING_BPF_OP code, which's ->prep(), ->issue(), ..., are hooked with struct_ops prog, so all io_uring core code is used, just the exact IORING_BPF_OP behavior is defined by struct_ops prog.

Right, but I'm talking about what the io_uring BPF program is capable of doing.

There can be many types of io_uring BPF progs from function viewpoint, we are not talking about same type.

Quoted 39 lines
Quoted 24 lines
Quoted 6 lines

the outside was looking much better. At least it covered a bunch of needs without extra changes. I was just wiring up io_uring changes I wanted to make BPF writer lifes easier. Let me resend the bpf series with it.

It makes me wonder if they are complementary, but I'm not sure what

I think the two are orthogonal in function, and they can co-exist.

Quoted 1 line

your use cases are and what capabilities it might need.

The main use cases are described in cover letter and the 3rd patch, please find the details there.

So far the main case is to access the registered (kernel)buffer from issue() callback of struct_ops, because the buffer doesn't have userspace mapping. The last two patches adds support to provide two buffers(fixed, plain) for IORING_BPF_OP, and in future vectored buffer will be added too, so IORING_BPF_OP can handle buffer flexibly, such as:

- use exported compress kfunc to compress data from kernel buffer into another buffer or inplace, then the following linked SQE can be submitted to write the built compressed data into storage

Quoted 6 lines

- in raid use case, calculate IO data parity from kernel buffer, and store the parity data to another plain user buffer, then the following linked SQE can be submitted to write the built parity data to storage

Even for userspace buffer, the BPF_OP can support similar handling for saving one extra io_uring_enter() syscall.

Sure, registered buffer handling was one of the use cases for that recent re-itarations as well, and David Wei had some thoughts for it as well. Though, it was not exactly about copying.

Quoted 3 lines
Quoted 1 line

[1] https://lore.kernel.org/io-uring/[email protected]/

I looked at your patches, in which SQE is generated in bpf prog(kernel),

Quoted 7 lines

Quick note: userspace and BPF are both allowed to submit requests / generate SQEs.

Quoted 1 line

and it can't be used in my case.

Hmm, how so? Let's say ublk registers a buffer and posts a completion. Then BPF runs, it sees the completion and does the necessary processing, probably using some kfuncs like the ones

It is easy to say, how can the BPF prog know the next completion is exactly waiting for? You have to rely on bpf map to communicate with userspace to understanding what completion is what you are interested in, also need all information from userpace for preparing the SQE for submission from bpf prog. Tons of userspace and kernel communication.

Quoted 2 lines

you introduced. After it can optionally queue up requests writing it to the storage or anything else.

Again, I do not want to move userspace logic into bpf prog(kernel), what IORING_BPF_OP provides is to define one operation, then userspace can use it just like in-kernel operations.

Then existing application can apply IORING_BPF_OP just with little small change. If submitting SQE from bpf prog, ublk application need re-write for supporting register buffer based zero copy.

Quoted 4 lines

The reason I'm asking is because it's supposed to be able to do anything the userspace can already achieve (and more). So, if it can't be used for this use cases, there should be some problem in my design.

BPF prog programming is definitely much more limited compared with userspace application because it is safe kernel programming.

Thanks,
Ming

#38 lore raw
Quoted 20 lines · Ming Lei
Quoted 9 lines · Pavel Begunkov
Quoted 8 lines · Ming Lei
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Ming Lei

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

BPF requests were tried long time ago and it wasn't great. Performance

Quoted 7 lines
Quoted 2 lines

Care to share the link so I can learn from the lesson? Maybe things have changed now...

https://lore.kernel.org/io-uring/[email protected]/T/#m31d0a2ac6e2213f912a200f5e8d88bd74f81406b

There were some extra features and testing from folks, but I don't think it was ever posted to the list.

Thanks for sharing the link:

Quoted 7 lines
```
The main problem solved is feeding completion information of other
requests in a form of CQEs back into BPF. I decided to wire up support
for multiple completion queues (aka CQs) and give BPF programs access to
them, so leaving userspace in control over synchronisation that should
be much more flexible that the link-based approach.
```

FWIW, and those extensions were the sign telling that the approach wasn't flexible enough.

Quoted 7 lines

Looks it is totally different with my patch in motivation and policy.

I do _not_ want to move application logic into kernel by building SQE from kernel prog. With IORING_OP_BPF, the whole io_uring application is built & maintained completely in userspace, so I needn't to do cumbersome kernel/user communication just for setting up one SQE in prog, not mention maintaining SQE's relation with userspace side's.

It's built and maintained in userspace in either case, and in both cases you have bpf implementing some logic that was previously done in userspace. To emphasize, you can do the desired parts of handling in BPF, and I'm not suggesting moving the entirety of request processing in there.

Quoted 11 lines
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases

What is the overhead? In this patch, OP's prep() and issue() are defined in

The overhead of creating, freeing and executing a request. If you use it with links, it's also overhead of that. That prototype could also optionally wait for completions, and it wasn't free either.

IORING_OP_BPF is same with existing normal io_uring request and link, wrt all above you mentioned.

It is, but it's an extra request, and in previous testing overhead for that extra request was affecting total performance, that's why linking or not is also important.

Quoted 17 lines

IORING_OP_BPF's motivation is for being io_uring's supplementary or extention in function, not for improving performance.

Quoted 9 lines
Quoted 6 lines

bpf prog, but in typical use case, the code size is pretty small, and bpf prog code is supposed to run in fast path.>

Quoted 1 line

were looking pretty ugly, internally, and for BPF writers as well.

I am not sure what simple use cases you are talking about.

As an example, creating a loop reading a file:
read N bytes; wait for completion; repeat

IORING_OP_BPF isn't supposed to implement FS operation in bpf prog.

It doesn't mean IORING_OP_BPF can't support async issuing:

Quoted 7 lines

- issue_wait() can be added for offload in io-wq context

OR

- for typical FS AIO, in theory it can be supported too, just the struct_ops need to define one completion callback, and the callback can be called from ->ki_complete().

There is more to IO than read/write, and I'm afraid each new type of operation would need some extra kfunc glue. And even then there is enough of handling for rw requests in io_uring than just calling the callback. It's nicer to be able to reuse all io_uring request handling, which wouldn't even need extra kfuncs.

...

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

and it can't be used in my case.

Hmm, how so? Let's say ublk registers a buffer and posts a completion. Then BPF runs, it sees the completion and does the necessary processing, probably using some kfuncs like the ones

It is easy to say, how can the BPF prog know the next completion is exactly waiting for? You have to rely on bpf map to communicate with userspace

By taking a peek at and maybe dereferencing cqe->user_data.

Quoted 3 lines

to understanding what completion is what you are interested in, also need all information from userpace for preparing the SQE for submission from bpf prog. Tons of userspace and kernel communication.

You can setup a BPF arena, and all that comm will be working with a block of shared memory. Or same but via io_uring parameter region. That sounds pretty simple.

Quoted 6 lines
Quoted 2 lines

you introduced. After it can optionally queue up requests writing it to the storage or anything else.

Again, I do not want to move userspace logic into bpf prog(kernel), what IORING_BPF_OP provides is to define one operation, then userspace can use it just like in-kernel operations.

Right, but that's rather limited. I want to cover all those use cases with one implementation instead of fragmenting users, if that can be achieved.

Quoted 11 lines

Then existing application can apply IORING_BPF_OP just with little small change. If submitting SQE from bpf prog, ublk application need re-write for supporting register buffer based zero copy.

Quoted 4 lines

The reason I'm asking is because it's supposed to be able to do anything the userspace can already achieve (and more). So, if it can't be used for this use cases, there should be some problem in my design.

BPF prog programming is definitely much more limited compared with userspace application because it is safe kernel programming.

Signature · 1 line
Pavel Begunkov
author
#39 lore raw
Quoted 34 lines · Pavel Begunkov
Quoted 20 lines · Ming Lei
Quoted 9 lines · Pavel Begunkov
Quoted 8 lines · Ming Lei
Quoted 6 lines · Pavel Begunkov
Quoted 3 lines · Ming Lei

Hello,

Add IORING_OP_BPF for extending io_uring operations, follows typical cases:

BPF requests were tried long time ago and it wasn't great. Performance

Quoted 7 lines
Quoted 2 lines

Care to share the link so I can learn from the lesson? Maybe things have changed now...

https://lore.kernel.org/io-uring/[email protected]/T/#m31d0a2ac6e2213f912a200f5e8d88bd74f81406b

There were some extra features and testing from folks, but I don't think it was ever posted to the list.

Thanks for sharing the link:

Quoted 7 lines
```
The main problem solved is feeding completion information of other
requests in a form of CQEs back into BPF. I decided to wire up support
for multiple completion queues (aka CQs) and give BPF programs access to
them, so leaving userspace in control over synchronisation that should
be much more flexible that the link-based approach.
```

FWIW, and those extensions were the sign telling that the approach wasn't flexible enough.

Quoted 2 lines

Looks it is totally different with my patch in motivation and policy.

Quoted 7 lines
Quoted 5 lines

I do _not_ want to move application logic into kernel by building SQE from kernel prog. With IORING_OP_BPF, the whole io_uring application is built & maintained completely in userspace, so I needn't to do cumbersome kernel/user communication just for setting up one SQE in prog, not mention maintaining SQE's relation with userspace side's.

It's built and maintained in userspace in either case, and in

No.

BPF prog is not userspace, it is definitely kernel stuff, but it belongs to application scope.

Quoted 4 lines

both cases you have bpf implementing some logic that was previously done in userspace. To emphasize, you can do the desired parts of handling in BPF, and I'm not suggesting moving the entirety of request processing in there.

The problem with your patch is that SQE is built in bpf prog(kernel), then inevitable application logic is moved to bpf prog, which isn't good at handling complicated logic.

Then people have to run kernel<->user communication for setting up the SQE.

And the SQE in bpf prog may need to be linked with previous and following SQEs in usersapce, which basically partitions application logic into two parts: one is in userspace, another is in bpf prog(kernel).

The patch I am suggesting doesn't have this problem, all SQEs are built in userspace, and just the minimized part(standalone and well defined function) is done in bpf prog.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases

What is the overhead? In this patch, OP's prep() and issue() are defined in

The overhead of creating, freeing and executing a request. If you use it with links, it's also overhead of that. That prototype could also optionally wait for completions, and it wasn't free either.

Quoted 6 lines
Quoted 2 lines

IORING_OP_BPF is same with existing normal io_uring request and link, wrt all above you mentioned.

It is, but it's an extra request, and in previous testing overhead for that extra request was affecting total performance, that's why linking or not is also important.

Yes, but does the extra request matters for whole performance?

I did have such test:

1) in tools/testing/selftests/ublk/null.c

- for zero copy test, one extra nop is submitted

2) rublk test

- for zero copy test, it simply returns without submitting nop

The IOPS gap is pretty small.

Also in your approach, without allocating one new SQE in bpf, how to provide generic interface for bpf prog to work on different functions, such as, memory copy or raid5 parity or compression ..., all require flexible handling, such as, variable parameters, buffer could be plain user memory , fixed, vectored or fixed vectored,..., so one SQE or new operation is the easiest way for providing the abstraction and generic bpf prog interface.

Quoted 26 lines
Quoted 18 lines

IORING_OP_BPF's motivation is for being io_uring's supplementary or extention in function, not for improving performance.

Quoted 9 lines
Quoted 6 lines

bpf prog, but in typical use case, the code size is pretty small, and bpf prog code is supposed to run in fast path.>

Quoted 1 line

were looking pretty ugly, internally, and for BPF writers as well.

I am not sure what simple use cases you are talking about.

As an example, creating a loop reading a file:
read N bytes; wait for completion; repeat

IORING_OP_BPF isn't supposed to implement FS operation in bpf prog.

It doesn't mean IORING_OP_BPF can't support async issuing:

Quoted 7 lines

- issue_wait() can be added for offload in io-wq context

OR

- for typical FS AIO, in theory it can be supported too, just the struct_ops need to define one completion callback, and the callback can be called from ->ki_complete().

Quoted 5 lines

There is more to IO than read/write, and I'm afraid each new type of operation would need some extra kfunc glue. And even then there is enough of handling for rw requests in io_uring than just calling the callback. It's nicer to be able to reuse all io_uring request handling, which wouldn't even need extra kfuncs.

Looks you are trying to propose generic bpf io_uring request, which is ambitious goal, :-)

But that isn't my patchset's motivation, which just serves as supplement or extention of existing io_uring.

Another big case could be network IO, which could be covered -EAGAIN, or other main cases?

Quoted 11 lines

...

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

and it can't be used in my case.

Hmm, how so? Let's say ublk registers a buffer and posts a completion. Then BPF runs, it sees the completion and does the necessary processing, probably using some kfuncs like the ones

It is easy to say, how can the BPF prog know the next completion is exactly waiting for? You have to rely on bpf map to communicate with userspace

By taking a peek at and maybe dereferencing cqe->user_data.

Yes, but you have to pass the interested ->user_data to bpf prog first.

There could be many inflight interested IOs, how to query them efficiently?

Scan each one after every CQE is posted? But ebpf just support bound loops, the complexity may be run out of easily[1].

https://docs.ebpf.io/linux/concepts/loops/

Quoted 8 lines
Quoted 4 lines

to understanding what completion is what you are interested in, also need all information from userpace for preparing the SQE for submission from bpf prog. Tons of userspace and kernel communication.

You can setup a BPF arena, and all that comm will be working with a block of shared memory. Or same but via io_uring parameter region. That sounds pretty simple.

But application logic has to splitted into two parts, both two have to rely on the shared memory to communicate.

The exiting io_uring application has been complicated enough, adding one extra shared memory communication for holding application logic just makes things worse. Even in userspace programming, it is horrible to model logic into data, that is why state machine pattern is usually not readable.

Think about writing high performance raid5 application based on ublk zero copy & io_uring, for example, handling one simple write:

- one ublk write command comes for raid5

- suppose the command just writes data to one single stripe exactly

- submitting each write to N - 1 disks

- When all N writes are done, the new SQE needs to work:

- calculate parity by reading buffers from the N request kernel buffer and writing resulted XOR parity to one user specified buffer

- then new FS IO need to be submitted to write the parity data to one calculated disk(N)

So the involved things for bpf prog SQE:

- monitoring N - 1 writes
- do the parity calculation job, which has to define one kfunc
- mark parity is ready & notify userspace for writing parity(how to notify?)

Now there can be variable(many) such WRITEs to handle concurrently, and the bpf prog has to cover them all.

The above just the simplest case, the write command may not align with stripe, so parity calculation may need to read data from other stripes.

If you think it is pretty simple, care to provide one example to show your approach is workable?

Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

you introduced. After it can optionally queue up requests writing it to the storage or anything else.

Again, I do not want to move userspace logic into bpf prog(kernel), what IORING_BPF_OP provides is to define one operation, then userspace can use it just like in-kernel operations.

Right, but that's rather limited. I want to cover all those use cases with one implementation instead of fragmenting users, if that can be achieved.

I don't know when your ambitious plan can land or be doable.

I am going to write V2 with the approach of IORING_BPF_OP which is at least workable for some cases, and much easier to take in userspace. Also it doesn't conflict with your approach.

Thanks,
Ming

#40 lore raw

Hey Ming,

Sorry for a late reply

On 11/13/25 04:18, Ming Lei wrote:
...

Quoted 6 lines
Quoted 4 lines

both cases you have bpf implementing some logic that was previously done in userspace. To emphasize, you can do the desired parts of handling in BPF, and I'm not suggesting moving the entirety of request processing in there.

The problem with your patch is that SQE is built in bpf prog(kernel), then

It's an option, not a requirement. It should be perfectly fine, for example, to only process CQEs and run some kfuncs, and return back to userspace.

Quoted 8 lines

inevitable application logic is moved to bpf prog, which isn't good at handling complicated logic.

Then people have to run kernel<->user communication for setting up the SQE.

And the SQE in bpf prog may need to be linked with previous and following SQEs in usersapce, which basically partitions application logic into two parts: one is in userspace, another is in bpf prog(kernel).

I'm not a huge fan of links. They add enough of complexity to the kernel. I'd rather see them gone / sidelined out of the normal execution paths if there is an alternative.

Quoted 14 lines

The patch I am suggesting doesn't have this problem, all SQEs are built in userspace, and just the minimized part(standalone and well defined function) is done in bpf prog.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines

for short BPF programs is not great because of io_uring request handling overhead. And flexibility was severely lacking, so even simple use cases

What is the overhead? In this patch, OP's prep() and issue() are defined in

The overhead of creating, freeing and executing a request. If you use it with links, it's also overhead of that. That prototype could also optionally wait for completions, and it wasn't free either.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

IORING_OP_BPF is same with existing normal io_uring request and link, wrt all above you mentioned.

It is, but it's an extra request, and in previous testing overhead for that extra request was affecting total performance, that's why linking or not is also important.

Yes, but does the extra request matters for whole performance?

It did in previous tests with small pre-buffered IO, but that depends on how well ammortised it is with other requests and BPF execution.

Quoted 10 lines

I did have such test:

1) in tools/testing/selftests/ublk/null.c

- for zero copy test, one extra nop is submitted

2) rublk test

- for zero copy test, it simply returns without submitting nop

Quoted 8 lines

The IOPS gap is pretty small.

Also in your approach, without allocating one new SQE in bpf, how to provide generic interface for bpf prog to work on different functions, such as, memory copy or raid5 parity or compression ..., all require flexible handling, such as, variable parameters, buffer could be plain user memory , fixed, vectored or fixed vectored,..., so one SQE or new operation is the easiest way for providing the abstraction and generic bpf prog interface.

Or it can be a kfunc

...

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

It is easy to say, how can the BPF prog know the next completion is exactly waiting for? You have to rely on bpf map to communicate with userspace

By taking a peek at and maybe dereferencing cqe->user_data.

Yes, but you have to pass the interested ->user_data to bpf prog first.

It can be looked it up from the CQ

Quoted 6 lines

There could be many inflight interested IOs, how to query them efficiently?

Scan each one after every CQE is posted? But ebpf just support bound loops, the complexity may be run out of easily[1].

https://docs.ebpf.io/linux/concepts/loops/

Good point, I need to take a look at the looping.

Quoted 33 lines
Quoted 7 lines
Quoted 3 lines

to understanding what completion is what you are interested in, also need all information from userpace for preparing the SQE for submission from bpf prog. Tons of userspace and kernel communication.

You can setup a BPF arena, and all that comm will be working with a block of shared memory. Or same but via io_uring parameter region. That sounds pretty simple.

But application logic has to splitted into two parts, both two have to rely on the shared memory to communicate.

The exiting io_uring application has been complicated enough, adding one extra shared memory communication for holding application logic just makes things worse. Even in userspace programming, it is horrible to model logic into data, that is why state machine pattern is usually not readable.

Think about writing high performance raid5 application based on ublk zero copy & io_uring, for example, handling one simple write:

- one ublk write command comes for raid5

- suppose the command just writes data to one single stripe exactly

- submitting each write to N - 1 disks

- When all N writes are done, the new SQE needs to work:

- calculate parity by reading buffers from the N request kernel buffer and writing resulted XOR parity to one user specified buffer

- then new FS IO need to be submitted to write the parity data to one calculated disk(N)

Quoted 6 lines

So the involved things for bpf prog SQE:

- monitoring N - 1 writes
- do the parity calculation job, which has to define one kfunc
- mark parity is ready & notify userspace for writing parity(how to notify?)

And something still needs to do all that. The only silver lining for userspace handling is that there is more language sugar helping with it like coroutines.

Quoted 26 lines

Now there can be variable(many) such WRITEs to handle concurrently, and the bpf prog has to cover them all.

The above just the simplest case, the write command may not align with stripe, so parity calculation may need to read data from other stripes.

If you think it is pretty simple, care to provide one example to show your approach is workable?

Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

you introduced. After it can optionally queue up requests writing it to the storage or anything else.

Again, I do not want to move userspace logic into bpf prog(kernel), what IORING_BPF_OP provides is to define one operation, then userspace can use it just like in-kernel operations.

Right, but that's rather limited. I want to cover all those use cases with one implementation instead of fragmenting users, if that can be achieved.

I don't know when your ambitious plan can land or be doable.

I am going to write V2 with the approach of IORING_BPF_OP which is at least workable for some cases, and much easier to take in userspace. Also it doesn't conflict with your approach.

Signature · 1 line
Pavel Begunkov