Skip to contentSkip to conversation

ublk: support bpf

RFC PATCH In review Ming Lei 3 participants 28 messages Jan 7, 2025 – Jan 15, 2025

Subsystems (5)
lore
Share & tools
lore

Ming Lei

author maintainer
#0 lore raw

cover ublk: support bpf

Hello,

Patch 1~6 cleans up & prepares for supporting ublk-bpf, which should be ready to go.

Patch 7~14 supports ublk-bpf over struct_ops and selftests code. And please see detailed motivation in commit log of "ublk: bpf: add bpf struct_ops" and the last document patch.

Patch 15~21 adds bpf aio over struct_ops and applies it for ublk-bpf, and selftests code.

Patch 22 adds document for ublk-bpf.

Git tree:

https://github.com/ming1/linux.git ublk_bpf_rfc
https://github.com/ming1/linux/commits/ublk_bpf_rfc/

Kernel selftest:

make -C tools/testing/selftests TARGETS=ublk run_tests

Comments are welcome!

Ming Lei (22):
ublk: remove two unused fields from 'struct ublk_queue'
ublk: convert several bool type fields into bitfield of ublk_queue
ublk: add helper of ublk_need_map_io()
ublk: move ublk into one standalone directory
ublk: move private definitions into private header
ublk: move several helpers to private header
ublk: bpf: add bpf prog attach helpers
ublk: bpf: add bpf struct_ops
ublk: bpf: attach bpf prog to ublk device
ublk: bpf: add kfunc for ublk bpf prog
ublk: bpf: enable ublk-bpf
selftests: ublk: add tests for the ublk-bpf initial implementation
selftests: ublk: add tests for covering io split
selftests: ublk: add tests for covering redirecting to userspace
ublk: bpf: add bpf aio kfunc
ublk: bpf: add bpf aio struct_ops
ublk: bpf: attach bpf aio prog to ublk device
ublk: bpf: add several ublk bpf aio kfuncs
ublk: bpf: wire bpf aio with ublk io handling
selftests: add tests for ublk bpf aio
selftests: add tests for covering both bpf aio and split
ublk: document ublk-bpf & bpf-aio

 Documentation/block/ublk.rst                  |  170 ++
 MAINTAINERS                                   |    3 +-
 drivers/block/Kconfig                         |   32 +-
 drivers/block/Makefile                        |    2 +-
 drivers/block/ublk/Kconfig                    |   52 +
 drivers/block/ublk/Makefile                   |   10 +
 drivers/block/ublk/bpf.c                      |  370 ++++
 drivers/block/ublk/bpf.h                      |  231 +++
 drivers/block/ublk/bpf_aio.c                  |  266 +++
 drivers/block/ublk/bpf_aio.h                  |  118 ++
 drivers/block/ublk/bpf_aio_ops.c              |  174 ++
 drivers/block/ublk/bpf_ops.c                  |  344 ++++
 drivers/block/ublk/bpf_reg.h                  |   77 +
 drivers/block/{ublk_drv.c => ublk/main.c}     |  267 +--
 drivers/block/ublk/ublk.h                     |  237 +++
 include/uapi/linux/ublk_cmd.h                 |   16 +-
 tools/testing/selftests/Makefile              |    1 +
 tools/testing/selftests/ublk/.gitignore       |    4 +
 tools/testing/selftests/ublk/Makefile         |  236 +++
 tools/testing/selftests/ublk/config           |    2 +
 tools/testing/selftests/ublk/progs/ublk_bpf.h |   13 +
 .../selftests/ublk/progs/ublk_bpf_kfunc.h     |   44 +
 .../testing/selftests/ublk/progs/ublk_loop.c  |  166 ++
 .../testing/selftests/ublk/progs/ublk_null.c  |  177 ++
 .../selftests/ublk/progs/ublk_stripe.c        |  319 ++++
 tools/testing/selftests/ublk/test_common.sh   |  119 ++
 tools/testing/selftests/ublk/test_loop_01.sh  |   33 +
 tools/testing/selftests/ublk/test_loop_02.sh  |   24 +
 tools/testing/selftests/ublk/test_null_01.sh  |   19 +
 tools/testing/selftests/ublk/test_null_02.sh  |   23 +
 tools/testing/selftests/ublk/test_null_03.sh  |   21 +
 tools/testing/selftests/ublk/test_null_04.sh  |   21 +
 .../testing/selftests/ublk/test_stripe_01.sh  |   35 +
 .../testing/selftests/ublk/test_stripe_02.sh  |   26 +
 tools/testing/selftests/ublk/ublk_bpf.c       | 1673 +++++++++++++++++
 35 files changed, 5101 insertions(+), 224 deletions(-)
 create mode 100644 drivers/block/ublk/Kconfig
 create mode 100644 drivers/block/ublk/Makefile
 create mode 100644 drivers/block/ublk/bpf.c
 create mode 100644 drivers/block/ublk/bpf.h
 create mode 100644 drivers/block/ublk/bpf_aio.c
 create mode 100644 drivers/block/ublk/bpf_aio.h
 create mode 100644 drivers/block/ublk/bpf_aio_ops.c
 create mode 100644 drivers/block/ublk/bpf_ops.c
 create mode 100644 drivers/block/ublk/bpf_reg.h
 rename drivers/block/{ublk_drv.c => ublk/main.c} (93%)
 create mode 100644 drivers/block/ublk/ublk.h
 create mode 100644 tools/testing/selftests/ublk/.gitignore
 create mode 100644 tools/testing/selftests/ublk/Makefile
 create mode 100644 tools/testing/selftests/ublk/config
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_bpf.h
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_bpf_kfunc.h
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_loop.c
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_null.c
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_stripe.c
 create mode 100755 tools/testing/selftests/ublk/test_common.sh
 create mode 100755 tools/testing/selftests/ublk/test_loop_01.sh
 create mode 100755 tools/testing/selftests/ublk/test_loop_02.sh
 create mode 100755 tools/testing/selftests/ublk/test_null_01.sh
 create mode 100755 tools/testing/selftests/ublk/test_null_02.sh
 create mode 100755 tools/testing/selftests/ublk/test_null_03.sh
 create mode 100755 tools/testing/selftests/ublk/test_null_04.sh
 create mode 100755 tools/testing/selftests/ublk/test_stripe_01.sh
 create mode 100755 tools/testing/selftests/ublk/test_stripe_02.sh
 create mode 100644 tools/testing/selftests/ublk/ublk_bpf.c
Signature · 1 line
2.47.0
author maintainer
#1 lore raw

RFC PATCH 1/22 ublk: remove two unused fields from 'struct ublk_queue'

Remove two unused fields(io_addr & max_io_sz) from struct ublk_queue.

---
 drivers/block/ublk_drv.c | 2 --
 1 file changed, 2 deletions(-)
drivers/block/ublk_drv.c +0 -2
index 934ab9332c80..77ce3231eba4 100644
@@ -143,8 +143,6 @@ struct ublk_queue {
 
 	struct llist_head	io_cmds;
 
-	unsigned long io_addr;	/* mapped vm address */
-	unsigned int max_io_sz;
 	bool force_abort;
 	bool timeout;
 	bool canceling;
Signature · 1 line
2.47.0
author maintainer
#2 lore raw

RFC PATCH 2/22 ublk: convert several bool type fields into bitfield of `ublk_queue`

In reply to Ming Lei

Convert several bool type fields into bitfields of ublk_queue, so that we can remove one padding and save one 4 bytes in ublk_queue.

---
 drivers/block/ublk_drv.c | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)
drivers/block/ublk_drv.c +11 -11
index 77ce3231eba4..00363e8affc6 100644
@@ -143,10 +143,10 @@ struct ublk_queue {
 
 	struct llist_head	io_cmds;
 
-	bool force_abort;
-	bool timeout;
-	bool canceling;
-	bool fail_io; /* copy of dev->state == UBLK_S_DEV_FAIL_IO */
+	unsigned short force_abort:1;
+	unsigned short timeout:1;
+	unsigned short canceling:1;
+	unsigned short fail_io:1; /* copy of dev->state == UBLK_S_DEV_FAIL_IO */
 	unsigned short nr_io_ready;	/* how many ios setup */
 	spinlock_t		cancel_lock;
 	struct ublk_device *dev;
@@ -1257,7 +1257,7 @@ static enum blk_eh_timer_return ublk_timeout(struct request *rq)
 	if (ubq->flags & UBLK_F_UNPRIVILEGED_DEV) {
 		if (!ubq->timeout) {
 			send_sig(SIGKILL, ubq->ubq_daemon, 0);
-			ubq->timeout = true;
+			ubq->timeout = 1;
 		}
 
 		return BLK_EH_DONE;
@@ -1459,7 +1459,7 @@ static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq)
 		spin_unlock(&ubq->cancel_lock);
 		return false;
 	}
-	ubq->canceling = true;
+	ubq->canceling = 1;
 	spin_unlock(&ubq->cancel_lock);
 
 	spin_lock(&ub->lock);
@@ -1609,7 +1609,7 @@ static void ublk_unquiesce_dev(struct ublk_device *ub)
 	 * can move on.
 	 */
 	for (i = 0; i < ub->dev_info.nr_hw_queues; i++)
-		ublk_get_queue(ub, i)->force_abort = true;
+		ublk_get_queue(ub, i)->force_abort = 1;
 
 	blk_mq_unquiesce_queue(ub->ub_disk->queue);
 	/* We may have requeued some rqs in ublk_quiesce_queue() */
@@ -1672,7 +1672,7 @@ static void ublk_nosrv_work(struct work_struct *work)
 		blk_mq_quiesce_queue(ub->ub_disk->queue);
 		ub->dev_info.state = UBLK_S_DEV_FAIL_IO;
 		for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
-			ublk_get_queue(ub, i)->fail_io = true;
+			ublk_get_queue(ub, i)->fail_io = 1;
 		}
 		blk_mq_unquiesce_queue(ub->ub_disk->queue);
 	}
@@ -2744,8 +2744,8 @@ static void ublk_queue_reinit(struct ublk_device *ub, struct ublk_queue *ubq)
 	put_task_struct(ubq->ubq_daemon);
 	/* We have to reset it to NULL, otherwise ub won't accept new FETCH_REQ */
 	ubq->ubq_daemon = NULL;
-	ubq->timeout = false;
-	ubq->canceling = false;
+	ubq->timeout = 0;
+	ubq->canceling = 0;
 
 	for (i = 0; i < ubq->q_depth; i++) {
 		struct ublk_io *io = &ubq->ios[i];
@@ -2844,7 +2844,7 @@ static int ublk_ctrl_end_recovery(struct ublk_device *ub,
 		blk_mq_quiesce_queue(ub->ub_disk->queue);
 		ub->dev_info.state = UBLK_S_DEV_LIVE;
 		for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
-			ublk_get_queue(ub, i)->fail_io = false;
+			ublk_get_queue(ub, i)->fail_io = 0;
 		}
 		blk_mq_unquiesce_queue(ub->ub_disk->queue);
 	}
Signature · 1 line
2.47.0
author maintainer
#3 lore raw

RFC PATCH 3/22 ublk: add helper of ublk_need_map_io()

In reply to Ming Lei

ublk_need_map_io() is more readable, and it can cover the coming UBLK_BPF.

---
 drivers/block/ublk_drv.c | 13 +++++++++----
 1 file changed, 9 insertions(+), 4 deletions(-)
drivers/block/ublk_drv.c +9 -4
index 00363e8affc6..1a63a1aa99ed 100644
@@ -582,6 +582,11 @@ static inline bool ublk_support_user_copy(const struct ublk_queue *ubq)
 	return ubq->flags & UBLK_F_USER_COPY;
 }
 
+static inline bool ublk_need_map_io(const struct ublk_queue *ubq)
+{
+	return !ublk_support_user_copy(ubq);
+}
+
 static inline bool ublk_need_req_ref(const struct ublk_queue *ubq)
 {
 	/*
@@ -909,7 +914,7 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req,
 {
 	const unsigned int rq_bytes = blk_rq_bytes(req);
 
-	if (ublk_support_user_copy(ubq))
+	if (!ublk_need_map_io(ubq))
 		return rq_bytes;
 
 	/*
@@ -933,7 +938,7 @@ static int ublk_unmap_io(const struct ublk_queue *ubq,
 {
 	const unsigned int rq_bytes = blk_rq_bytes(req);
 
-	if (ublk_support_user_copy(ubq))
+	if (!ublk_need_map_io(ubq))
 		return rq_bytes;
 
 	if (ublk_need_unmap_req(req)) {
@@ -1809,7 +1814,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,
 		if (io->flags & UBLK_IO_FLAG_OWNED_BY_SRV)
 			goto out;
 
-		if (!ublk_support_user_copy(ubq)) {
+		if (ublk_need_map_io(ubq)) {
 			/*
 			 * FETCH_RQ has to provide IO buffer if NEED GET
 			 * DATA is not enabled
@@ -1831,7 +1836,7 @@ static int __ublk_ch_uring_cmd(struct io_uring_cmd *cmd,
 		if (!(io->flags & UBLK_IO_FLAG_OWNED_BY_SRV))
 			goto out;
 
-		if (!ublk_support_user_copy(ubq)) {
+		if (ublk_need_map_io(ubq)) {
 			/*
 			 * COMMIT_AND_FETCH_REQ has to provide IO buffer if
 			 * NEED GET DATA is not enabled or it is Read IO.
Signature · 1 line
2.47.0
author maintainer
#4 lore raw

RFC PATCH 4/22 ublk: move ublk into one standalone directory

In reply to Ming Lei

Prepare for supporting ublk-bpf, which has to add more source files, so create ublk/ for avoiding to pollute drivers/block/

Meantime rename the source file as ublk/main.c

---
 MAINTAINERS                               |  2 +-
 drivers/block/Kconfig                     | 32 +-------------------
 drivers/block/Makefile                    |  2 +-
 drivers/block/ublk/Kconfig                | 36 +++++++++++++++++++++++
 drivers/block/ublk/Makefile               |  7 +++++
 drivers/block/{ublk_drv.c => ublk/main.c} |  0
 6 files changed, 46 insertions(+), 33 deletions(-)
 create mode 100644 drivers/block/ublk/Kconfig
 create mode 100644 drivers/block/ublk/Makefile
 rename drivers/block/{ublk_drv.c => ublk/main.c} (100%)
MAINTAINERS +1 -1
index c575de4903db..890f6195d03f 100644
@@ -23982,7 +23982,7 @@ M:	Ming Lei <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	Documentation/block/ublk.rst
-F:	drivers/block/ublk_drv.c
+F:	drivers/block/ublk/
 F:	include/uapi/linux/ublk_cmd.h
 
 UBSAN
drivers/block/Kconfig +1 -31
index a97f2c40c640..4e5144183ade 100644
@@ -379,37 +379,7 @@ config BLK_DEV_RBD
 
 	  If unsure, say N.
 
-config BLK_DEV_UBLK
-	tristate "Userspace block driver (Experimental)"
-	select IO_URING
-	help
-	  io_uring based userspace block driver. Together with ublk server, ublk
-	  has been working well, but interface with userspace or command data
-	  definition isn't finalized yet, and might change according to future
-	  requirement, so mark is as experimental now.
-
-	  Say Y if you want to get better performance because task_work_add()
-	  can be used in IO path for replacing io_uring cmd, which will become
-	  shared between IO tasks and ubq daemon, meantime task_work_add() can
-	  can handle batch more effectively, but task_work_add() isn't exported
-	  for module, so ublk has to be built to kernel.
-
-config BLKDEV_UBLK_LEGACY_OPCODES
-	bool "Support legacy command opcode"
-	depends on BLK_DEV_UBLK
-	default y
-	help
-	  ublk driver started to take plain command encoding, which turns out
-	  one bad way. The traditional ioctl command opcode encodes more
-	  info and basically defines each code uniquely, so opcode conflict
-	  is avoided, and driver can handle wrong command easily, meantime it
-	  may help security subsystem to audit io_uring command.
-
-	  Say Y if your application still uses legacy command opcode.
-
-	  Say N if you don't want to support legacy command opcode. It is
-	  suggested to enable N if your application(ublk server) switches to
-	  ioctl command encoding.
+source "drivers/block/ublk/Kconfig"
 
 source "drivers/block/rnbd/Kconfig"
 
drivers/block/Makefile +1 -1
index 1105a2d4fdcb..a6fdc62b817c 100644
@@ -40,6 +40,6 @@ obj-$(CONFIG_BLK_DEV_RNBD)	+= rnbd/
 
 obj-$(CONFIG_BLK_DEV_NULL_BLK)	+= null_blk/
 
-obj-$(CONFIG_BLK_DEV_UBLK)			+= ublk_drv.o
+obj-$(CONFIG_BLK_DEV_UBLK)			+= ublk/
 
 swim_mod-y	:= swim.o swim_asm.o
drivers/block/ublk/Kconfig new file +36 -0
new file mode 100644
index 000000000000..b06e3df09779
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# ublkl block device driver configuration
+#
+
+config BLK_DEV_UBLK
+	tristate "Userspace block driver (Experimental)"
+	select IO_URING
+	help
+	  io_uring based userspace block driver. Together with ublk server, ublk
+	  has been working well, but interface with userspace or command data
+	  definition isn't finalized yet, and might change according to future
+	  requirement, so mark is as experimental now.
+
+	  Say Y if you want to get better performance because task_work_add()
+	  can be used in IO path for replacing io_uring cmd, which will become
+	  shared between IO tasks and ubq daemon, meantime task_work_add() can
+	  can handle batch more effectively, but task_work_add() isn't exported
+	  for module, so ublk has to be built to kernel.
+
+config BLKDEV_UBLK_LEGACY_OPCODES
+	bool "Support legacy command opcode"
+	depends on BLK_DEV_UBLK
+	default y
+	help
+	  ublk driver started to take plain command encoding, which turns out
+	  one bad way. The traditional ioctl command opcode encodes more
+	  info and basically defines each code uniquely, so opcode conflict
+	  is avoided, and driver can handle wrong command easily, meantime it
+	  may help security subsystem to audit io_uring command.
+
+	  Say Y if your application still uses legacy command opcode.
+
+	  Say N if you don't want to support legacy command opcode. It is
+	  suggested to enable N if your application(ublk server) switches to
+	  ioctl command encoding.
drivers/block/ublk/Makefile new file +7 -0
new file mode 100644
index 000000000000..30e06b74dd82
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: GPL-2.0
+
+# needed for trace events
+ccflags-y			+= -I$(src)
+
+ublk_drv-$(CONFIG_BLK_DEV_UBLK)	:= main.o
+obj-$(CONFIG_BLK_DEV_UBLK)	+= ublk_drv.o
drivers/block/ublk/main.c renamed from drivers/block/ublk_drv.c +0 -0
similarity index 100%
rename from drivers/block/ublk_drv.c
rename to drivers/block/ublk/main.c
Signature · 1 line
2.47.0
author maintainer
#5 lore raw

RFC PATCH 5/22 ublk: move private definitions into private header

In reply to Ming Lei

Add one private header file and move private definitions into this file.

---
 drivers/block/ublk/main.c | 150 +-----------------------------------
 drivers/block/ublk/ublk.h | 157 ++++++++++++++++++++++++++++++++++++++
 2 files changed, 158 insertions(+), 149 deletions(-)
 create mode 100644 drivers/block/ublk/ublk.h
drivers/block/ublk/main.c +1 -149
index 1a63a1aa99ed..2510193303bb 100644
@@ -19,7 +19,6 @@
 #include <linux/errno.h>
 #include <linux/major.h>
 #include <linux/wait.h>
-#include <linux/blkdev.h>
 #include <linux/init.h>
 #include <linux/swap.h>
 #include <linux/slab.h>
@@ -35,162 +34,15 @@
 #include <linux/ioprio.h>
 #include <linux/sched/mm.h>
 #include <linux/uaccess.h>
-#include <linux/cdev.h>
 #include <linux/io_uring/cmd.h>
-#include <linux/blk-mq.h>
 #include <linux/delay.h>
 #include <linux/mm.h>
 #include <asm/page.h>
 #include <linux/task_work.h>
 #include <linux/namei.h>
 #include <linux/kref.h>
-#include <uapi/linux/ublk_cmd.h>
-
-#define UBLK_MINORS		(1U << MINORBITS)
-
-/* private ioctl command mirror */
-#define UBLK_CMD_DEL_DEV_ASYNC	_IOC_NR(UBLK_U_CMD_DEL_DEV_ASYNC)
-
-/* All UBLK_F_* have to be included into UBLK_F_ALL */
-#define UBLK_F_ALL (UBLK_F_SUPPORT_ZERO_COPY \
-		| UBLK_F_URING_CMD_COMP_IN_TASK \
-		| UBLK_F_NEED_GET_DATA \
-		| UBLK_F_USER_RECOVERY \
-		| UBLK_F_USER_RECOVERY_REISSUE \
-		| UBLK_F_UNPRIVILEGED_DEV \
-		| UBLK_F_CMD_IOCTL_ENCODE \
-		| UBLK_F_USER_COPY \
-		| UBLK_F_ZONED \
-		| UBLK_F_USER_RECOVERY_FAIL_IO)
-
-#define UBLK_F_ALL_RECOVERY_FLAGS (UBLK_F_USER_RECOVERY \
-		| UBLK_F_USER_RECOVERY_REISSUE \
-		| UBLK_F_USER_RECOVERY_FAIL_IO)
-
-/* All UBLK_PARAM_TYPE_* should be included here */
-#define UBLK_PARAM_TYPE_ALL                                \
-	(UBLK_PARAM_TYPE_BASIC | UBLK_PARAM_TYPE_DISCARD | \
-	 UBLK_PARAM_TYPE_DEVT | UBLK_PARAM_TYPE_ZONED)
-
-struct ublk_rq_data {
-	struct llist_node node;
-
-	struct kref ref;
-};
-
-struct ublk_uring_cmd_pdu {
-	struct ublk_queue *ubq;
-	u16 tag;
-};
-
-/*
- * io command is active: sqe cmd is received, and its cqe isn't done
- *
- * If the flag is set, the io command is owned by ublk driver, and waited
- * for incoming blk-mq request from the ublk block device.
- *
- * If the flag is cleared, the io command will be completed, and owned by
- * ublk server.
- */
-#define UBLK_IO_FLAG_ACTIVE	0x01
-
-/*
- * IO command is completed via cqe, and it is being handled by ublksrv, and
- * not committed yet
- *
- * Basically exclusively with UBLK_IO_FLAG_ACTIVE, so can be served for
- * cross verification
- */
-#define UBLK_IO_FLAG_OWNED_BY_SRV 0x02
-
-/*
- * IO command is aborted, so this flag is set in case of
- * !UBLK_IO_FLAG_ACTIVE.
- *
- * After this flag is observed, any pending or new incoming request
- * associated with this io command will be failed immediately
- */
-#define UBLK_IO_FLAG_ABORTED 0x04
-
-/*
- * UBLK_IO_FLAG_NEED_GET_DATA is set because IO command requires
- * get data buffer address from ublksrv.
- *
- * Then, bio data could be copied into this data buffer for a WRITE request
- * after the IO command is issued again and UBLK_IO_FLAG_NEED_GET_DATA is unset.
- */
-#define UBLK_IO_FLAG_NEED_GET_DATA 0x08
-
-/* atomic RW with ubq->cancel_lock */
-#define UBLK_IO_FLAG_CANCELED	0x80000000
 
-struct ublk_io {
-	/* userspace buffer address from io cmd */
-	__u64	addr;
-	unsigned int flags;
-	int res;
-
-	struct io_uring_cmd *cmd;
-};
-
-struct ublk_queue {
-	int q_id;
-	int q_depth;
-
-	unsigned long flags;
-	struct task_struct	*ubq_daemon;
-	char *io_cmd_buf;
-
-	struct llist_head	io_cmds;
-
-	unsigned short force_abort:1;
-	unsigned short timeout:1;
-	unsigned short canceling:1;
-	unsigned short fail_io:1; /* copy of dev->state == UBLK_S_DEV_FAIL_IO */
-	unsigned short nr_io_ready;	/* how many ios setup */
-	spinlock_t		cancel_lock;
-	struct ublk_device *dev;
-	struct ublk_io ios[];
-};
-
-struct ublk_device {
-	struct gendisk		*ub_disk;
-
-	char	*__queues;
-
-	unsigned int	queue_size;
-	struct ublksrv_ctrl_dev_info	dev_info;
-
-	struct blk_mq_tag_set	tag_set;
-
-	struct cdev		cdev;
-	struct device		cdev_dev;
-
-#define UB_STATE_OPEN		0
-#define UB_STATE_USED		1
-#define UB_STATE_DELETED	2
-	unsigned long		state;
-	int			ub_number;
-
-	struct mutex		mutex;
-
-	spinlock_t		lock;
-	struct mm_struct	*mm;
-
-	struct ublk_params	params;
-
-	struct completion	completion;
-	unsigned int		nr_queues_ready;
-	unsigned int		nr_privileged_daemon;
-
-	struct work_struct	nosrv_work;
-};
-
-/* header of ublk_params */
-struct ublk_params_header {
-	__u32	len;
-	__u32	types;
-};
+#include "ublk.h"
 
 static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq);
 
drivers/block/ublk/ublk.h new file +157 -0

157 lines not on this page · raw message

Signature · 1 line
2.47.0
author maintainer
#6 lore raw

RFC PATCH 6/22 ublk: move several helpers to private header

In reply to Ming Lei

Move several helpers into the private header so that make them visible to the whole driver, and prepare for supporting ublk-bpf.

---
 drivers/block/ublk/main.c | 16 +++-------------
 drivers/block/ublk/ublk.h | 11 +++++++++++
 2 files changed, 14 insertions(+), 13 deletions(-)
drivers/block/ublk/main.c +3 -13
index 2510193303bb..aefb414ebf6c 100644
@@ -47,8 +47,6 @@
 static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq);
 
 static inline unsigned int ublk_req_build_flags(struct request *req);
-static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq,
-						   int tag);
 static inline bool ublk_dev_is_user_copy(const struct ublk_device *ub)
 {
 	return ub->dev_info.flags & UBLK_F_USER_COPY;
@@ -325,7 +323,6 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq,
 
 #endif
 
-static inline void __ublk_complete_rq(struct request *req);
 static void ublk_complete_rq(struct kref *ref);
 
 static dev_t ublk_chr_devt;
@@ -496,7 +493,7 @@ static noinline struct ublk_device *ublk_get_device(struct ublk_device *ub)
 }
 
 /* Called in slow path only, keep it noinline for trace purpose */
-static noinline void ublk_put_device(struct ublk_device *ub)
+void ublk_put_device(struct ublk_device *ub)
 {
 	put_device(&ub->cdev_dev);
 }
@@ -512,13 +509,6 @@ static inline bool ublk_rq_has_data(const struct request *rq)
 	return bio_has_data(rq->bio);
 }
 
-static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq,
-		int tag)
-{
-	return (struct ublksrv_io_desc *)
-		&(ubq->io_cmd_buf[tag * sizeof(struct ublksrv_io_desc)]);
-}
-
 static inline char *ublk_queue_cmd_buf(struct ublk_device *ub, int q_id)
 {
 	return ublk_get_queue(ub, q_id)->io_cmd_buf;
@@ -887,7 +877,7 @@ static inline bool ubq_daemon_is_dying(struct ublk_queue *ubq)
 }
 
 /* todo: handle partial completion */
-static inline void __ublk_complete_rq(struct request *req)
+void __ublk_complete_rq(struct request *req)
 {
 	struct ublk_queue *ubq = req->mq_hctx->driver_data;
 	struct ublk_io *io = &ubq->ios[req->tag];
@@ -2082,7 +2072,7 @@ static void ublk_remove(struct ublk_device *ub)
 	ublks_added--;
 }
 
-static struct ublk_device *ublk_get_device_from_id(int idx)
+struct ublk_device *ublk_get_device_from_id(int idx)
 {
 	struct ublk_device *ub = NULL;
 
drivers/block/ublk/ublk.h +11 -0
index 12e39a33015a..76aee4225c78 100644
@@ -154,4 +154,15 @@ struct ublk_params_header {
 };
 
 
+static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq,
+		int tag)
+{
+	return (struct ublksrv_io_desc *)
+		&(ubq->io_cmd_buf[tag * sizeof(struct ublksrv_io_desc)]);
+}
+
+struct ublk_device *ublk_get_device_from_id(int idx);
+void ublk_put_device(struct ublk_device *ub);
+void __ublk_complete_rq(struct request *req);
+
 #endif
Signature · 1 line
2.47.0
author maintainer
#7 lore raw

RFC PATCH 7/22 ublk: bpf: add bpf prog attach helpers

In reply to Ming Lei

Add bpf prog attach helpers and prepare for supporting ublk bpf, in which multiple ublk device may attach to same bpf prog, and there can be multiple bpf progs.

bpf_prog_consumer will be embedded in the bpf prog user side, such as ublk device, bpf_prog_provider will be embedded in the bpf struct_ops prog side.

---
 drivers/block/ublk/bpf_reg.h | 77 ++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)
 create mode 100644 drivers/block/ublk/bpf_reg.h
drivers/block/ublk/bpf_reg.h new file +77 -0
new file mode 100644
index 000000000000..79d02e93aea8
@@ -0,0 +1,77 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#ifndef UBLK_INT_BPF_REG_HEADER
+#define UBLK_INT_BPF_REG_HEADER
+
+#include <linux/types.h>
+
+struct bpf_prog_consumer;
+struct bpf_prog_provider;
+
+typedef int (*bpf_prog_attach_t)(struct bpf_prog_consumer *consumer,
+				 struct bpf_prog_provider *provider);
+typedef void (*bpf_prog_detach_t)(struct bpf_prog_consumer *consumer,
+				  bool unreg);
+
+struct bpf_prog_consumer_ops {
+	bpf_prog_attach_t		attach_fn;
+	bpf_prog_detach_t		detach_fn;
+};
+
+struct bpf_prog_consumer {
+	const struct bpf_prog_consumer_ops	*ops;
+	unsigned int				prog_id;
+	struct list_head			node;
+	struct bpf_prog_provider		*provider;
+};
+
+struct bpf_prog_provider {
+	struct list_head	list;
+};
+
+static inline void bpf_prog_provider_init(struct bpf_prog_provider *provider)
+{
+	INIT_LIST_HEAD(&provider->list);
+}
+
+static inline bool bpf_prog_provider_is_empty(
+		struct bpf_prog_provider *provider)
+{
+	return list_empty(&provider->list);
+}
+
+static inline int bpf_prog_consumer_attach(struct bpf_prog_consumer *consumer,
+					   struct bpf_prog_provider *provider)
+{
+	const struct bpf_prog_consumer_ops *ops = consumer->ops;
+
+	if (!ops || !ops->attach_fn)
+		return -EINVAL;
+
+	if (ops->attach_fn) {
+		int ret = ops->attach_fn(consumer, provider);
+
+		if (ret)
+			return ret;
+	}
+	consumer->provider = provider;
+	list_add(&consumer->node, &provider->list);
+	return 0;
+}
+
+static inline void bpf_prog_consumer_detach(struct bpf_prog_consumer *consumer,
+					    bool unreg)
+{
+	const struct bpf_prog_consumer_ops *ops = consumer->ops;
+
+	if (!consumer->provider)
+		return;
+
+	if (!list_empty(&consumer->node)) {
+		if (ops && ops->detach_fn)
+			ops->detach_fn(consumer, unreg);
+		list_del_init(&consumer->node);
+		consumer->provider = NULL;
+	}
+}
+
+#endif
Signature · 1 line
2.47.0
author maintainer
#8 lore raw

RFC PATCH 8/22 ublk: bpf: add bpf struct_ops

In reply to Ming Lei

Add struct_ops support for ublk, so we can use struct_ops bpf prog to handle ublk IO command with application defined struct_ops.

Follows the motivation for ublk-bpf:

1) support stacking ublk

- there are many 3rd party volume manager, ublk may be built over ublk device for simplifying implementation, however, multiple userspace-kernel context switch for handling one single IO can't be accepted from performance view of point

- ublk-bpf can avoid user-kernel context switch in most fast io path, so makes ublk over ublk possible

2) complicated virtual block device

- many complicated virtual block devices have admin&meta code path and normal io fast path; meta & admin IO handling is usually complicated, so it can be moved to userspace for relieving development burden; meantime IO fast path can be kept in kernel space for the sake of high performance.

- bpf provides rich maps, which can help a lot for communication between userspace and prog or between prog and prog.

- one typical example is qcow2, which meta io handling can be moved to userspace, and fast io path is implemented with ublk-bpf in which one efficient bpf map can be looked up first and see if this virtual LBA & host LBA is found in the map, handle the IO with ublk-bpf if the mapping is hit, otherwise forward to userspace to deal with meta IO.

3) some simple high performance virtual devices

- such as null & loop, the whole implementation can be done in bpf prog

Export struct ublk_bpf_ops as bpf struct_ops, so that bpf prog can implement callbacks for handling ublk io commands:

- if UBLK_BPF_IO_QUEUED is returned from ->queue_io_cmd() or ->queue_io_cmd_daemon(), this io command has been queued in bpf prog, so it won't be forwarded to userspace

- if UBLK_BPF_IO_REDIRECT is returned from ->queue_io_cmd() or ->queue_io_cmd_daemon(), this io command will be forwarded to userspace

- if UBLK_BPF_IO_CONTINUE is returned from ->queue_io_cmd() or ->queue_io_cmd_daemon(), part of this io command is queued, and ublk_bpf_return_t carries how many bytes queued, so ublk driver will continue to call the callback to queue remained bytes of this io command further, this way is helpful for implementing stacking devices by splitting io command.

Also ->release_io_cmd() is added for providing chance to notify bpf prog that this io command is going to be released.

---
 drivers/block/ublk/Kconfig   |  16 +++
 drivers/block/ublk/Makefile  |   3 +
 drivers/block/ublk/bpf.h     | 184 ++++++++++++++++++++++++
 drivers/block/ublk/bpf_ops.c | 261 +++++++++++++++++++++++++++++++++++
 drivers/block/ublk/main.c    |  29 +++-
 drivers/block/ublk/ublk.h    |  33 +++++
 6 files changed, 524 insertions(+), 2 deletions(-)
 create mode 100644 drivers/block/ublk/bpf.h
 create mode 100644 drivers/block/ublk/bpf_ops.c
drivers/block/ublk/Kconfig +16 -0
index b06e3df09779..23aa97d51956 100644
@@ -34,3 +34,19 @@ config BLKDEV_UBLK_LEGACY_OPCODES
 	  Say N if you don't want to support legacy command opcode. It is
 	  suggested to enable N if your application(ublk server) switches to
 	  ioctl command encoding.
+
+config UBLK_BPF
+	bool "UBLK-BPF support"
+	depends on BPF
+	depends on BLK_DEV_UBLK
+	help
+	This option allows to support eBPF programs on the UBLK subsystem.
+	eBPF programs can handle fast IO code path directly in kernel space,
+	and avoid to switch to ublk daemon userspace conext, meantime zero
+	copy can be supported directly.
+
+	Usually target code need to partition into two parts: fast io code path
+	which is run as eBPF prog in kernel context, and slow & complicated
+	meta/admin code path which is run in ublk daemon userspace context.
+	And use efficient bpf map for communication between user mode and
+	kernel bpf prog.
drivers/block/ublk/Makefile +3 -0
index 30e06b74dd82..7058b0fc13bf 100644
@@ -4,4 +4,7 @@
 ccflags-y			+= -I$(src)
 
 ublk_drv-$(CONFIG_BLK_DEV_UBLK)	:= main.o
+ifeq ($(CONFIG_UBLK_BPF), y)
+ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o
+endif
 obj-$(CONFIG_BLK_DEV_UBLK)	+= ublk_drv.o
drivers/block/ublk/bpf.h new file +184 -0

184 lines not on this page · raw message

drivers/block/ublk/bpf_ops.c new file +261 -0

261 lines not on this page · raw message

drivers/block/ublk/main.c +27 -2
index aefb414ebf6c..29d3e7f656a7 100644
@@ -43,6 +43,7 @@
 #include <linux/kref.h>
 
 #include "ublk.h"
+#include "bpf.h"
 
 static bool ublk_abort_requests(struct ublk_device *ub, struct ublk_queue *ubq);
 
@@ -1061,6 +1062,10 @@ static inline void __ublk_rq_task_work(struct request *req,
 			mapped_bytes >> 9;
 	}
 
+	if (ublk_support_bpf(ubq) && ublk_run_bpf_prog(ubq, req,
+				ublk_get_bpf_io_cb_daemon(ubq), true))
+		return;
+
 	ublk_init_req_ref(ubq, req);
 	ubq_complete_io_cmd(io, UBLK_IO_RES_OK, issue_flags);
 }
@@ -1088,6 +1093,10 @@ static void ublk_queue_cmd(struct ublk_queue *ubq, struct request *rq)
 {
 	struct ublk_rq_data *data = blk_mq_rq_to_pdu(rq);
 
+	if (ublk_support_bpf(ubq) && ublk_run_bpf_prog(ubq, rq,
+				ublk_get_bpf_io_cb(ubq), false))
+		return;
+
 	if (llist_add(&data->node, &ubq->io_cmds)) {
 		struct ublk_io *io = &ubq->ios[rq->tag];
 
@@ -1265,8 +1274,24 @@ static void ublk_commit_completion(struct ublk_device *ub,
 	if (req_op(req) == REQ_OP_ZONE_APPEND)
 		req->__sector = ub_cmd->zone_append_lba;
 
-	if (likely(!blk_should_fake_timeout(req->q)))
-		ublk_put_req_ref(ubq, req);
+	if (likely(!blk_should_fake_timeout(req->q))) {
+		/*
+		 * userspace may have setup everything, but still let bpf
+		 * prog to handle io by returning -EAGAIN, this way provides
+		 * single bpf io handle fast path, and should simplify things
+		 * a lot.
+		 */
+		if (ublk_support_bpf(ubq) && io->res == -EAGAIN) {
+			if(!ublk_run_bpf_prog(ubq, req,
+					ublk_get_bpf_any_io_cb(ubq), true)) {
+				/* give up now */
+				io->res = -EIO;
+				ublk_put_req_ref(ubq, req);
+			}
+		} else {
+			ublk_put_req_ref(ubq, req);
+		}
+	}
 }
 
 /*
drivers/block/ublk/ublk.h +33 -0
index 76aee4225c78..e9ceadbc616d 100644
@@ -33,10 +33,26 @@
 	(UBLK_PARAM_TYPE_BASIC | UBLK_PARAM_TYPE_DISCARD | \
 	 UBLK_PARAM_TYPE_DEVT | UBLK_PARAM_TYPE_ZONED)
 
+enum {
+	UBLK_BPF_IO_PREP	= 0,
+	UBLK_BPF_IO_COMPLETED   = 1,
+};
+
+struct ublk_bpf_io {
+	const struct ublksrv_io_desc	*iod;
+	unsigned long			flags;
+	refcount_t                      ref;
+	int				res;
+};
+
 struct ublk_rq_data {
 	struct llist_node node;
 
 	struct kref ref;
+
+#ifdef CONFIG_UBLK_BPF
+	struct ublk_bpf_io	bpf_data;
+#endif
 };
 
 struct ublk_uring_cmd_pdu {
@@ -104,6 +120,10 @@ struct ublk_queue {
 
 	struct llist_head	io_cmds;
 
+#ifdef CONFIG_UBLK_BPF
+	struct ublk_bpf_ops     *bpf_ops;
+#endif
+
 	unsigned short force_abort:1;
 	unsigned short timeout:1;
 	unsigned short canceling:1;
@@ -161,8 +181,21 @@ static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq,
 		&(ubq->io_cmd_buf[tag * sizeof(struct ublksrv_io_desc)]);
 }
 
+static inline bool ublk_support_bpf(const struct ublk_queue *ubq)
+{
+	return false;
+}
+
 struct ublk_device *ublk_get_device_from_id(int idx);
 void ublk_put_device(struct ublk_device *ub);
 void __ublk_complete_rq(struct request *req);
 
+static inline void __ublk_complete_rq_with_res(struct request *req, int res)
+{
+	struct ublk_queue *ubq = req->mq_hctx->driver_data;
+	struct ublk_io *io = &ubq->ios[req->tag];
+
+	io->res = res;
+	__ublk_complete_rq(req);
+}
 #endif
Signature · 1 line
2.47.0
#9 lore raw

Re: [RFC PATCH 08/22] ublk: bpf: add bpf struct_ops

Quoted 12 lines · Ming Lei
drivers/block/ublk/bpf_ops.c new file +12 -0
+
+/* Return true if io cmd is queued, otherwise forward it to userspace */
+bool ublk_run_bpf_handler(struct ublk_queue *ubq, struct request *req,
+                         queue_io_cmd_t cb)
+{
+       ublk_bpf_return_t ret;
+       struct ublk_rq_data *data = blk_mq_rq_to_pdu(req);
+       struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
+       struct ublk_bpf_io *bpf_io = &data->bpf_data;
+       const unsigned long total = iod->nr_sectors << 9;
+       unsigned int done = 0;
+       bool res = true;

drivers/block/ublk/bpf_ops.c line 167

Quoted 10 lines
drivers/block/ublk/bpf_ops.c new file +10 -0
+       int err;
+
+       if (!test_bit(UBLK_BPF_IO_PREP, &bpf_io->flags))
+               ublk_bpf_prep_io(bpf_io, iod);
+
+       do {
+               enum ublk_bpf_disposition rc;
+               unsigned int bytes;
+
+               ret = cb(bpf_io, done);

High level observation...
I suspect forcing all sturct_ops callbacks to have only these two arguments and packing args into ublk_bpf_io will be limiting in the long term.

And this part of api would need to be redesigned,
but since it's not an uapi... not a big deal.

drivers/block/ublk/bpf_ops.c line 174

Quoted 7 lines
drivers/block/ublk/bpf_ops.c new file +7 -0
+               rc = ublk_bpf_get_disposition(ret);
+
+               if (rc == UBLK_BPF_IO_QUEUED)
+                       goto exit;
+
+               if (rc == UBLK_BPF_IO_REDIRECT)
+                       break;

Same point about return value processing...
Each struct_ops callback could have had its own meaning of retvals.
I suspect it would have been more flexible and more powerful this way.

Other than that bpf plumbing looks good.

There is an issue with leaking allocated memory in bpf_aio_alloc kfunc (it probably should be KF_ACQUIRE)
and a few other things, but before doing any in depth review from bpf pov I'd like to hear what block folks think.

Motivation looks useful,
but the claim of performance gains without performance numbers is a leap of faith.

Quoted 19 lines
drivers/block/ublk/bpf_ops.c new file +19 -0
+
+               if (unlikely(rc != UBLK_BPF_IO_CONTINUE)) {
+                       printk_ratelimited(KERN_ERR "%s: unknown rc code %d\n",
+                                       __func__, rc);
+                       err = -EINVAL;
+                       goto fail;
+               }
+
+               bytes = ublk_bpf_get_return_bytes(ret);
+               if (unlikely((bytes & 511) || !bytes)) {
+                       err = -EREMOTEIO;
+                       goto fail;
+               } else if (unlikely(bytes > total - done)) {
+                       err = -ENOSPC;
+                       goto fail;
+               } else {
+                       done += bytes;
+               }
+       } while (done < total);
author maintainer
#10 lore raw

Re: [RFC PATCH 08/22] ublk: bpf: add bpf struct_ops

Hello Alexei,

Thanks for your comments!

Quoted 20 lines · Alexei Starovoitov
Quoted 19 lines · Ming Lei
drivers/block/ublk/bpf_ops.c new file +19 -0
+
+/* Return true if io cmd is queued, otherwise forward it to userspace */
+bool ublk_run_bpf_handler(struct ublk_queue *ubq, struct request *req,
+                         queue_io_cmd_t cb)
+{
+       ublk_bpf_return_t ret;
+       struct ublk_rq_data *data = blk_mq_rq_to_pdu(req);
+       struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
+       struct ublk_bpf_io *bpf_io = &data->bpf_data;
+       const unsigned long total = iod->nr_sectors << 9;
+       unsigned int done = 0;
+       bool res = true;
+       int err;
+
+       if (!test_bit(UBLK_BPF_IO_PREP, &bpf_io->flags))
+               ublk_bpf_prep_io(bpf_io, iod);
+
+       do {
+               enum ublk_bpf_disposition rc;
Quoted 8 lines

drivers/block/ublk/bpf_ops.c line 167

Quoted 3 lines
drivers/block/ublk/bpf_ops.c new file +3 -0
+               unsigned int bytes;
+
+               ret = cb(bpf_io, done);

High level observation...
I suspect forcing all sturct_ops callbacks to have only these two arguments and packing args into ublk_bpf_io will be limiting in the long term.

There are three callbacks defined, and only the two with same type for queuing io commands are covered in this function.

But yes, callback type belongs to API, which should be designed carefully, and I will think about further.

Quoted 4 lines

And this part of api would need to be redesigned,
but since it's not an uapi... not a big deal.

Quoted 13 lines

drivers/block/ublk/bpf_ops.c line 174

Quoted 7 lines
drivers/block/ublk/bpf_ops.c new file +7 -0
+               rc = ublk_bpf_get_disposition(ret);
+
+               if (rc == UBLK_BPF_IO_QUEUED)
+                       goto exit;
+
+               if (rc == UBLK_BPF_IO_REDIRECT)
+                       break;

Same point about return value processing...
Each struct_ops callback could have had its own meaning of retvals.
I suspect it would have been more flexible and more powerful this way.

Yeah, I agree, just the 3rd callback of release_io_cmd_t isn't covered in this function.

Quoted 5 lines

Other than that bpf plumbing looks good.

There is an issue with leaking allocated memory in bpf_aio_alloc kfunc (it probably should be KF_ACQUIRE)

It is one problem which troubles me too:

- another callback of struct_ops/bpf_aio_complete_cb is guaranteed to be called after the 'struct bpf_aio' instance is submitted via kfunc bpf_aio_submit(), and it is supposed to be freed from struct_ops/bpf_aio_complete_cb

- but the following verifier failure is triggered if bpf_aio_alloc and bpf_aio_release are marked as KF_ACQUIRE & KF_RELEASE.

```
libbpf: prog 'ublk_loop_comp_cb': -- BEGIN PROG LOAD LOG --
Global function ublk_loop_comp_cb() doesn't return scalar. Only those are supported.
```

Here 'struct bpf_aio' instance isn't stored in map, and it is provided from struct_ops callback(bpf_aio_complete_cb), I appreciate you may share any idea about how to let KF_ACQUIRE/KF_RELEASE cover the usage here.

Quoted 2 lines

and a few other things, but before doing any in depth review from bpf pov I'd like to hear what block folks think.

Me too, look forward to comments from our block guys.

Quoted 4 lines

Motivation looks useful,
but the claim of performance gains without performance numbers is a leap of faith.

Follows some data:

1) ublk-null bpf vs. ublk-null with bpf

- 1.97M IOPS vs. 3.7M IOPS

- setup ublk-null

cd tools/testing/selftests/ublk
./ublk_bpf add -t null -q 2

- setup ublk-null wit bpf

	cd tools/testing/selftests/ublk
	./ublk_bpf reg -t null ./ublk_null.bpf.o
	./ublk_bpf add -t null -q 2 --bpf_prog 0

- run fio/t/io_uring -p 0 /dev/ublkb0

2) ublk-loop

The built-in utility of ublk_bpf only supports bpf io handling, but compared with ublksrv, the improvement isn't so big, still with ~10%. One reason is that bpf aio is just started, not optimized, in theory:

- it saves one kernel-user context switch
- save one time of user-kernel IO buffer copy
- much less io handling code footprint compared with userspace io handling

The improvement is supposed to be big especially in big chunk size IO workload.

Thanks,
Ming

#11 lore raw

Re: [RFC PATCH 08/22] ublk: bpf: add bpf struct_ops

Quoted 75 lines · Ming Lei

Hello Alexei,

Thanks for your comments!

Quoted 20 lines · Alexei Starovoitov
Quoted 19 lines · Ming Lei
drivers/block/ublk/bpf_ops.c new file +19 -0
+
+/* Return true if io cmd is queued, otherwise forward it to userspace */
+bool ublk_run_bpf_handler(struct ublk_queue *ubq, struct request *req,
+                         queue_io_cmd_t cb)
+{
+       ublk_bpf_return_t ret;
+       struct ublk_rq_data *data = blk_mq_rq_to_pdu(req);
+       struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
+       struct ublk_bpf_io *bpf_io = &data->bpf_data;
+       const unsigned long total = iod->nr_sectors << 9;
+       unsigned int done = 0;
+       bool res = true;
+       int err;
+
+       if (!test_bit(UBLK_BPF_IO_PREP, &bpf_io->flags))
+               ublk_bpf_prep_io(bpf_io, iod);
+
+       do {
+               enum ublk_bpf_disposition rc;
Quoted 8 lines

drivers/block/ublk/bpf_ops.c line 167

Quoted 3 lines
drivers/block/ublk/bpf_ops.c new file +3 -0
+               unsigned int bytes;
+
+               ret = cb(bpf_io, done);

High level observation...
I suspect forcing all sturct_ops callbacks to have only these two arguments and packing args into ublk_bpf_io will be limiting in the long term.

There are three callbacks defined, and only the two with same type for queuing io commands are covered in this function.

But yes, callback type belongs to API, which should be designed carefully, and I will think about further.

Quoted 4 lines

And this part of api would need to be redesigned,
but since it's not an uapi... not a big deal.

Quoted 13 lines

drivers/block/ublk/bpf_ops.c line 174

Quoted 7 lines
drivers/block/ublk/bpf_ops.c new file +7 -0
+               rc = ublk_bpf_get_disposition(ret);
+
+               if (rc == UBLK_BPF_IO_QUEUED)
+                       goto exit;
+
+               if (rc == UBLK_BPF_IO_REDIRECT)
+                       break;

Same point about return value processing...
Each struct_ops callback could have had its own meaning of retvals.
I suspect it would have been more flexible and more powerful this way.

Yeah, I agree, just the 3rd callback of release_io_cmd_t isn't covered in this function.

Quoted 5 lines

Other than that bpf plumbing looks good.

There is an issue with leaking allocated memory in bpf_aio_alloc kfunc (it probably should be KF_ACQUIRE)

It is one problem which troubles me too:

- another callback of struct_ops/bpf_aio_complete_cb is guaranteed to be called after the 'struct bpf_aio' instance is submitted via kfunc bpf_aio_submit(), and it is supposed to be freed from struct_ops/bpf_aio_complete_cb

Quoted 7 lines

- but the following verifier failure is triggered if bpf_aio_alloc and bpf_aio_release are marked as KF_ACQUIRE & KF_RELEASE.

```
libbpf: prog 'ublk_loop_comp_cb': -- BEGIN PROG LOAD LOG --
Global function ublk_loop_comp_cb() doesn't return scalar. Only those are supported.
```

That's odd.
Adding KF_ACQ/REL to bpf_aio_alloc/release kfuncs shouldn't affect verification of ublk_loop_comp_cb() prog. It's fine for it to stay 'void' return.
You probably made it global function and that's was the reason for this verifier error. Global funcs have to return scalar for now. We can relax this restriction if necessary.

Quoted 4 lines

Here 'struct bpf_aio' instance isn't stored in map, and it is provided from struct_ops callback(bpf_aio_complete_cb), I appreciate you may share any idea about how to let KF_ACQUIRE/KF_RELEASE cover the usage here.

This is so that:

ublk_loop_comp_cb ->
ublk_loop_comp_and_release_aio ->
bpf_aio_release

would properly recognize that ref to aio is dropped?

Currently the verifier doesn't support that, but there is work in progress to add this feature:

https://lore.kernel.org/bpf/[email protected]/

then in cfi_stabs annotated bio argument in bpf_aio_complete_cb() as "struct bpf_aio *aio__ref"

Then the verifier will recognize that callback argument comes refcounted and the prog has to call KF_RELEASE kfunc on it.

Quoted 46 lines
Quoted 3 lines

and a few other things, but before doing any in depth review from bpf pov I'd like to hear what block folks think.

Me too, look forward to comments from our block guys.

Quoted 4 lines

Motivation looks useful,
but the claim of performance gains without performance numbers is a leap of faith.

Follows some data:

1) ublk-null bpf vs. ublk-null with bpf

- 1.97M IOPS vs. 3.7M IOPS

- setup ublk-null

cd tools/testing/selftests/ublk
./ublk_bpf add -t null -q 2

- setup ublk-null wit bpf

cd tools/testing/selftests/ublk
./ublk_bpf reg -t null ./ublk_null.bpf.o
./ublk_bpf add -t null -q 2 --bpf_prog 0

- run fio/t/io_uring -p 0 /dev/ublkb0

2) ublk-loop

The built-in utility of ublk_bpf only supports bpf io handling, but compared with ublksrv, the improvement isn't so big, still with ~10%. One reason is that bpf aio is just started, not optimized, in theory:

- it saves one kernel-user context switch
- save one time of user-kernel IO buffer copy
- much less io handling code footprint compared with userspace io handling

The improvement is supposed to be big especially in big chunk size IO workload.

Thanks,
Ming

author maintainer
#12 lore raw

Re: [RFC PATCH 08/22] ublk: bpf: add bpf struct_ops

Hello Alexei,

Quoted 84 lines · Alexei Starovoitov
Quoted 75 lines · Ming Lei

Hello Alexei,

Thanks for your comments!

Quoted 20 lines · Alexei Starovoitov
Quoted 19 lines · Ming Lei
drivers/block/ublk/bpf_ops.c new file +19 -0
+
+/* Return true if io cmd is queued, otherwise forward it to userspace */
+bool ublk_run_bpf_handler(struct ublk_queue *ubq, struct request *req,
+                         queue_io_cmd_t cb)
+{
+       ublk_bpf_return_t ret;
+       struct ublk_rq_data *data = blk_mq_rq_to_pdu(req);
+       struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
+       struct ublk_bpf_io *bpf_io = &data->bpf_data;
+       const unsigned long total = iod->nr_sectors << 9;
+       unsigned int done = 0;
+       bool res = true;
+       int err;
+
+       if (!test_bit(UBLK_BPF_IO_PREP, &bpf_io->flags))
+               ublk_bpf_prep_io(bpf_io, iod);
+
+       do {
+               enum ublk_bpf_disposition rc;
Quoted 8 lines

drivers/block/ublk/bpf_ops.c line 167

Quoted 3 lines
drivers/block/ublk/bpf_ops.c new file +3 -0
+               unsigned int bytes;
+
+               ret = cb(bpf_io, done);

High level observation...
I suspect forcing all sturct_ops callbacks to have only these two arguments and packing args into ublk_bpf_io will be limiting in the long term.

There are three callbacks defined, and only the two with same type for queuing io commands are covered in this function.

But yes, callback type belongs to API, which should be designed carefully, and I will think about further.

Quoted 4 lines

And this part of api would need to be redesigned,
but since it's not an uapi... not a big deal.

Quoted 13 lines

drivers/block/ublk/bpf_ops.c line 174

Quoted 7 lines
drivers/block/ublk/bpf_ops.c new file +7 -0
+               rc = ublk_bpf_get_disposition(ret);
+
+               if (rc == UBLK_BPF_IO_QUEUED)
+                       goto exit;
+
+               if (rc == UBLK_BPF_IO_REDIRECT)
+                       break;

Same point about return value processing...
Each struct_ops callback could have had its own meaning of retvals.
I suspect it would have been more flexible and more powerful this way.

Yeah, I agree, just the 3rd callback of release_io_cmd_t isn't covered in this function.

Quoted 5 lines

Other than that bpf plumbing looks good.

There is an issue with leaking allocated memory in bpf_aio_alloc kfunc (it probably should be KF_ACQUIRE)

It is one problem which troubles me too:

- another callback of struct_ops/bpf_aio_complete_cb is guaranteed to be called after the 'struct bpf_aio' instance is submitted via kfunc bpf_aio_submit(), and it is supposed to be freed from struct_ops/bpf_aio_complete_cb

Quoted 7 lines

- but the following verifier failure is triggered if bpf_aio_alloc and bpf_aio_release are marked as KF_ACQUIRE & KF_RELEASE.

```
libbpf: prog 'ublk_loop_comp_cb': -- BEGIN PROG LOAD LOG --
Global function ublk_loop_comp_cb() doesn't return scalar. Only those are supported.
```
Quoted 7 lines

That's odd.
Adding KF_ACQ/REL to bpf_aio_alloc/release kfuncs shouldn't affect verification of ublk_loop_comp_cb() prog. It's fine for it to stay 'void' return.
You probably made it global function and that's was the reason for this verifier error. Global funcs have to return scalar for now. We can relax this restriction if necessary.

Looks marking ublk_loop_comp_cb() as static doesn't work:

[root@ktest-40 ublk]# make
CLNG-BPF ublk_loop.bpf.o
GEN-SKEL ublk_loop.skel.h
libbpf: relocation against STT_SECTION in non-exec section is not supported!
Error: failed to link '/root/git/linux/tools/testing/selftests/ublk/ublk_loop.bpf.o': Invalid argument (22)

But seems not big deal because we can change its return type to 'int'.

Quoted 17 lines
Quoted 5 lines

Here 'struct bpf_aio' instance isn't stored in map, and it is provided from struct_ops callback(bpf_aio_complete_cb), I appreciate you may share any idea about how to let KF_ACQUIRE/KF_RELEASE cover the usage here.

This is so that:

ublk_loop_comp_cb ->
ublk_loop_comp_and_release_aio ->
bpf_aio_release

would properly recognize that ref to aio is dropped?

Currently the verifier doesn't support that, but there is work in progress to add this feature:

Quoted 7 lines

https://lore.kernel.org/bpf/[email protected]/

then in cfi_stabs annotated bio argument in bpf_aio_complete_cb() as "struct bpf_aio *aio__ref"

Then the verifier will recognize that callback argument comes refcounted and the prog has to call KF_RELEASE kfunc on it.

This looks one very nice feature, thanks for sharing it!

I tried to apply the above patch and patch 3 on next tree and pass 'aio__ref' to the callback cfi_stabs, but still failed:

[root@ktest-40 ublk]# ./test_loop_01.sh
libbpf: prog 'ublk_loop_comp_cb': BPF program load failed: -EINVAL
libbpf: prog 'ublk_loop_comp_cb': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int BPF_PROG(ublk_loop_comp_cb, struct bpf_aio *aio, long ret) @ ublk_loop.c:34
0: (79) r7 = *(u64 *)(r1 +8) ; R1=ctx() R7_w=scalar()
1: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_aio_complete_cb' arg0 has btf_id 37354 type STRUCT 'bpf_aio'
2: R1=ctx() R6_w=trusted_ptr_bpf_aio()
; struct ublk_bpf_io *io = ublk_bpf_acquire_io_from_aio(aio); @ ublk_loop.c:24
2: (bf) r1 = r6 ; R1_w=trusted_ptr_bpf_aio() R6_w=trusted_ptr_bpf_aio()
3: (85) call ublk_bpf_acquire_io_from_aio#43231 ; R0_w=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
4: (bf) r8 = r0 ; R0_w=ptr_ublk_bpf_io(ref_obj_id=1) R8_w=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
; ublk_bpf_complete_io(io, ret); @ ublk_loop.c:26
5: (bf) r1 = r8 ; R1_w=ptr_ublk_bpf_io(ref_obj_id=1) R8_w=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
6: (bc) w2 = w7 ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R7_w=scalar() refs=1
7: (85) call ublk_bpf_complete_io#43241 ; refs=1
; ublk_bpf_release_io_from_aio(io); @ ublk_loop.c:27
8: (bf) r1 = r8 ; R1_w=ptr_ublk_bpf_io(ref_obj_id=1) R8=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
9: (85) call ublk_bpf_release_io_from_aio#43257 ;
; ublk_bpf_dettach_and_complete_aio(aio); @ ublk_loop.c:29
10: (bf) r1 = r6 ; R1_w=trusted_ptr_bpf_aio() R6=trusted_ptr_bpf_aio()
11: (85) call ublk_bpf_dettach_and_complete_aio#43245 ;
; bpf_aio_release(aio); @ ublk_loop.c:30
12: (bf) r1 = r6 ; R1_w=trusted_ptr_bpf_aio() R6=trusted_ptr_bpf_aio()
13: (85) call bpf_aio_release#95841
release kernel function bpf_aio_release expects refcounted PTR_TO_BTF_ID processed 14 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1 -- END PROG LOAD LOG --
libbpf: prog 'ublk_loop_comp_cb': failed to load: -EINVAL
libbpf: failed to load object 'ublk_loop.bpf.o'
fail to load bpf obj from ublk_loop.bpf.o
fail to register bpf prog loop ublk_loop.bpf.o

Thanks,
Ming

#13 lore raw

Re: [RFC PATCH 08/22] ublk: bpf: add bpf struct_ops

Quoted 161 lines · Ming Lei

Hello Alexei,

Quoted 84 lines · Alexei Starovoitov
Quoted 75 lines · Ming Lei

Hello Alexei,

Thanks for your comments!

Quoted 20 lines · Alexei Starovoitov
Quoted 19 lines · Ming Lei
drivers/block/ublk/bpf_ops.c new file +19 -0
+
+/* Return true if io cmd is queued, otherwise forward it to userspace */
+bool ublk_run_bpf_handler(struct ublk_queue *ubq, struct request *req,
+                         queue_io_cmd_t cb)
+{
+       ublk_bpf_return_t ret;
+       struct ublk_rq_data *data = blk_mq_rq_to_pdu(req);
+       struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag);
+       struct ublk_bpf_io *bpf_io = &data->bpf_data;
+       const unsigned long total = iod->nr_sectors << 9;
+       unsigned int done = 0;
+       bool res = true;
+       int err;
+
+       if (!test_bit(UBLK_BPF_IO_PREP, &bpf_io->flags))
+               ublk_bpf_prep_io(bpf_io, iod);
+
+       do {
+               enum ublk_bpf_disposition rc;
Quoted 8 lines

drivers/block/ublk/bpf_ops.c line 167

Quoted 3 lines
drivers/block/ublk/bpf_ops.c new file +3 -0
+               unsigned int bytes;
+
+               ret = cb(bpf_io, done);

High level observation...
I suspect forcing all sturct_ops callbacks to have only these two arguments and packing args into ublk_bpf_io will be limiting in the long term.

There are three callbacks defined, and only the two with same type for queuing io commands are covered in this function.

But yes, callback type belongs to API, which should be designed carefully, and I will think about further.

Quoted 4 lines

And this part of api would need to be redesigned,
but since it's not an uapi... not a big deal.

Quoted 13 lines

drivers/block/ublk/bpf_ops.c line 174

Quoted 7 lines
drivers/block/ublk/bpf_ops.c new file +7 -0
+               rc = ublk_bpf_get_disposition(ret);
+
+               if (rc == UBLK_BPF_IO_QUEUED)
+                       goto exit;
+
+               if (rc == UBLK_BPF_IO_REDIRECT)
+                       break;

Same point about return value processing...
Each struct_ops callback could have had its own meaning of retvals.
I suspect it would have been more flexible and more powerful this way.

Yeah, I agree, just the 3rd callback of release_io_cmd_t isn't covered in this function.

Quoted 5 lines

Other than that bpf plumbing looks good.

There is an issue with leaking allocated memory in bpf_aio_alloc kfunc (it probably should be KF_ACQUIRE)

It is one problem which troubles me too:

- another callback of struct_ops/bpf_aio_complete_cb is guaranteed to be called after the 'struct bpf_aio' instance is submitted via kfunc bpf_aio_submit(), and it is supposed to be freed from struct_ops/bpf_aio_complete_cb

Quoted 7 lines

- but the following verifier failure is triggered if bpf_aio_alloc and bpf_aio_release are marked as KF_ACQUIRE & KF_RELEASE.

```
libbpf: prog 'ublk_loop_comp_cb': -- BEGIN PROG LOAD LOG --
Global function ublk_loop_comp_cb() doesn't return scalar. Only those are supported.
```
Quoted 7 lines

That's odd.
Adding KF_ACQ/REL to bpf_aio_alloc/release kfuncs shouldn't affect verification of ublk_loop_comp_cb() prog. It's fine for it to stay 'void' return.
You probably made it global function and that's was the reason for this verifier error. Global funcs have to return scalar for now. We can relax this restriction if necessary.

Looks marking ublk_loop_comp_cb() as static doesn't work:

[root@ktest-40 ublk]# make
CLNG-BPF ublk_loop.bpf.o
GEN-SKEL ublk_loop.skel.h
libbpf: relocation against STT_SECTION in non-exec section is not supported!
Error: failed to link '/root/git/linux/tools/testing/selftests/ublk/ublk_loop.bpf.o': Invalid argument (22)

But seems not big deal because we can change its return type to 'int'.

Quoted 17 lines
Quoted 5 lines

Here 'struct bpf_aio' instance isn't stored in map, and it is provided from struct_ops callback(bpf_aio_complete_cb), I appreciate you may share any idea about how to let KF_ACQUIRE/KF_RELEASE cover the usage here.

This is so that:

ublk_loop_comp_cb ->
ublk_loop_comp_and_release_aio ->
bpf_aio_release

would properly recognize that ref to aio is dropped?

Currently the verifier doesn't support that, but there is work in progress to add this feature:

Quoted 7 lines

https://lore.kernel.org/bpf/[email protected]/

then in cfi_stabs annotated bio argument in bpf_aio_complete_cb() as "struct bpf_aio *aio__ref"

Then the verifier will recognize that callback argument comes refcounted and the prog has to call KF_RELEASE kfunc on it.

This looks one very nice feature, thanks for sharing it!

I tried to apply the above patch and patch 3 on next tree and pass 'aio__ref' to the callback cfi_stabs, but still failed:

[root@ktest-40 ublk]# ./test_loop_01.sh
libbpf: prog 'ublk_loop_comp_cb': BPF program load failed: -EINVAL
libbpf: prog 'ublk_loop_comp_cb': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
; int BPF_PROG(ublk_loop_comp_cb, struct bpf_aio *aio, long ret) @ ublk_loop.c:34
0: (79) r7 = *(u64 *)(r1 +8) ; R1=ctx() R7_w=scalar()
1: (79) r6 = *(u64 *)(r1 +0)
func 'bpf_aio_complete_cb' arg0 has btf_id 37354 type STRUCT 'bpf_aio' 2: R1=ctx() R6_w=trusted_ptr_bpf_aio()
; struct ublk_bpf_io *io = ublk_bpf_acquire_io_from_aio(aio); @ ublk_loop.c:24
2: (bf) r1 = r6 ; R1_w=trusted_ptr_bpf_aio() R6_w=trusted_ptr_bpf_aio()
3: (85) call ublk_bpf_acquire_io_from_aio#43231 ; R0_w=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
4: (bf) r8 = r0 ; R0_w=ptr_ublk_bpf_io(ref_obj_id=1) R8_w=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
; ublk_bpf_complete_io(io, ret); @ ublk_loop.c:26
5: (bf) r1 = r8 ; R1_w=ptr_ublk_bpf_io(ref_obj_id=1) R8_w=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
6: (bc) w2 = w7 ; R2_w=scalar(smin=0,smax=umax=0xffffffff,var_off=(0x0; 0xffffffff)) R7_w=scalar() refs=1
7: (85) call ublk_bpf_complete_io#43241 ; refs=1
; ublk_bpf_release_io_from_aio(io); @ ublk_loop.c:27
8: (bf) r1 = r8 ; R1_w=ptr_ublk_bpf_io(ref_obj_id=1) R8=ptr_ublk_bpf_io(ref_obj_id=1) refs=1
9: (85) call ublk_bpf_release_io_from_aio#43257 ;
; ublk_bpf_dettach_and_complete_aio(aio); @ ublk_loop.c:29
10: (bf) r1 = r6 ; R1_w=trusted_ptr_bpf_aio() R6=trusted_ptr_bpf_aio()
11: (85) call ublk_bpf_dettach_and_complete_aio#43245 ;
; bpf_aio_release(aio); @ ublk_loop.c:30
12: (bf) r1 = r6 ; R1_w=trusted_ptr_bpf_aio() R6=trusted_ptr_bpf_aio()

Quoted 9 lines

13: (85) call bpf_aio_release#95841
release kernel function bpf_aio_release expects refcounted PTR_TO_BTF_ID processed 14 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1 -- END PROG LOAD LOG --
libbpf: prog 'ublk_loop_comp_cb': failed to load: -EINVAL
libbpf: failed to load object 'ublk_loop.bpf.o'
fail to load bpf obj from ublk_loop.bpf.o
fail to register bpf prog loop ublk_loop.bpf.o

Hi Ming,

Your stub function signature does not look quite right.

It should be <struct_ops_name>__<op_name>, hence:
static void bpf_aio_complete_ops__bpf_aio_complete_cb(struct bpf_aio *io__ref, long ret)

For more detail, look at find_stub_func_proto().

Thanks,
Amery

Quoted 4 lines

Thanks,
Ming

author maintainer
#14 lore raw

RFC PATCH 9/22 ublk: bpf: attach bpf prog to ublk device

In reply to Ming Lei

Attach bpf program to ublk device before adding ublk disk, and detach it after the disk is removed.

ublk device needs to provide the struct_ops ID for attaching the specific prog, and each ublk device has to attach to only single bpf prog.

So that we can use the attached bpf prog for handling ublk IO command.

Meantime add two ublk bpf callbacks for prog to attach & detach ublk device.

---
 drivers/block/ublk/Makefile  |  2 +-
 drivers/block/ublk/bpf.c     | 99 ++++++++++++++++++++++++++++++++++++
 drivers/block/ublk/bpf.h     | 33 ++++++++++++
 drivers/block/ublk/bpf_ops.c | 34 +++++++++++++
 drivers/block/ublk/main.c    | 25 ++++++---
 drivers/block/ublk/ublk.h    | 16 ++++++
 6 files changed, 200 insertions(+), 9 deletions(-)
 create mode 100644 drivers/block/ublk/bpf.c
drivers/block/ublk/Makefile +1 -1
index 7058b0fc13bf..f843a9005cdb 100644
@@ -5,6 +5,6 @@ ccflags-y			+= -I$(src)
 
 ublk_drv-$(CONFIG_BLK_DEV_UBLK)	:= main.o
 ifeq ($(CONFIG_UBLK_BPF), y)
-ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o
+ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o bpf.o
 endif
 obj-$(CONFIG_BLK_DEV_UBLK)	+= ublk_drv.o
drivers/block/ublk/bpf.c new file +99 -0

99 lines not on this page · raw message

drivers/block/ublk/bpf.h +33 -0
index e3505c9ab86a..4e178cbecb74 100644
@@ -7,6 +7,8 @@
 typedef unsigned long ublk_bpf_return_t;
 typedef ublk_bpf_return_t (*queue_io_cmd_t)(struct ublk_bpf_io *io, unsigned int);
 typedef void (*release_io_cmd_t)(struct ublk_bpf_io *io);
+typedef int (*attach_dev_t)(int dev_id);
+typedef void (*detach_dev_t)(int dev_id);
 
 #ifdef CONFIG_UBLK_BPF
 #include <linux/filter.h>
@@ -47,6 +49,12 @@ struct ublk_bpf_ops {
 	/* called when the io command reference drops to zero, can't be sleepable */
 	release_io_cmd_t	release_io_cmd;
 
+	/* called when attaching bpf prog to this ublk dev */
+	attach_dev_t		attach_dev;
+
+	/* called when detaching bpf prog from this ublk dev */
+	detach_dev_t		detach_dev;
+
 	/* private: don't show in doc, must be the last field */
 	struct bpf_prog_provider	provider;
 };
@@ -149,7 +157,12 @@ static inline queue_io_cmd_t ublk_get_bpf_any_io_cb(struct ublk_queue *ubq)
 	return ublk_get_bpf_io_cb_daemon(ubq);
 }
 
+int ublk_bpf_init(void);
 int ublk_bpf_struct_ops_init(void);
+int ublk_bpf_prog_attach(struct bpf_prog_consumer *consumer);
+void ublk_bpf_prog_detach(struct bpf_prog_consumer *consumer);
+int ublk_bpf_attach(struct ublk_device *ub);
+void ublk_bpf_detach(struct ublk_device *ub);
 
 #else
 
@@ -176,9 +189,29 @@ static inline queue_io_cmd_t ublk_get_bpf_any_io_cb(struct ublk_queue *ubq)
 	return NULL;
 }
 
+static inline int ublk_bpf_init(void)
+{
+	return 0;
+}
+
 static inline int ublk_bpf_struct_ops_init(void)
 {
 	return 0;
 }
+
+static inline int ublk_bpf_prog_attach(struct bpf_prog_consumer *consumer)
+{
+	return 0;
+}
+static inline void ublk_bpf_prog_detach(struct bpf_prog_consumer *consumer)
+{
+}
+static inline int ublk_bpf_attach(struct ublk_device *ub)
+{
+	return 0;
+}
+static inline void ublk_bpf_detach(struct ublk_device *ub)
+{
+}
 #endif
 #endif
drivers/block/ublk/bpf_ops.c +34 -0
index 6ac2aebd477e..05d8d415b30d 100644
@@ -133,6 +133,29 @@ static void ublk_bpf_unreg(void *kdata, struct bpf_link *link)
 	kfree(curr);
 }
 
+int ublk_bpf_prog_attach(struct bpf_prog_consumer *consumer)
+{
+	unsigned id = consumer->prog_id;
+	struct ublk_bpf_ops *ops;
+	int ret = -EINVAL;
+
+	mutex_lock(&ublk_bpf_ops_lock);
+	ops = xa_load(&ublk_ops, id);
+	if (ops && ops->id == id)
+		ret = bpf_prog_consumer_attach(consumer, &ops->provider);
+	mutex_unlock(&ublk_bpf_ops_lock);
+
+	return ret;
+}
+
+void ublk_bpf_prog_detach(struct bpf_prog_consumer *consumer)
+{
+	mutex_lock(&ublk_bpf_ops_lock);
+	bpf_prog_consumer_detach(consumer, false);
+	mutex_unlock(&ublk_bpf_ops_lock);
+}
+
+
 static void ublk_bpf_prep_io(struct ublk_bpf_io *io,
 		const struct ublksrv_io_desc *iod)
 {
@@ -231,10 +254,21 @@ static void ublk_bpf_release_io_cmd(struct ublk_bpf_io *io)
 {
 }
 
+static int ublk_bpf_attach_dev(int dev_id)
+{
+	return 0;
+}
+
+static void ublk_bpf_detach_dev(int dev_id)
+{
+}
+
 static struct ublk_bpf_ops __bpf_ublk_bpf_ops = {
 	.queue_io_cmd = ublk_bpf_queue_io_cmd,
 	.queue_io_cmd_daemon = ublk_bpf_run_io_task,
 	.release_io_cmd = ublk_bpf_release_io_cmd,
+	.attach_dev	= ublk_bpf_attach_dev,
+	.detach_dev	= ublk_bpf_detach_dev,
 };
 
 static struct bpf_struct_ops bpf_ublk_bpf_ops = {
drivers/block/ublk/main.c +17 -8
index 29d3e7f656a7..0b136bc5247f 100644
@@ -486,7 +486,7 @@ static inline bool ublk_need_get_data(const struct ublk_queue *ubq)
 }
 
 /* Called in slow path only, keep it noinline for trace purpose */
-static noinline struct ublk_device *ublk_get_device(struct ublk_device *ub)
+struct ublk_device *ublk_get_device(struct ublk_device *ub)
 {
 	if (kobject_get_unless_zero(&ub->cdev_dev.kobj))
 		return ub;
@@ -499,12 +499,6 @@ void ublk_put_device(struct ublk_device *ub)
 	put_device(&ub->cdev_dev);
 }
 
-static inline struct ublk_queue *ublk_get_queue(struct ublk_device *dev,
-		int qid)
-{
-       return (struct ublk_queue *)&(dev->__queues[qid * dev->queue_size]);
-}
-
 static inline bool ublk_rq_has_data(const struct request *rq)
 {
 	return bio_has_data(rq->bio);
@@ -1492,6 +1486,8 @@ static struct gendisk *ublk_detach_disk(struct ublk_device *ub)
 {
 	struct gendisk *disk;
 
+	ublk_bpf_detach(ub);
+
 	/* Sync with ublk_abort_queue() by holding the lock */
 	spin_lock(&ub->lock);
 	disk = ub->ub_disk;
@@ -2206,12 +2202,19 @@ static int ublk_ctrl_start_dev(struct ublk_device *ub, struct io_uring_cmd *cmd)
 			goto out_put_cdev;
 	}
 
-	ret = add_disk(disk);
+	ret = ublk_bpf_attach(ub);
 	if (ret)
 		goto out_put_cdev;
 
+	ret = add_disk(disk);
+	if (ret)
+		goto out_put_bpf;
+
 	set_bit(UB_STATE_USED, &ub->state);
 
+out_put_bpf:
+	if (ret)
+		ublk_bpf_detach(ub);
 out_put_cdev:
 	if (ret) {
 		ublk_detach_disk(ub);
@@ -2967,8 +2970,14 @@ static int __init ublk_init(void)
 	if (ret)
 		goto free_chrdev_region;
 
+	ret = ublk_bpf_init();
+	if (ret)
+		goto unregister_class;
+
 	return 0;
 
+unregister_class:
+	class_unregister(&ublk_chr_class);
 free_chrdev_region:
 	unregister_chrdev_region(ublk_chr_devt, UBLK_MINORS);
 unregister_mis:
drivers/block/ublk/ublk.h +16 -0
index e9ceadbc616d..7579b0032a3c 100644
@@ -7,6 +7,8 @@
 #include <linux/cdev.h>
 #include <uapi/linux/ublk_cmd.h>
 
+#include "bpf_reg.h"
+
 #define UBLK_MINORS		(1U << MINORBITS)
 
 /* private ioctl command mirror */
@@ -153,6 +155,9 @@ struct ublk_device {
 	unsigned long		state;
 	int			ub_number;
 
+#ifdef CONFIG_UBLK_BPF
+	struct bpf_prog_consumer prog;
+#endif
 	struct mutex		mutex;
 
 	spinlock_t		lock;
@@ -173,6 +178,11 @@ struct ublk_params_header {
 	__u32	types;
 };
 
+static inline struct ublk_queue *ublk_get_queue(struct ublk_device *dev,
+		int qid)
+{
+       return (struct ublk_queue *)&(dev->__queues[qid * dev->queue_size]);
+}
 
 static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq,
 		int tag)
@@ -186,6 +196,12 @@ static inline bool ublk_support_bpf(const struct ublk_queue *ubq)
 	return false;
 }
 
+static inline bool ublk_dev_support_bpf(const struct ublk_device *ub)
+{
+	return false;
+}
+
+struct ublk_device *ublk_get_device(struct ublk_device *ub);
 struct ublk_device *ublk_get_device_from_id(int idx);
 void ublk_put_device(struct ublk_device *ub);
 void __ublk_complete_rq(struct request *req);
Signature · 1 line
2.47.0
author maintainer
#15 lore raw

RFC PATCH 10/22 ublk: bpf: add kfunc for ublk bpf prog

In reply to Ming Lei

Define some kfunc for ublk bpf prog for handling ublk IO command in application code.

---
 drivers/block/ublk/bpf.c | 78 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 78 insertions(+)
drivers/block/ublk/bpf.c +78 -0
index 479045a5f0d9..4179b7f61e92 100644
@@ -93,7 +93,85 @@ void ublk_bpf_detach(struct ublk_device *ub)
 	ublk_bpf_prog_detach(&ub->prog);
 }
 
+
+__bpf_kfunc_start_defs();
+__bpf_kfunc const struct ublksrv_io_desc *
+ublk_bpf_get_iod(const struct ublk_bpf_io *io)
+{
+	if (io)
+		return io->iod;
+	return NULL;
+}
+
+__bpf_kfunc unsigned int
+ublk_bpf_get_io_tag(const struct ublk_bpf_io *io)
+{
+	if (io) {
+		const struct request *req = ublk_bpf_get_req(io);
+
+		return req->tag;
+	}
+	return -1;
+}
+
+__bpf_kfunc unsigned int
+ublk_bpf_get_queue_id(const struct ublk_bpf_io *io)
+{
+	if (io) {
+		const struct request *req = ublk_bpf_get_req(io);
+
+		if (req->mq_hctx) {
+			const struct ublk_queue *ubq = req->mq_hctx->driver_data;
+
+			return ubq->q_id;
+		}
+	}
+	return -1;
+}
+
+__bpf_kfunc unsigned int
+ublk_bpf_get_dev_id(const struct ublk_bpf_io *io)
+{
+	if (io) {
+		const struct request *req = ublk_bpf_get_req(io);
+
+		if (req->mq_hctx) {
+			const struct ublk_queue *ubq = req->mq_hctx->driver_data;
+
+			return ubq->dev->dev_info.dev_id;
+		}
+	}
+	return -1;
+}
+
+__bpf_kfunc void
+ublk_bpf_complete_io(struct ublk_bpf_io *io, int res)
+{
+	ublk_bpf_complete_io_cmd(io, res);
+}
+
+BTF_KFUNCS_START(ublk_bpf_kfunc_ids)
+BTF_ID_FLAGS(func, ublk_bpf_complete_io, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, ublk_bpf_get_iod, KF_TRUSTED_ARGS | KF_RET_NULL)
+BTF_ID_FLAGS(func, ublk_bpf_get_io_tag, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, ublk_bpf_get_queue_id, KF_TRUSTED_ARGS)
+BTF_ID_FLAGS(func, ublk_bpf_get_dev_id, KF_TRUSTED_ARGS)
+BTF_KFUNCS_END(ublk_bpf_kfunc_ids)
+
+static const struct btf_kfunc_id_set ublk_bpf_kfunc_set = {
+	.owner = THIS_MODULE,
+	.set   = &ublk_bpf_kfunc_ids,
+};
+
 int __init ublk_bpf_init(void)
 {
+	int err;
+
+	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
+					&ublk_bpf_kfunc_set);
+	if (err) {
+		pr_warn("error while setting UBLK BPF tracing kfuncs: %d", err);
+		return err;
+	}
 	return ublk_bpf_struct_ops_init();
 }
Signature · 1 line
2.47.0
author maintainer
#16 lore raw

RFC PATCH 11/22 ublk: bpf: enable ublk-bpf

In reply to Ming Lei

Add feature flag of UBLK_F_BPF, meantime pass bpf struct_ops prog id via ublk parameter from userspace.

ublk-bpf needs to copy data between ublk request pages and userspace buffer any more, so let ublk_need_map_io() return false for UBLK_F_BPF too.

---
 drivers/block/ublk/bpf.c      |  3 +--
 drivers/block/ublk/main.c     | 15 ++++++++++++++-
 drivers/block/ublk/ublk.h     | 10 ++++++----
 include/uapi/linux/ublk_cmd.h | 14 +++++++++++++-
 4 files changed, 34 insertions(+), 8 deletions(-)
drivers/block/ublk/bpf.c +1 -2
index 4179b7f61e92..ef1546a7ccda 100644
@@ -79,8 +79,7 @@ int ublk_bpf_attach(struct ublk_device *ub)
 	if (!ublk_dev_support_bpf(ub))
 		return 0;
 
-	/* todo: ublk device need to provide struct_ops prog id */
-	ub->prog.prog_id = 0;
+	ub->prog.prog_id = ub->params.bpf.ops_id;
 	ub->prog.ops = &ublk_prog_consumer_ops;
 
 	return ublk_bpf_prog_attach(&ub->prog);
drivers/block/ublk/main.c +14 -1
index 0b136bc5247f..3c2ed9bf924d 100644
@@ -416,6 +416,19 @@ static int ublk_validate_params(const struct ublk_device *ub)
 	else if (ublk_dev_is_zoned(ub))
 		return -EINVAL;
 
+	if (ub->params.types & UBLK_PARAM_TYPE_BPF) {
+		const struct ublk_param_bpf *p = &ub->params.bpf;
+
+		if (!ublk_dev_support_bpf(ub))
+			return -EINVAL;
+
+		if (!(p->flags & UBLK_BPF_HAS_OPS_ID))
+			return -EINVAL;
+	} else {
+		if (ublk_dev_support_bpf(ub))
+			return -EINVAL;
+	}
+
 	return 0;
 }
 
@@ -434,7 +447,7 @@ static inline bool ublk_support_user_copy(const struct ublk_queue *ubq)
 
 static inline bool ublk_need_map_io(const struct ublk_queue *ubq)
 {
-	return !ublk_support_user_copy(ubq);
+	return !(ublk_support_user_copy(ubq) || ublk_support_bpf(ubq));
 }
 
 static inline bool ublk_need_req_ref(const struct ublk_queue *ubq)
drivers/block/ublk/ublk.h +6 -4
index 7579b0032a3c..8343e70bd723 100644
@@ -24,7 +24,8 @@
 		| UBLK_F_CMD_IOCTL_ENCODE \
 		| UBLK_F_USER_COPY \
 		| UBLK_F_ZONED \
-		| UBLK_F_USER_RECOVERY_FAIL_IO)
+		| UBLK_F_USER_RECOVERY_FAIL_IO \
+		| UBLK_F_BPF)
 
 #define UBLK_F_ALL_RECOVERY_FLAGS (UBLK_F_USER_RECOVERY \
 		| UBLK_F_USER_RECOVERY_REISSUE \
@@ -33,7 +34,8 @@
 /* All UBLK_PARAM_TYPE_* should be included here */
 #define UBLK_PARAM_TYPE_ALL                                \
 	(UBLK_PARAM_TYPE_BASIC | UBLK_PARAM_TYPE_DISCARD | \
-	 UBLK_PARAM_TYPE_DEVT | UBLK_PARAM_TYPE_ZONED)
+	 UBLK_PARAM_TYPE_DEVT | UBLK_PARAM_TYPE_ZONED | \
+	 UBLK_PARAM_TYPE_BPF)
 
 enum {
 	UBLK_BPF_IO_PREP	= 0,
@@ -193,12 +195,12 @@ static inline struct ublksrv_io_desc *ublk_get_iod(struct ublk_queue *ubq,
 
 static inline bool ublk_support_bpf(const struct ublk_queue *ubq)
 {
-	return false;
+	return ubq->flags & UBLK_F_BPF;
 }
 
 static inline bool ublk_dev_support_bpf(const struct ublk_device *ub)
 {
-	return false;
+	return ub->dev_info.flags & UBLK_F_BPF;
 }
 
 struct ublk_device *ublk_get_device(struct ublk_device *ub);
include/uapi/linux/ublk_cmd.h +13 -1
index a8bc98bb69fc..27cf14e65cbc 100644
@@ -207,6 +207,9 @@
  */
 #define UBLK_F_USER_RECOVERY_FAIL_IO (1ULL << 9)
 
+/* ublk IO is handled by bpf prog */
+#define UBLK_F_BPF		(1ULL << 10)
+
 /* device state */
 #define UBLK_S_DEV_DEAD	0
 #define UBLK_S_DEV_LIVE	1
@@ -401,6 +404,13 @@ struct ublk_param_zoned {
 	__u8	reserved[20];
 };
 
+struct ublk_param_bpf {
+#define UBLK_BPF_HAS_OPS_ID            (1 << 0)
+	__u8	flags;
+	__u8	ops_id;
+	__u8	reserved[6];
+};
+
 struct ublk_params {
 	/*
 	 * Total length of parameters, userspace has to set 'len' for both
@@ -413,12 +423,14 @@ struct ublk_params {
 #define UBLK_PARAM_TYPE_DISCARD         (1 << 1)
 #define UBLK_PARAM_TYPE_DEVT            (1 << 2)
 #define UBLK_PARAM_TYPE_ZONED           (1 << 3)
+#define UBLK_PARAM_TYPE_BPF             (1 << 4)
 	__u32	types;			/* types of parameter included */
 
 	struct ublk_param_basic		basic;
 	struct ublk_param_discard	discard;
 	struct ublk_param_devt		devt;
-	struct ublk_param_zoned	zoned;
+	struct ublk_param_zoned		zoned;
+	struct ublk_param_bpf		bpf;
 };
 
 #endif
Signature · 1 line
2.47.0
author maintainer
#17 lore raw

RFC PATCH 12/22 selftests: ublk: add tests for the ublk-bpf initial implementation

In reply to Ming Lei

Create one ublk null target over ublk-bpf, in which every block IO is handled by the ublk_null bpf prog. And the whole ublk implementation requires liburing.

Meantime add basic read/write IO test over this ublk null disk, and make sure basic IO function works as expected.

ublk/Makefile is stolen from tools/testing/selftests/hid/Makefile

---
 MAINTAINERS                                   |    1 +
 tools/testing/selftests/Makefile              |    1 +
 tools/testing/selftests/ublk/.gitignore       |    4 +
 tools/testing/selftests/ublk/Makefile         |  228 +++
 tools/testing/selftests/ublk/config           |    2 +
 tools/testing/selftests/ublk/progs/ublk_bpf.h |   13 +
 .../selftests/ublk/progs/ublk_bpf_kfunc.h     |   23 +
 .../testing/selftests/ublk/progs/ublk_null.c  |   63 +
 tools/testing/selftests/ublk/test_common.sh   |   72 +
 tools/testing/selftests/ublk/test_null_01.sh  |   19 +
 tools/testing/selftests/ublk/test_null_02.sh  |   23 +
 tools/testing/selftests/ublk/ublk_bpf.c       | 1429 +++++++++++++++++
 12 files changed, 1878 insertions(+)
 create mode 100644 tools/testing/selftests/ublk/.gitignore
 create mode 100644 tools/testing/selftests/ublk/Makefile
 create mode 100644 tools/testing/selftests/ublk/config
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_bpf.h
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_bpf_kfunc.h
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_null.c
 create mode 100755 tools/testing/selftests/ublk/test_common.sh
 create mode 100755 tools/testing/selftests/ublk/test_null_01.sh
 create mode 100755 tools/testing/selftests/ublk/test_null_02.sh
 create mode 100644 tools/testing/selftests/ublk/ublk_bpf.c
MAINTAINERS +1 -0
index 890f6195d03f..8ff8773377c4 100644
@@ -23984,6 +23984,7 @@ S:	Maintained
 F:	Documentation/block/ublk.rst
 F:	drivers/block/ublk/
 F:	include/uapi/linux/ublk_cmd.h
+F:	tools/testing/selftests/ublk/
 
 UBSAN
 M:	Kees Cook <[email protected]>
tools/testing/selftests/Makefile +1 -0
index 2401e973c359..1c20256e662b 100644
@@ -111,6 +111,7 @@ endif
 TARGETS += tmpfs
 TARGETS += tpm2
 TARGETS += tty
+TARGETS += ublk
 TARGETS += uevent
 TARGETS += user_events
 TARGETS += vDSO
tools/testing/selftests/ublk/.gitignore new file +4 -0
new file mode 100644
index 000000000000..865dca93cf75
@@ -0,0 +1,4 @@
+ublk_bpf
+*.skel.h
+/tools
+*-verify.state
tools/testing/selftests/ublk/Makefile new file +228 -0

228 lines not on this page · raw message

tools/testing/selftests/ublk/config new file +2 -0
new file mode 100644
index 000000000000..295b1f5c6c6c
@@ -0,0 +1,2 @@
+CONFIG_BLK_DEV_UBLK=m
+CONFIG_UBLK_BPF=y
tools/testing/selftests/ublk/progs/ublk_bpf.h new file +13 -0
new file mode 100644
index 000000000000..a302a645b096
@@ -0,0 +1,13 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef UBLK_BPF_GEN_H
+#define UBLK_BPF_GEN_H
+
+#include "ublk_bpf_kfunc.h"
+
+#ifdef DEBUG
+#define BPF_DBG(...) bpf_printk(__VA_ARGS__)
+#else
+#define BPF_DBG(...)
+#endif
+
+#endif
tools/testing/selftests/ublk/progs/ublk_bpf_kfunc.h new file +23 -0
new file mode 100644
index 000000000000..acab490d933c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0
+#ifndef UBLK_BPF_INTERNAL_H
+#define UBLK_BPF_INTERNAL_H
+
+#ifndef BITS_PER_LONG
+#define BITS_PER_LONG	(sizeof(unsigned long) * 8)
+#endif
+
+#define UBLK_BPF_DISPOSITION_BITS       (4)
+#define UBLK_BPF_DISPOSITION_SHIFT      (BITS_PER_LONG - UBLK_BPF_DISPOSITION_BITS)
+
+static inline ublk_bpf_return_t ublk_bpf_return_val(enum ublk_bpf_disposition rc,
+                unsigned int bytes)
+{
+	return (ublk_bpf_return_t) ((unsigned long)rc << UBLK_BPF_DISPOSITION_SHIFT) | bytes;
+}
+
+extern const struct ublksrv_io_desc *ublk_bpf_get_iod(const struct ublk_bpf_io *io) __ksym;
+extern void ublk_bpf_complete_io(const struct ublk_bpf_io *io, int res) __ksym;
+extern int ublk_bpf_get_dev_id(const struct ublk_bpf_io *io) __ksym;
+extern int ublk_bpf_get_queue_id(const struct ublk_bpf_io *io) __ksym;
+extern int ublk_bpf_get_io_tag(const struct ublk_bpf_io *io) __ksym;
+#endif
tools/testing/selftests/ublk/progs/ublk_null.c new file +63 -0

63 lines not on this page · raw message

tools/testing/selftests/ublk/test_common.sh new file +72 -0

72 lines not on this page · raw message

tools/testing/selftests/ublk/test_null_01.sh new file +19 -0
new file mode 100755
index 000000000000..eecb4278e894
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="null_01"
+ERR_CODE=0
+
+_prep_test
+
+# add single ublk null disk without bpf prog
+_add_ublk_dev -t null -n 0 --quiet
+
+# run fio over the two disks
+fio --name=job1 --filename=/dev/ublkb0 --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1
+ERR_CODE=$?
+
+_cleanup_test
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_null_02.sh new file +23 -0
new file mode 100755
index 000000000000..eb0da89f3461
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="null_02"
+ERR_CODE=0
+
+# prepare & register and pin bpf prog
+_prep_bpf_test "null" ublk_null.bpf.o
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t null -n 0 --bpf_prog 0 --quiet
+_add_ublk_dev -t null -n 1 --bpf_prog 0 --quiet
+
+# run fio over the two disks
+fio --name=job1 --filename=/dev/ublkb0 --rw=readwrite --size=256M \
+	--name=job2 --filename=/dev/ublkb1 --rw=readwrite --size=256M > /dev/null 2>&1
+ERR_CODE=$?
+
+# cleanup & unregister and unpin the bpf prog
+_cleanup_bpf_test "null"
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/ublk_bpf.c new file +1429 -0

1,429 lines not on this page · raw message

Signature · 1 line
2.47.0
author maintainer
#18 lore raw

RFC PATCH 13/22 selftests: ublk: add tests for covering io split

In reply to Ming Lei

One io command can be queued in split way, add test case for covering this way:

- split the io command into two sub-io if the io size is bigger than 512

- the 1st sub-io size is 512byte, and the 2nd sub-io is the remained bytes

Complete the whole io command until the two sub-io are queued.

---
 tools/testing/selftests/ublk/Makefile         |  1 +
 .../testing/selftests/ublk/progs/ublk_null.c  | 46 +++++++++++++++++++
 tools/testing/selftests/ublk/test_null_03.sh  | 21 +++++++++
 3 files changed, 68 insertions(+)
 create mode 100755 tools/testing/selftests/ublk/test_null_03.sh
tools/testing/selftests/ublk/Makefile +1 -0
index a95f317211e7..5a940bae9cbb 100644
@@ -21,6 +21,7 @@ endif
 
 TEST_PROGS := test_null_01.sh
 TEST_PROGS += test_null_02.sh
+TEST_PROGS += test_null_03.sh
 
 # Order correspond to 'make run_tests' order
 TEST_GEN_PROGS_EXTENDED = ublk_bpf
tools/testing/selftests/ublk/progs/ublk_null.c +46 -0

56 lines not on this page · raw message

tools/testing/selftests/ublk/test_null_03.sh new file +21 -0
new file mode 100755
index 000000000000..c0b3a4d941c9
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="null_03"
+ERR_CODE=0
+
+# prepare and register & pin bpf prog
+_prep_bpf_test "null" ublk_null.bpf.o
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t null -n 0 --bpf_prog 1 --quiet
+
+# run fio over the ublk disk
+fio --name=job1 --filename=/dev/ublkb0 --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1
+ERR_CODE=$?
+
+# clean and unregister & unpin the bpf prog
+_cleanup_bpf_test "null"
+
+_show_result $TID $ERR_CODE
Signature · 1 line
2.47.0
author maintainer
#19 lore raw

RFC PATCH 14/22 selftests: ublk: add tests for covering redirecting to userspace

In reply to Ming Lei

Reuse ublk-null for testing UBLK_BPF_IO_REDIRECT:

- queue & complete io with odd tag number
- redirect io with even tag number, and let userspace handle their queueing & completion
- also select some ios, and returns -EAGAIN from userspace & marking it as ready for bpf prog to handle, then finally completed with bpf prog in 2nd time

So we can cover code path for UBLK_BPF_IO_REDIRECT.

---
 tools/testing/selftests/ublk/Makefile         |  1 +
 .../selftests/ublk/progs/ublk_bpf_kfunc.h     | 10 +++
 .../testing/selftests/ublk/progs/ublk_null.c  | 68 +++++++++++++++++++
 tools/testing/selftests/ublk/test_null_04.sh  | 21 ++++++
 tools/testing/selftests/ublk/ublk_bpf.c       | 39 ++++++++++-
 5 files changed, 136 insertions(+), 3 deletions(-)
 create mode 100755 tools/testing/selftests/ublk/test_null_04.sh
tools/testing/selftests/ublk/Makefile +1 -0
index 5a940bae9cbb..38903f05d99d 100644
@@ -22,6 +22,7 @@ endif
 TEST_PROGS := test_null_01.sh
 TEST_PROGS += test_null_02.sh
 TEST_PROGS += test_null_03.sh
+TEST_PROGS += test_null_04.sh
 
 # Order correspond to 'make run_tests' order
 TEST_GEN_PROGS_EXTENDED = ublk_bpf
tools/testing/selftests/ublk/progs/ublk_bpf_kfunc.h +10 -0
index acab490d933c..1db8870b57d6 100644
@@ -20,4 +20,14 @@ extern void ublk_bpf_complete_io(const struct ublk_bpf_io *io, int res) __ksym;
 extern int ublk_bpf_get_dev_id(const struct ublk_bpf_io *io) __ksym;
 extern int ublk_bpf_get_queue_id(const struct ublk_bpf_io *io) __ksym;
 extern int ublk_bpf_get_io_tag(const struct ublk_bpf_io *io) __ksym;
+
+static inline unsigned long long build_io_key(const struct ublk_bpf_io *io)
+{
+	unsigned long long dev_id = (unsigned short)ublk_bpf_get_dev_id(io);
+	unsigned long long q_id = (unsigned short)ublk_bpf_get_queue_id(io);
+	unsigned long long tag = ublk_bpf_get_io_tag(io);
+
+	return (dev_id << 32) | (q_id << 16) | tag;
+}
+
 #endif
tools/testing/selftests/ublk/progs/ublk_null.c +68 -0

84 lines not on this page · raw message

tools/testing/selftests/ublk/test_null_04.sh new file +21 -0
new file mode 100755
index 000000000000..f175e2ddb5cd
@@ -0,0 +1,21 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="null_04"
+ERR_CODE=0
+
+# prepare and register & pin bpf prog
+_prep_bpf_test "null" ublk_null.bpf.o
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t null -n 0 --bpf_prog 2 --quiet
+
+# run fio over the ublk disk
+fio --name=job1 --filename=/dev/ublkb0 --ioengine=libaio --rw=readwrite --iodepth=32 --size=256M > /dev/null 2>&1
+ERR_CODE=$?
+
+# clean and unregister & unpin the bpf prog
+_cleanup_bpf_test "null"
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/ublk_bpf.c +36 -3
index 2d923e42845d..e2c2e92268e1 100644
@@ -1283,6 +1283,16 @@ static int cmd_dev_help(char *exe)
 }
 
 /****************** part 2: target implementation ********************/
+//extern int bpf_map_update_elem(int fd, const void *key, const void *value,
+//                                   __u64 flags);
+
+static inline unsigned long long build_io_key(struct ublk_queue *q, int tag)
+{
+       unsigned long long dev_id = (unsigned short)q->dev->dev_info.dev_id;
+       unsigned long long q_id = (unsigned short)q->q_id;
+
+       return (dev_id << 32) | (q_id << 16) | tag;
+}
 
 static int ublk_null_tgt_init(struct ublk_dev *dev)
 {
@@ -1314,12 +1324,35 @@ static int ublk_null_tgt_init(struct ublk_dev *dev)
 static int ublk_null_queue_io(struct ublk_queue *q, int tag)
 {
 	const struct ublksrv_io_desc *iod = ublk_get_iod(q, tag);
+	bool bpf = q->dev->dev_info.flags & UBLK_F_BPF;
 
-	/* won't be called for UBLK_F_BPF */
-	assert(!(q->dev->dev_info.flags & UBLK_F_BPF));
+	/* either !UBLK_F_BPF or UBLK_F_BPF with redirect */
+	assert(!bpf || (bpf && !(tag & 0x1)));
 
-	ublk_complete_io(q, tag, iod->nr_sectors << 9);
+	if (bpf && (tag % 4)) {
+		unsigned long long key = build_io_key(q, tag);
+		int map_fd;
+		int err;
+		int val = 1;
+
+		map_fd = bpf_obj_get("/sys/fs/bpf/ublk/null/io_map");
+		if (map_fd < 0) {
+			ublk_err("Error finding BPF map fd from pinned path\n");
+			goto exit;
+		}
+
+		/* make this io ready for bpf prog to handle */
+		err = bpf_map_update_elem(map_fd, &key, &val, BPF_ANY);
+		if (err) {
+			ublk_err("Error updating map element: %d\n", errno);
+			goto exit;
+		}
+		ublk_complete_io(q, tag, -EAGAIN);
+		return 0;
+	}
 
+exit:
+	ublk_complete_io(q, tag, iod->nr_sectors << 9);
 	return 0;
 }
 
Signature · 1 line
2.47.0
author maintainer
#20 lore raw

RFC PATCH 15/22 ublk: bpf: add bpf aio kfunc

In reply to Ming Lei

Define bpf aio kfunc for bpf prog to submit AIO, so far it begins with filesystem IO only, and in the future, it may be extended for network IO.

Only bvec buffer is covered for doing FS IO over this buffer, but it is easy to cover UBUF because we have the great iov iter.

With bpf aio, not only user-kernel context switch is avoided, but also user-kernel buffer copy is saved. It is very similar with loop's direct IO implementation.

These kfuncs can be used for other subsystems, and should have belong to lib/, but let's start from ublk first. When it becomes mature or gets more use cases, it can be moved to /lib.

Define bpf struct_ops of bpf_aio_complete_ops which needs to be implemented by the caller for completing bpf aio via bpf prog, which will be done in the following patches.

---
 drivers/block/ublk/Makefile  |   2 +-
 drivers/block/ublk/bpf.c     |  40 +++++-
 drivers/block/ublk/bpf.h     |   1 +
 drivers/block/ublk/bpf_aio.c | 251 +++++++++++++++++++++++++++++++++++
 drivers/block/ublk/bpf_aio.h |  66 +++++++++
 5 files changed, 358 insertions(+), 2 deletions(-)
 create mode 100644 drivers/block/ublk/bpf_aio.c
 create mode 100644 drivers/block/ublk/bpf_aio.h
drivers/block/ublk/Makefile +1 -1
index f843a9005cdb..7094607c040d 100644
@@ -5,6 +5,6 @@ ccflags-y			+= -I$(src)
 
 ublk_drv-$(CONFIG_BLK_DEV_UBLK)	:= main.o
 ifeq ($(CONFIG_UBLK_BPF), y)
-ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o bpf.o
+ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o bpf.o bpf_aio.o
 endif
 obj-$(CONFIG_BLK_DEV_UBLK)	+= ublk_drv.o
drivers/block/ublk/bpf.c +39 -1
index ef1546a7ccda..d5880d61abe5 100644
@@ -155,8 +155,23 @@ BTF_ID_FLAGS(func, ublk_bpf_get_iod, KF_TRUSTED_ARGS | KF_RET_NULL)
 BTF_ID_FLAGS(func, ublk_bpf_get_io_tag, KF_TRUSTED_ARGS)
 BTF_ID_FLAGS(func, ublk_bpf_get_queue_id, KF_TRUSTED_ARGS)
 BTF_ID_FLAGS(func, ublk_bpf_get_dev_id, KF_TRUSTED_ARGS)
+
+/* bpf aio kfunc */
+BTF_ID_FLAGS(func, bpf_aio_alloc, KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_aio_alloc_sleepable, KF_RET_NULL)
+BTF_ID_FLAGS(func, bpf_aio_release)
+BTF_ID_FLAGS(func, bpf_aio_submit)
 BTF_KFUNCS_END(ublk_bpf_kfunc_ids)
 
+__bpf_kfunc void bpf_aio_release_dtor(void *aio)
+{
+	bpf_aio_release(aio);
+}
+CFI_NOSEAL(bpf_aio_release_dtor);
+BTF_ID_LIST(bpf_aio_dtor_ids)
+BTF_ID(struct, bpf_aio)
+BTF_ID(func, bpf_aio_release_dtor)
+
 static const struct btf_kfunc_id_set ublk_bpf_kfunc_set = {
 	.owner = THIS_MODULE,
 	.set   = &ublk_bpf_kfunc_ids,
@@ -164,6 +179,12 @@ static const struct btf_kfunc_id_set ublk_bpf_kfunc_set = {
 
 int __init ublk_bpf_init(void)
 {
+	const struct btf_id_dtor_kfunc aio_dtors[] = {
+		{
+			.btf_id	      = bpf_aio_dtor_ids[0],
+			.kfunc_btf_id = bpf_aio_dtor_ids[1]
+		},
+	};
 	int err;
 
 	err = register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS,
@@ -172,5 +193,22 @@ int __init ublk_bpf_init(void)
 		pr_warn("error while setting UBLK BPF tracing kfuncs: %d", err);
 		return err;
 	}
-	return ublk_bpf_struct_ops_init();
+
+	err = ublk_bpf_struct_ops_init();
+	if (err) {
+		pr_warn("error while initializing ublk bpf struct_ops: %d", err);
+		return err;
+	}
+
+	err = register_btf_id_dtor_kfuncs(aio_dtors, ARRAY_SIZE(aio_dtors),
+			THIS_MODULE);
+	if (err) {
+		pr_warn("error while registering aio destructor: %d", err);
+		return err;
+	}
+
+	err = bpf_aio_init();
+	if (err)
+		pr_warn("error while initializing bpf aio kfunc: %d", err);
+	return err;
 }
drivers/block/ublk/bpf.h +1 -0
index 4e178cbecb74..0ab25743ae7d 100644
@@ -3,6 +3,7 @@
 #define UBLK_INT_BPF_HEADER
 
 #include "bpf_reg.h"
+#include "bpf_aio.h"
 
 typedef unsigned long ublk_bpf_return_t;
 typedef ublk_bpf_return_t (*queue_io_cmd_t)(struct ublk_bpf_io *io, unsigned int);
drivers/block/ublk/bpf_aio.c new file +251 -0

251 lines not on this page · raw message

drivers/block/ublk/bpf_aio.h new file +66 -0

66 lines not on this page · raw message

Signature · 1 line
2.47.0
author maintainer
#21 lore raw

RFC PATCH 16/22 ublk: bpf: add bpf aio struct_ops

In reply to Ming Lei

Add bpf aio struct_ops, so that application code can provide bpf aio completion callback in the struct_ops prog, then bpf aio can be supported.

---
 drivers/block/ublk/Makefile      |   2 +-
 drivers/block/ublk/bpf_aio.c     |   7 ++
 drivers/block/ublk/bpf_aio.h     |  12 +++
 drivers/block/ublk/bpf_aio_ops.c | 152 +++++++++++++++++++++++++++++++
 4 files changed, 172 insertions(+), 1 deletion(-)
 create mode 100644 drivers/block/ublk/bpf_aio_ops.c
drivers/block/ublk/Makefile +1 -1
index 7094607c040d..a47f65eb97f8 100644
@@ -5,6 +5,6 @@ ccflags-y			+= -I$(src)
 
 ublk_drv-$(CONFIG_BLK_DEV_UBLK)	:= main.o
 ifeq ($(CONFIG_UBLK_BPF), y)
-ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o bpf.o bpf_aio.o
+ublk_drv-$(CONFIG_BLK_DEV_UBLK)	+= bpf_ops.o bpf.o bpf_aio.o bpf_aio_ops.o
 endif
 obj-$(CONFIG_BLK_DEV_UBLK)	+= ublk_drv.o
drivers/block/ublk/bpf_aio.c +7 -0
index 65013fe8054f..6e93f28f389b 100644
@@ -243,9 +243,16 @@ __bpf_kfunc int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos,
 
 int __init bpf_aio_init(void)
 {
+	int err;
+
 	bpf_aio_cachep = KMEM_CACHE(bpf_aio, SLAB_PANIC);
 	bpf_aio_work_cachep = KMEM_CACHE(bpf_aio_work, SLAB_PANIC);
 	bpf_aio_wq = alloc_workqueue("bpf_aio", WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
 
+	err = bpf_aio_struct_ops_init();
+	if (err) {
+		pr_warn("error while initializing bpf aio struct_ops: %d", err);
+		return err;
+	}
 	return 0;
 }
drivers/block/ublk/bpf_aio.h +12 -0
index 625737965c90..07fcd43fd2ac 100644
@@ -3,6 +3,8 @@
 #ifndef UBLK_BPF_AIO_HEADER
 #define UBLK_BPF_AIO_HEADER
 
+#include "bpf_reg.h"
+
 #define	BPF_AIO_OP_BITS		8
 #define	BPF_AIO_OP_MASK		((1 << BPF_AIO_OP_BITS) - 1)
 
@@ -47,9 +49,18 @@ struct bpf_aio {
 
 typedef void (*bpf_aio_complete_t)(struct bpf_aio *io, long ret);
 
+/**
+ * struct bpf_aio_complete_ops - A BPF struct_ops of callbacks allowing to
+ * 	complete `bpf_aio` submitted by `bpf_aio_submit()`
+ * @id: id used by bpf aio consumer, defined by globally
+ * @bpf_aio_complete_cb: callback for completing submitted `bpf_aio`
+ * @provider: holding all consumers of this struct_ops prog, used by
+ * 	kernel only
+ */
 struct bpf_aio_complete_ops {
 	unsigned int		id;
 	bpf_aio_complete_t	bpf_aio_complete_cb;
+	struct bpf_prog_provider provider;
 };
 
 static inline unsigned int bpf_aio_get_op(const struct bpf_aio *aio)
@@ -58,6 +69,7 @@ static inline unsigned int bpf_aio_get_op(const struct bpf_aio *aio)
 }
 
 int bpf_aio_init(void);
+int bpf_aio_struct_ops_init(void);
 struct bpf_aio *bpf_aio_alloc(unsigned int op, enum bpf_aio_flag aio_flags);
 struct bpf_aio *bpf_aio_alloc_sleepable(unsigned int op, enum bpf_aio_flag aio_flags);
 void bpf_aio_release(struct bpf_aio *aio);
drivers/block/ublk/bpf_aio_ops.c new file +152 -0

152 lines not on this page · raw message

Signature · 1 line
2.47.0
author maintainer
#22 lore raw

RFC PATCH 17/22 ublk: bpf: attach bpf aio prog to ublk device

In reply to Ming Lei

Attach bpf aio program to ublk device before adding ublk disk, and detach it after the disk is removed. And when the bpf aio prog is unregistered, all devices will detach from the prog automatically.

ublk device needs to provide the bpf aio struct_ops ID for attaching the specific prog, and each ublk device has to attach to only single bpf prog.

So that we can use the attached bpf aio prog to submit bpf aio for handling ublk IO.

Given bpf aio prog is attached to ublk device, ublk bpf prog has to provide one kfunc to assign 'bpf_aio_complete_ops *' to 'struct bpf_aio' instance.

---
 drivers/block/ublk/bpf.c         | 81 +++++++++++++++++++++++++++++++-
 drivers/block/ublk/bpf_aio.c     |  4 ++
 drivers/block/ublk/bpf_aio.h     |  4 ++
 drivers/block/ublk/bpf_aio_ops.c | 22 +++++++++
 drivers/block/ublk/ublk.h        | 10 ++++
 include/uapi/linux/ublk_cmd.h    |  4 +-
 6 files changed, 123 insertions(+), 2 deletions(-)
drivers/block/ublk/bpf.c +80 -1
index d5880d61abe5..921bbbcf4d9e 100644
@@ -19,6 +19,79 @@ static int ublk_set_bpf_ops(struct ublk_device *ub,
 	return 0;
 }
 
+static int ublk_set_bpf_aio_op(struct ublk_device *ub,
+		struct bpf_aio_complete_ops *ops)
+{
+	int i;
+
+	for (i = 0; i < ub->dev_info.nr_hw_queues; i++) {
+		if (ops && ublk_get_queue(ub, i)->bpf_aio_ops) {
+			ublk_set_bpf_aio_op(ub, NULL);
+			return -EBUSY;
+		}
+		ublk_get_queue(ub, i)->bpf_aio_ops = ops;
+	}
+	return 0;
+}
+
+static int ublk_bpf_aio_prog_attach_cb(struct bpf_prog_consumer *consumer,
+				       struct bpf_prog_provider *provider)
+{
+	struct ublk_device *ub = container_of(consumer, struct ublk_device,
+					      aio_prog);
+	struct bpf_aio_complete_ops *ops = container_of(provider,
+			struct bpf_aio_complete_ops, provider);
+	int ret = -ENODEV;
+
+	if (ublk_get_device(ub)) {
+		ret = ublk_set_bpf_aio_op(ub, ops);
+		if (ret)
+			ublk_put_device(ub);
+	}
+
+	return ret;
+}
+
+static void ublk_bpf_aio_prog_detach_cb(struct bpf_prog_consumer *consumer,
+					bool unreg)
+{
+	struct ublk_device *ub = container_of(consumer, struct ublk_device,
+					      aio_prog);
+
+	if (unreg) {
+		blk_mq_freeze_queue(ub->ub_disk->queue);
+		ublk_set_bpf_aio_op(ub, NULL);
+		blk_mq_unfreeze_queue(ub->ub_disk->queue);
+	} else {
+		ublk_set_bpf_aio_op(ub, NULL);
+	}
+	ublk_put_device(ub);
+}
+
+static const struct bpf_prog_consumer_ops ublk_aio_prog_consumer_ops = {
+	.attach_fn	= ublk_bpf_aio_prog_attach_cb,
+	.detach_fn	= ublk_bpf_aio_prog_detach_cb,
+};
+
+static int ublk_bpf_aio_attach(struct ublk_device *ub)
+{
+	if (!ublk_dev_support_bpf_aio(ub))
+		return 0;
+
+	ub->aio_prog.prog_id = ub->params.bpf.aio_ops_id;
+	ub->aio_prog.ops = &ublk_aio_prog_consumer_ops;
+
+	return bpf_aio_prog_attach(&ub->aio_prog);
+}
+
+static void ublk_bpf_aio_detach(struct ublk_device *ub)
+{
+	if (!ublk_dev_support_bpf_aio(ub))
+		return;
+	bpf_aio_prog_detach(&ub->aio_prog);
+}
+
+
 static int ublk_bpf_prog_attach_cb(struct bpf_prog_consumer *consumer,
 				   struct bpf_prog_provider *provider)
 {
@@ -76,19 +149,25 @@ static const struct bpf_prog_consumer_ops ublk_prog_consumer_ops = {
 
 int ublk_bpf_attach(struct ublk_device *ub)
 {
+	int ret;
+
 	if (!ublk_dev_support_bpf(ub))
 		return 0;
 
 	ub->prog.prog_id = ub->params.bpf.ops_id;
 	ub->prog.ops = &ublk_prog_consumer_ops;
 
-	return ublk_bpf_prog_attach(&ub->prog);
+	ret = ublk_bpf_prog_attach(&ub->prog);
+	if (ret)
+		return ret;
+	return ublk_bpf_aio_attach(ub);
 }
 
 void ublk_bpf_detach(struct ublk_device *ub)
 {
 	if (!ublk_dev_support_bpf(ub))
 		return;
+	ublk_bpf_aio_detach(ub);
 	ublk_bpf_prog_detach(&ub->prog);
 }
 
drivers/block/ublk/bpf_aio.c +4 -0
index 6e93f28f389b..da050be4b710 100644
@@ -213,6 +213,10 @@ __bpf_kfunc int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos,
 {
 	struct file *file;
 
+	/*
+	 * ->ops has to assigned by kfunc of consumer subsystem because
+	 * bpf prog lifetime is aligned with the consumer subsystem
+	 */
 	if (!aio->ops)
 		return -EINVAL;
 
drivers/block/ublk/bpf_aio.h +4 -0
index 07fcd43fd2ac..d144c5e20dcb 100644
@@ -75,4 +75,8 @@ struct bpf_aio *bpf_aio_alloc_sleepable(unsigned int op, enum bpf_aio_flag aio_f
 void bpf_aio_release(struct bpf_aio *aio);
 int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos, unsigned bytes,
 		unsigned io_flags);
+
+int bpf_aio_prog_attach(struct bpf_prog_consumer *consumer);
+void bpf_aio_prog_detach(struct bpf_prog_consumer *consumer);
+
 #endif
drivers/block/ublk/bpf_aio_ops.c +22 -0
index 12757f634dbd..04ad45fd24e6 100644
@@ -120,6 +120,28 @@ static void bpf_aio_unreg(void *kdata, struct bpf_link *link)
 	kfree(curr);
 }
 
+int bpf_aio_prog_attach(struct bpf_prog_consumer *consumer)
+{
+	unsigned id = consumer->prog_id;
+	struct bpf_aio_complete_ops *ops;
+	int ret = -EINVAL;
+
+	mutex_lock(&bpf_aio_ops_lock);
+	ops = xa_load(&bpf_aio_all_ops, id);
+	if (ops && ops->id == id)
+		ret = bpf_prog_consumer_attach(consumer, &ops->provider);
+	mutex_unlock(&bpf_aio_ops_lock);
+
+	return ret;
+}
+
+void bpf_aio_prog_detach(struct bpf_prog_consumer *consumer)
+{
+	mutex_lock(&bpf_aio_ops_lock);
+	bpf_prog_consumer_detach(consumer, false);
+	mutex_unlock(&bpf_aio_ops_lock);
+}
+
 static void bpf_aio_cb(struct bpf_aio *io, long ret)
 {
 }
drivers/block/ublk/ublk.h +10 -0
index 8343e70bd723..2c33f6a94bf2 100644
@@ -126,6 +126,7 @@ struct ublk_queue {
 
 #ifdef CONFIG_UBLK_BPF
 	struct ublk_bpf_ops     *bpf_ops;
+	struct bpf_aio_complete_ops     *bpf_aio_ops;
 #endif
 
 	unsigned short force_abort:1;
@@ -159,6 +160,7 @@ struct ublk_device {
 
 #ifdef CONFIG_UBLK_BPF
 	struct bpf_prog_consumer prog;
+	struct bpf_prog_consumer aio_prog;
 #endif
 	struct mutex		mutex;
 
@@ -203,6 +205,14 @@ static inline bool ublk_dev_support_bpf(const struct ublk_device *ub)
 	return ub->dev_info.flags & UBLK_F_BPF;
 }
 
+static inline bool ublk_dev_support_bpf_aio(const struct ublk_device *ub)
+{
+	if (!ublk_dev_support_bpf(ub))
+		return false;
+
+	return ub->params.bpf.flags & UBLK_BPF_HAS_AIO_OPS_ID;
+}
+
 struct ublk_device *ublk_get_device(struct ublk_device *ub);
 struct ublk_device *ublk_get_device_from_id(int idx);
 void ublk_put_device(struct ublk_device *ub);
include/uapi/linux/ublk_cmd.h +3 -1
index 27cf14e65cbc..ed6df4d61e89 100644
@@ -406,9 +406,11 @@ struct ublk_param_zoned {
 
 struct ublk_param_bpf {
 #define UBLK_BPF_HAS_OPS_ID            (1 << 0)
+#define UBLK_BPF_HAS_AIO_OPS_ID        (1 << 1)
 	__u8	flags;
 	__u8	ops_id;
-	__u8	reserved[6];
+	__u16	aio_ops_id;
+	__u8	reserved[4];
 };
 
 struct ublk_params {
Signature · 1 line
2.47.0
author maintainer
#23 lore raw

RFC PATCH 18/22 ublk: bpf: add several ublk bpf aio kfuncs

In reply to Ming Lei

Add ublk bpf aio kfuncs for bpf prog to do:

- prepare buffer
- assign bpf aio struct_ops
- submit bpf aios for handle ublk io command
- deal with ublk io and bpf aio lifetime, and make sure that ublk io won't be completed until all bpf aios are completed

---
 drivers/block/ublk/bpf.c     | 77 ++++++++++++++++++++++++++++++++++++
 drivers/block/ublk/bpf_aio.c |  6 ++-
 drivers/block/ublk/bpf_aio.h | 38 +++++++++++++++++-
 drivers/block/ublk/ublk.h    |  2 +
 4 files changed, 121 insertions(+), 2 deletions(-)
drivers/block/ublk/bpf.c +77 -0
index 921bbbcf4d9e..c0babf6d5868 100644
@@ -228,6 +228,77 @@ ublk_bpf_complete_io(struct ublk_bpf_io *io, int res)
 	ublk_bpf_complete_io_cmd(io, res);
 }
 
+/*
+ * Called before submitting one bpf aio in prog, and this ublk IO's
+ * reference is increased.
+ *
+ * Grab reference of `io` for this `aio`, and the reference will be dropped
+ * by ublk_bpf_dettach_and_complete_aio()
+ */
+__bpf_kfunc int
+ublk_bpf_attach_and_prep_aio(const struct ublk_bpf_io *_io, unsigned off,
+		unsigned bytes, struct bpf_aio *aio)
+{
+	struct ublk_bpf_io *io = (struct ublk_bpf_io *)_io;
+	const struct request *req;
+	const struct ublk_rq_data *data;
+	const struct ublk_bpf_io *bpf_io;
+
+	if (!io || !aio)
+		return -EINVAL;
+
+	req = ublk_bpf_get_req(io);
+	if (!req)
+		return -EINVAL;
+
+	if (off + bytes > blk_rq_bytes(req))
+		return -EINVAL;
+
+	if (req->mq_hctx) {
+		const struct ublk_queue *ubq = req->mq_hctx->driver_data;
+
+		bpf_aio_assign_cb(aio, ubq->bpf_aio_ops);
+	}
+
+	data = blk_mq_rq_to_pdu((struct request *)req);
+	bpf_io = &data->bpf_data;
+	bpf_aio_assign_buf(aio, &bpf_io->buf, off, bytes);
+
+	refcount_inc(&io->ref);
+	aio->private_data = (void *)io;
+
+	return 0;
+}
+
+/*
+ * Called after this attached aio is completed, and the associated ublk IO's
+ * reference is decreased, and if the reference is dropped to zero, complete
+ * this ublk IO.
+ *
+ * Return -EIOCBQUEUED if this `io` is being handled, and 0 is returned
+ * if it can be completed now.
+ */
+__bpf_kfunc void
+ublk_bpf_dettach_and_complete_aio(struct bpf_aio *aio)
+{
+	struct ublk_bpf_io *io = aio->private_data;
+
+	if (io) {
+		ublk_bpf_io_dec_ref(io);
+		aio->private_data = NULL;
+	}
+}
+
+__bpf_kfunc struct ublk_bpf_io *ublk_bpf_acquire_io_from_aio(struct bpf_aio *aio)
+{
+	return aio->private_data;
+}
+
+__bpf_kfunc void ublk_bpf_release_io_from_aio(struct ublk_bpf_io *io)
+{
+}
+
+
 BTF_KFUNCS_START(ublk_bpf_kfunc_ids)
 BTF_ID_FLAGS(func, ublk_bpf_complete_io, KF_TRUSTED_ARGS)
 BTF_ID_FLAGS(func, ublk_bpf_get_iod, KF_TRUSTED_ARGS | KF_RET_NULL)
@@ -240,6 +311,12 @@ BTF_ID_FLAGS(func, bpf_aio_alloc, KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_aio_alloc_sleepable, KF_RET_NULL)
 BTF_ID_FLAGS(func, bpf_aio_release)
 BTF_ID_FLAGS(func, bpf_aio_submit)
+
+/* ublk bpf aio kfuncs */
+BTF_ID_FLAGS(func, ublk_bpf_attach_and_prep_aio)
+BTF_ID_FLAGS(func, ublk_bpf_dettach_and_complete_aio)
+BTF_ID_FLAGS(func, ublk_bpf_acquire_io_from_aio, KF_ACQUIRE)
+BTF_ID_FLAGS(func, ublk_bpf_release_io_from_aio, KF_RELEASE)
 BTF_KFUNCS_END(ublk_bpf_kfunc_ids)
 
 __bpf_kfunc void bpf_aio_release_dtor(void *aio)
drivers/block/ublk/bpf_aio.c +5 -1
index da050be4b710..06a6cc8f38b1 100644
@@ -211,6 +211,7 @@ __bpf_kfunc void bpf_aio_release(struct bpf_aio *aio)
 __bpf_kfunc int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos,
 		unsigned bytes, unsigned io_flags)
 {
+	unsigned op = bpf_aio_get_op(aio);
 	struct file *file;
 
 	/*
@@ -220,6 +221,9 @@ __bpf_kfunc int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos,
 	if (!aio->ops)
 		return -EINVAL;
 
+	if (unlikely((bytes > aio->buf_size) && bpf_aio_is_rw(op)))
+		return -EINVAL;
+
 	file = fget(fd);
 	if (!file)
 		return -EINVAL;
@@ -232,7 +236,7 @@ __bpf_kfunc int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos,
 	aio->iocb.ki_filp = file;
 	aio->iocb.ki_flags = io_flags;
 	aio->bytes = bytes;
-	if (bpf_aio_is_rw(bpf_aio_get_op(aio))) {
+	if (bpf_aio_is_rw(op)) {
 		if (file->f_flags & O_DIRECT)
 			aio->iocb.ki_flags |= IOCB_DIRECT;
 		else
drivers/block/ublk/bpf_aio.h +37 -1
index d144c5e20dcb..0683139f5354 100644
@@ -40,11 +40,15 @@ struct bpf_aio_buf {
 
 struct bpf_aio {
 	unsigned int opf;
-	unsigned int bytes;
+	union {
+		unsigned int bytes;
+		unsigned int buf_size;
+	};
 	struct bpf_aio_buf	buf;
 	struct bpf_aio_work	*work;
 	const struct bpf_aio_complete_ops *ops;
 	struct kiocb iocb;
+	void	*private_data;
 };
 
 typedef void (*bpf_aio_complete_t)(struct bpf_aio *io, long ret);
@@ -68,6 +72,38 @@ static inline unsigned int bpf_aio_get_op(const struct bpf_aio *aio)
 	return aio->opf & BPF_AIO_OP_MASK;
 }
 
+/* Must be called from kfunc defined in consumer subsystem */
+static inline void bpf_aio_assign_cb(struct bpf_aio *aio,
+		const struct bpf_aio_complete_ops *ops)
+{
+	aio->ops = ops;
+}
+
+/*
+ * Skip `skip` bytes and assign the advanced source buffer for `aio`, so
+ * we can cover this part of source buffer by this `aio`
+ */
+static inline void bpf_aio_assign_buf(struct bpf_aio *aio,
+		const struct bpf_aio_buf *src, unsigned skip,
+		unsigned bytes)
+{
+	const struct bio_vec *bvec, *end;
+	struct bpf_aio_buf *abuf = &aio->buf;
+
+	skip += src->bvec_off;
+	for (bvec = src->bvec, end = bvec + src->nr_bvec; bvec < end; bvec++) {
+		if (likely(skip < bvec->bv_len))
+			break;
+		skip -= bvec->bv_len;
+	}
+
+	aio->buf_size = bytes;
+	abuf->bvec_off = skip;
+	abuf->nr_bvec = src->nr_bvec - (bvec - src->bvec);
+	abuf->bvec = bvec;
+}
+
+
 int bpf_aio_init(void);
 int bpf_aio_struct_ops_init(void);
 struct bpf_aio *bpf_aio_alloc(unsigned int op, enum bpf_aio_flag aio_flags);
drivers/block/ublk/ublk.h +2 -0
index 2c33f6a94bf2..4bd04512c894 100644
@@ -8,6 +8,7 @@
 #include <uapi/linux/ublk_cmd.h>
 
 #include "bpf_reg.h"
+#include "bpf_aio.h"
 
 #define UBLK_MINORS		(1U << MINORBITS)
 
@@ -47,6 +48,7 @@ struct ublk_bpf_io {
 	unsigned long			flags;
 	refcount_t                      ref;
 	int				res;
+	struct bpf_aio_buf		buf;
 };
 
 struct ublk_rq_data {
Signature · 1 line
2.47.0
author maintainer
#24 lore raw

RFC PATCH 19/22 ublk: bpf: wire bpf aio with ublk io handling

In reply to Ming Lei

Add ublk_bpf_aio_prep_io_buf() and call it before running ublk bpf prog, so wire everything together.

---
 drivers/block/ublk/bpf.h     | 13 +++++++++
 drivers/block/ublk/bpf_ops.c | 51 +++++++++++++++++++++++++++++++++++-
 drivers/block/ublk/main.c    |  5 ----
 drivers/block/ublk/ublk.h    |  6 +++++
 4 files changed, 69 insertions(+), 6 deletions(-)
drivers/block/ublk/bpf.h +13 -0
index 0ab25743ae7d..a3d238bc707d 100644
@@ -99,6 +99,9 @@ static inline void ublk_bpf_io_dec_ref(struct ublk_bpf_io *io)
 				ubq->bpf_ops->release_io_cmd(io);
 		}
 
+		if (test_bit(UBLK_BPF_BVEC_ALLOCATED, &io->flags))
+			kvfree(io->buf.bvec);
+
 		if (test_bit(UBLK_BPF_IO_COMPLETED, &io->flags)) {
 			smp_rmb();
 			__clear_bit(UBLK_BPF_IO_PREP, &io->flags);
@@ -158,6 +161,11 @@ static inline queue_io_cmd_t ublk_get_bpf_any_io_cb(struct ublk_queue *ubq)
 	return ublk_get_bpf_io_cb_daemon(ubq);
 }
 
+static inline bool ublk_support_bpf_aio(const struct ublk_queue *ubq)
+{
+	return ublk_support_bpf(ubq) && ubq->bpf_aio_ops;
+}
+
 int ublk_bpf_init(void);
 int ublk_bpf_struct_ops_init(void);
 int ublk_bpf_prog_attach(struct bpf_prog_consumer *consumer);
@@ -190,6 +198,11 @@ static inline queue_io_cmd_t ublk_get_bpf_any_io_cb(struct ublk_queue *ubq)
 	return NULL;
 }
 
+static inline bool ublk_support_bpf_aio(const struct ublk_queue *ubq)
+{
+	return false;
+}
+
 static inline int ublk_bpf_init(void)
 {
 	return 0;
drivers/block/ublk/bpf_ops.c +50 -1

64 lines not on this page · raw message

drivers/block/ublk/main.c +0 -5
index 3c2ed9bf924d..1974ebd33ce0 100644
@@ -512,11 +512,6 @@ void ublk_put_device(struct ublk_device *ub)
 	put_device(&ub->cdev_dev);
 }
 
-static inline bool ublk_rq_has_data(const struct request *rq)
-{
-	return bio_has_data(rq->bio);
-}
-
 static inline char *ublk_queue_cmd_buf(struct ublk_device *ub, int q_id)
 {
 	return ublk_get_queue(ub, q_id)->io_cmd_buf;
drivers/block/ublk/ublk.h +6 -0
index 4bd04512c894..00b09589d95c 100644
@@ -41,6 +41,7 @@
 enum {
 	UBLK_BPF_IO_PREP	= 0,
 	UBLK_BPF_IO_COMPLETED   = 1,
+	UBLK_BPF_BVEC_ALLOCATED	= 2,
 };
 
 struct ublk_bpf_io {
@@ -215,6 +216,11 @@ static inline bool ublk_dev_support_bpf_aio(const struct ublk_device *ub)
 	return ub->params.bpf.flags & UBLK_BPF_HAS_AIO_OPS_ID;
 }
 
+static inline bool ublk_rq_has_data(const struct request *rq)
+{
+	return bio_has_data(rq->bio);
+}
+
 struct ublk_device *ublk_get_device(struct ublk_device *ub);
 struct ublk_device *ublk_get_device_from_id(int idx);
 void ublk_put_device(struct ublk_device *ub);
Signature · 1 line
2.47.0
author maintainer
#25 lore raw

RFC PATCH 20/22 selftests: add tests for ublk bpf aio

In reply to Ming Lei

Create ublk loop target which uses bpf aio to submit & complete FS IO, then run write & read & verify on the ublk loop disk for making sure ublk bpf aio works as expected.

---
 tools/testing/selftests/ublk/Makefile         |   3 +
 .../selftests/ublk/progs/ublk_bpf_kfunc.h     |  11 ++
 .../testing/selftests/ublk/progs/ublk_loop.c  | 166 ++++++++++++++++++
 tools/testing/selftests/ublk/test_common.sh   |  47 +++++
 tools/testing/selftests/ublk/test_loop_01.sh  |  33 ++++
 tools/testing/selftests/ublk/test_loop_02.sh  |  24 +++
 tools/testing/selftests/ublk/ublk_bpf.c       | 141 ++++++++++++++-
 7 files changed, 419 insertions(+), 6 deletions(-)
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_loop.c
 create mode 100755 tools/testing/selftests/ublk/test_loop_01.sh
 create mode 100755 tools/testing/selftests/ublk/test_loop_02.sh
tools/testing/selftests/ublk/Makefile +3 -0
index 38903f05d99d..2540ae7a75a3 100644
@@ -24,6 +24,9 @@ TEST_PROGS += test_null_02.sh
 TEST_PROGS += test_null_03.sh
 TEST_PROGS += test_null_04.sh
 
+TEST_PROGS += test_loop_01.sh
+TEST_PROGS += test_loop_02.sh
+
 # Order correspond to 'make run_tests' order
 TEST_GEN_PROGS_EXTENDED = ublk_bpf
 
tools/testing/selftests/ublk/progs/ublk_bpf_kfunc.h +11 -0
index 1db8870b57d6..9fb134e40d49 100644
@@ -21,6 +21,17 @@ extern int ublk_bpf_get_dev_id(const struct ublk_bpf_io *io) __ksym;
 extern int ublk_bpf_get_queue_id(const struct ublk_bpf_io *io) __ksym;
 extern int ublk_bpf_get_io_tag(const struct ublk_bpf_io *io) __ksym;
 
+extern void ublk_bpf_dettach_and_complete_aio(struct bpf_aio *aio) __ksym;
+extern int ublk_bpf_attach_and_prep_aio(const struct ublk_bpf_io *_io, unsigned off, unsigned bytes, struct bpf_aio *aio) __ksym;
+extern struct ublk_bpf_io *ublk_bpf_acquire_io_from_aio(struct bpf_aio *aio) __ksym;
+extern void ublk_bpf_release_io_from_aio(struct ublk_bpf_io *io) __ksym;
+
+extern struct bpf_aio *bpf_aio_alloc(unsigned int op, enum bpf_aio_flag flags) __ksym;
+extern struct bpf_aio *bpf_aio_alloc_sleepable(unsigned int op, enum bpf_aio_flag flags) __ksym;
+extern void bpf_aio_release(struct bpf_aio *aio) __ksym;
+extern int bpf_aio_submit(struct bpf_aio *aio, int fd, loff_t pos,
+                unsigned bytes, unsigned io_flags) __ksym;
+
 static inline unsigned long long build_io_key(const struct ublk_bpf_io *io)
 {
 	unsigned long long dev_id = (unsigned short)ublk_bpf_get_dev_id(io);
tools/testing/selftests/ublk/progs/ublk_loop.c new file +166 -0

166 lines not on this page · raw message

tools/testing/selftests/ublk/test_common.sh +47 -0

50 lines not on this page · raw message

tools/testing/selftests/ublk/test_loop_01.sh new file +33 -0
new file mode 100755
index 000000000000..10c73ec0a01a
@@ -0,0 +1,33 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="loop_01"
+ERR_CODE=0
+
+# prepare & register and pin bpf prog
+_prep_bpf_test "loop" ublk_loop.bpf.o
+
+backfile_0=`_create_backfile 256M`
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t loop -n 0 --bpf_prog 16 --bpf_aio_prog 16 --quiet $backfile_0
+
+# run fio over the ublk disk
+fio --name=write_and_verify \
+    --filename=/dev/ublkb0 \
+    --ioengine=libaio --iodepth=4 \
+    --rw=write \
+    --size=256M \
+    --direct=1 \
+    --verify=crc32c \
+    --do_verify=1 \
+    --bs=4k > /dev/null 2>&1
+ERR_CODE=$?
+
+# cleanup & unregister and unpin the bpf prog
+_cleanup_bpf_test "loop"
+
+_remove_backfile $backfile_0
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_loop_02.sh new file +24 -0
new file mode 100755
index 000000000000..05c3a863f517
@@ -0,0 +1,24 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="loop_02"
+ERR_CODE=0
+
+# prepare & register and pin bpf prog
+_prep_bpf_test "loop" ublk_loop.bpf.o
+
+backfile_0=`_create_backfile 256M`
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t loop -n 0 --bpf_prog 16 --bpf_aio_prog 16 --quiet $backfile_0
+
+_mkfs_mount_test /dev/ublkb0
+ERR_CODE=$?
+
+# cleanup & unregister and unpin the bpf prog
+_cleanup_bpf_test "loop"
+
+_remove_backfile $backfile_0
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/ublk_bpf.c +135 -6

217 lines not on this page · raw message

Signature · 1 line
2.47.0
author maintainer
#26 lore raw

RFC PATCH 21/22 selftests: add tests for covering both bpf aio and split

In reply to Ming Lei

Add ublk-stripe for covering both bpf aio and io split features.

---
 tools/testing/selftests/ublk/Makefile         |   3 +
 .../selftests/ublk/progs/ublk_stripe.c        | 319 ++++++++++++++++++
 .../testing/selftests/ublk/test_stripe_01.sh  |  35 ++
 .../testing/selftests/ublk/test_stripe_02.sh  |  26 ++
 tools/testing/selftests/ublk/ublk_bpf.c       |  88 ++++-
 5 files changed, 468 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/ublk/progs/ublk_stripe.c
 create mode 100755 tools/testing/selftests/ublk/test_stripe_01.sh
 create mode 100755 tools/testing/selftests/ublk/test_stripe_02.sh
tools/testing/selftests/ublk/Makefile +3 -0
index 2540ae7a75a3..7c30c5728694 100644
@@ -27,6 +27,9 @@ TEST_PROGS += test_null_04.sh
 TEST_PROGS += test_loop_01.sh
 TEST_PROGS += test_loop_02.sh
 
+TEST_PROGS += test_stripe_01.sh
+TEST_PROGS += test_stripe_02.sh
+
 # Order correspond to 'make run_tests' order
 TEST_GEN_PROGS_EXTENDED = ublk_bpf
 
tools/testing/selftests/ublk/progs/ublk_stripe.c new file +319 -0

319 lines not on this page · raw message

tools/testing/selftests/ublk/test_stripe_01.sh new file +35 -0
new file mode 100755
index 000000000000..3c21f7db495a
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="stripe_01"
+ERR_CODE=0
+
+# prepare & register and pin bpf prog
+_prep_bpf_test "stripe" ublk_stripe.bpf.o
+
+backfile_0=`_create_backfile 256M`
+backfile_1=`_create_backfile 256M`
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t stripe -n 0 --bpf_prog 32 --bpf_aio_prog 32 --quiet $backfile_0 $backfile_1
+
+# run fio over the ublk disk
+fio --name=write_and_verify \
+    --filename=/dev/ublkb0 \
+    --ioengine=libaio --iodepth=4 \
+    --rw=write \
+    --size=256M \
+    --direct=1 \
+    --verify=crc32c \
+    --do_verify=1 \
+    --bs=4k > /dev/null 2>&1
+ERR_CODE=$?
+
+# cleanup & unregister and unpin the bpf prog
+_cleanup_bpf_test "stripe"
+
+_remove_backfile $backfile_0
+_remove_backfile $backfile_1
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/test_stripe_02.sh new file +26 -0
new file mode 100755
index 000000000000..fdbb81dc53d8
@@ -0,0 +1,26 @@
+#!/bin/bash
+
+. test_common.sh
+
+TID="stripe_02"
+ERR_CODE=0
+
+# prepare & register and pin bpf prog
+_prep_bpf_test "stripe" ublk_stripe.bpf.o
+
+backfile_0=`_create_backfile 256M`
+backfile_1=`_create_backfile 256M`
+
+# add two ublk null disks with the pinned bpf prog
+_add_ublk_dev -t stripe -n 0 --bpf_prog 32 --bpf_aio_prog 32 --quiet $backfile_0 $backfile_1
+
+_mkfs_mount_test /dev/ublkb0
+ERR_CODE=$?
+
+# cleanup & unregister and unpin the bpf prog
+_cleanup_bpf_test "stripe"
+
+_remove_backfile $backfile_0
+_remove_backfile $backfile_1
+
+_show_result $TID $ERR_CODE
tools/testing/selftests/ublk/ublk_bpf.c +85 -3

111 lines not on this page · raw message

Signature · 1 line
2.47.0
author maintainer
#27 lore raw

RFC PATCH 22/22 ublk: document ublk-bpf & bpf-aio

In reply to Ming Lei

Document ublk-bpf motivation and implementation.

Document bpf-aio implementation.

Document ublk-bpf selftests.

---
 Documentation/block/ublk.rst | 170 +++++++++++++++++++++++++++++++++++
 1 file changed, 170 insertions(+)
Documentation/block/ublk.rst +170 -0
index 51665a3e6a50..bf7a3df48036 100644
@@ -309,6 +309,176 @@ with specified IO tag in the command data:
   ``UBLK_IO_COMMIT_AND_FETCH_REQ`` to the server, ublkdrv needs to copy
   the server buffer (pages) read to the IO request pages.
 
+
+UBLK-BPF support
+================
+
+Motivation
+----------
+
+- support stacking ublk
+
+  There are many 3rd party volume manager, ublk may be built over ublk device
+  for simplifying implementation, however, multiple userspace-kernel context
+  switchs for handling one single IO can't be accepted from performance view
+  of point
+
+  ublk-bpf can avoid user-kernel context switch in most fast io path, so ublk
+  over ublk becomes possible
+
+- complicated virtual block device
+
+  Many complicated virtual block devices have admin&meta code path and normal
+  IO fast path; meta & admin IO handling is usually complicated, so it can be
+  moved to ublk server for relieving development burden; meantime IO fast path
+  can be kept in kernel space for the sake of high performance.
+
+  Bpf provides rich maps, which helps a lot for communication between
+  userspace and prog or between prog and prog.
+
+  One typical example is qcow2, which meta IO handling can be kept in
+  ublk server, and fast IO path is moved to bpf prog. Efficient bpf map can be
+  looked up first and see if this virtual LBA & host LBA mapping is hit in
+  the map. If yes, handle the IO with ublk-bpf directly, otherwise forward to
+  ublk server to populate the mapping first.
+
+- some simple high performance virtual devices
+
+  Such as null & loop, the whole implementation can be moved to bpf prog
+  completely.
+
+- provides chance to get similar performance with kernel driver
+
+  One round of kernel/user context switch is avoided, and one extra IO data
+  copy is saved
+
+bpf aio
+-------
+
+bpf aio exports kfuncs for bpf prog to submit & complete IO in async way.
+IO completion handler is provided by the bpf aio user, which is still
+defined in bpf prog(such as ublk bpf prog) as `struct bpf_aio_complete_ops`
+of bpf struct_ops.
+
+bpf aio is designed as generic interface, which can be used for any bpf prog
+in theory, and it may be move to `/lib/` in future if the interface becomes
+mature and stable enough.
+
+- bpf_aio_alloc()
+
+  Allocate one bpf aio instance of `struct bpf_aio`
+
+- bpf_aio_release()
+
+  Free one bpf aio instance of `struct bpf_aio`
+
+- bpf_aio_submit()
+
+  Submit one bpf aio instance of `struct bpf_aio` in async way.
+
+- `struct bpf_aio_complete_ops`
+
+  Define bpf aio completion callback implemented as bpf struct_ops, and
+  it is called when the submitted bpf aio is completed.
+
+
+ublk bpf implementation
+-----------------------
+
+Export `struct ublk_bpf_ops` as bpf struct_ops, so that ublk IO command
+can be queued or handled in the callback defined in the ublk bpf struct_ops,
+see the whole logic in `ublk_run_bpf_handler`:
+
+- `UBLK_BPF_IO_QUEUED`
+
+  If ->queue_io_cmd() or ->queue_io_cmd_daemon() returns `UBLK_BPF_IO_QUEUED`,
+  this IO command has been queued by bpf prog, so it won't be forwarded to
+  ublk server
+
+- `UBLK_BPF_IO_REDIRECT`
+
+  If ->queue_io_cmd() or ->queue_io_cmd_daemon() returns `UBLK_BPF_IO_REDIRECT`,
+  this IO command will be forwarded to ublk server
+
+- `UBLK_BPF_IO_CONTINUE`
+
+  If ->queue_io_cmd() or ->queue_io_cmd_daemon() returns `UBLK_BPF_IO_CONTINUE`,
+  part of this io command is queued, and `ublk_bpf_return_t` carries how many
+  bytes queued, so ublk driver will continue to call the callback to queue
+  remained bytes of this io command further, this way is helpful for
+  implementing stacking devices by allowing IO command split.
+
+ublk bpf provides kfuncs for ublk bpf prog to queue and handle ublk IO command:
+
+- ublk_bpf_complete_io()
+
+  Complete this ublk IO command
+
+- ublk_bpf_get_io_tag()
+
+  Get tag of this ublk IO command
+
+- ublk_bpf_get_queue_id()
+
+  Get queue id of this ublk IO command
+
+- ublk_bpf_get_dev_id()
+
+  Get device id of this ublk IO command
+
+- ublk_bpf_attach_and_prep_aio()
+
+  Attach & prepare bpf aio to this ublk IO command, bpf aio buffer is
+  prepared, and aio's complete callback is setup, so the user prog can
+  get notified when the bpf aio is completed
+
+- ublk_bpf_dettach_and_complete_aio()
+
+  Detach bpf aio from this IO command, and it is usually called from bpf
+  aio's completion callback.
+
+- ublk_bpf_acquire_io_from_aio()
+
+  Acquire ublk IO command from the aio, one typical use is for calling
+  ublk_bpf_complete_io() to complete ublk IO command
+
+- ublk_bpf_release_io_from_aio()
+
+  Release ublk IO command which is acquired from `ublk_bpf_acquire_io_from_aio`
+
+
+Test
+----
+
+- Build kernel & install kernel headers & reboot & test
+
+  enable CONFIG_BLK_DEV_UBLK & CONFIG_UBLK_BPF
+
+  make
+
+  make headers_install INSTALL_HDR_PATH=/usr
+
+  reboot
+
+  make -C tools/testing/selftests TARGETS=ublk run_test
+
+ublk selftests implements null, loop and stripe targets for covering all
+bpf features:
+
+- complete bpf IO handling
+
+- complete ublk server IO handling
+
+- mixed bpf prog and ublk server IO handling
+
+- bpf aio for loop & stripe
+
+- IO split via `UBLK_BPF_IO_CONTINUE` for implementing ublk-stripe
+
+Write & read verify, and mkfs.ext4 & mount & umount are run in the
+selftest.
+
+
 Future development
 ==================
 
Signature · 1 line
2.47.0