Skip to contentSkip to conversation

configurable block error injection

PATCH In review Christoph Hellwig 4 participants 18 messages Jun 2 – Jun 5

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

Hi all,

this series adds a new configurable block error injection facility. We already have a few to inject block errors, but unfortunately most of them are either not very useful or hard to use, or both:

- The fail_make_request failure injection point can't distinguish different commands, different ranges in the file and can only injection plain I/O errors.
- the should_fail_bio 'dynamic' failure injection has all the same issues as fail_make_request
- dm-error can only fail all command in the table using BLK_STS_IOERR and requires setting up a new block device
- dm-flakey and dm-dust allow all kinds of configurability, but still don't have good error selection, no good support for non-read/write commands and are limited to the dm table alignment requirements, which for zoned devices enforces setting them up for an entire zone. They also once again require setting up a stacked block device, which is really annoying in harnesses like xfstests

This series adds a new debugfs-based block layer error injection that allows to configure what operations and ranges the injection applied to, and what status to return. It also allows to configure a failure ratio similar to the xfs errortag injection.

As part of this the should_fail_bio is removed, as the should_fail_bio function really gets in the way in it's current form, and the injection of an errno which always gets turned into BLK_STS_IOERR doesn't make much sense. This required adjusting the bpf test suite as it attached to this function as it assumed it to be present.

Diffstat:

 Documentation/block/error-injection.rst                    |   59 ++
 Documentation/block/index.rst                              |    1 
 block/Kconfig                                              |    6 
 block/Makefile                                             |    1 
 block/blk-core.c                                           |  128 ++---
 block/blk-merge.c                                          |    5 
 block/blk-mq.c                                             |    3 
 block/blk-sysfs.c                                          |    4 
 block/blk.h                                                |   24 
 block/error-injection.c                                    |  329 +++++++++++++
 block/genhd.c                                              |    4 
 include/linux/blk_types.h                                  |    2 
 include/linux/blkdev.h                                     |    5 
 lib/Kconfig.debug                                          |    6 
 tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c |    7 
 15 files changed, 507 insertions(+), 77 deletions(-)
#1 lore raw

PATCH 1/9 block: remove ALLOW_ERROR_INJECTION for should_fail_bio

Allow error injection for should_fail_bio is a bit misguided. It allows inserting an errno, which is then ignored, but it forced and out of line call for something that should not exist when error injection is disabled.

Remove the error injection flag in preparation for adding better block layer error injection, and switch the bpf test to use a MM error injection site instead.

---
 block/blk-core.c                                           | 1 -
 tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c | 7 ++++---
 2 files changed, 4 insertions(+), 4 deletions(-)
block/blk-core.c +0 -1
index b0f0a304ea0b..3a23af3e26a9 100644
@@ -545,7 +545,6 @@ int should_fail_bio(struct bio *bio)
 		return -EIO;
 	return 0;
 }
-ALLOW_ERROR_INJECTION(should_fail_bio, ERRNO);
 
 /*
  * Check whether this bio extends beyond the end of the device or partition.
tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c +4 -3
index 2e0ddef77ba5..6c8b161cdd7b 100644
@@ -588,12 +588,13 @@ static void test_attach_override(void)
 		goto cleanup;
 	}
 
-	/* The should_fail_bio function is on error injection list,
+	/* The __filemap_add_folio function is on error injection list,
 	 * attach should succeed.
 	 */
 	link = bpf_program__attach_kprobe_multi_opts(skel->progs.test_override,
-						     "should_fail_bio", NULL);
-	if (!ASSERT_OK_PTR(link, "override_attached_should_fail_bio"))
+						     "__filemap_add_folio,",
+						     NULL);
+	if (!ASSERT_OK_PTR(link, "override_attached___filemap_add_folio,"))
 		goto cleanup;
 
 	bpf_link__destroy(link);
Signature · 1 line
2.53.0
#2 lore raw

PATCH 2/9 block: consolidate the calls to should_fail_bio

In reply to Christoph Hellwig

Delay the point of error injection a bit so that we have a single site in blk-core.c after more of the submission side checks and blkcg throttling. This allows to make should_fail_bio static in blk-core.c.

---
 block/blk-core.c  | 6 +++++-
 block/blk-merge.c | 5 +----
 block/blk.h       | 1 -
 3 files changed, 6 insertions(+), 6 deletions(-)
block/blk-core.c +5 -1
index 3a23af3e26a9..f35e0d3fb127 100644
@@ -539,7 +539,7 @@ static inline void bio_check_ro(struct bio *bio)
 	}
 }
 
-int should_fail_bio(struct bio *bio)
+static int should_fail_bio(struct bio *bio)
 {
 	if (should_fail_request(bdev_whole(bio->bi_bdev), bio->bi_iter.bi_size))
 		return -EIO;
@@ -723,6 +723,10 @@ static void __submit_bio_noacct_mq(struct bio *bio)
 
 void submit_bio_noacct_nocheck(struct bio *bio, bool split)
 {
+	if (should_fail_bio(bio)) {
+		bio_io_error(bio);
+		return;
+	}
 	blk_cgroup_bio_start(bio);
 
 	if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) {
block/blk-merge.c +1 -4
index 7cc82a7a6f4e..b44f8ae849b8 100644
@@ -130,11 +130,8 @@ struct bio *bio_submit_split_bioset(struct bio *bio, unsigned int split_sectors,
 	trace_block_split(split, bio->bi_iter.bi_sector);
 	WARN_ON_ONCE(bio_zone_write_plugging(bio));
 
-	if (should_fail_bio(bio))
-		bio_io_error(bio);
-	else if (!blk_throtl_bio(bio))
+	if (!blk_throtl_bio(bio))
 		submit_bio_noacct_nocheck(bio, true);
-
 	return split;
 }
 EXPORT_SYMBOL_GPL(bio_submit_split_bioset);
block/blk.h +0 -1
index bf1a80493ff1..889a39589356 100644
@@ -646,7 +646,6 @@ extern const struct address_space_operations def_blk_aops;
 int disk_register_independent_access_ranges(struct gendisk *disk);
 void disk_unregister_independent_access_ranges(struct gendisk *disk);
 
-int should_fail_bio(struct bio *bio);
 #ifdef CONFIG_FAIL_MAKE_REQUEST
 bool should_fail_request(struct block_device *part, unsigned int bytes);
 #else /* CONFIG_FAIL_MAKE_REQUEST */
Signature · 1 line
2.53.0
#3 lore raw

PATCH 3/9 block: refactor should_fail_bio and should_fail_request

In reply to Christoph Hellwig

Move the bdev flag checks into a helper and the blk-mq clone insers so that we can do only a single actual error injection for I/O to partitions instead of doing it twice.

---
 block/blk-core.c          | 28 ++++++++++++++--------------
 block/blk-mq.c            |  3 ++-
 block/blk.h               |  5 ++---
 include/linux/blk_types.h |  2 --
 4 files changed, 18 insertions(+), 20 deletions(-)
block/blk-core.c +14 -14
index f35e0d3fb127..644888b66f33 100644
@@ -502,10 +502,9 @@ static int __init setup_fail_make_request(char *str)
 }
 __setup("fail_make_request=", setup_fail_make_request);
 
-bool should_fail_request(struct block_device *part, unsigned int bytes)
+bool should_fail_request(unsigned int bytes)
 {
-	return bdev_test_flag(part, BD_MAKE_IT_FAIL) &&
-	       should_fail(&fail_make_request, bytes);
+	return should_fail(&fail_make_request, bytes);
 }
 
 static int __init fail_make_request_debugfs(void)
@@ -539,11 +538,13 @@ static inline void bio_check_ro(struct bio *bio)
 	}
 }
 
-static int should_fail_bio(struct bio *bio)
+static inline bool may_fail_bio(struct bio *bio)
 {
-	if (should_fail_request(bdev_whole(bio->bi_bdev), bio->bi_iter.bi_size))
-		return -EIO;
-	return 0;
+	if (!IS_ENABLED(CONFIG_FAIL_MAKE_REQUEST))
+		return false;
+	return bdev_test_flag(bio->bi_bdev, BD_MAKE_IT_FAIL) ||
+		(bio_flagged(bio, BIO_REMAPPED) &&
+		 bdev_test_flag(bdev_whole(bio->bi_bdev), BD_MAKE_IT_FAIL));
 }
 
 /*
@@ -577,8 +578,6 @@ static int blk_partition_remap(struct bio *bio)
 {
 	struct block_device *p = bio->bi_bdev;
 
-	if (unlikely(should_fail_request(p, bio->bi_iter.bi_size)))
-		return -EIO;
 	if (bio_sectors(bio)) {
 		bio->bi_iter.bi_sector += p->bd_start_sect;
 		trace_block_bio_remap(bio, p->bd_dev,
@@ -723,10 +722,13 @@ static void __submit_bio_noacct_mq(struct bio *bio)
 
 void submit_bio_noacct_nocheck(struct bio *bio, bool split)
 {
-	if (should_fail_bio(bio)) {
-		bio_io_error(bio);
-		return;
+	if (unlikely(may_fail_bio(bio))) {
+		if (should_fail_request(bio->bi_iter.bi_size)) {
+			bio_io_error(bio);
+			return;
+		}
 	}
+
 	blk_cgroup_bio_start(bio);
 
 	if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) {
@@ -799,8 +801,6 @@ void submit_bio_noacct(struct bio *bio)
 			goto not_supported;
 	}
 
-	if (should_fail_bio(bio))
-		goto end_io;
 	bio_check_ro(bio);
 	if (!bio_flagged(bio, BIO_REMAPPED)) {
 		if (unlikely(bio_check_eod(bio)))
block/blk-mq.c +2 -1
index 629e16003eb7..bf66645622df 100644
@@ -3275,7 +3275,8 @@ blk_status_t blk_insert_cloned_request(struct request *rq)
 		return BLK_STS_IOERR;
 	}
 
-	if (q->disk && should_fail_request(q->disk->part0, blk_rq_bytes(rq)))
+	if (q->disk && bdev_test_flag(q->disk->part0, BD_MAKE_IT_FAIL) &&
+	    should_fail_request(blk_rq_bytes(rq)))
 		return BLK_STS_IOERR;
 
 	ret = blk_crypto_rq_get_keyslot(rq);
block/blk.h +2 -3
index 889a39589356..250a6eee700a 100644
@@ -647,10 +647,9 @@ int disk_register_independent_access_ranges(struct gendisk *disk);
 void disk_unregister_independent_access_ranges(struct gendisk *disk);
 
 #ifdef CONFIG_FAIL_MAKE_REQUEST
-bool should_fail_request(struct block_device *part, unsigned int bytes);
+bool should_fail_request(unsigned int bytes);
 #else /* CONFIG_FAIL_MAKE_REQUEST */
-static inline bool should_fail_request(struct block_device *part,
-					unsigned int bytes)
+static inline bool should_fail_request(unsigned int bytes)
 {
 	return false;
 }
include/linux/blk_types.h +0 -2
index 8808ee76e73c..4a3cfa857637 100644
@@ -51,9 +51,7 @@ struct block_device {
 #define BD_WRITE_HOLDER		(1u<<9)
 #define BD_HAS_SUBMIT_BIO	(1u<<10)
 #define BD_RO_WARNED		(1u<<11)
-#ifdef CONFIG_FAIL_MAKE_REQUEST
 #define BD_MAKE_IT_FAIL		(1u<<12)
-#endif
 	dev_t			bd_dev;
 	struct address_space	*bd_mapping;	/* page cache */
 
Signature · 1 line
2.53.0
#4 lore raw

PATCH 4/9 block: move the FAIL_MAKE_REQUEST symbol from lib/ to block/

In reply to Christoph Hellwig

Keep the Kconfig symbol together with the code that it guards.

---
 block/Kconfig     | 6 ++++++
 lib/Kconfig.debug | 6 ------
 2 files changed, 6 insertions(+), 6 deletions(-)
block/Kconfig +6 -0
index 15027963472d..6c942391f65e 100644
@@ -209,6 +209,12 @@ config BLK_INLINE_ENCRYPTION_FALLBACK
 	  by falling back to the kernel crypto API when inline
 	  encryption hardware is not present.
 
+config FAIL_MAKE_REQUEST
+	bool "Fault-injection capability for disk IO"
+	depends on FAULT_INJECTION
+	help
+	  Provide fault-injection capability for disk IO.
+
 source "block/partitions/Kconfig"
 
 config BLK_PM
lib/Kconfig.debug +0 -6
index 8ff5adcfe1e0..fb085963ec5e 100644
@@ -2116,12 +2116,6 @@ config FAULT_INJECTION_USERCOPY
 	  Provides fault-injection capability to inject failures
 	  in usercopy functions (copy_from_user(), get_user(), ...).
 
-config FAIL_MAKE_REQUEST
-	bool "Fault-injection capability for disk IO"
-	depends on FAULT_INJECTION && BLOCK
-	help
-	  Provide fault-injection capability for disk IO.
-
 config FAIL_IO_TIMEOUT
 	bool "Fault-injection capability for faking disk interrupts"
 	depends on FAULT_INJECTION && BLOCK
Signature · 1 line
2.53.0
#5 lore raw

PATCH 5/9 block: add a macro to initialize the status table

In reply to Christoph Hellwig

Prepare for adding a new value to the error table by adding a macro to fill it.

---
 block/blk-core.c | 45 +++++++++++++++++++++++++--------------------
 1 file changed, 25 insertions(+), 20 deletions(-)
block/blk-core.c +25 -20
index 644888b66f33..1ab666fc2e27 100644
@@ -132,39 +132,44 @@ inline const char *blk_op_str(enum req_op op)
 }
 EXPORT_SYMBOL_GPL(blk_op_str);
 
+#define ENT(_tag, _errno, _desc)	\
+[BLK_STS_##_tag] = {				\
+	.errno		= _errno,		\
+	.name		= _desc,		\
+}
 static const struct {
 	int		errno;
 	const char	*name;
 } blk_errors[] = {
-	[BLK_STS_OK]		= { 0,		"" },
-	[BLK_STS_NOTSUPP]	= { -EOPNOTSUPP, "operation not supported" },
-	[BLK_STS_TIMEOUT]	= { -ETIMEDOUT,	"timeout" },
-	[BLK_STS_NOSPC]		= { -ENOSPC,	"critical space allocation" },
-	[BLK_STS_TRANSPORT]	= { -ENOLINK,	"recoverable transport" },
-	[BLK_STS_TARGET]	= { -EREMOTEIO,	"critical target" },
-	[BLK_STS_RESV_CONFLICT]	= { -EBADE,	"reservation conflict" },
-	[BLK_STS_MEDIUM]	= { -ENODATA,	"critical medium" },
-	[BLK_STS_PROTECTION]	= { -EILSEQ,	"protection" },
-	[BLK_STS_RESOURCE]	= { -ENOMEM,	"kernel resource" },
-	[BLK_STS_DEV_RESOURCE]	= { -EBUSY,	"device resource" },
-	[BLK_STS_AGAIN]		= { -EAGAIN,	"nonblocking retry" },
-	[BLK_STS_OFFLINE]	= { -ENODEV,	"device offline" },
+	ENT(OK,			0,		""),
+	ENT(NOTSUPP,		-EOPNOTSUPP,	"operation not supported"),
+	ENT(TIMEOUT,		-ETIMEDOUT,	"timeout"),
+	ENT(NOSPC,		-ENOSPC,	"critical space allocation"),
+	ENT(TRANSPORT,		-ENOLINK,	"recoverable transport"),
+	ENT(TARGET,		-EREMOTEIO,	"critical target"),
+	ENT(RESV_CONFLICT,	-EBADE,		"reservation conflict"),
+	ENT(MEDIUM,		-ENODATA,	"critical medium"),
+	ENT(PROTECTION,		-EILSEQ,	"protection"),
+	ENT(RESOURCE,		-ENOMEM,	"kernel resource"),
+	ENT(DEV_RESOURCE,	-EBUSY,		"device resource"),
+	ENT(AGAIN,		-EAGAIN,	"nonblocking retry"),
+	ENT(OFFLINE,		-ENODEV,	"device offline"),
 
 	/* device mapper special case, should not leak out: */
-	[BLK_STS_DM_REQUEUE]	= { -EREMCHG, "dm internal retry" },
+	ENT(DM_REQUEUE,		-EREMCHG,	"dm internal retry"),
 
 	/* zone device specific errors */
-	[BLK_STS_ZONE_OPEN_RESOURCE]	= { -ETOOMANYREFS, "open zones exceeded" },
-	[BLK_STS_ZONE_ACTIVE_RESOURCE]	= { -EOVERFLOW, "active zones exceeded" },
+	ENT(ZONE_OPEN_RESOURCE, -ETOOMANYREFS,	"open zones exceeded"),
+	ENT(ZONE_ACTIVE_RESOURCE, -EOVERFLOW,	"active zones exceeded"),
 
 	/* Command duration limit device-side timeout */
-	[BLK_STS_DURATION_LIMIT]	= { -ETIME, "duration limit exceeded" },
-
-	[BLK_STS_INVAL]		= { -EINVAL,	"invalid" },
+	ENT(DURATION_LIMIT,	-ETIME,		"duration limit exceeded"),
+	ENT(INVAL,		-EINVAL,	"invalid"),
 
 	/* everything else not covered above: */
-	[BLK_STS_IOERR]		= { -EIO,	"I/O" },
+	ENT(IOERR,		-EIO,		"I/O"),
 };
+#undef ENT
 
 blk_status_t errno_to_blk_status(int errno)
 {
Signature · 1 line
2.53.0
#6 lore raw

PATCH 6/9 block: add a "tag" for block status codes

In reply to Christoph Hellwig

The full name of the status codes is not good for user interfaces as it can contain white spaces. Add the name of the status code without the BLK_STS_ prefix as a tag so that it can be used for user interfaces.

---
 block/blk-core.c | 24 ++++++++++++++++++++++++
 block/blk.h      |  2 ++
 2 files changed, 26 insertions(+)
block/blk-core.c +24 -0
index 1ab666fc2e27..19a4d0672b3d 100644
@@ -135,10 +135,12 @@ EXPORT_SYMBOL_GPL(blk_op_str);
 #define ENT(_tag, _errno, _desc)	\
 [BLK_STS_##_tag] = {				\
 	.errno		= _errno,		\
+	.tag		= __stringify(_tag),	\
 	.name		= _desc,		\
 }
 static const struct {
 	int		errno;
+	const char	*tag;
 	const char	*name;
 } blk_errors[] = {
 	ENT(OK,			0,		""),
@@ -203,6 +205,28 @@ const char *blk_status_to_str(blk_status_t status)
 	return blk_errors[idx].name;
 }
 
+const char *blk_status_to_tag(blk_status_t status)
+{
+	int idx = (__force int)status;
+
+	if (WARN_ON_ONCE(idx >= ARRAY_SIZE(blk_errors)))
+		return "<null>";
+	return blk_errors[idx].tag;
+}
+
+blk_status_t tag_to_blk_status(const char *tag)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(blk_errors); i++) {
+		if (blk_errors[i].tag &&
+		    !strcmp(blk_errors[i].tag, tag))
+			return (__force blk_status_t)i;
+	}
+
+	return BLK_STS_OK;
+}
+
 /**
  * blk_sync_queue - cancel any pending callbacks on a queue
  * @q: the queue
block/blk.h +2 -0
index 250a6eee700a..1e80338af858 100644
@@ -50,6 +50,8 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,
 void blk_free_flush_queue(struct blk_flush_queue *q);
 
 const char *blk_status_to_str(blk_status_t status);
+const char *blk_status_to_tag(blk_status_t status);
+blk_status_t tag_to_blk_status(const char *tag);
 
 bool __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic);
 bool blk_queue_start_drain(struct request_queue *q);
Signature · 1 line
2.53.0
#7 lore raw

PATCH 7/9 block: add a str_to_blk_op helper

In reply to Christoph Hellwig

Add a helper to find the REQ_OP_XYZ constant from the "XYZ" string. This will be used for the error injection debugfs interface.

---
 block/blk-core.c | 13 +++++++++++++
 block/blk.h      |  1 +
 2 files changed, 14 insertions(+)
block/blk-core.c +13 -0
index 19a4d0672b3d..8bbc03ce924f 100644
@@ -132,6 +132,19 @@ inline const char *blk_op_str(enum req_op op)
 }
 EXPORT_SYMBOL_GPL(blk_op_str);
 
+enum req_op str_to_blk_op(const char *op)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(blk_op_name); i++) {
+		if (blk_op_name[i] &&
+		    !strcmp(blk_op_name[i], op))
+			return i;
+	}
+
+	return REQ_OP_LAST;
+}
+
 #define ENT(_tag, _errno, _desc)	\
 [BLK_STS_##_tag] = {				\
 	.errno		= _errno,		\
block/blk.h +1 -0
index 1e80338af858..4857b899e2b6 100644
@@ -52,6 +52,7 @@ void blk_free_flush_queue(struct blk_flush_queue *q);
 const char *blk_status_to_str(blk_status_t status);
 const char *blk_status_to_tag(blk_status_t status);
 blk_status_t tag_to_blk_status(const char *tag);
+enum req_op str_to_blk_op(const char *op);
 
 bool __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic);
 bool blk_queue_start_drain(struct request_queue *q);
Signature · 1 line
2.53.0
#8 lore raw

PATCH 8/9 block: add configurable error injection

In reply to Christoph Hellwig

Add a new block error injection interface that allows to inject specific status code for specific ranges.

---
 Documentation/block/error-injection.rst |  59 +++++
 Documentation/block/index.rst           |   1 +
 block/Makefile                          |   1 +
 block/blk-core.c                        |   2 +
 block/blk-sysfs.c                       |   4 +
 block/blk.h                             |  15 ++
 block/error-injection.c                 | 299 ++++++++++++++++++++++++
 block/genhd.c                           |   4 +
 include/linux/blkdev.h                  |   5 +
 9 files changed, 390 insertions(+)
 create mode 100644 Documentation/block/error-injection.rst
 create mode 100644 block/error-injection.c
Documentation/block/error-injection.rst new file +59 -0
new file mode 100644
index 000000000000..be87091b5330
@@ -0,0 +1,59 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+============================
+Configurable Error Injection
+============================
+
+Overview
+--------
+
+Configurable error injection allows injecting specific block layer status codes
+for ranges of a block device.  Error can be injected unconditional, or with a
+given probability.
+
+To use configurable error injection, CONFIG_FAIL_MAKE_REQUEST must be enabled.
+
+The only interface is the error_injection debugfs file, which is created for
+each registered gendisk.  Writes to this file are used to create or delete rules
+and reads return a list of the current error injection sites.
+
+Options
+-------
+
+The following options specify the operations:
+
+===================	=======================================================
+add			add a new rule
+removeall		remove all existing rules
+===================	=======================================================
+
+The following options specify the details of the rule for the add operation:
+
+===================	=======================================================
+op=%s			block layer operation this rule applies to, e.g. READ
+			or WRITE.
+			Mandatory.
+start=%u		First block layer sector the rule applies to.
+			Optional, defaults to 0.
+nr_sectors=%u		Number of sectors this rule applies.
+			Optional, defaults to the remainder of the device.
+status=%s		Status to return.
+			Mandatory.
+chance=%u		Only return a failure with a likelihood of 1/chance.
+			Optional, defaults to 1 (always).
+===================	=======================================================
+
+Example
+-------
+
+Return BLK_STS_IOERR for one in 10 reads of sector 0 of /dev/nvme0n1:
+
+	$ echo 'add,op=READ,start=0,status=IOERR,chance=10' > /sys/kernel/debug/block/nvme0n1/error_injection
+
+Return BLK_STS_MEDIUM for every write to /dev/nvme0n1:
+
+	$ echo 'add,op=WRITE,start=0,status=MEDIUM' > /sys/kernel/debug/block/nvme0n1/error_injection
+
+Remove all rules for /dev/nvme0n1:
+
+	$ echo 'removeall' > /sys/kernel/debug/block/nvme0n1/error_injection
Documentation/block/index.rst +1 -0
index 9fea696f9daa..bfa1bbd31ddf 100644
@@ -22,3 +22,4 @@ Block
    switching-sched
    writeback_cache_control
    ublk
+   error-injection
block/Makefile +1 -0
index 7dce2e44276c..d223b6b7d72f 100644
@@ -11,6 +11,7 @@ obj-y		:= bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \
 			genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \
 			disk-events.o blk-ia-ranges.o early-lookup.o
 
+obj-$(CONFIG_FAIL_MAKE_REQUEST)	+= error-injection.o
 obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o
 obj-$(CONFIG_BLK_DEV_BSGLIB)	+= bsg-lib.o
 obj-$(CONFIG_BLK_CGROUP)	+= blk-cgroup.o
block/blk-core.c +2 -0
index 8bbc03ce924f..04a392849ab0 100644
@@ -765,6 +765,8 @@ static void __submit_bio_noacct_mq(struct bio *bio)
 void submit_bio_noacct_nocheck(struct bio *bio, bool split)
 {
 	if (unlikely(may_fail_bio(bio))) {
+		if (blk_error_inject(bio))
+			return;
 		if (should_fail_request(bio->bi_iter.bi_size)) {
 			bio_io_error(bio);
 			return;
block/blk-sysfs.c +4 -0
index f22c1f253eb3..43f909c7f0c9 100644
@@ -933,6 +933,8 @@ static void blk_debugfs_remove(struct gendisk *disk)
 
 	blk_debugfs_lock_nomemsave(q);
 	blk_trace_shutdown(q);
+	if (IS_ENABLED(CONFIG_FAIL_MAKE_REQUEST))
+		blk_error_injection_exit(disk);
 	debugfs_remove_recursive(q->debugfs_dir);
 	q->debugfs_dir = NULL;
 	q->sched_debugfs_dir = NULL;
@@ -963,6 +965,8 @@ int blk_register_queue(struct gendisk *disk)
 
 	memflags = blk_debugfs_lock(q);
 	q->debugfs_dir = debugfs_create_dir(disk->disk_name, blk_debugfs_root);
+	if (IS_ENABLED(CONFIG_FAIL_MAKE_REQUEST))
+		blk_error_injection_init(disk);
 	if (queue_is_mq(q))
 		blk_mq_debugfs_register(q);
 	blk_debugfs_unlock(q, memflags);
block/blk.h +15 -0
index 4857b899e2b6..19f925d8f39d 100644
@@ -781,4 +781,19 @@ static inline void blk_debugfs_unlock(struct request_queue *q,
 	memalloc_noio_restore(memflags);
 }
 
+void blk_error_injection_init(struct gendisk *disk);
+void blk_error_injection_exit(struct gendisk *disk);
+
+bool __blk_error_inject(struct bio *bio);
+static inline bool blk_error_inject(struct bio *bio)
+{
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	struct gendisk *disk = bio->bi_bdev->bd_disk;
+
+	if (!list_empty_careful(&disk->error_injection_list))
+		return __blk_error_inject(bio);
+#endif
+	return false;
+}
+
 #endif /* BLK_INTERNAL_H */
block/error-injection.c new file +299 -0

299 lines not on this page · raw message

block/genhd.c +4 -0
index 7d6854fd28e9..30f42461d895 100644
@@ -1485,6 +1485,10 @@ struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id,
 	lockdep_init_map(&disk->lockdep_map, "(bio completion)", lkclass, 0);
 #ifdef CONFIG_BLOCK_HOLDER_DEPRECATED
 	INIT_LIST_HEAD(&disk->slave_bdevs);
+#endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	mutex_init(&disk->error_injection_lock);
+	INIT_LIST_HEAD(&disk->error_injection_list);
 #endif
 	mutex_init(&disk->rqos_state_mutex);
 	kobject_init(&disk->queue_kobj, &blk_queue_ktype);
include/linux/blkdev.h +5 -0
index 17270a28c66d..8743ad616b7f 100644
@@ -227,6 +227,11 @@ struct gendisk {
 	 */
 	struct blk_independent_access_ranges *ia_ranges;
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	struct mutex		error_injection_lock;
+	struct list_head	error_injection_list;
+#endif
+
 	struct mutex rqos_state_mutex;	/* rqos state change mutex */
 };
 
Signature · 1 line
2.53.0
#9 lore raw

Re: [PATCH 8/9] block: add configurable error injection

Quoted 16 lines · Christoph Hellwig
block/error-injection.c new file +16 -0
+static int error_inject_add(struct gendisk *disk, enum req_op op,
+		sector_t start, u64 nr_sectors, blk_status_t status,
+		unsigned int chance)
+{
+	struct blk_error_inject *inj;
+
+	if (op == REQ_OP_LAST)
+		return -EINVAL;
+	if (status == BLK_STS_OK)
+		return -EINVAL;
+	if (U64_MAX - nr_sectors < start)
+		return -EINVAL;
+
+	if (!nr_sectors)
+		nr_sectors = U64_MAX;
+

...

block/error-injection.c line 80

Quoted 4 lines
block/error-injection.c new file +4 -0
+
+	inj->op = op;
+	inj->start = start;
+	inj->end = start + nr_sectors - 1;

When nr_sectors is 0, it is reset to U64_MAX so overflows if start > 1. I think you want to remove overriding nr_sectors to U64_MAX and do:

	if (!nr_sectors)
		inj->end = U64_MAX;
	else if (U64_MAX - nr_sectors < start )
		return -EINVAL;
	else
		inj->end = start + nr_sectors - 1;

block/error-injection.c line 92

Quoted 12 lines
block/error-injection.c new file +12 -0
+	inj->status = status;
+	inj->chance = chance;
+
+	/*
+	 * Add to the front of the list so that newer entries can partially
+	 * override other entries.  This also intentional allows duplicate
+	 * entries as there is no real reason to reject them.
+	 */
+	mutex_lock(&disk->error_injection_lock);
+	if (!disk_live(disk)) {
+		mutex_unlock(&disk->error_injection_lock);
+		return -EINVAL;

I think we've leaked 'inj' in this error case.

Review

Quoted 2 lines
(quoted diff) +2 -0
+	}
+	list_add(&inj->entry, &disk->error_injection_list);

The __blk_error_inject interates this list with "list_for_each_entry_rcu", so shouldn't this be list_add_rcu to match?

Quoted 5 lines
block/error-injection.c new file +5 -0
+	mutex_unlock(&disk->error_injection_lock);
+
+	bdev_set_flag(disk->part0, BD_MAKE_IT_FAIL);
+	return 0;
+}

<snip>

block/error-injection.c line 139

Quoted 6 lines
block/error-injection.c new file +6 -0
+static const match_table_t opt_tokens = {
+	{ Opt_add,			"add",			},
+	{ Opt_removeall,		"removeall",		},
+	{ Opt_op,			"op=%s",		},
+	{ Opt_start,			"start=%u"		},
+	{ Opt_nr_sectors,		"nr_sectors=%u"		},

Shouldn't start and nr_sectors use %llu?

Quoted 3 lines
block/error-injection.c new file +3 -0
+static ssize_t blk_error_injection_write(struct file *file,
+		const char __user *ubuf, size_t count, loff_t *pos)
+{

...

block/error-injection.c line 189

Quoted 4 lines
block/error-injection.c new file +4 -0
+	options = memdup_user_nul(ubuf, count);
+	if (!options)
+		return -ENOMEM;
+

On failure, memdup_user_nul returns an ERR_PTR rather than NULL.

	if (IS_ERR(options))
		return PTR_ERR(options);

block/error-injection.c line 242

Quoted 3 lines
block/error-injection.c new file +3 -0
+	case Removeall:
+		if (option_mask & ~Opt_removeall)
+			return -EINVAL;

Leaking "options"? Should this be:

		if (option_mask & ~Opt_removeall) {
			ret = -EINVAL;
			goto out_free_options;
		}

?

Quoted 12 lines
block/error-injection.c new file +12 -0
+		error_inject_removall(disk);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	if (!ret)
+		ret = count;
+out_free_options:
+	kfree(options);
+	return ret;
+}
#10 lore raw

Re: [PATCH 8/9] block: add configurable error injection

Quoted 1 line · Keith Busch

When nr_sectors is 0, it is reset to U64_MAX so overflows if start > 1.

Yeah.

Quoted 8 lines

I think you want to remove overriding nr_sectors to U64_MAX and do:

	if (!nr_sectors)
		inj->end = U64_MAX;
	else if (U64_MAX - nr_sectors < start )
		return -EINVAL;
	else
		inj->end = start + nr_sectors - 1;

I ended up ordering a bit differently for better readability, but yes.

Quoted 6 lines

block/error-injection.c line 92

Quoted 4 lines
block/error-injection.c new file +4 -0
+	mutex_lock(&disk->error_injection_lock);
+	if (!disk_live(disk)) {
+		mutex_unlock(&disk->error_injection_lock);
+		return -EINVAL;

I think we've leaked 'inj' in this error case.

Yes.

Quoted 6 lines

Review

Quoted 3 lines
(quoted diff) +2 -0
+	}
+	list_add(&inj->entry, &disk->error_injection_list);

The __blk_error_inject interates this list with "list_for_each_entry_rcu", so shouldn't this be list_add_rcu to match?

Yes.

Quoted 8 lines

block/error-injection.c line 139

Quoted 6 lines
block/error-injection.c new file +6 -0
+static const match_table_t opt_tokens = {
+	{ Opt_add,			"add",			},
+	{ Opt_removeall,		"removeall",		},
+	{ Opt_op,			"op=%s",		},
+	{ Opt_start,			"start=%u"		},
+	{ Opt_nr_sectors,		"nr_sectors=%u"		},

Shouldn't start and nr_sectors use %llu?

lib/parser.c doesn't use those prefixes, it's a bit weird.

Quoted 8 lines

block/error-injection.c line 189

Quoted 3 lines
block/error-injection.c new file +3 -0
+	if (!options)
+		return -ENOMEM;
+

On failure, memdup_user_nul returns an ERR_PTR rather than NULL.

	if (IS_ERR(options))
		return PTR_ERR(options);

Aarg, annoying. Because memdup_user does return NULL :(

Quoted 7 lines

block/error-injection.c line 242

Quoted 4 lines
block/error-injection.c new file +3 -0
+	case Removeall:
+		if (option_mask & ~Opt_removeall)
+			return -EINVAL;

Leaking "options"? Should this be:

Quoted 6 lines
		if (option_mask & ~Opt_removeall) {
			ret = -EINVAL;
			goto out_free_options;
		}

?

Yes.

reviewer
#11 lore raw

Re: [PATCH 8/9] block: add configurable error injection

In reply to Christoph Hellwig
Quoted 7 lines · Christoph Hellwig
Documentation/block/error-injection.rst new file +1 -0
new file mode 100644
index 000000000000..be87091b5330
@@ -0,0 +1,59 @@
+.. SPDX-License-Identifier: GPL-2.0

Documentation/block/error-injection.rst line 11

Quoted 10 lines
Documentation/block/error-injection.rst new file +10 -0
+
+============================
+Configurable Error Injection
+============================
+
+Overview
+--------
+
+Configurable error injection allows injecting specific block layer status codes
+for ranges of a block device.  Error can be injected unconditional, or with a

Errors can be injected unconditionally or with a

Quoted 7 lines
Documentation/block/error-injection.rst new file +7 -0
+given probability.
+
+To use configurable error injection, CONFIG_FAIL_MAKE_REQUEST must be enabled.
+
+The only interface is the error_injection debugfs file, which is created for
+each registered gendisk.  Writes to this file are used to create or delete rules
+and reads return a list of the current error injection sites.

[snip]

Signature · 1 line
~Randy
#12 lore raw

PATCH 9/9 block: move the fail request code

In reply to Christoph Hellwig

Keep all error injection in one place, and out of line for the main I/O submission fast path.

---
 block/blk-core.c        | 37 ++-----------------------------------
 block/error-injection.c | 30 ++++++++++++++++++++++++++++++
 2 files changed, 32 insertions(+), 35 deletions(-)
block/blk-core.c +2 -35
index 04a392849ab0..7465dd291272 100644
@@ -29,7 +29,6 @@
 #include <linux/swap.h>
 #include <linux/writeback.h>
 #include <linux/task_io_accounting_ops.h>
-#include <linux/fault-inject.h>
 #include <linux/list_sort.h>
 #include <linux/delay.h>
 #include <linux/ratelimit.h>
@@ -534,32 +533,6 @@ bool blk_get_queue(struct request_queue *q)
 }
 EXPORT_SYMBOL(blk_get_queue);
 
-#ifdef CONFIG_FAIL_MAKE_REQUEST
-
-static DECLARE_FAULT_ATTR(fail_make_request);
-
-static int __init setup_fail_make_request(char *str)
-{
-	return setup_fault_attr(&fail_make_request, str);
-}
-__setup("fail_make_request=", setup_fail_make_request);
-
-bool should_fail_request(unsigned int bytes)
-{
-	return should_fail(&fail_make_request, bytes);
-}
-
-static int __init fail_make_request_debugfs(void)
-{
-	struct dentry *dir = fault_create_debugfs_attr("fail_make_request",
-						NULL, &fail_make_request);
-
-	return PTR_ERR_OR_ZERO(dir);
-}
-
-late_initcall(fail_make_request_debugfs);
-#endif /* CONFIG_FAIL_MAKE_REQUEST */
-
 static inline void bio_check_ro(struct bio *bio)
 {
 	if (op_is_write(bio_op(bio)) && bdev_read_only(bio->bi_bdev)) {
@@ -764,14 +737,8 @@ static void __submit_bio_noacct_mq(struct bio *bio)
 
 void submit_bio_noacct_nocheck(struct bio *bio, bool split)
 {
-	if (unlikely(may_fail_bio(bio))) {
-		if (blk_error_inject(bio))
-			return;
-		if (should_fail_request(bio->bi_iter.bi_size)) {
-			bio_io_error(bio);
-			return;
-		}
-	}
+	if (unlikely(may_fail_bio(bio)) && blk_error_inject(bio))
+		return;
 
 	blk_cgroup_bio_start(bio);
 
block/error-injection.c +30 -0
index dc0420c4eb58..45f2454d0bca 100644
@@ -4,6 +4,7 @@
  */
 #include <linux/debugfs.h>
 #include <linux/blkdev.h>
+#include <linux/fault-inject.h>
 #include <linux/parser.h>
 #include <linux/seq_file.h>
 #include "blk.h"
@@ -47,6 +48,13 @@ bool __blk_error_inject(struct bio *bio)
 		}
 	}
 	rcu_read_unlock();
+
+	/* legacy I/O error injection */
+	if (should_fail_request(bio->bi_iter.bi_size)) {
+		bio_io_error(bio);
+		return true;
+	}
+
 	return false;
 }
 
@@ -297,3 +305,25 @@ void blk_error_injection_exit(struct gendisk *disk)
 {
 	error_inject_removall(disk);
 }
+
+static DECLARE_FAULT_ATTR(fail_make_request);
+
+bool should_fail_request(unsigned int bytes)
+{
+	return should_fail(&fail_make_request, bytes);
+}
+
+static int __init setup_fail_make_request(char *str)
+{
+	return setup_fault_attr(&fail_make_request, str);
+}
+__setup("fail_make_request=", setup_fail_make_request);
+
+static int __init fail_make_request_debugfs(void)
+{
+	struct dentry *dir = fault_create_debugfs_attr("fail_make_request",
+						NULL, &fail_make_request);
+
+	return PTR_ERR_OR_ZERO(dir);
+}
+late_initcall(fail_make_request_debugfs);
Signature · 1 line
2.53.0
Reviewed-by
#13 lore raw
In reply to Christoph Hellwig
Quoted 5 lines · Christoph Hellwig

Hi all,

this series adds a new configurable block error injection facility. We already have a few to inject block errors, but unfortunately most of them are either not very useful or hard to use, or both:

Looks great! I just have some comments on patch 8/9, but for the rest:

#14 lore raw
In reply to Christoph Hellwig
Quoted 20 lines · Christoph Hellwig

Hi all,

this series adds a new configurable block error injection facility. We already have a few to inject block errors, but unfortunately most of them are either not very useful or hard to use, or both:

- The fail_make_request failure injection point can't distinguish different commands, different ranges in the file and can only injection plain I/O errors.
- the should_fail_bio 'dynamic' failure injection has all the same issues as fail_make_request
- dm-error can only fail all command in the table using BLK_STS_IOERR and requires setting up a new block device
- dm-flakey and dm-dust allow all kinds of configurability, but still don't have good error selection, no good support for non-read/write commands and are limited to the dm table alignment requirements, which for zoned devices enforces setting them up for an entire zone. They also once again require setting up a stacked block device, which is really annoying in harnesses like xfstests

Quoted 4 lines

This series adds a new debugfs-based block layer error injection that allows to configure what operations and ranges the injection applied to, and what status to return. It also allows to configure a failure ratio similar to the xfs errortag injection.

I wonder if the block layer would be interested in moving block error injection off the should_fail() fault injection framework and extending the ALLOW_ERROR_INJECTION annotation instead and offloading all the debugfs configuration logic (block/error-injection.c) into eBPF?

I talked about moderr [1] at LPC 2025. It's a simple error injection tool in eBPF for the module subsystem. The suggested direction there was to generalize the tool to ideally to no tool at all, and leverage bpftrace to describe the error injection conditions a given subsystem needs to be tested under. That would let blktests, for example, absorb that and simplify the configuration logic this series adds in the kernel for debugfs.

A previous attempt to add inline error injection [2] was rejected as too intrusive / source-polluting; the eBPF approach solves that, since the injection logic lives in a standalone tool/script rather than in the kernel sources.

What do you guys think?

[1] https://lpc.events/event/19/contributions/2204/
[2] https://lore.kernel.org/all/[email protected]/

#15 lore raw
Quoted 4 lines · Daniel Gomez

I wonder if the block layer would be interested in moving block error injection off the should_fail() fault injection framework and extending the ALLOW_ERROR_INJECTION annotation instead and offloading all the debugfs configuration logic (block/error-injection.c) into eBPF?

I've looked into plain ALLOW_ERROR_INJECTION-based injection and it is not very useful. I didn't even now eBPF could use it, but I looked into other eBPF injections and at least for my uses cases it was a bit of a mess. I'd have to allow access to certain bio fields and would have create a stable UAPI for commands and status using the fake BTF struct access which really would not be a good idea here as we need to be able to change internals. Additionally having fully BTF-enabled toolchains in test VMs is not great either.

I've also not actually found any good map type for range lookups, which is kinda essential here.

Quoted 7 lines

I talked about moderr [1] at LPC 2025. It's a simple error injection tool in eBPF for the module subsystem. The suggested direction there was to generalize the tool to ideally to no tool at all, and leverage bpftrace to describe the error injection conditions a given subsystem needs to be tested under. That would let blktests, for example, absorb that and simplify the configuration logic this series adds in the kernel for debugfs.

I don't think pulling in ebpftrace for simple error injection is a winning proposition..

Quoted 3 lines

A previous attempt to add inline error injection [2] was rejected as too intrusive / source-polluting;

I'm not sure a single hand waivy comment counts as rejection, although I'm not a huge fan of setup_fault_attr - it makes a mess of debugfs and creates a lot of boilerplate for a single not very much configurable call site. That might be ok for something like the make_request case (although I think we can do better as shown in this patch), but for making random functions fail it is a lot of overhead. These injections points also are not anywhere near stable enough to be exposed.

#16 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Daniel Gomez

I wonder if the block layer would be interested in moving block error injection off the should_fail() fault injection framework and extending the ALLOW_ERROR_INJECTION annotation instead and offloading all the debugfs configuration logic (block/error-injection.c) into eBPF?

I've looked into plain ALLOW_ERROR_INJECTION-based injection and it is not very useful. I didn't even now eBPF could use it,

For context: Josef Bacik introduced it first for BPF only (BPF_ALLOW_ERROR_INJECTION). Masami Hiramatsu then generalized it into the error injection framework, renaming it to ALLOW_ERROR_INJECTION and adding the fail_function debugfs interface (which calls should_fail()). So annotating a function with ALLOW_ERROR_INJECTION gives you both backends at once: debugfs (fail_function) and eBPF (bpf_override_return()).

Quoted 3 lines

but I
looked into other eBPF injections and at least for my uses cases it was a bit of a mess.

Agreed, and I had not looked closely enough at the series before proposing the wrong primitives. ALLOW_ERROR_INJECTION / bpf_override_return() are not sufficient here: bpf_override_return() cannot set bio->bi_status or call bio_endio(), which I think is the key operation here.

IIU the series correctly, and oversimplifying: when injection is enabled and a bio matches, the block layer completes the bio inline with the chosen blk_status_t (the status= rule from debugfs) via bio_endio_status(). The submission path returns to the caller immediately, with the bio already in the error state. Nothing is ever sent to the device, but the completion path sees the injected error.

submit_bio() -> submit_bio_noacct() -> submit_bio_noacct_nocheck()

      +-- Path 1: no match -> continue normal IO submission
      +-- Path 2: match (diskN) -> blk_error_inject()
                     -> bio_endio_status(bio, inj->status)
                          -> bio_endio()
                     // error injected. bio completed

So this is bio mutating + a bio_endio() call, not a return override. That can't be solved with ALLOW_ERROR_INJECTION. But we can use BPF_PROG_TYPE_STRUCT_OPS instead: the kernel keeps ownership of the bio_endio_status() call, and the eBPF program only drives the policy, ie. which blk_status_t to return for a given bio, based on whatever heuristics it implements.

Quoted 1 line

I'd have to allow access to certain bio

With struct_ops the bio is passed to the ebpf side as read-only, bio fields can be read to decide the policy but cannot write them. Is read-only access to bio fields itself a concern?

Quoted 3 lines

fields and would have create a stable UAPI for commands and status using the fake BTF struct access which really would not be a good idea here as we need to be able to change internals.

That should not be a problem at all. With CO-RE (compile once, run everywhere) the program resolves the bio field offsets against the BTF of the kernel it loads on, so it adapts dynamically if the layout changes. The contract is just the struct_ops callback signature: a struct bio * argument and a blk_status_t return. And that doesn't imply any UAPI commitment AFAIK.

Quoted 2 lines

Additionally
having fully BTF-enabled toolchains in test VMs is not great either.

Are you referring to the old BCC toolchain requirements [1]? This is solved in CO-RE [2]. The toolchain (Clang/LLVM, pahole) stays on the build host; the test VM only needs the prebuilt BPF object, libbpf at runtime, and the kernel's own BTF (CONFIG_DEBUG_INFO_BTF). No compiler or BTF toolchain is required inside the VM. Clang/LLVM 10+ is enough to build CO-RE libbpf tools [3].

Quoted 3 lines

I've also not actually found any good map type for range lookups, which is kinda essential here.

Are you referring to the bio sector range comparison in __blk_error_inject()? I don't think that needs to be delegated to a BPF map (Documentation/bpf/maps.rst). The ebpf side has direct access to the bio fields, so it can apply the same sector/op filtering __blk_error_inject() does today. That match is already a linear list walk, so the ebpf program just runs the same [start, end] condition check.

In summary: what do you think of evolving this series
into eBPF, but BPF_PROG_TYPE_STRUCT_OPS instead of ALLOW_ERROR_INJECTION/bpf_override_return()?. The configurable debugfs injection is a nice enhancement over what we had, but the matching is static (op/start/nr_sectors/status/chance) per gendisk. A struct_ops hook would expand the failure model to anything derivable from the bio (plus any state the program keeps), with the kernel still owning the bio_endio_status() mutation and only the policy moving out of the tree. The benefit is the flexibility to express the policy without hard-coding the model in the kernel.

The open question is whether programmable injection is something we want to support in-tree, or whether more debugfs knobs suffice. Thoughts?

#17 lore raw
Quoted 6 lines · Daniel Gomez

IIU the series correctly, and oversimplifying: when injection is enabled and a bio matches, the block layer completes the bio inline with the chosen blk_status_t (the status= rule from debugfs) via bio_endio_status(). The submission path returns to the caller immediately, with the bio already in the error state. Nothing is ever sent to the device, but the completion path sees the injected error.

Yes.

Quoted 5 lines
Quoted 1 line

I'd have to allow access to certain bio

With struct_ops the bio is passed to the ebpf side as read-only, bio fields can be read to decide the policy but cannot write them. Is read-only access to bio fields itself a concern?

Yes, for the reason state below.

Quoted 11 lines
Quoted 4 lines

fields and would have create a stable UAPI for commands and status using the fake BTF struct access which really would not be a good idea here as we need to be able to change internals.

That should not be a problem at all. With CO-RE (compile once, run everywhere) the program resolves the bio field offsets against the BTF of the kernel it loads on, so it adapts dynamically if the layout changes. The contract is just the struct_ops callback signature: a struct bio * argument and a blk_status_t return. And that doesn't imply any UAPI commitment AFAIK.

That assumes the fields even exists in this form. And we definitively need to do things about the iter going ahead. In other words we have to build up a huge abstraction here first.

Quoted 9 lines
Quoted 2 lines

Additionally
having fully BTF-enabled toolchains in test VMs is not great either.

Are you referring to the old BCC toolchain requirements [1]? This is solved in CO-RE [2]. The toolchain (Clang/LLVM, pahole) stays on the build host; the test VM only needs the prebuilt BPF object, libbpf at runtime, and the kernel's own BTF (CONFIG_DEBUG_INFO_BTF). No compiler or BTF toolchain is required inside the VM. Clang/LLVM 10+ is enough to build CO-RE libbpf tools [3].

libbpf and BTF support is what I mean. And of course whatever code you write to communicate with it as a simple shell script won't work any more.

Quoted 2 lines

Are you referring to the bio sector range comparison in __blk_error_inject()?

Yes.

Quoted 6 lines

I don't think that needs to be delegated to a
BPF map (Documentation/bpf/maps.rst). The ebpf side has direct access to the bio fields, so it can apply the same sector/op filtering __blk_error_inject() does today. That match is already a linear list walk, so the ebpf program just runs the same [start, end] condition check.

Which means I need to write eBPF code for every specific range I want to inject, which makes the thing very hard to use.

Quoted 3 lines

In summary: what do you think of evolving this series into eBPF, but BPF_PROG_TYPE_STRUCT_OPS instead of ALLOW_ERROR_INJECTION/bpf_override_return()?.

As said before, I prototyped it and it was a mess. Having about 300 lines of simple code that can be directly used from a shell script seems much preferable.