configurable block error injection v2
Subsystems (2)
Hi all,
this series adds a new configurable block error injection facility. We already have a few to inject block errors, but unfortunately most of them are either not very useful or hard to use, or both:
- The fail_make_request failure injection point can't distinguish different commands, different ranges in the file and can only injection plain I/O errors.
- the should_fail_bio 'dynamic' failure injection has all the same issues as fail_make_request
- dm-error can only fail all command in the table using BLK_STS_IOERR and requires setting up a new block device
- dm-flakey and dm-dust allow all kinds of configurability, but still don't have good error selection, no good support for non-read/write commands and are limited to the dm table alignment requirements, which for zoned devices enforces setting them up for an entire zone. They also once again require setting up a stacked block device, which is really annoying in harnesses like xfstests
This series adds a new debugfs-based block layer error injection that allows to configure what operations and ranges the injection applied to, and what status to return. It also allows to configure a failure ratio similar to the xfs errortag injection.
Changes since v1:
- drop the should_fail_bio removal and cleanup depending on it, as it's used by eBPF programs and thus a hidden UABI.
- as a result split the code out to it's own Kconfig symbol
- various error handling fixed pointed out by Keith
- documentation spelling fixes pointed out by Randy
Diffstat:
Documentation/block/error-injection.rst | 59 ++++++ Documentation/block/index.rst | 1 block/Kconfig | 7 block/Makefile | 1 block/blk-core.c | 86 ++++++-- block/blk-sysfs.c | 4 block/blk.h | 15 + block/error-injection.c | 308 ++++++++++++++++++++++++++++++++ block/genhd.c | 4 include/linux/blkdev.h | 6 10 files changed, 471 insertions(+), 20 deletions(-)
PATCH 1/4 block: add a macro to initialize the status table
Prepare for adding a new value to the error table by adding a macro to fill it.
--- block/blk-core.c | 45 +++++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 20 deletions(-)
block/blk-core.c +25 -20
@@ -132,39 +132,44 @@ inline const char *blk_op_str(enum req_op op) }EXPORT_SYMBOL_GPL (blk_op_str); +#define ENT (_tag, _errno, _desc) \ +[BLK_STS_##_tag] = { \ + .errno = _errno, \ + .name = _desc, \ +}static const struct {int errno;const char *name; } blk_errors[] = { - [BLK_STS_OK ] = {0 ,"" }, - [BLK_STS_NOTSUPP ] = { -EOPNOTSUPP ,"operation not supported" }, - [BLK_STS_TIMEOUT ] = { -ETIMEDOUT ,"timeout" }, - [BLK_STS_NOSPC ] = { -ENOSPC ,"critical space allocation" }, - [BLK_STS_TRANSPORT ] = { -ENOLINK ,"recoverable transport" }, - [BLK_STS_TARGET ] = { -EREMOTEIO ,"critical target" }, - [BLK_STS_RESV_CONFLICT ] = { -EBADE ,"reservation conflict" }, - [BLK_STS_MEDIUM ] = { -ENODATA ,"critical medium" }, - [BLK_STS_PROTECTION ] = { -EILSEQ ,"protection" }, - [BLK_STS_RESOURCE ] = { -ENOMEM ,"kernel resource" }, - [BLK_STS_DEV_RESOURCE ] = { -EBUSY ,"device resource" }, - [BLK_STS_AGAIN ] = { -EAGAIN ,"nonblocking retry" }, - [BLK_STS_OFFLINE ] = { -ENODEV ,"device offline" }, +ENT (OK ,0 ,"" ), +ENT (NOTSUPP , -EOPNOTSUPP ,"operation not supported" ), +ENT (TIMEOUT , -ETIMEDOUT ,"timeout" ), +ENT (NOSPC , -ENOSPC ,"critical space allocation" ), +ENT (TRANSPORT , -ENOLINK ,"recoverable transport" ), +ENT (TARGET , -EREMOTEIO ,"critical target" ), +ENT (RESV_CONFLICT , -EBADE ,"reservation conflict" ), +ENT (MEDIUM , -ENODATA ,"critical medium" ), +ENT (PROTECTION , -EILSEQ ,"protection" ), +ENT (RESOURCE , -ENOMEM ,"kernel resource" ), +ENT (DEV_RESOURCE , -EBUSY ,"device resource" ), +ENT (AGAIN , -EAGAIN ,"nonblocking retry" ), +ENT (OFFLINE , -ENODEV ,"device offline" ),/* device mapper special case, should not leak out: */ - [BLK_STS_DM_REQUEUE ] = { -EREMCHG ,"dm internal retry" }, +ENT (DM_REQUEUE , -EREMCHG ,"dm internal retry" ),/* zone device specific errors */ - [BLK_STS_ZONE_OPEN_RESOURCE ] = { -ETOOMANYREFS ,"open zones exceeded" }, - [BLK_STS_ZONE_ACTIVE_RESOURCE ] = { -EOVERFLOW ,"active zones exceeded" }, +ENT (ZONE_OPEN_RESOURCE , -ETOOMANYREFS ,"open zones exceeded" ), +ENT (ZONE_ACTIVE_RESOURCE , -EOVERFLOW ,"active zones exceeded" ),/* Command duration limit device-side timeout */ - [BLK_STS_DURATION_LIMIT ] = { -ETIME ,"duration limit exceeded" }, - - [BLK_STS_INVAL ] = { -EINVAL ,"invalid" }, +ENT (DURATION_LIMIT , -ETIME ,"duration limit exceeded" ), +ENT (INVAL , -EINVAL ,"invalid" ),/* everything else not covered above: */ - [BLK_STS_IOERR ] = { -EIO ,"I/O" }, +ENT (IOERR , -EIO ,"I/O" ), }; +#undef ENTblk_status_t errno_to_blk_status (int errno) {
Signature · 1 line
2.53.0
Re: [PATCH 1/4] block: add a macro to initialize the status table
block/blk-core.c inline const char *blk_op_str(enum req_op op) · line 138
Quoted 7 lines · Christoph Hellwig
Prepare for adding a new value to the error table by adding a macro to fill it.
block/blk-core.c +4
-0+#define ENT (_tag, _errno, _desc) \ +[BLK_STS_##_tag] = { \ + .errno = _errno, \ + .name = _desc, \
Bleh. I hate this. Before, I can grep for BLK_STS_NOSPC and find it. After, I can't. Yes, I know we have a lot of such things already, but I don't like adding more.
Re: [PATCH 1/4] block: add a macro to initialize the status table
Quoted 10 lines · Matthew Wilcox
block/blk-core.c inline const char *blk_op_str(enum req_op op) · line 138
Quoted 7 lines · Christoph Hellwig
Prepare for adding a new value to the error table by adding a macro to fill it.
block/blk-core.c +4
-0+#define ENT (_tag, _errno, _desc) \ +[BLK_STS_##_tag] = { \ + .errno = _errno, \ + .name = _desc, \Bleh. I hate this. Before, I can grep for
BLK_STS_NOSPCand find it.
You will still find BLK_STS_NOSPC itself in include/linux/blk_types.h
Quoted 2 lines
After, I can't. Yes, I know we have a lot of such things already, but I don't like adding more.
I'm not a huge fan off CPP pasting, and especially thing we should never use it to define global symbols (hi page/folio flag helpers!), and in general try to avoid using them as much as possible. But I think here the need to keep the names in sync with the tags exposed in debugfs is more important than the grepability. Especially as this sits in _the_ core block file, so it can't be easily missed.
Re: [PATCH 1/4] block: add a macro to initialize the status table
In reply to Christoph HellwigQuoted 5 lines · Christoph Hellwig
Prepare for adding a new value to the error table by adding a macro to fill it.
Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Keith Busch <[email protected]>
Looks good to me.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 1/4] block: add a macro to initialize the status table
In reply to Christoph HellwigQuoted 9 lines · Christoph Hellwig
Prepare for adding a new value to the error table by adding a macro to fill it.
Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Keith Busch <[email protected]>--- block/blk-core.c | 45 +++++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 20 deletions(-)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH 2/4 block: add a "tag" for block status codes
In reply to Christoph HellwigThe full name of the status codes is not good for user interfaces as it can contain white spaces. Add the name of the status code without the BLK_STS_ prefix as a tag so that it can be used for user interfaces.
--- block/blk-core.c | 28 ++++++++++++++++++++++++++++ block/blk.h | 2 ++ 2 files changed, 30 insertions(+)
block/blk-core.c +28 -0
@@ -135,10 +135,12 @@ EXPORT_SYMBOL_GPL(blk_op_str);#define ENT (_tag, _errno, _desc) \ [BLK_STS_##_tag] = { \ .errno = _errno, \ + .tag = __stringify(_tag), \ .name = _desc, \ }static const struct {int errno; +const char *tag;const char *name; } blk_errors[] = {ENT (OK ,0 ,"" ),
@@ -203,6 +205,32 @@ const char *blk_status_to_str(blk_status_t status)return blk_errors[idx].name; } +const char *blk_status_to_tag (blk_status_t status) +{ +int idx = (__force int )status; + +if (WARN_ON_ONCE (idx >=ARRAY_SIZE (blk_errors))) +return "<null>" ; +return blk_errors[idx].tag; +} + +blk_status_t tag_to_blk_status (const char *tag) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (blk_errors); i++) { +if (blk_errors[i].tag && + !strcmp (blk_errors[i].tag, tag)) +return (__force blk_status_t )i; + } + +/* +* Return BLK_STS_OK for mismatches as this function is intended to +* parse error status values. +*/ +return BLK_STS_OK ; +} +/** * blk_sync_queue - cancel any pending callbacks on a queue * @q: the queue
block/blk.h +2 -0
@@ -50,6 +50,8 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size,void blk_free_flush_queue (struct blk_flush_queue *q);const char *blk_status_to_str (blk_status_t status); +const char *blk_status_to_tag (blk_status_t status); +blk_status_t tag_to_blk_status (const char *tag);bool __blk_mq_unfreeze_queue (struct request_queue *q,bool force_atomic);bool blk_queue_start_drain (struct request_queue *q);
Signature · 1 line
2.53.0
Re: [PATCH 2/4] block: add a "tag" for block status codes
Quoted 6 lines · Christoph Hellwig
The full name of the status codes is not good for user interfaces as it can contain white spaces. Add the name of the status code without the
BLK_STS_prefix as a tag so that it can be used for user interfaces.Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Keith Busch <[email protected]>
Looks good to me.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 2/4] block: add a "tag" for block status codes
In reply to Christoph HellwigQuoted 11 lines · Christoph Hellwig
The full name of the status codes is not good for user interfaces as it can contain white spaces. Add the name of the status code without the
BLK_STS_prefix as a tag so that it can be used for user interfaces.Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Keith Busch <[email protected]>--- block/blk-core.c | 28 ++++++++++++++++++++++++++++ block/blk.h | 2 ++ 2 files changed, 30 insertions(+)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH 3/4 block: add a str_to_blk_op helper
In reply to Christoph HellwigAdd a helper to find the REQ_OP_XYZ constant from the "XYZ" string. This will be used for the error injection debugfs interface.
--- block/blk-core.c | 10 ++++++++++ block/blk.h | 1 + 2 files changed, 11 insertions(+)
block/blk-core.c +10 -0
@@ -132,6 +132,16 @@ inline const char *blk_op_str(enum req_op op) }EXPORT_SYMBOL_GPL (blk_op_str); +enum req_op str_to_blk_op (const char *op) +{ +int i; + +for (i =0 ; i <ARRAY_SIZE (blk_op_name); i++) +if (blk_op_name[i] && !strcmp (blk_op_name[i], op)) +return (enum req_op )i; +return REQ_OP_LAST ; +} +#define ENT (_tag, _errno, _desc) \ [BLK_STS_##_tag] = { \ .errno = _errno, \
block/blk.h +1 -0
@@ -52,6 +52,7 @@ void blk_free_flush_queue(struct blk_flush_queue *q);const char *blk_status_to_str (blk_status_t status);const char *blk_status_to_tag (blk_status_t status);blk_status_t tag_to_blk_status (const char *tag); +enum req_op str_to_blk_op (const char *op);bool __blk_mq_unfreeze_queue (struct request_queue *q,bool force_atomic);bool blk_queue_start_drain (struct request_queue *q);
Signature · 1 line
2.53.0
Re: [PATCH 3/4] block: add a str_to_blk_op helper
Quoted 5 lines · Christoph Hellwig
Add a helper to find the
REQ_OP_XYZconstant from the "XYZ" string. This will be used for the error injection debugfs interface.Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Keith Busch <[email protected]>
Looks good to me.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 3/4] block: add a str_to_blk_op helper
In reply to Christoph HellwigQuoted 10 lines · Christoph Hellwig
Add a helper to find the
REQ_OP_XYZconstant from the "XYZ" string. This will be used for the error injection debugfs interface.Signed-off-by Christoph Hellwig <[email protected]>Reviewed-by Keith Busch <[email protected]>--- block/blk-core.c | 10 ++++++++++ block/blk.h | 1 + 2 files changed, 11 insertions(+)
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
PATCH 4/4 block: add configurable error injection
In reply to Christoph HellwigAdd a new block error injection interface that allows to inject specific status code for specific ranges.
--- Documentation/block/error-injection.rst | 59 +++++ Documentation/block/index.rst | 1 + block/Kconfig | 7 + block/Makefile | 1 + block/blk-core.c | 3 + block/blk-sysfs.c | 4 + block/blk.h | 12 + block/error-injection.c | 308 ++++++++++++++++++++++++ block/genhd.c | 4 + include/linux/blkdev.h | 6 + 10 files changed, 405 insertions(+) create mode 100644 Documentation/block/error-injection.rst create mode 100644 block/error-injection.c
Documentation/block/error-injection.rst new file +59 -0
@@ -0,0 +1,59 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================ +Configurable Error Injection +============================ + +Overview +-------- + +Configurable error injection allows injecting specific block layer status codes +for ranges of a block device. Errors can be injected unconditionally, or with a +given probability. + +To use configurable error injection, CONFIG_BLK_ERROR_INJECTION must be enabled. + +The only interface is the error_injection debugfs file, which is created for +each registered gendisk. Writes to this file are used to create or delete rules +and reads return a list of the current error injection sites. + +Options +------- + +The following options specify the operations: + +=================== ======================================================= +add add a new rule +removeall remove all existing rules +=================== ======================================================= + +The following options specify the details of the rule for the add operation: + +=================== ======================================================= +op=%s block layer operation this rule applies to, e.g. READ + or WRITE. + Mandatory. +start=%u First block layer sector the rule applies to. + Optional, defaults to 0. +nr_sectors=%u Number of sectors this rule applies. + Optional, defaults to the remainder of the device. +status=%s Status to return. + Mandatory. +chance=%u Only return a failure with a likelihood of 1/chance. + Optional, defaults to 1 (always). +=================== ======================================================= + +Example +------- + +Return BLK_STS_IOERR for one in 10 reads of sector 0 of /dev/nvme0n1: + + $ echo 'add,op=READ,start=0,status=IOERR,chance=10' > /sys/kernel/debug/block/nvme0n1/error_injection + +Return BLK_STS_MEDIUM for every write to /dev/nvme0n1: + + $ echo 'add,op=WRITE,start=0,status=MEDIUM' > /sys/kernel/debug/block/nvme0n1/error_injection + +Remove all rules for /dev/nvme0n1: + + $ echo 'removeall' > /sys/kernel/debug/block/nvme0n1/error_injection
Documentation/block/index.rst +1 -0
@@ -22,3 +22,4 @@ Block switching-sched writeback_cache_control ublk + error-injection
block/Kconfig +7 -0
@@ -221,6 +221,13 @@ config BLOCK_HOLDER_DEPRECATED config BLK_MQ_STACKING bool +config BLK_ERROR_INJECTION + bool "Enable block layer error injection" + help + Enable inserting arbitrary block errors through a debugfs interface. + + See Documentation/block/error-injection.rst for details. + source "block/Kconfig.iosched" endif # BLOCK
block/Makefile +1 -0
@@ -11,6 +11,7 @@ obj-y := bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \ genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \ disk-events.o blk-ia-ranges.o early-lookup.o +obj-$(CONFIG_BLK_ERROR_INJECTION) += error-injection.o obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o obj-$(CONFIG_BLK_DEV_BSGLIB) += bsg-lib.o obj-$(CONFIG_BLK_CGROUP) += blk-cgroup.o
block/blk-core.c +3 -0
@@ -767,6 +767,9 @@ static void __submit_bio_noacct_mq(struct bio *bio)void submit_bio_noacct_nocheck (struct bio *bio,bool split) { +if (unlikely (blk_error_inject (bio))) +return ; +blk_cgroup_bio_start (bio);if (!bio_flagged (bio,BIO_TRACE_COMPLETION )) {
block/blk-sysfs.c +4 -0
@@ -933,6 +933,8 @@ static void blk_debugfs_remove(struct gendisk *disk)blk_debugfs_lock_nomemsave (q);blk_trace_shutdown (q); +if (IS_ENABLED (CONFIG_BLK_ERROR_INJECTION )) +blk_error_injection_exit (disk);debugfs_remove_recursive (q->debugfs_dir); q->debugfs_dir =NULL ; q->sched_debugfs_dir =NULL ;
@@ -963,6 +965,8 @@ int blk_register_queue(struct gendisk *disk) memflags =blk_debugfs_lock (q); q->debugfs_dir =debugfs_create_dir (disk->disk_name, blk_debugfs_root); +if (IS_ENABLED (CONFIG_BLK_ERROR_INJECTION )) +blk_error_injection_init (disk);if (queue_is_mq (q))blk_mq_debugfs_register (q);blk_debugfs_unlock (q, memflags);
block/blk.h +12 -0
@@ -660,6 +660,18 @@ static inline bool should_fail_request(struct block_device *part, }#endif /* CONFIG_FAIL_MAKE_REQUEST */ +void blk_error_injection_init (struct gendisk *disk); +void blk_error_injection_exit (struct gendisk *disk); +bool __blk_error_inject (struct bio *bio); +static inline bool blk_error_inject (struct bio *bio) +{ +if (!IS_ENABLED (CONFIG_BLK_ERROR_INJECTION )) +return false; +if (!test_bit (GD_ERROR_INJECT , &bio->bi_bdev->bd_disk->state)) +return false; +return __blk_error_inject (bio); +} +/* * Optimized request reference counting. Ideally we'd make timeouts be more * clever, as that's the only reason we need references at all... But until
block/error-injection.c new file +308 -0
308 lines not on this page · raw message
block/genhd.c +4 -0
@@ -1485,6 +1485,10 @@ struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id,lockdep_init_map (&disk->lockdep_map,"(bio completion)" , lkclass,0 );#ifdef CONFIG_BLOCK_HOLDER_DEPRECATED INIT_LIST_HEAD (&disk->slave_bdevs); +#endif +#ifdef CONFIG_BLK_ERROR_INJECTION +mutex_init (&disk->error_injection_lock); +INIT_LIST_HEAD (&disk->error_injection_list);#endif mutex_init (&disk->rqos_state_mutex);kobject_init (&disk->queue_kobj, &blk_queue_ktype);
include/linux/blkdev.h +6 -0
@@ -176,6 +176,7 @@ struct gendisk {#define GD_SUPPRESS_PART_SCAN 5#define GD_OWNS_QUEUE 6#define GD_ZONE_APPEND_USED 7 +#define GD_ERROR_INJECT 8struct mutex open_mutex;/* open/close mutex */ unsigned open_partitions;/* number of open partitions */
@@ -227,6 +228,11 @@ struct gendisk {*/ struct blk_independent_access_ranges *ia_ranges; +#ifdef CONFIG_BLK_ERROR_INJECTION +struct mutex error_injection_lock; +struct list_head error_injection_list; +#endif +struct mutex rqos_state_mutex;/* rqos state change mutex */ };
Signature · 1 line
2.53.0
Re: [PATCH 4/4] block: add configurable error injection
Quoted 16 lines · Christoph Hellwig
Add a new block error injection interface that allows to inject specific status code for specific ranges.
Signed-off-by Christoph Hellwig <[email protected]>--- Documentation/block/error-injection.rst | 59 +++++ Documentation/block/index.rst | 1 + block/Kconfig | 7 + block/Makefile | 1 + block/blk-core.c | 3 + block/blk-sysfs.c | 4 + block/blk.h | 12 + block/error-injection.c | 308 ++++++++++++++++++++++++ block/genhd.c | 4 + include/linux/blkdev.h | 6 + 10 files changed, 405 insertions(+)
Quoted 3 lines
create mode 100644
Documentation/block/error-injection.rstcreate mode 100644block/error-injection.c
Cheers,
Hannes
Signature · 4 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
Re: [PATCH 4/4] block: add configurable error injection
In reply to Christoph HellwigQuoted 4 lines · Christoph Hellwig
Add a new block error injection interface that allows to inject specific status code for specific ranges.
Signed-off-by Christoph Hellwig <[email protected]>
[...]
Documentation/block/error-injection.rst line 34
Quoted 3 lines
+=================== =======================================================
Documentation/block/error-injection.rst new file +2
-0+op=%s block layer operation this rule applies to, e.g. READ + or WRITE.
Like you did in the commit message of patch 3, maybe mention that this should match "XYZ" of one of the defined REQ_OP_XYZ operation ?
Documentation/block/error-injection.rst line 40
Quoted 6 lines
Documentation/block/error-injection.rst new file +6
-0+ Mandatory. +start=%u First block layer sector the rule applies to. + Optional, defaults to 0. +nr_sectors=%u Number of sectors this rule applies. + Optional, defaults to the remainder of the device. +status=%s Status to return.
Maybe mention that this should match XYZ for one one of the defined BLK_STS_XYZ ?
Quoted 4 lines
Documentation/block/error-injection.rst new file +4
-0+ Mandatory. +chance=%u Only return a failure with a likelihood of 1/chance. + Optional, defaults to 1 (always). +=================== =======================================================
[...]
block/error-injection.c line 89
Quoted 3 lines
block/error-injection.c new file +3
-0+/* +* Add to the front of the list so that newer entries can partially +* override other entries. This also intentional allows duplicate
s/intentional/intentionally
block/error-injection.c line 91
Quoted 2 lines
block/error-injection.c new file +2
-0+* entries as there is no real reason to reject them. +*/
Beside these nits, looks good to me.
Signature · 2 lines
Damien Le Moal Western Digital Research