nvme: Add fault injection feature
Subsystems (3)
v3 cover nvme: Add fault injection feature
Linux's fault injection framework provides a systematic way to support error injection via debugfs in the /sys/kernel/debug directory. This patch uses the framework to add error injection to NVMe driver.
Change log since last version:
V2:
- add user changable status and no_retry debugfs attribute
- add document to show how to use nvme fault injection
V1:
- move error injection point into nvme_end_request()
- Add expected result in the commit log
Thank you,
Thomas
PATCH v3 1/2 nvme: Add fault injection feature
Linux's fault injection framework provides a systematic way to support error injection via debugfs in the /sys/kernel/debug directory. This patch uses the framework to add error injection to NVMe driver. The fault injection source code is stored in a separate file and only linked if CONFIG_FAULT_INJECTION_DEBUG_FS kernel config is selected.
Once the error injection is enabled, NVME_SC_INVALID_OPCODE with no retry will be injected into the nvme_end_request. Users can change the default status code and no retry flag via debufs. Following example shows how to enable and inject an error. For more examples, refer to Documentation/fault-injection/nvme-fault-injection.txt
How to enable nvme fault injection:
First, enable CONFIG_FAULT_INJECTION_DEBUG_FS kernel config, recompile the kernel. After booting up the kernel, do the following.
How to inject an error:
mount /dev/nvme0n1 /mnt
echo 1 > /sys/kernel/debug/nvme0n1/fault_inject/times echo 100 > /sys/kernel/debug/nvme0n1/fault_inject/probability cp a.file /mnt
Expected Result:
cp: cannot stat ?/mnt/a.file?: Input/output error
Message from dmesg:
FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0, times 1
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.15.0-rc8+ #2 Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 Call Trace: <IRQ> dump_stack+0x5c/0x7d should_fail+0x148/0x170 nvme_should_fail+0x2f/0x50 [nvme_core] nvme_process_cq+0xe7/0x1d0 [nvme] nvme_irq+0x1e/0x40 [nvme] __handle_irq_event_percpu+0x3a/0x190 handle_irq_event_percpu+0x30/0x70 handle_irq_event+0x36/0x60 handle_fasteoi_irq+0x78/0x120 handle_irq+0xa7/0x130 ? tick_irq_enter+0xa8/0xc0 do_IRQ+0x43/0xc0 common_interrupt+0xa2/0xa2 </IRQ> RIP: 0010:native_safe_halt+0x2/0x10 RSP: 0018:ffffffff82003e90 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffdd RAX: ffffffff817a10c0 RBX: ffffffff82012480 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000000000000 R08: 000000008e38ce64 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff82012480 R13: ffffffff82012480 R14: 0000000000000000 R15: 0000000000000000 ? __sched_text_end+0x4/0x4 default_idle+0x18/0xf0 do_idle+0x150/0x1d0 cpu_startup_entry+0x6f/0x80 start_kernel+0x4c4/0x4e4 ? set_init_arg+0x55/0x55 secondary_startup_64+0xa5/0xb0 print_req_error: I/O error, dev nvme0n1, sector 9240
EXT4-fs error (device nvme0n1): ext4_find_entry:1436:
inode #2: comm cp: reading directory lblock 0
--- drivers/nvme/host/Makefile | 1 + drivers/nvme/host/core.c | 2 + drivers/nvme/host/fault_inject.c | 79 ++++++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 27 ++++++++++++++ 4 files changed, 109 insertions(+) create mode 100644 drivers/nvme/host/fault_inject.c
drivers/nvme/host/Makefile +1 -0
@@ -8,6 +8,7 @@ obj-$(CONFIG_NVME_FC) += nvme-fc.o nvme-core-y := core.o nvme-core-$(CONFIG_NVME_MULTIPATH) += multipath.o nvme-core-$(CONFIG_NVM) += lightnvm.o +nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS) += fault_inject.o nvme-y += pci.o
drivers/nvme/host/core.c +2 -0
@@ -2951,6 +2951,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid)if (new)nvme_mpath_add_disk (ns->head);nvme_mpath_add_disk_links (ns); +nvme_fault_inject_init (ns);return ; out_unlink_ns:mutex_lock (&ctrl->subsys->lock);
@@ -2969,6 +2970,7 @@ static void nvme_ns_remove(struct nvme_ns *ns)if (test_and_set_bit (NVME_NS_REMOVING , &ns->flags))return ; +nvme_fault_inject_fini (ns);if (ns->disk && ns->disk->flags &GENHD_FL_UP ) {nvme_mpath_remove_disk_links (ns); sysfs_remove_group(&disk_to_dev(ns->disk)->kobj,
drivers/nvme/host/fault_inject.c new file +79 -0
79 lines not on this page · raw message
drivers/nvme/host/nvme.h +27 -0
@@ -21,6 +21,7 @@#include <linux/blk-mq.h> #include <linux/lightnvm.h> #include <linux/sed-opal.h> +#include <linux/fault-inject.h> extern unsigned int nvme_io_timeout;#define NVME_IO_TIMEOUT (nvme_io_timeout * HZ)
@@ -257,6 +258,15 @@ struct nvme_ns_head {int instance; }; +#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS +struct nvme_fault_inject { +struct fault_attr attr; +struct dentry *parent; +bool dont_retry;/* DNR, do not retry */ +u16 status;/* status code */ +}; +#endif +struct nvme_ns {struct list_head list;
@@ -278,6 +288,11 @@ struct nvme_ns {#define NVME_NS_REMOVING 0#define NVME_NS_DEAD 1u16 noiob; + +#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS +struct nvme_fault_inject fault_inject; +#endif + };struct nvme_ctrl_ops {
@@ -296,6 +311,16 @@ struct nvme_ctrl_ops {int (*reinit_request)(void *data,struct request *rq); }; +#ifdef CONFIG_FAULT_INJECTION_DEBUG_FS +void nvme_fault_inject_init (struct nvme_ns *ns); +void nvme_fault_inject_fini (struct nvme_ns *ns); +void nvme_should_fail (struct request *req); +#else +static inline void nvme_fault_inject_init (struct nvme_ns *ns) {} +static inline void nvme_fault_inject_fini (struct nvme_ns *ns) {} +static inline void nvme_should_fail (struct request *req) {} +#endif +static inline bool nvme_ctrl_ready (struct nvme_ctrl *ctrl) {u32 val =0 ;
@@ -332,6 +357,8 @@ static inline void nvme_end_request(struct request *req, __le16 status, rq->status =le16_to_cpu (status) >>1 ; rq->result = result; +/* inject error when permitted by fault injection framework */ +nvme_should_fail (req);blk_mq_complete_request (req); }
Signature · 1 line
1.8.3.1
PATCH v3 1/2 nvme: Add fault injection feature
Looks fine to me.
PATCH v3 1/2 nvme: Add fault injection feature
Quoted 3 lines · Keith Busch
Looks fine to me.
Reviewed-by Keith Busch <keith.busch at intel.com>
Thank you Keith.
Quoted 6 lines
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
PATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
In reply to Thomas TaiAdd examples to show how to use nvme fault injection.
--- Documentation/fault-injection/fault-injection.txt | 5 + .../fault-injection/nvme-fault-injection.txt | 116 +++++++++++++++++++++ 2 files changed, 121 insertions(+) create mode 100644 Documentation/fault-injection/nvme-fault-injection.txt
Documentation/fault-injection/fault-injection.txt +5 -0
@@ -30,6 +30,11 @@ o fail_mmc_request injects MMC data errors on devices permitted by setting debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request +o NVMe fault injection + + inject NVMe NVME_SC_INVALID_OPCODE with no retry on devices permitted + by setting debugfs entries under /sys/kernel/debug/nvme*/fault_inject + Configure fault-injection capabilities behavior -----------------------------------------------
Documentation/fault-injection/nvme-fault-injection.txt new file +116 -0
116 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Also looks fine to me.
PATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Quoted 3 lines · Keith Busch
Also looks fine to me.
Reviewed-by Keith Busch <keith.busch at intel.com>
Thank you Keith.
Quoted 6 lines
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerPATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Quoted 6 lines
Quoted 4 lines
Also looks fine to me.
Reviewed-by Keith Busch <keith.busch at intel.com>Thank you Keith.
Will pick it up for 4.17
Sagi Grimberg
maintainerPATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
In reply to Thomas TaiDocumentation/fault-injection/fault-injection.txt o fail_mmc_request · line 37
Quoted 6 lines
Documentation/fault-injection/fault-injection.txt +5
-0+o NVMe fault injection + + inject NVMe NVME_SC_INVALID_OPCODE with no retry on devices permitted + by setting debugfs entries under /sys/kernel/debug/nvme*/fault_inject +
Actually, can you update the documentation entry here?
PATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Quoted 8 lines · Sagi Grimberg
Documentation/fault-injection/fault-injection.txt o fail_mmc_request · line 37
Quoted 6 lines
Documentation/fault-injection/fault-injection.txt +5
-0+o NVMe fault injection + +? inject NVMe NVME_SC_INVALID_OPCODE with no retry on devices permitted +? by setting debugfs entries under /sys/kernel/debug/nvme*/fault_inject +Actually, can you update the documentation entry here?
Yes Sagi. I will submit the documentation to the fault injection maintainer. Thank you for your help.
Thomas
Quoted 5 lines
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerPATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Quoted 11 lines
Quoted 8 lines
Documentation/fault-injection/fault-injection.txt o fail_mmc_request · line 37
Quoted 6 lines
Documentation/fault-injection/fault-injection.txt +5
-0+o NVMe fault injection + +? inject NVMe NVME_SC_INVALID_OPCODE with no retry on devices permitted +? by setting debugfs entries under /sys/kernel/debug/nvme*/fault_inject +Actually, can you update the documentation entry here?
Yes Sagi. I will submit the documentation to the fault injection maintainer. Thank you for your help.
It's going to 4.17 (we have time), so I'd prefer to get it right once instead of having another fixup on top. Would it be possible to re-spin with the minor fix?
PATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
Quoted 10 lines
Quoted 8 lines
Documentation/fault-injection/fault-injection.txt +3
-0+o NVMe fault injection + +? inject NVMe NVME_SC_INVALID_OPCODE with no retry on devicespermitted
+? by setting debugfs entries under/sys/kernel/debug/nvme*/fault_inject
+Actually, can you update the documentation entry here?
Quoted 6 lines
Quoted 2 lines
Yes Sagi. I will submit the documentation to the fault injection maintainer. Thank you for your help.
It's going to 4.17 (we have time), so I'd prefer to get it right once instead of having another fixup on top. Would it be possible to re-spin with the minor fix?
Sure. Do you mean I should resend the patch set which include the fault injection maintainer in the email?
Sagi Grimberg
maintainerPATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 9 lines
Quoted 7 lines
Documentation/fault-injection/fault-injection.txt +3
-0+o NVMe fault injection + +? inject NVMe NVME_SC_INVALID_OPCODE with no retry on devicespermitted
+? by setting debugfs entries under/sys/kernel/debug/nvme*/fault_inject
+Actually, can you update the documentation entry here?
Yes Sagi. I will submit the documentation to the fault injection maintainer. Thank you for your help.
Quoted 6 lines
Quoted 3 lines
It's going to 4.17 (we have time), so I'd prefer to get it right once instead of having another fixup on top. Would it be possible to re-spin with the minor fix?
Sure. Do you mean I should resend the patch set which include the fault injection maintainer in the email?
No, just fix the documentation to say not only NVME_SC_INVALID_OPCODE can be injected (the entry in fault-injection.txt)
PATCH v3 2/2 Documentation: nvme: Documentation for nvme fault injection
[ ... ]
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
It's going to 4.17 (we have time), so I'd prefer to get it right once instead of having another fixup on top. Would it be possible to re-spin with the minor fix?
Sure. Do you mean I should resend the patch set which include the fault injection maintainer in the email?
No, just fix the documentation to say not only
NVME_SC_INVALID_OPCODEcan be injected (the entry in fault-injection.txt)
Oh yes. Sorry I missed that. will update the doc. Thank you.
Sagi Grimberg
maintainerv3 cover nvme: Add fault injection feature
In reply to Thomas TaiThomas,
This looks good to me, Keith/Christoph any feedback? Or should I take this forward?
v3 cover nvme: Add fault injection feature
Quoted 4 lines · Sagi Grimberg
Thomas,
This looks good to me, Keith/Christoph any feedback? Or should I take this forward?
Hi Sagi,
Thank you for your review. Any feedback from Keith/Christoph that I can help answer?
Thank you,
Thomas
Quoted 5 lines
_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme