Skip to contentSkip to conversation

modules: allow error injection

PATCH In review Daniel Gomez 7 participants 18 messages Jan 22, 2025 – Mar 6, 2025 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): CGME20250122131157eucas1p14bc56b73e8598908574aab12ecdfc245@eucas1p1.samsung.com

#1 lore raw

cover modules: allow error injection

Allow error injection during module initialization for testing.

This adds ALLOW_ERROR_INJECTION() annotation for complete_formation(), do_init_module() and module_enable_rodata_ro_after_init() so we can test the error path in these functions.

In addition, add moderr support, an eBPF based tool to inject module errors in any of the above functions for a specific module name.

The following example is added to the moderr commit. Adding it also here for convenience:

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:
[ 89.463690] brd: module loaded
[ 89.463855] brd: module_enable_rodata_ro_after_init() returned -22,
ro_after_init data might still be writable

The tool is based on libbpf and is located under tools/bpf.

Here some specific questions for discussion.

* I've noticed that some users of ALLOW_ERROR_INJECTION() in the kernel add the option to fail behind a Kconfig (Fault-injection framework, FAULT_INJECTION) in lib/Kconfig.debug, such as FAILSLAB or FAIL_PAGE_ALLOC. Shall we add similar condition for error injection in modules?

* What other functions may people be interested in adding to the list of suported for error injection?

* I found a more generic error injection tool (inject.py) [1] in IOVisor/BCC (Ccing IOVisor mailing list as well). The tool seems to be maintained and supports error injection based on a specified call chain and optional predicates. However, it relies on the old and deprecated Python eBPF/BCC infrastructure. Since there are other users of ALLOW_ERROR_INJECTION(), I'm curious: is there a different generic tool to generate error injection in the kernel already converted to libbpf? Do users of ALLOW_ERROR_INJECTION() still depend on inject.py tool?

[1] https://github.com/iovisor/bcc/blob/master/tools/inject.py

Additionally, would it make sense to develop an in-tree tool with libbpf support for this purpose? This approach might require rewriting inject.py in libbpf and/or extending moderr to include inject.py features (probably replace and rework some of the current functionality in moderr).

moderr uses an enum to detect which kprobe function to override the return value. I think adding bpf_get_stack()/bpf_get_stackid() support would avoid the need for hardcoding the kprobe detection mechanism. This approach could also be used for navigation of the call stack as in inject.py. Are there any alternative methods to achieve this that avoid relying on enumerating the fuctions to probe?

---
Daniel Gomez (2):
module: allow for module error injection
moderr: add module error injection tool

 kernel/module/main.c          |   3 +
 kernel/module/strict_rwx.c    |   2 +
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 8 files changed, 515 insertions(+), 3 deletions(-)

---
base-commit: 232f121837ad8b1c21cc80f2c8842a4090c5a2a0
change-id: 20250121-modules-error-injection-176c525791b0

Best regards,

Signature · 1 line
Daniel Gomez <[email protected]>
#2 lore raw

PATCH 1/2 module: allow for module error injection

Allow to test kernel module initialization with eBPF error injection by forcing errors when any of the below annotated functions with ALLOW_ERROR_INJECTION() return.

Allow to debug and test module load error behaviour when:

complete_formation(): when module initialization switches from MODULE_STATE_UNFORMED to MODULE_STATE_COMING stage.

do_init_module(): when an error occurs during module initialization and before we switch from MODULE_STATE_COMING to MODULE_STATE_LIVE stage.

module_enable_rodata_ro_after_init(): when an error occurs while setting memory to RO after module initialization. This is when module initialization reaches MODULE_STATE_LIVE stage.

---
 kernel/module/main.c       | 3 +++
 kernel/module/strict_rwx.c | 2 ++
 2 files changed, 5 insertions(+)
kernel/module/main.c +3 -0
index 1fb9ad289a6f8f328fc37c6d93730882d0e6e613..54e6c4d0aab23ae5001a52c26417e7e7957ea3fd 100644
@@ -59,6 +59,7 @@
 #include <linux/codetag.h>
 #include <linux/debugfs.h>
 #include <linux/execmem.h>
+#include <linux/bpf.h>
 #include <uapi/linux/module.h>
 #include "internal.h"
 
@@ -3089,6 +3090,7 @@ static noinline int do_init_module(struct module *mod)
 
 	return ret;
 }
+ALLOW_ERROR_INJECTION(do_init_module, ERRNO);
 
 static int may_init_module(void)
 {
@@ -3225,6 +3227,7 @@ static int complete_formation(struct module *mod, struct load_info *info)
 	mutex_unlock(&module_mutex);
 	return err;
 }
+ALLOW_ERROR_INJECTION(complete_formation, ERRNO);
 
 static int prepare_coming_module(struct module *mod)
 {
kernel/module/strict_rwx.c +2 -0
index 74834ba15615fa02d1fa72b913feedd7400af779..21936f5cc1e679e4de90331ef4920597d9093f35 100644
@@ -9,6 +9,7 @@
 #include <linux/mm.h>
 #include <linux/vmalloc.h>
 #include <linux/set_memory.h>
+#include <linux/bpf.h>
 #include "internal.h"
 
 static int module_set_memory(const struct module *mod, enum mod_mem_type type,
@@ -71,6 +72,7 @@ int module_enable_rodata_ro_after_init(const struct module *mod)
 
 	return module_set_memory(mod, MOD_RO_AFTER_INIT, set_memory_ro);
 }
+ALLOW_ERROR_INJECTION(module_enable_rodata_ro_after_init, ERRNO);
 
 int module_enable_data_nx(const struct module *mod)
 {
Signature · 1 line
2.39.5
#3 lore raw

Re: [PATCH 1/2] module: allow for module error injection

Quoted 27 lines · Daniel Gomez

Allow to test kernel module initialization with eBPF error injection by forcing errors when any of the below annotated functions with ALLOW_ERROR_INJECTION() return.

Allow to debug and test module load error behaviour when:

complete_formation(): when module initialization switches from MODULE_STATE_UNFORMED to MODULE_STATE_COMING stage.

do_init_module(): when an error occurs during module initialization and before we switch from MODULE_STATE_COMING to MODULE_STATE_LIVE stage.

module_enable_rodata_ro_after_init(): when an error occurs while setting memory to RO after module initialization. This is when module initialization reaches MODULE_STATE_LIVE stage.

---
 kernel/module/main.c       | 3 +++
 kernel/module/strict_rwx.c | 2 ++
 2 files changed, 5 insertions(+)
kernel/module/main.c +0 -0
index 1fb9ad289a6f8f328fc37c6d93730882d0e6e613..54e6c4d0aab23ae5001a52c26417e7e7957ea3fd 100644

kernel/module/main.c line 62

Quoted 4 lines
kernel/module/main.c +1 -0
@@ -59,6 +59,7 @@
 #include <linux/codetag.h>
 #include <linux/debugfs.h>
 #include <linux/execmem.h>
+#include <linux/bpf.h>

This should be <linux/error-injection.h>, no?

Quoted 19 lines
kernel/module/main.c +2 -0
 #include <uapi/linux/module.h>
 #include "internal.h"
 
@@ -3089,6 +3090,7 @@ static noinline int do_init_module(struct module *mod)
 
 	return ret;
 }
+ALLOW_ERROR_INJECTION(do_init_module, ERRNO);
 
 static int may_init_module(void)
 {
@@ -3225,6 +3227,7 @@ static int complete_formation(struct module *mod, struct load_info *info)
 	mutex_unlock(&module_mutex);
 	return err;
 }
+ALLOW_ERROR_INJECTION(complete_formation, ERRNO);
 
 static int prepare_coming_module(struct module *mod)
 {

<snip>

#4 lore raw

PATCH 2/2 moderr: add module error injection tool

In reply to Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
tools/bpf/Makefile +10 -3
index 243b79f2b451e52ca196f79dc46befd1b3dab458..018cab5102e7e42b8b7b2749f4f463bf55c5119b 100644
@@ -38,7 +38,7 @@ FEATURE_TESTS = libbfd disassembler-four-args disassembler-init-styled
 FEATURE_DISPLAY = libbfd
 
 check_feat := 1
-NON_CHECK_FEAT_TARGETS := clean bpftool_clean runqslower_clean resolve_btfids_clean
+NON_CHECK_FEAT_TARGETS := clean bpftool_clean moderr_clean runqslower_clean resolve_btfids_clean
 ifdef MAKECMDGOALS
 ifeq ($(filter-out $(NON_CHECK_FEAT_TARGETS),$(MAKECMDGOALS)),)
   check_feat := 0
@@ -76,7 +76,7 @@ $(OUTPUT)%.lex.o: $(OUTPUT)%.lex.c
 
 PROGS = $(OUTPUT)bpf_jit_disasm $(OUTPUT)bpf_dbg $(OUTPUT)bpf_asm
 
-all: $(PROGS) bpftool runqslower
+all: $(PROGS) bpftool moderr runqslower
 
 $(OUTPUT)bpf_jit_disasm: CFLAGS += -DPACKAGE='bpf_jit_disasm'
 $(OUTPUT)bpf_jit_disasm: $(OUTPUT)bpf_jit_disasm.o
@@ -92,7 +92,7 @@ $(OUTPUT)bpf_exp.lex.c: $(OUTPUT)bpf_exp.yacc.c
 $(OUTPUT)bpf_exp.yacc.o: $(OUTPUT)bpf_exp.yacc.c
 $(OUTPUT)bpf_exp.lex.o: $(OUTPUT)bpf_exp.lex.c
 
-clean: bpftool_clean runqslower_clean resolve_btfids_clean
+clean: bpftool_clean moderr_clean runqslower_clean resolve_btfids_clean
 	$(call QUIET_CLEAN, bpf-progs)
 	$(Q)$(RM) -r -- $(OUTPUT)*.o $(OUTPUT)bpf_jit_disasm $(OUTPUT)bpf_dbg \
 	       $(OUTPUT)bpf_asm $(OUTPUT)bpf_exp.yacc.* $(OUTPUT)bpf_exp.lex.*
@@ -118,6 +118,12 @@ bpftool_install:
 bpftool_clean:
 	$(call descend,bpftool,clean)
 
+moderr:
+	$(call descend,moderr)
+
+moderr_clean:
+	$(call descend,moderr,clean)
+
 runqslower:
 	$(call descend,runqslower)
 
@@ -131,5 +137,6 @@ resolve_btfids_clean:
 	$(call descend,resolve_btfids,clean)
 
 .PHONY: all install clean bpftool bpftool_install bpftool_clean \
+	moderr moderr_clean \
 	runqslower runqslower_clean \
 	resolve_btfids resolve_btfids_clean
tools/bpf/moderr/.gitignore new file +2 -0
new file mode 100644
index 0000000000000000000000000000000000000000..ffdb70230c8bc308efcc8b7d2084856e2225da91
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+/.output
tools/bpf/moderr/Makefile new file +95 -0

95 lines not on this page · raw message

tools/bpf/moderr/moderr.bpf.c new file +127 -0

127 lines not on this page · raw message

tools/bpf/moderr/moderr.c new file +236 -0

236 lines not on this page · raw message

tools/bpf/moderr/moderr.h new file +40 -0
new file mode 100644
index 0000000000000000000000000000000000000000..e17440cf4bd5fe09b927cb83807a88f66861bba5
@@ -0,0 +1,40 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Copyright (c) 2025 Samsung */
+#ifndef __MODERR_H
+#define __MODERR_H
+
+#define MAX_PARAM_PREFIX_LEN (64 - sizeof(unsigned long))
+#define MODULE_NAME_LEN MAX_PARAM_PREFIX_LEN
+#define MODULE_FUNC_LEN 128
+#define MESSAGE_LEN 128
+
+enum modfunc {
+	UNKNOWN,
+	COMPLETE_FORMATION = 1,
+	DO_INIT_MODULE,
+	MODULE_ENABLE_RODATA_AFTER_INIT,
+};
+
+struct event {
+	char modname[MODULE_NAME_LEN];
+	int err;
+	int func;
+	char msg[MESSAGE_LEN];
+	int dbg;
+};
+
+static inline const char *modfunc_to_string(enum modfunc fc)
+{
+	switch (fc) {
+	case COMPLETE_FORMATION:
+		return "complete_formation()";
+	case DO_INIT_MODULE:
+		return "do_init_module()";
+	case MODULE_ENABLE_RODATA_AFTER_INIT:
+		return "module_enable_rodata_after_init()";
+	default:
+		return "unknown";
+	}
+}
+
+#endif /* __MODERR_H */
Signature · 1 line
2.39.5
#5 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

The value of error injection is not clear to me. Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.
These 3 functions don't seem to be in this category.

maintainer
#6 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

[0] https://www.spinics.net/lists/linux-block/msg68159.html

Luis

#7 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 63 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

Quoted 8 lines

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific? Based on its current implementation and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  
	23

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate. Although I think having it in kernel tree is simpler maintenance-wise.

Lucas De Marchi

Quoted 4 lines

[0] https://www.spinics.net/lists/linux-block/msg68159.html

Luis

maintainer
#8 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

Luis

maintainer
#9 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 97 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Quoted 8 lines

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

Quoted 2 lines

Luis

#10 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 103 lines · Daniel Gomez
Quoted 102 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

Quoted 8 lines
Quoted 3 lines

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

I don't think we use them... we just set them to "always trigger" and then create the conditions for that to happen. But my question still
remains: what is the benefit of using the bpf approach over writing these files?

Lucas De Marchi

Quoted 3 lines
Quoted 3 lines

Luis

maintainer
#11 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 113 lines · Lucas De Marchi
Quoted 103 lines · Daniel Gomez
Quoted 102 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

Quoted 8 lines
Quoted 3 lines

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

Quoted 4 lines

I don't think we use them... we just set them to "always trigger" and then create the conditions for that to happen. But my question still
remains: what is the benefit of using the bpf approach over writing these files?

The code to trigger the error injection still needs to exist with both approaches. My understanding from debugfs and the comment from Luis earlier in the thread is that with eBPF you can mitigate how intrusive in-kernel code changes are to allow error injection. Luis added the example here [1] for debugfs.

[1] https://lore.kernel.org/all/[email protected]/

To compare patch 8 in [1] with eBPF approach: the patch describes all the necessary changes required to allow error injection on the add_disk() path. With eBPF one would simply annotate the function(s) with ALLOW_ERROR_INJECTION(), e.g. device_add() and replace the return value in eBPF code with bpf_override_return() as implemented in moderr tool for module_enable_rdata_after_init() for example.

New error injection users such modules or block/disk would benefit of the eBPF approach with having a more simpler in-kernel code for the same purpose. Current users of debugfs/errorinj would have to remove all the upstream intrusive error injection related code if they want to adopt the eBPF approach.

Daniel

Quoted 6 lines

Lucas De Marchi

Quoted 3 lines
Quoted 3 lines

Luis

#12 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 125 lines · Daniel Gomez
Quoted 113 lines · Lucas De Marchi
Quoted 103 lines · Daniel Gomez
Quoted 102 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

Quoted 8 lines
Quoted 3 lines

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

Quoted 4 lines

I don't think we use them... we just set them to "always trigger" and then create the conditions for that to happen. But my question still
remains: what is the benefit of using the bpf approach over writing these files?

The code to trigger the error injection still needs to exist with both approaches. My understanding from debugfs and the comment from Luis earlier in the thread is that with eBPF you can mitigate how intrusive in-kernel code changes are to allow error injection. Luis added the example here [1] for debugfs.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/

To compare patch 8 in [1] with eBPF approach: the patch describes all the necessary changes required to allow error injection on the add_disk() path. With eBPF one would simply annotate the function(s) with ALLOW_ERROR_INJECTION(), e.g. device_add() and replace the return value in eBPF code with bpf_override_return() as implemented in moderr tool for module_enable_rdata_after_init() for example.

but that is all that we need with the fail_function in debugfs too:

$ git grep ALLOW_ERROR_INJECTION -- drivers/gpu/drm/xe
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(xe_device_create, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(wait_for_lmem_ready, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_exec_queue.c:ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
drivers/gpu/drm/xe/xe_ggtt.c:ALLOW_ERROR_INJECTION(xe_ggtt_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ads.c:ALLOW_ERROR_INJECTION(xe_guc_ads_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ct.c:ALLOW_ERROR_INJECTION(xe_guc_ct_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_log.c:ALLOW_ERROR_INJECTION(xe_guc_log_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_relay.c:ALLOW_ERROR_INJECTION(xe_guc_relay_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() is used to conditionally skip function execution
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() macro but this is acceptable because for those
drivers/gpu/drm/xe/xe_pm.c:ALLOW_ERROR_INJECTION(xe_pm_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_create, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_prepare, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_run, ERRNO);
drivers/gpu/drm/xe/xe_sriov.c:ALLOW_ERROR_INJECTION(xe_sriov_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_sync.c:ALLOW_ERROR_INJECTION(xe_sync_entry_parse, ERRNO);
drivers/gpu/drm/xe/xe_tile.c:ALLOW_ERROR_INJECTION(xe_tile_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_uc_fw.c:ALLOW_ERROR_INJECTION(xe_uc_fw_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vma_ops_alloc, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vm_create_scratch, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(vm_bind_ioctl_ops_create, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(vm_bind_ioctl_ops_execute, ERRNO);
drivers/gpu/drm/xe/xe_wa.c:ALLOW_ERROR_INJECTION(xe_wa_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_wopcm.c:ALLOW_ERROR_INJECTION(xe_wopcm_init, ERRNO); /* See xe_pci_probe() */

That is different from the patch you are pointing to because that patch is trying to add arbitrary/named error injection points throughout the code. However via debugfs it's still possible to add error injection to the beginning of a function by annotating that function with ALLOW_ERROR_INJECTION. If a function is annotated with that, then if you do e.g.

echo xe_device_create > /sys/kernel/debug/fail_function/inject

it will cause that function to fail. There are some additional files to control _when_ that function should fail, but I'm failing to see a clear benefit. See this example in the docs:

Documentation/fault-injection/fault-injection.rst:- Inject open_ctree error while btrfs mount::

Lucas De Marchi

Quoted 14 lines

New error injection users such modules or block/disk would benefit of the eBPF approach with having a more simpler in-kernel code for the same purpose. Current users of debugfs/errorinj would have to remove all the upstream intrusive error injection related code if they want to adopt the eBPF approach.

Daniel

Quoted 6 lines

Lucas De Marchi

Quoted 3 lines
Quoted 3 lines

Luis

maintainer
#13 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 163 lines · Lucas De Marchi
Quoted 125 lines · Daniel Gomez
Quoted 113 lines · Lucas De Marchi
Quoted 103 lines · Daniel Gomez
Quoted 102 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

Quoted 8 lines
Quoted 3 lines

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

Quoted 4 lines

I don't think we use them... we just set them to "always trigger" and then create the conditions for that to happen. But my question still
remains: what is the benefit of using the bpf approach over writing these files?

The code to trigger the error injection still needs to exist with both approaches. My understanding from debugfs and the comment from Luis earlier in the thread is that with eBPF you can mitigate how intrusive in-kernel code changes are to allow error injection. Luis added the example here [1] for debugfs.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/

To compare patch 8 in [1] with eBPF approach: the patch describes all the necessary changes required to allow error injection on the add_disk() path. With eBPF one would simply annotate the function(s) with ALLOW_ERROR_INJECTION(), e.g. device_add() and replace the return value in eBPF code with bpf_override_return() as implemented in moderr tool for module_enable_rdata_after_init() for example.

but that is all that we need with the fail_function in debugfs too:

$ git grep ALLOW_ERROR_INJECTION -- drivers/gpu/drm/xe
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(xe_device_create, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(wait_for_lmem_ready, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_exec_queue.c:ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
drivers/gpu/drm/xe/xe_ggtt.c:ALLOW_ERROR_INJECTION(xe_ggtt_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ads.c:ALLOW_ERROR_INJECTION(xe_guc_ads_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ct.c:ALLOW_ERROR_INJECTION(xe_guc_ct_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_log.c:ALLOW_ERROR_INJECTION(xe_guc_log_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_relay.c:ALLOW_ERROR_INJECTION(xe_guc_relay_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() is used to conditionally skip function execution
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() macro but this is acceptable because for those
drivers/gpu/drm/xe/xe_pm.c:ALLOW_ERROR_INJECTION(xe_pm_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_create, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_prepare, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_run, ERRNO);
drivers/gpu/drm/xe/xe_sriov.c:ALLOW_ERROR_INJECTION(xe_sriov_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_sync.c:ALLOW_ERROR_INJECTION(xe_sync_entry_parse, ERRNO);
drivers/gpu/drm/xe/xe_tile.c:ALLOW_ERROR_INJECTION(xe_tile_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_uc_fw.c:ALLOW_ERROR_INJECTION(xe_uc_fw_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vma_ops_alloc, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vm_create_scratch, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(vm_bind_ioctl_ops_create, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(vm_bind_ioctl_ops_execute, ERRNO);
drivers/gpu/drm/xe/xe_wa.c:ALLOW_ERROR_INJECTION(xe_wa_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_wopcm.c:ALLOW_ERROR_INJECTION(xe_wopcm_init, ERRNO); /* See xe_pci_probe() */
Quoted 3 lines

That is different from the patch you are pointing to because that patch is trying to add arbitrary/named error injection points throughout the code. However via debugfs it's still possible to add error injection to

When reading the patch I assumed the block/failure-injection.c was needed for the knobs in sysfs/debugfs. But I see I was wrong and these are only needed for the arbitrary error injection points?

I see mm/fail_page_alloc.c has a similar approach with should_fail_alloc_page().

Quoted 9 lines

the beginning of a function by annotating that function with ALLOW_ERROR_INJECTION. If a function is annotated with that, then if you do e.g.

echo xe_device_create > /sys/kernel/debug/fail_function/inject

it will cause that function to fail. There are some additional files to control _when_ that function should fail, but I'm failing to see a clear benefit. See this example in the docs:

Can you clarify if _when_ (in debugfs) allows you to access function arguments of a given annotated function with ALLOW_ERROR_INJECTION()? It seems that might be the only part that can be moved out of the kernel and handled in eBPF. Other than that, I don't see either a benefit of using one approach over the other.

Quoted 4 lines

Documentation/fault-injection/fault-injection.rst:- Inject open_ctree error while btrfs mount::

Lucas De Marchi

#14 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

+Francois who added most of the error injection points in xe

Quoted 164 lines · Daniel Gomez
Quoted 163 lines · Lucas De Marchi
Quoted 125 lines · Daniel Gomez
Quoted 113 lines · Lucas De Marchi
Quoted 103 lines · Daniel Gomez
Quoted 102 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

Quoted 8 lines
Quoted 3 lines

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

Quoted 4 lines

I don't think we use them... we just set them to "always trigger" and then create the conditions for that to happen. But my question still
remains: what is the benefit of using the bpf approach over writing these files?

The code to trigger the error injection still needs to exist with both approaches. My understanding from debugfs and the comment from Luis earlier in the thread is that with eBPF you can mitigate how intrusive in-kernel code changes are to allow error injection. Luis added the example here [1] for debugfs.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/

To compare patch 8 in [1] with eBPF approach: the patch describes all the necessary changes required to allow error injection on the add_disk() path. With eBPF one would simply annotate the function(s) with ALLOW_ERROR_INJECTION(), e.g. device_add() and replace the return value in eBPF code with bpf_override_return() as implemented in moderr tool for module_enable_rdata_after_init() for example.

but that is all that we need with the fail_function in debugfs too:

$ git grep ALLOW_ERROR_INJECTION -- drivers/gpu/drm/xe
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(xe_device_create, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(wait_for_lmem_ready, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_exec_queue.c:ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
drivers/gpu/drm/xe/xe_ggtt.c:ALLOW_ERROR_INJECTION(xe_ggtt_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ads.c:ALLOW_ERROR_INJECTION(xe_guc_ads_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ct.c:ALLOW_ERROR_INJECTION(xe_guc_ct_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_log.c:ALLOW_ERROR_INJECTION(xe_guc_log_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_relay.c:ALLOW_ERROR_INJECTION(xe_guc_relay_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() is used to conditionally skip function execution
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() macro but this is acceptable because for those
drivers/gpu/drm/xe/xe_pm.c:ALLOW_ERROR_INJECTION(xe_pm_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_create, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_prepare, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_run, ERRNO);
drivers/gpu/drm/xe/xe_sriov.c:ALLOW_ERROR_INJECTION(xe_sriov_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_sync.c:ALLOW_ERROR_INJECTION(xe_sync_entry_parse, ERRNO);
drivers/gpu/drm/xe/xe_tile.c:ALLOW_ERROR_INJECTION(xe_tile_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_uc_fw.c:ALLOW_ERROR_INJECTION(xe_uc_fw_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vma_ops_alloc, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vm_create_scratch, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERRORALLOW_ERROR_INJECTION_INJECTION(vm_bind_ioctl_ops_create, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(vm_bind_ioctl_ops_execute, ERRNO);
drivers/gpu/drm/xe/xe_wa.c:ALLOW_ERROR_INJECTION(xe_wa_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_wopcm.c:ALLOW_ERROR_INJECTION(xe_wopcm_init, ERRNO); /* See xe_pci_probe() */
Quoted 7 lines
Quoted 3 lines

That is different from the patch you are pointing to because that patch is trying to add arbitrary/named error injection points throughout the code. However via debugfs it's still possible to add error injection to

When reading the patch I assumed the block/failure-injection.c was needed for the knobs in sysfs/debugfs. But I see I was wrong and these are only needed for the arbitrary error injection points?

yeah. For working with ALLOW_ERROR_INJECTION() we have to refactor the code so the functions follow its requirements. When that is true, then we can simply use the fail_function/inject to trigger it.

Quoted 9 lines

I see mm/fail_page_alloc.c has a similar approach with should_fail_alloc_page().

Quoted 6 lines

the beginning of a function by annotating that function with ALLOW_ERROR_INJECTION. If a function is annotated with that, then if you do e.g.

echo xe_device_create > /sys/kernel/debug/fail_function/inject

Quoted 8 lines
Quoted 3 lines

it will cause that function to fail. There are some additional files to control _when_ that function should fail, but I'm failing to see a clear benefit. See this example in the docs:

Can you clarify if _when_ (in debugfs) allows you to access function arguments of a given annotated function with ALLOW_ERROR_INJECTION()? It seems that might be the only part that can be moved out of the kernel and handled in eBPF. Other than that, I don't see either a benefit of using one approach over the other.

afaik we can't change the behavior based on arguments when using the debugfs approach.

Lucas De Marchi

Quoted 6 lines
Quoted 5 lines

Documentation/fault-injection/fault-injection.rst:- Inject open_ctree error while btrfs mount::

Lucas De Marchi

#15 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 174 lines · Lucas De Marchi

+Francois who added most of the error injection points in xe

Quoted 171 lines · Daniel Gomez
Quoted 163 lines · Lucas De Marchi
Quoted 125 lines · Daniel Gomez
Quoted 113 lines · Lucas De Marchi
Quoted 103 lines · Daniel Gomez
Quoted 102 lines · Luis Chamberlain
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 5 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific?

Gotta start somewhere.

Quoted 8 lines

Based on its current implementation
and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23
Quoted 8 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate.

Would we install this upon install target?

Danny can decide on this :)

Quoted 2 lines

Although I think having it in kernel tree is
simpler maintenance-wise.

Quoted 8 lines
Quoted 3 lines

I think we have at least two users upstream who can make use of it. If we end up going through tools/testing/selftests/module/ first, can't you make use of it later?

What are the features in debugfs required to be useful for xe that we can port to an eBPF version? I see from the link provided the use of probability, interval, times and space but these are configured to allways trigger the error. Is that right?

Quoted 4 lines

I don't think we use them... we just set them to "always trigger" and then create the conditions for that to happen. But my question still
remains: what is the benefit of using the bpf approach over writing these files?

The code to trigger the error injection still needs to exist with both approaches. My understanding from debugfs and the comment from Luis earlier in the thread is that with eBPF you can mitigate how intrusive in-kernel code changes are to allow error injection. Luis added the example here [1] for debugfs.

Quoted 8 lines

[1] https://lore.kernel.org/all/[email protected]/

To compare patch 8 in [1] with eBPF approach: the patch describes all the necessary changes required to allow error injection on the add_disk() path. With eBPF one would simply annotate the function(s) with ALLOW_ERROR_INJECTION(), e.g. device_add() and replace the return value in eBPF code with bpf_override_return() as implemented in moderr tool for module_enable_rdata_after_init() for example.

but that is all that we need with the fail_function in debugfs too:

$ git grep ALLOW_ERROR_INJECTION -- drivers/gpu/drm/xe
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(xe_device_create, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_device.c:ALLOW_ERROR_INJECTION(wait_for_lmem_ready, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_exec_queue.c:ALLOW_ERROR_INJECTION(xe_exec_queue_create_bind, ERRNO);
drivers/gpu/drm/xe/xe_ggtt.c:ALLOW_ERROR_INJECTION(xe_ggtt_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ads.c:ALLOW_ERROR_INJECTION(xe_guc_ads_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_ct.c:ALLOW_ERROR_INJECTION(xe_guc_ct_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_log.c:ALLOW_ERROR_INJECTION(xe_guc_log_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_guc_relay.c:ALLOW_ERROR_INJECTION(xe_guc_relay_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() is used to conditionally skip function execution
drivers/gpu/drm/xe/xe_pci.c: * ALLOW_ERROR_INJECTION() macro but this is acceptable because for those
drivers/gpu/drm/xe/xe_pm.c:ALLOW_ERROR_INJECTION(xe_pm_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_create, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_prepare, ERRNO);
drivers/gpu/drm/xe/xe_pt.c:ALLOW_ERROR_INJECTION(xe_pt_update_ops_run, ERRNO);
drivers/gpu/drm/xe/xe_sriov.c:ALLOW_ERROR_INJECTION(xe_sriov_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_sync.c:ALLOW_ERROR_INJECTION(xe_sync_entry_parse, ERRNO);
drivers/gpu/drm/xe/xe_tile.c:ALLOW_ERROR_INJECTION(xe_tile_init_early, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_uc_fw.c:ALLOW_ERROR_INJECTION(xe_uc_fw_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vma_ops_alloc, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(xe_vm_create_scratch, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERRORALLOW_ERROR_INJECTION_INJECTION(vm_bind_ioctl_ops_create, ERRNO);
drivers/gpu/drm/xe/xe_vm.c:ALLOW_ERROR_INJECTION(vm_bind_ioctl_ops_execute, ERRNO);
drivers/gpu/drm/xe/xe_wa.c:ALLOW_ERROR_INJECTION(xe_wa_init, ERRNO); /* See xe_pci_probe() */
drivers/gpu/drm/xe/xe_wopcm.c:ALLOW_ERROR_INJECTION(xe_wopcm_init, ERRNO); /* See xe_pci_probe() */
Quoted 3 lines

That is different from the patch you are pointing to because that patch is trying to add arbitrary/named error injection points throughout the code. However via debugfs it's still possible to add error injection to

When reading the patch I assumed the block/failure-injection.c was needed for the knobs in sysfs/debugfs. But I see I was wrong and these are only needed for the arbitrary error injection points?

Correct, ALLOW_ERROR_INJECTION() is sufficient to make a function error injectable and controllable with the debugfs knobs. Arbitrary error injection points might also be added using only ALLOW_ERROR_INJECTION() and extra functions:

    static int should_X_fail(void)
    {
      return 0;
    }
    ALLOW_ERROR_INJECTION(should_X_fail, ERRNO);

    int foo(...)
    {
      ...
      if (should_X_fail()) {
        // error
      }
      ...
    }

But this would introduce a lot of dummy code, which the block/failure-injection.c approach prevents.

Until now, fault injection in xe has mostly been used to test error handling during probe and during IOCTLs calls. Errors are unconditionnally injected in each function one by one during tests. This is why probability, interval, times and space are configured to always trigger.

Besides, function-level error injection points have been a good fit in the driver, hence the absence of arbitrary error injection points.

Francois

Quoted 34 lines

yeah. For working with ALLOW_ERROR_INJECTION() we have to refactor the code so the functions follow its requirements. When that is true, then we can simply use the fail_function/inject to trigger it.

Quoted 9 lines

I see mm/fail_page_alloc.c has a similar approach with should_fail_alloc_page().

Quoted 6 lines

the beginning of a function by annotating that function with ALLOW_ERROR_INJECTION. If a function is annotated with that, then if you do e.g.

echo xe_device_create > /sys/kernel/debug/fail_function/inject

Quoted 8 lines
Quoted 3 lines

it will cause that function to fail. There are some additional files to control _when_ that function should fail, but I'm failing to see a clear benefit. See this example in the docs:

Can you clarify if _when_ (in debugfs) allows you to access function arguments of a given annotated function with ALLOW_ERROR_INJECTION()? It seems that might be the only part that can be moved out of the kernel and handled in eBPF. Other than that, I don't see either a benefit of using one approach over the other.

afaik we can't change the behavior based on arguments when using the debugfs approach.

Lucas De Marchi

Quoted 6 lines
Quoted 5 lines

Documentation/fault-injection/fault-injection.rst:- Inject open_ctree error while btrfs mount::

Lucas De Marchi

maintainer
#16 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

In reply to Lucas De Marchi
Quoted 69 lines · Lucas De Marchi
Quoted 68 lines · Luis Chamberlain
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

For selftests we do add random tools.

Quoted 1 line

The value of error injection is not clear to me.

It is of great value, since it deals with corner cases which are otherwise hard to reproduce in places which a real error can be catostrophic.

Quoted 2 lines

Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

Right.

Quoted 1 line

These 3 functions don't seem to be in this category.

That's the key here we should focus on. The problem is when a maintainer *does* agree that adding an error injection entry is useful for testing, and we have a developer willing to do the work to help test / validate it. In this case, this error case is rare but we do want to strive to test this as we ramp up and extend our modules selftests.

Then there is the aspect of how to mitigate how instrusive code changes to allow error injection are. In 2021 we evaluated the prospect of error injection in-kernel long ago for other areas like the block layer for add_disk() failures [0] but the minimal interface to enable this from userspace with debugfs was considered just too intrusive.

This effort tried to evaluate what this could look like with eBPF to mitigate the required in-kernel code, and I believe the light weight nature of it by just requiring a sprinkle with ALLOW_ERROR_INJECTION() suffices to my taste.

Quoted 7 lines
Quoted 3 lines

So, perhaps the tools aspect can just go in:

tools/testing/selftests/module/

but why would it be module-specific? Based on its current implementation and discussion about inject.py it seems to be generic enough to be useful to test any function annotated with ALLOW_ERROR_INJECTION().

Right, but inject.py is based on the old/deprecated Python eBPF/BCC infrastructure (although I think it's still a working and maintained tool based on commit history). However, as I noted in the cover letter, I think it would be useful to port it to use libbpf instead.

Quoted 4 lines

As xe driver maintainer, it may be interesting to use such a tool:

	$ git grep ALLOW_ERROR_INJECT -- drivers/gpu/drm/xe | wc -l  	23

I was wondering if users of ALLOW_ERROR_INJECTION() still depend on inject.py tool, or other tools were used. Or perhaps all are using debugfs?

Quoted 6 lines

How does this approach compare to writing the function name on debugfs (the current approach in xe's testsuite)?

fail_function @ https://docs.kernel.org/fault-injection/fault-injection.html#fault-injection-capabilities-infrastructure
https://gitlab.freedesktop.org/drm/igt-gpu-tools/-/blob/master/tests/intel/xe_fault_injection.c?ref_type=heads#L108

IMHO and looking at the references above, looks "simpler" to handle these cases in eBPF. Specially because the error injection logic does not have to live along with the code but in a separate tool. Link from Luis [0] (also [1]) is a good example of when debugfs may be seen a bit too intrusive and how eBPF may resolve the problems maintainers have.

[1] https://lore.kernel.org/all/[email protected]/

In summary, a function annotated with the error injection tag can modify its return value in eBPF code by using the bpf_override_return() helper. The logic for deciding when to inject the error is likely similar to the current implementation in debugfs. This can be seen in patch 2, file tools/bpf/ moderr/moderr.bpf.c where error is injected based on module name and target function.

Quoted 3 lines

If you decide to have the tool to live somewhere else, then kmod repo could be a candidate. Although I think having it in kernel tree is

Thanks Lucas.

Quoted 1 line

simpler maintenance-wise.

I agree with you that having the tool (or maybe tools if we decide to split) in tree it may be easier and may be a way to showcase its usage more effectively.

Quoted 7 lines

Lucas De Marchi

Quoted 4 lines

[0] https://www.spinics.net/lists/linux-block/msg68159.html

Luis

maintainer
#17 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

In reply to Alexei Starovoitov
Quoted 32 lines · Alexei Starovoitov
Quoted 22 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

Quoted 8 lines
---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

Can you clarify what do you mean? There are other tools under tools/ and tools/ bpf [1].

[1] https://lore.kernel.org/bpf/20200114184230.GA204154@krava/

I will anyway move the tool to the suggested location in the other thread.

Quoted 4 lines

The value of error injection is not clear to me. Other places in the kernel use it to test paths in the kernel that are difficult to do otherwise.

By any chance do you know which tool are they using?

Quoted 1 line

These 3 functions don't seem to be in this category.

Luis already answered this.

#18 lore raw

Re: [PATCH 2/2] moderr: add module error injection tool

Quoted 34 lines · Daniel Gomez
Quoted 32 lines · Alexei Starovoitov
Quoted 30 lines · Daniel Gomez

Add support for a module error injection tool. The tool
can inject errors in the annotated module kernel functions such as complete_formation(), do_init_module() and module_enable_rodata_after_init(). Module name and module function are required parameters to have control over the error injection.

Example: Inject error -22 to module_enable_rodata_ro_after_init for brd module:

sudo moderr --modname=brd --modfunc=module_enable_rodata_ro_after_init \
--error=-22 --trace
Monitoring module error injection... Hit Ctrl-C to end.
MODULE ERROR FUNCTION
brd -22 module_enable_rodata_after_init()

Kernel messages:

[   89.463690] brd: module loaded
[   89.463855] brd: module_enable_rodata_ro_after_init() returned -22,

ro_after_init data might still be writable

---
 tools/bpf/Makefile            |  13 ++-
 tools/bpf/moderr/.gitignore   |   2 +
 tools/bpf/moderr/Makefile     |  95 +++++++++++++++++
 tools/bpf/moderr/moderr.bpf.c | 127 +++++++++++++++++++++++
 tools/bpf/moderr/moderr.c     | 236 ++++++++++++++++++++++++++++++++++++++++++
 tools/bpf/moderr/moderr.h     |  40 +++++++
 6 files changed, 510 insertions(+), 3 deletions(-)
Quoted 7 lines
Quoted 2 lines

The tool looks useful, but we don't add tools to the kernel repo. It has to stay out of tree.

Can you clarify what do you mean? There are other tools under tools/ and tools/ bpf [1].

[1] https://lore.kernel.org/bpf/20200114184230.GA204154@krava/

As you noticed we added only one tool out of many and recently discussed removing it, since the value of keeping it in the tree is minimal.

Quoted 1 line

I will anyway move the tool to the suggested location in the other thread.

I don't think it's a good idea.
Keep it outside. There is no reason for it to live in the tree.