Skip to contentSkip to conversation

fault-injection capabilities (v3)

PATCH In review Akinobu Mita 2 participants 25 messages Sep 14, 2006 – Sep 20, 2006

Subsystems (8)
lore
Share & tools
lore
#0 lore raw

cover fault-injection capabilities (v3)

Changes from last version

- add stacktrace filter
- drop process-filter from boot-time setup
- build fixes from Don Mullis
- add documentation and some scripts

---

This patch set provides some fault-injection capabilities.

- kmalloc() failures

- alloc_pages() failures

- disk IO errors

We can see what really happens if those failures happen.

In order to enable these fault-injection capabilities:

1. Enable relevant config options (CONFIG_FAILSLAB, CONFIG_PAGE_ALLOC, CONFIG_MAKE_REQUEST) and runtime configuration kernel module (CONFIG_FAULT_INJECTION_DEBUGFS)

2. build and boot with this kernel

3. modprobe fault-inject-debugfs

4. configure fault-injection capabilities behavior by debugfs

For example about kmalloc failures:

/debug/failslab/probability

specifies how often it should fail in percent.

/debug/failslab/interval

specifies the interval of failures.

/debug/failslab/times

specifies how many times failures may happen at most.

/debug/failslab/space

specifies the size of free space where memory can be allocated safely in bytes.

/debug/failslab/process-filter

specifies whether process filter is enabled or not.
it allows failing only permitted processes by /proc/<pid>/make-it-fail

/debug/failslab/stacktrace-depth

specifies the maximum stacktrace depth walking allowed. a value '0' means stacktrace filter is disabled.

/debug/failslab/address-start
/debug/failslab/address-end

specifies the range of virtual address.
it allows failing only if the stacktrace hits in this range.

5. see what really happens.

#1 lore raw

PATCH 1/8 documentation and scripts

This patch provides the documentation and helper/testing scripts for fault-injection capabilities.

 Documentation/fault-injection/failcmd.sh          |    4 
 Documentation/fault-injection/failmodule.sh       |   32 ++++
 Documentation/fault-injection/fault-injection.txt |  150 ++++++++++++++++++++++
 3 files changed, 186 insertions(+)
Documentation/fault-injection/failcmd.sh new file +4 -0
Index: work-shouldfail/Documentation/fault-injection/failcmd.sh
===================================================================
@@ -0,0 +1,4 @@
+#!/bin/bash
+
+echo 1 > /proc/self/make-it-fail
+exec $*
Documentation/fault-injection/failmodule.sh new file +32 -0
Index: work-shouldfail/Documentation/fault-injection/failmodule.sh
===================================================================
@@ -0,0 +1,32 @@
+#!/bin/bash
+#
+# Usage: failmodule <failname> <modulename> [stacktrace-depth]
+#
+#	<failname>: "failslab", "fail_alloc_page", or "fail_make_request"
+#
+#	<modulename>: module name that you want to inject faults.
+#
+#	[stacktrace-depth]: the maximum number of stacktrace walking allowed
+#
+
+STACKTRACE_DEPTH=5
+if [ $# -gt 2 ]; then
+	STACKTRACE_DEPTH=$3
+fi
+
+if [ ! -d /debug/$1 ]; then
+	echo "Fault-injection $1 does not exist" >&2
+	exit 1
+fi
+if [ ! -d /sys/module/$2 ]; then
+	echo "Module $2 does not exist" >&2
+	exit 1
+fi
+
+# Disable stacktrace filter at first
+echo 0 > /debug/$1/address-end
+echo 1 > /debug/$1/stacktrace-depth
+
+echo `cat /sys/module/$2/sections/.text` > /debug/$1/address-start
+echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/address-end
+echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth
Documentation/fault-injection/fault-injection.txt new file +150 -0

150 lines not on this page · raw message

--

#4 lore raw

PATCH 2/8 fault-injection capabilities infrastructure

In reply to Akinobu Mita

This patch provides some functions for implement fault-injection capabilities.

- Lightweight random simulator is taken from crasher module for SUSE kernel
- The function should_fail() is taken from failmalloc-1.0
(http://www.nongnu.org/failmalloc/)

 include/linux/fault-inject.h |   41 ++++++++++++++++++++
 lib/Kconfig.debug            |    4 ++
 lib/Makefile                 |    1 
 lib/fault-inject.c           |   84 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 130 insertions(+)
lib/Kconfig.debug +4 -0
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
@@ -368,3 +368,7 @@ config RCU_TORTURE_TEST
 	  at boot time (you probably don't).
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
+
+config FAULT_INJECTION
+	bool
+
lib/Makefile +1 -0
Index: work-shouldfail/lib/Makefile
===================================================================
@@ -52,6 +52,7 @@ obj-$(CONFIG_SMP) += percpu_counter.o
 obj-$(CONFIG_AUDIT_GENERIC) += audit.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
+obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
 
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h
include/linux/fault-inject.h new file +41 -0

41 lines not on this page · raw message

lib/fault-inject.c new file +84 -0

84 lines not on this page · raw message

--

#7 lore raw

PATCH 3/8 fault-injection capability for kmalloc

In reply to Akinobu Mita

This patch provides fault-injection capability for kmalloc.

Boot option:

failslab=<interval>,<probability>,<space>,<times>

<interval> -- specifies the interval of failures.

<probability> -- specifies how often it should fail in percent.

<space> -- specifies the size of free space where memory can be
allocated safely in bytes.

<times> -- specifies how many times failures may happen at most.

Example:

	failslab=10,100,0,-1

slab allocation (kmalloc(), kmem_cache_alloc(),..) fails once per 10 times.

 include/linux/fault-inject.h |    4 ++++
 lib/Kconfig.debug            |    7 +++++++
 mm/slab.c                    |   34 ++++++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+)
mm/slab.c +34 -0
Index: work-shouldfail/mm/slab.c
===================================================================
@@ -108,6 +108,7 @@
 #include	<linux/mempolicy.h>
 #include	<linux/mutex.h>
 #include	<linux/rtmutex.h>
+#include	<linux/fault-inject.h>
 
 #include	<asm/uaccess.h>
 #include	<asm/cacheflush.h>
@@ -2963,11 +2964,44 @@ static void *cache_alloc_debugcheck_afte
 #define cache_alloc_debugcheck_after(a,b,objp,d) (objp)
 #endif
 
+#ifdef CONFIG_FAILSLAB
+
+static DEFINE_FAULT_ATTR(failslab_attr);
+
+static int __init setup_failslab(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_fault_attr(&failslab_attr, str);
+}
+__setup("failslab=", setup_failslab);
+
+struct fault_attr *failslab = &failslab_attr;
+EXPORT_SYMBOL_GPL(failslab);
+
+static int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+	if ((flags & __GFP_NOFAIL) || cachep == &cache_cache)
+		return 0;
+	return should_fail(failslab, obj_size(cachep));
+}
+
+#else
+
+static inline int should_failslab(struct kmem_cache *cachep, gfp_t flags)
+{
+	return 0;
+}
+
+#endif
+
 static inline void *____cache_alloc(struct kmem_cache *cachep, gfp_t flags)
 {
 	void *objp;
 	struct array_cache *ac;
 
+	if (should_failslab(cachep, flags))
+		return NULL;
+
 #ifdef CONFIG_NUMA
 	if (unlikely(current->flags & (PF_SPREAD_SLAB | PF_MEMPOLICY))) {
 		objp = alternate_node_alloc(cachep, flags);
lib/Kconfig.debug +7 -0
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
@@ -372,3 +372,10 @@ config RCU_TORTURE_TEST
 config FAULT_INJECTION
 	bool
 
+config FAILSLAB
+	bool "fault-injection capabilitiy for kmalloc"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capabilitiy for kmalloc.
+
include/linux/fault-inject.h +4 -0
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
@@ -36,6 +36,10 @@ int setup_fault_attr(struct fault_attr *
 void should_fail_srandom(unsigned long entropy);
 int should_fail(struct fault_attr *attr, ssize_t size);
 
+#ifdef CONFIG_FAILSLAB
+extern struct fault_attr *failslab;
+#endif
+
 #endif /* CONFIG_FAULT_INJECTION */
 
 #endif /* _LINUX_FAULT_INJECT_H */

--

#17 lore raw

PATCH 8/8 stacktrace filtering for fault-injection capabilities

In reply to Akinobu Mita

This patch provides stacktrace filtering feature.
The stacktrace filter allows failing only for the caller you are interested in.

stacktrace filter is enabled by setting the value of /debugfs/*/stacktrace-depth more than 0.
and specify the range of the virtual address
by the /debugfs/*/address-start and /debugfs/*/address-end

Please see the example that demostrates how to inject slab allocation failures only for a specific module
in Documentation/fault-injection/fault-injection.txt

 include/linux/fault-inject.h |   15 +++++++++
 lib/Kconfig.debug            |    2 +
 lib/fault-inject-debugfs.c   |   32 +++++++++++++++++++
 lib/fault-inject.c           |   71 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 120 insertions(+)
lib/fault-inject.c +71 -0
Index: work-shouldfail/lib/fault-inject.c
===================================================================
@@ -6,6 +6,9 @@
 #include <linux/fs.h>
 #include <linux/module.h>
 #include <linux/interrupt.h>
+#include <linux/unwind.h>
+#include <linux/stacktrace.h>
+#include <linux/kallsyms.h>
 #include <linux/fault-inject.h>
 
 int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -59,6 +62,72 @@ static int fail_process(struct fault_att
 	return !in_interrupt() && task->make_it_fail;
 }
 
+#ifdef CONFIG_STACK_UNWIND
+
+static asmlinkage int fail_stacktrace_callback(struct unwind_frame_info *info,
+						void *arg)
+{
+	int depth;
+	struct fault_attr *attr = arg;
+
+	for (depth = 0; depth < attr->stacktrace_depth
+			&& unwind(info) == 0 && UNW_PC(info); depth++) {
+		if (arch_unw_user_mode(info))
+			break;
+		if (attr->address_start <= UNW_PC(info) &&
+			       UNW_PC(info) < attr->address_end)
+			return 1;
+	}
+	return 0;
+}
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+	struct unwind_frame_info info;
+
+	/* stacktrace filter is disabled */
+	if (attr->stacktrace_depth == 0)
+		return 1;
+
+	return unwind_init_running(&info, fail_stacktrace_callback, attr);
+}
+
+#elif defined(CONFIG_STACKTRACE)
+
+#include <linux/stacktrace.h>
+
+#define MAX_STACK_TRACE_DEPTH 10
+
+static int fail_stacktrace(struct fault_attr *attr)
+{
+	struct stack_trace trace;
+	int depth = attr->stacktrace_depth;
+	unsigned long entries[MAX_STACK_TRACE_DEPTH];
+	int n;
+
+	/* stacktrace filter is disabled */
+	if (depth == 0)
+		return 1;
+
+	trace.nr_entries = 0;
+	trace.entries = entries;
+	trace.max_entries = (depth < MAX_STACK_TRACE_DEPTH) ?
+				depth : MAX_STACK_TRACE_DEPTH;
+
+	save_stack_trace(&trace, NULL, 0, 1);
+	for (n = 0; n < trace.nr_entries; n++)
+		if (attr->address_start <= entries[n] &&
+			       entries[n] < attr->address_end)
+			return 1;
+	return 0;
+}
+
+#else
+
+#define fail_stacktrace(attr)	(0)
+
+#endif
+
 /*
  * This code is stolen from failmalloc-1.0
  * http://www.nongnu.org/failmalloc/
@@ -68,6 +137,8 @@ int should_fail(struct fault_attr *attr,
 {
 	if (!fail_process(attr, current))
 		return 0;
+	if (!fail_stacktrace(attr))
+		return 0;
 
 	if (atomic_read(&max_failures(attr)) == 0)
 		return 0;
include/linux/fault-inject.h +15 -0
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
@@ -30,6 +30,21 @@ struct fault_attr {
 
 	/* A value of '0' means process filter is disabled. */
 	u32 process_filter;
+
+	/*
+	 * maximam number of stacktrace depth walking allowed
+	 * A value of '0' means stacktrace filter is disabled.
+	 */
+	unsigned long stacktrace_depth;
+
+	/*
+	 * If stacktrace_depth is enabled, it allows failing only when it
+	 * has been called from the virtual addresses in the range
+	 * 'address_start' to 'address_end-1'
+	 */ 
+	unsigned long address_start;
+	unsigned long address_end;
+
 };
 
 #define DEFINE_FAULT_ATTR(name) \
lib/fault-inject-debugfs.c +32 -0
Index: work-shouldfail/lib/fault-inject-debugfs.c
===================================================================
@@ -9,6 +9,9 @@ struct fault_attr_entries {
 	struct dentry *times_file;
 	struct dentry *space_file;
 	struct dentry *process_filter_file;
+	struct dentry *stacktrace_depth_file;
+	struct dentry *address_start_file;
+	struct dentry *address_end_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -71,6 +74,18 @@ static void cleanup_fault_attr_entries(s
 			debugfs_remove(entries->process_filter_file);
 			entries->process_filter_file = NULL;
 		}
+		if (entries->stacktrace_depth_file) {
+			debugfs_remove(entries->stacktrace_depth_file);
+			entries->stacktrace_depth_file = NULL;
+		}
+		if (entries->address_start_file) {
+			debugfs_remove(entries->address_start_file);
+			entries->address_start_file = NULL;
+		}
+		if (entries->address_end_file) {
+			debugfs_remove(entries->address_end_file);
+			entries->address_end_file = NULL;
+		}
 		debugfs_remove(entries->dir);
 		entries->dir = NULL;
 	}
@@ -116,6 +131,23 @@ static int init_fault_attr_entries(struc
 		goto fail;
 	entries->process_filter_file = file;
 
+	file = debugfs_create_ul("stacktrace-depth", mode, dir,
+				   &attr->stacktrace_depth);
+	if (!file)
+		goto fail;
+	entries->stacktrace_depth_file = file;
+
+	file = debugfs_create_ul("address-start", mode, dir,
+				   &attr->address_start);
+	if (!file)
+		goto fail;
+	entries->address_start_file = file;
+
+	file = debugfs_create_ul("address-end", mode, dir, &attr->address_end);
+	if (!file)
+		goto fail;
+	entries->address_end_file = file;
+
 	return 0;
 fail:
 	cleanup_fault_attr_entries(entries);
lib/Kconfig.debug +2 -0
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
@@ -371,6 +371,8 @@ config RCU_TORTURE_TEST
 
 config FAULT_INJECTION
 	bool
+	select STACKTRACE
+	select FRAME_POINTER
 
 config FAILSLAB
 	bool "fault-injection capabilitiy for kmalloc"

--

#9 lore raw

PATCH 5/8 fault-injection capability for disk IO

In reply to Akinobu Mita

This patch provides fault-injection capability for disk IO.

Boot option:

fail_make_request=<probability>,<interval>,<space>,<times>

<interval> -- specifies the interval of failures.

<probability> -- specifies how often it should fail in percent.

<space> -- specifies the size of free space where disk IO can be issued
safely in bytes.

<times> -- specifies how many times failures may happen at most.

Example:

	fail_make_request=10,100,0,-1
	echo 1 > /sys/blocks/hda/hda1/make-it-fail

generic_make_request() on /dev/hda1 fails once per 10 times.

 block/genhd.c                |   31 +++++++++++++++++++++++++++++++
 block/ll_rw_blk.c            |   35 +++++++++++++++++++++++++++++++++++
 fs/partitions/check.c        |   27 +++++++++++++++++++++++++++
 include/linux/fault-inject.h |    3 +++
 include/linux/genhd.h        |    4 ++++
 lib/Kconfig.debug            |    7 +++++++
 6 files changed, 107 insertions(+)
block/ll_rw_blk.c +35 -0
Index: work-shouldfail/block/ll_rw_blk.c
===================================================================
@@ -28,6 +28,7 @@
 #include <linux/interrupt.h>
 #include <linux/cpu.h>
 #include <linux/blktrace_api.h>
+#include <linux/fault-inject.h>
 
 /*
  * for max sense size
@@ -2993,6 +2994,37 @@ static void handle_bad_sector(struct bio
 	set_bit(BIO_EOF, &bio->bi_flags);
 }
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static DEFINE_FAULT_ATTR(fail_make_request_attr);
+
+static int __init setup_fail_make_request(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_fault_attr(&fail_make_request_attr, str);
+}
+__setup("fail_make_request=", setup_fail_make_request);
+
+struct fault_attr *fail_make_request = &fail_make_request_attr;
+EXPORT_SYMBOL_GPL(fail_make_request);
+
+static int should_fail_request(struct bio *bio)
+{
+	if ((bio->bi_bdev->bd_disk->flags & GENHD_FL_FAIL) ||
+	    (bio->bi_bdev->bd_part && bio->bi_bdev->bd_part->make_it_fail))
+		return should_fail(fail_make_request, bio->bi_size);
+
+	return 0;
+}
+#else
+
+static inline int should_fail_request(struct bio *bio)
+{
+	return 0;
+}
+
+#endif
+
 /**
  * generic_make_request: hand a buffer to its device driver for I/O
  * @bio:  The bio describing the location in memory and on the device.
@@ -3077,6 +3109,9 @@ end_io:
 		if (unlikely(test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
 			goto end_io;
 
+		if (should_fail_request(bio))
+			goto end_io;
+
 		/*
 		 * If this device has partitions, remap block n
 		 * of partition p to block n+start(p) of the disk.
lib/Kconfig.debug +7 -0
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
@@ -386,3 +386,10 @@ config FAIL_PAGE_ALLOC
 	help
 	  This option provides fault-injection capabilitiy for alloc_pages().
 
+config FAIL_MAKE_REQUEST
+	bool "fault-injection capabilitiy for disk IO"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION 
+	help
+	  This option provides fault-injection capabilitiy to disk IO.
+
block/genhd.c +31 -0
Index: work-shouldfail/block/genhd.c
===================================================================
@@ -412,6 +412,34 @@ static struct disk_attribute disk_attr_s
 	.show	= disk_stats_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t disk_fail_store(struct gendisk * disk,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0) {
+		if (i == 0)
+			disk->flags &= ~GENHD_FL_FAIL;
+		else
+			disk->flags |= GENHD_FL_FAIL;
+	}
+
+	return count;
+}
+static ssize_t disk_fail_read(struct gendisk * disk, char *page)
+{
+	return sprintf(page, "%d\n", disk->flags & GENHD_FL_FAIL ? 1 : 0);
+}
+static struct disk_attribute disk_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= disk_fail_store,
+	.show	= disk_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&disk_attr_uevent.attr,
 	&disk_attr_dev.attr,
@@ -419,6 +447,9 @@ static struct attribute * default_attrs[
 	&disk_attr_removable.attr,
 	&disk_attr_size.attr,
 	&disk_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&disk_attr_fail.attr,
+#endif
 	NULL,
 };
 
include/linux/genhd.h +4 -0
Index: work-shouldfail/include/linux/genhd.h
===================================================================
@@ -81,6 +81,9 @@ struct hd_struct {
 	struct kobject *holder_dir;
 	unsigned ios[2], sectors[2];	/* READs and WRITEs */
 	int policy, partno;
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	int make_it_fail;
+#endif
 };
 
 #define GENHD_FL_REMOVABLE			1
@@ -88,6 +91,7 @@ struct hd_struct {
 #define GENHD_FL_CD				8
 #define GENHD_FL_UP				16
 #define GENHD_FL_SUPPRESS_PARTITION_INFO	32
+#define GENHD_FL_FAIL				64
 
 struct disk_stats {
 	unsigned long sectors[2];	/* READs and WRITEs */
fs/partitions/check.c +27 -0
Index: work-shouldfail/fs/partitions/check.c
===================================================================
@@ -265,12 +265,39 @@ static struct part_attribute part_attr_s
 	.show	= part_stat_read
 };
 
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+
+static ssize_t part_fail_store(struct hd_struct * p,
+			       const char *buf, size_t count)
+{
+	int i;
+
+	if (count > 0 && sscanf(buf, "%d", &i) > 0)
+		p->make_it_fail = (i == 0) ? 0 : 1;
+
+	return count;
+}
+static ssize_t part_fail_read(struct hd_struct * p, char *page)
+{
+	return sprintf(page, "%d\n", p->make_it_fail);
+}
+static struct part_attribute part_attr_fail = {
+	.attr = {.name = "make-it-fail", .mode = S_IRUGO | S_IWUSR },
+	.store	= part_fail_store,
+	.show	= part_fail_read
+};
+
+#endif
+
 static struct attribute * default_attrs[] = {
 	&part_attr_uevent.attr,
 	&part_attr_dev.attr,
 	&part_attr_start.attr,
 	&part_attr_size.attr,
 	&part_attr_stat.attr,
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+	&part_attr_fail.attr,
+#endif
 	NULL,
 };
 
include/linux/fault-inject.h +3 -0
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
@@ -42,6 +42,9 @@ extern struct fault_attr *failslab;
 #ifdef CONFIG_FAIL_PAGE_ALLOC
 extern struct fault_attr *fail_page_alloc;
 #endif
+#ifdef CONFIG_FAIL_MAKE_REQUEST
+extern struct fault_attr *fail_make_request;
+#endif
 
 #endif /* CONFIG_FAULT_INJECTION */
 

--

#12 lore raw

PATCH 7/8 process filtering for fault-injection capabilities

In reply to Akinobu Mita

This patch provides process filtering feature. The process filter allows failing only permitted processes by /proc/<pid>/make-it-fail

Please see the example that demostrates how to inject slab allocation failures into module init/cleanup code
in Documentation/fault-injection/fault-injection.txt

 fs/proc/base.c               |   77 +++++++++++++++++++++++++++++++++++++++++++
 include/linux/fault-inject.h |    3 +
 include/linux/sched.h        |    3 +
 lib/fault-inject.c           |   13 +++++++
 4 files changed, 96 insertions(+)
fs/proc/base.c +77 -0
Index: work-shouldfail/fs/proc/base.c
===================================================================
@@ -138,6 +138,9 @@ enum pid_directory_inos {
 #endif
 	PROC_TGID_OOM_SCORE,
 	PROC_TGID_OOM_ADJUST,
+#ifdef CONFIG_FAULT_INJECTION
+	PROC_TGID_FAULT_INJECTION,
+#endif
 	PROC_TID_INO,
 	PROC_TID_STATUS,
 	PROC_TID_MEM,
@@ -181,6 +184,9 @@ enum pid_directory_inos {
 #endif
 	PROC_TID_OOM_SCORE,
 	PROC_TID_OOM_ADJUST,
+#ifdef CONFIG_FAULT_INJECTION
+	PROC_TID_FAULT_INJECTION,
+#endif
 
 	/* Add new entries before this */
 	PROC_TID_FD_DIR = 0x8000,	/* 0x8000-0xffff */
@@ -240,6 +246,9 @@ static struct pid_entry tgid_base_stuff[
 #ifdef CONFIG_AUDITSYSCALL
 	E(PROC_TGID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	E(PROC_TGID_FAULT_INJECTION, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
 	{0,0,NULL,0}
 };
 static struct pid_entry tid_base_stuff[] = {
@@ -282,6 +291,9 @@ static struct pid_entry tid_base_stuff[]
 #ifdef CONFIG_AUDITSYSCALL
 	E(PROC_TID_LOGINUID, "loginuid", S_IFREG|S_IWUSR|S_IRUGO),
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	E(PROC_TID_FAULT_INJECTION, "make-it-fail", S_IFREG|S_IWUSR|S_IRUGO),
+#endif
 	{0,0,NULL,0}
 };
 
@@ -992,6 +1004,65 @@ static struct file_operations proc_login
 };
 #endif
 
+#ifdef CONFIG_FAULT_INJECTION
+static ssize_t proc_fault_inject_read(struct file * file, char __user * buf,
+				      size_t count, loff_t *ppos)
+{
+	struct task_struct *task = get_proc_task(file->f_dentry->d_inode);
+	char buffer[PROC_NUMBUF];
+	size_t len;
+	int make_it_fail;
+	loff_t __ppos = *ppos;
+
+	if (!task)
+		return -ESRCH;
+	make_it_fail = task->make_it_fail;
+	put_task_struct(task);
+
+	len = snprintf(buffer, sizeof(buffer), "%i\n", make_it_fail);
+	if (__ppos >= len)
+		return 0;
+	if (count > len-__ppos)
+		count = len-__ppos;
+	if (copy_to_user(buf, buffer + __ppos, count))
+		return -EFAULT;
+	*ppos = __ppos + count;
+	return count;
+}
+
+static ssize_t proc_fault_inject_write(struct file * file,
+			const char __user * buf, size_t count, loff_t *ppos)
+{
+	struct task_struct *task;
+	char buffer[PROC_NUMBUF], *end;
+	int make_it_fail;
+
+	if (!capable(CAP_SYS_RESOURCE))
+		return -EPERM;
+	memset(buffer, 0, sizeof(buffer));
+	if (count > sizeof(buffer) - 1)
+		count = sizeof(buffer) - 1;
+	if (copy_from_user(buffer, buf, count))
+		return -EFAULT;
+	make_it_fail = simple_strtol(buffer, &end, 0);
+	if (*end == '\n')
+		end++;
+	task = get_proc_task(file->f_dentry->d_inode);
+	if (!task)
+		return -ESRCH;
+	task->make_it_fail = make_it_fail;
+	put_task_struct(task);
+	if (end - buffer == 0)
+		return -EIO;
+	return end - buffer;
+}
+
+static struct file_operations proc_fault_inject_operations = {
+	.read		= proc_fault_inject_read,
+	.write		= proc_fault_inject_write,
+};
+#endif
+
 #ifdef CONFIG_SECCOMP
 static ssize_t seccomp_read(struct file *file, char __user *buf,
 			    size_t count, loff_t *ppos)
@@ -1834,6 +1905,12 @@ static struct dentry *proc_pident_lookup
 			inode->i_fop = &proc_loginuid_operations;
 			break;
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+		case PROC_TID_FAULT_INJECTION:
+		case PROC_TGID_FAULT_INJECTION:
+			inode->i_fop = &proc_fault_inject_operations;
+			break;
+#endif
 		default:
 			printk("procfs: impossible type (%d)",p->type);
 			iput(inode);
include/linux/sched.h +3 -0
Index: work-shouldfail/include/linux/sched.h
===================================================================
@@ -996,6 +996,9 @@ struct task_struct {
 #ifdef	CONFIG_TASK_DELAY_ACCT
 	struct task_delay_info *delays;
 #endif
+#ifdef CONFIG_FAULT_INJECTION
+	int make_it_fail;
+#endif
 };
 
 static inline pid_t process_group(struct task_struct *tsk)
include/linux/fault-inject.h +3 -0
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
@@ -27,6 +27,9 @@ struct fault_attr {
 	atomic_t space;
 
 	unsigned long count;
+
+	/* A value of '0' means process filter is disabled. */
+	u32 process_filter;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
lib/fault-inject.c +13 -0
Index: work-shouldfail/lib/fault-inject.c
===================================================================
@@ -5,6 +5,7 @@
 #include <linux/types.h>
 #include <linux/fs.h>
 #include <linux/module.h>
+#include <linux/interrupt.h>
 #include <linux/fault-inject.h>
 
 int setup_fault_attr(struct fault_attr *attr, char *str)
@@ -49,6 +50,15 @@ void should_fail_srandom(unsigned long e
 	should_fail_random();
 }
 
+static int fail_process(struct fault_attr *attr, struct task_struct *task)
+{
+	/* process filter is disabled */
+	if (!attr->process_filter)
+		return 1;
+
+	return !in_interrupt() && task->make_it_fail;
+}
+
 /*
  * This code is stolen from failmalloc-1.0
  * http://www.nongnu.org/failmalloc/
@@ -56,6 +66,9 @@ void should_fail_srandom(unsigned long e
 
 int should_fail(struct fault_attr *attr, ssize_t size)
 {
+	if (!fail_process(attr, current))
+		return 0;
+
 	if (atomic_read(&max_failures(attr)) == 0)
 		return 0;
 

--

#8 lore raw

PATCH 4/8 fault-injection capability for alloc_pages()

In reply to Akinobu Mita

This patch provides fault-injection capability for alloc_pages()

Boot option:

fail_page_alloc=<interval>,<probability>,<space>,<times>

<interval> -- specifies the interval of failures.

<probability> -- specifies how often it should fail in percent.

<space> -- specifies the size of free space where memory can be
allocated safely in pages.

<times> -- specifies how many times failures may happen at most.

Example:

	fail_page_alloc=10,100,0,-1

The page allocation (alloc_pages(), ...) fails once per 10 times.

 include/linux/fault-inject.h |    3 +++
 lib/Kconfig.debug            |    7 +++++++
 mm/page_alloc.c              |   35 +++++++++++++++++++++++++++++++++++
 3 files changed, 45 insertions(+)
lib/Kconfig.debug +7 -0
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
@@ -379,3 +379,10 @@ config FAILSLAB
 	help
 	  This option provides fault-injection capabilitiy for kmalloc.
 
+config FAIL_PAGE_ALLOC
+	bool "fault-injection capabilitiy for alloc_pages()"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capabilitiy for alloc_pages().
+
mm/page_alloc.c +35 -0
Index: work-shouldfail/mm/page_alloc.c
===================================================================
@@ -37,6 +37,7 @@
 #include <linux/vmalloc.h>
 #include <linux/mempolicy.h>
 #include <linux/stop_machine.h>
+#include <linux/fault-inject.h>
 
 #include <asm/tlbflush.h>
 #include <asm/div64.h>
@@ -903,6 +904,37 @@ get_page_from_freelist(gfp_t gfp_mask, u
 	return page;
 }
 
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+
+static DEFINE_FAULT_ATTR(fail_page_alloc_attr);
+
+static int __init setup_fail_page_alloc(char *str)
+{
+	should_fail_srandom(jiffies);
+	return setup_fault_attr(&fail_page_alloc_attr, str);
+}
+__setup("fail_page_alloc=", setup_fail_page_alloc);
+
+struct fault_attr *fail_page_alloc = &fail_page_alloc_attr;
+EXPORT_SYMBOL_GPL(fail_page_alloc);
+
+static int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+	if (gfp_mask & __GFP_NOFAIL)
+		return 0;
+
+	return should_fail(fail_page_alloc, 1 << order);
+}
+
+#else
+
+static inline int should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
+{
+	return 0;
+}
+
+#endif
+
 /*
  * This is the 'heart' of the zoned buddy allocator.
  */
@@ -921,6 +953,9 @@ __alloc_pages(gfp_t gfp_mask, unsigned i
 
 	might_sleep_if(wait);
 
+	if (should_fail_alloc_page(gfp_mask, order))
+		return NULL;
+
 restart:
 	z = zonelist->zones;  /* the list of zones suitable for gfp_mask */
 
include/linux/fault-inject.h +3 -0
Index: work-shouldfail/include/linux/fault-inject.h
===================================================================
@@ -39,6 +39,9 @@ int should_fail(struct fault_attr *attr,
 #ifdef CONFIG_FAILSLAB
 extern struct fault_attr *failslab;
 #endif
+#ifdef CONFIG_FAIL_PAGE_ALLOC
+extern struct fault_attr *fail_page_alloc;
+#endif
 
 #endif /* CONFIG_FAULT_INJECTION */
 

--

#10 lore raw

PATCH 6/8 debugfs entries for configuration

In reply to Akinobu Mita

This kernel module provides debugfs entries to enable to configure fault-injection capabilities for failslab, fail_page_alloc, and fail_make_request.

The slab allocator, the page allocator, and the block layer are initalized before debugfs is available. and failslab, fail_page_alloc, and fail_make_request are also enabled at the same time.
So I put the initalization and cleanup for these debugfs entries into this kernel module.

This module provides the following entries so that we can configure by writing these files.

/debug/

|-- fail_make_request
|   |-- interval
|   |-- probability
|   |-- space
|   `-- times
|-- fail_page_alloc
|   |-- interval
|   |-- probability
|   |-- space
|   `-- times
`-- failslab
    |-- interval
    |-- probability
    |-- space
    `-- times
 lib/Kconfig.debug          |    8 ++
 lib/Makefile               |    1 
 lib/fault-inject-debugfs.c |  179 +++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 188 insertions(+)
lib/Kconfig.debug +8 -0
Index: work-shouldfail/lib/Kconfig.debug
===================================================================
@@ -393,3 +393,11 @@ config FAIL_MAKE_REQUEST
 	help
 	  This option provides fault-injection capabilitiy to disk IO.
 
+config FAULT_INJECTION_DEBUGFS 
+	tristate "runtime configuration for fault-injection capabilities"
+	depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
+	select DEBUG_FS
+	help
+	  This option provides kernel module that provides runtime
+	  configuration interface by debugfs.
+
lib/fault-inject-debugfs.c new file +179 -0

179 lines not on this page · raw message

lib/Makefile +1 -0
Index: work-shouldfail/lib/Makefile
===================================================================
@@ -53,6 +53,7 @@ obj-$(CONFIG_AUDIT_GENERIC) += audit.o
 
 obj-$(CONFIG_SWIOTLB) += swiotlb.o
 obj-$(CONFIG_FAULT_INJECTION) += fault-inject.o
+obj-$(CONFIG_FAULT_INJECTION_DEBUGFS) += fault-inject-debugfs.o
 
 hostprogs-y	:= gen_crc32table
 clean-files	:= crc32table.h

--

#5 lore raw

Re: [patch 2/8] fault-injection capabilities infrastructure

In reply to Akinobu Mita

Replace individual structure element comments with reference to Documentation/fault-injection/fault-injection.txt

Init "interval" to 1 (smallest useful value). Init "times" to 1 rather than -1 (infinity), for fewer accidental system lockups.

---
 include/linux/fault-inject.h |   20 +++++---------------
 1 file changed, 5 insertions(+), 15 deletions(-)
include/linux/fault-inject.h +5 -15
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
@@ -6,31 +6,21 @@
 #include <linux/types.h>
 #include <asm/atomic.h>
 
+/*
+ * For explanation of the elements of this struct, see
+ * Documentation/fault-injection/fault-injection.txt
+ */
 struct fault_attr {
-
-	/* how often it should fail in percent. */
 	unsigned long probability;
-
-	/* the interval of failures. */
 	unsigned long interval;
-
-	/*
-	 * how many times failures may happen at most.
-	 * A value of '-1' means infinity.
-	 */
 	atomic_t times;
-
-	/*
-	 * the size of free space where memory can be allocated safely.
-	 * A value of '0' means infinity.
-	 */
 	atomic_t space;
 
 	unsigned long count;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
-	struct fault_attr name = { .times = ATOMIC_INIT(-1), }
+	struct fault_attr name = { .interval=1, .times = ATOMIC_INIT(1), }
 
 int setup_fault_attr(struct fault_attr *attr, char *str);
 void should_fail_srandom(unsigned long entropy);
#2 lore raw

Re: [patch 1/8] documentation and scripts

In reply to Akinobu Mita

Add reference to similar functionality in drivers/md/faulty.c .

Clarify debugfs descriptions.

Add an entry for fault-injection to
Documentation/kernel-parameters.txt .

---
 Documentation/fault-injection/fault-injection.txt |   85 ++++++++++------------
 Documentation/kernel-parameters.txt               |    7 +
 2 files changed, 47 insertions(+), 45 deletions(-)
Documentation/fault-injection/fault-injection.txt +40 -45
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
@@ -1,6 +1,9 @@
 Fault injection capabilities infrastructure
 ===========================================
 
+See also drivers/md/faulty.c .
+
+
 Available fault injection capabilities
 --------------------------------------
 
@@ -14,58 +17,49 @@ o fail_page_alloc
 
 o fail_make_request
 
-  injects disk IO errors on permitted devices by /sys/block/<device>/make-it-fail
-  or /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
+  injects disk IO errors on permitted devices by
+  /sys/block/<device>/make-it-fail or
+  /sys/block/<device>/<partition>/make-it-fail. (generic_make_request())
 
 Configure fault-injection capabilities behavior
 -----------------------------------------------
 
-Example for failslab:
-
 o debugfs entries
 
 fault-inject-debugfs kernel module provides some debugfs entries for runtime
-configuration for fault-injection capabilities.
-
-- /debug/failslab/probability:
+configuration of fault-injection capabilities.
 
-	specifies how often it should fail in percent.
+- /debug/*/probability:
 
-- /debug/failslab/interval:
-
-	specifies the interval of failures.
-
-- /debug/failslab/times:
-
-	specifies how many times failures may happen at most.
+	likelihood of failure injection, in percent.
 
-- /debug/failslab/space:
+- /debug/*/interval:
 
-	specifies the size of free space where memory can be allocated
-	safely in bytes.
+	specifies the interval between failures, for calls to
+	should_fail() that pass all the other tests.
 
-- /debug/failslab/process-filter:
+	Note that if you enable this, by setting interval>1, you will
+	probably want to set probability=100.
 
-	specifies whether the process filter is enabled or not.
-	It allows failing only permitted processes by /proc/<pid>/make-it-fail
+- /debug/*/times:
 
-- /debug/failslab/stacktrace-depth:
-
-	specifies the maximum stacktrace depth walking allowed.
-	A value '0' means stacktrace filter is disabled.
+	specifies how many times failures may happen at most.
+	A value of -1 means "no limit".
 
-- /debug/failslab/address-start:
-- /debug/failslab/address-end:
+- /debug/*/space:
 
-	specifies the range of virtual address.
-	It allows failing only if the stacktrace hits in this range.
+	specifies an initial resource "budget", decremented by "size"
+	on each call to should_fail(,size).  Failure injection is
+	suppressed until "space" reaches zero.
 
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
-We can use boot option.
+use the boot option:
 
-- failslab=<interval>,<probability>,<space>,<times>
+    failslab=
+    fail_page_alloc=
+    fail_make_request=<interval>,<probability>,<space>,<times>
 
 How to add new fault injection capability
 -----------------------------------------
@@ -77,44 +71,42 @@ o define the fault attributes
   DEFINE_FAULT_INJECTION(name);
 
   Please see the definition of struct fault_attr in fault-inject.h
-  for the detail.
+  for details.
 
 o provide the way to configure fault attributes
 
 - boot option
 
-  If you need to enable the fault injection capability ealier boot time,
+  If you need to enable the fault injection capability from boot time,
   you can provide boot option to configure it. There is a helper function for it.
 
   setup_fault_attr(attr, str);
 
 - module parameters
 
-  If the scope of the fault injection capability is limited by a kernel module,
-  It is better to provide module parameters to configure the member of fault
-  attributes.
+  If the scope of the fault injection capability is limited to a
+  single kernel module, it is better to provide module parameters to
+  configure the fault attributes.
 
 - debugfs entries
 
   failslab, fail_page_alloc, and fail_make_request use this way.
-
-  But now there is no helper functions to provides debugfs entries for
+  But now there is no helper functions to provide debugfs entries for
   fault injection capabilities. Please refer to lib/fault-inject-debugfs.c
   to know how to do. And please try not to add new one into
   fault-inject-debugfs module.
-
-  Because failslab, fail_page_alloc, and fail_make_request are used ealier
-  boot time before debugfs is available and the slab allocator,
-  the page allocator, and the block layer cannot be built as module.
+  Because failslab, fail_page_alloc, and fail_make_request are used from
+  boot time, before debugfs is available, the slab allocator,
+  the page allocator, and the block layer cannot be built as modules.
 
 o add a hook to insert failures
 
   should_fail() returns 1 when failures should happen.
 
-  should_fail(attr);
+	should_fail(attr,size);
 
-Tests
------
+Application Examples
+--------------------
 
 o inject slab allocation failures into module init/cleanup code
 
@@ -148,3 +140,6 @@ echo 25 > /debug/failslab/probability
 
 ------------------------------------------------------------------------------
 
+
+
+
Documentation/kernel-parameters.txt +7 -0
Index: linux-2.6.17/Documentation/kernel-parameters.txt
===================================================================
@@ -544,6 +544,13 @@ running once the system is up.
 	eurwdt=		[HW,WDT] Eurotech CPU-1220/1410 onboard watchdog.
 			Format: <io>[,<irq>]
 
+	failslab=
+	fail_page_alloc=
+	fail_make_request=[KNL]
+			General fault injection mechanism.
+			Format: <interval>,<probability>,<space>,<times>
+			See also /Documentation/fault-injection/.
+
 	fd_mcs=		[HW,SCSI]
 			See header of drivers/scsi/fd_mcs.c.
 
#11 lore raw

Re: [patch 6/8] debugfs entries for configuration

In reply to Akinobu Mita

Factor out process-filter functionality, move later in series to process-filter-specific patch.

---
 lib/fault-inject-debugfs.c |   11 -----------
 1 file changed, 11 deletions(-)
lib/fault-inject-debugfs.c +0 -11
Index: linux-2.6.17/lib/fault-inject-debugfs.c
===================================================================
@@ -8,7 +8,6 @@ struct fault_attr_entries {
 	struct dentry *interval_file;
 	struct dentry *times_file;
 	struct dentry *space_file;
-	struct dentry *process_filter_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -67,10 +66,6 @@ static void cleanup_fault_attr_entries(s
 			debugfs_remove(entries->space_file);
 			entries->space_file = NULL;
 		}
-		if (entries->process_filter_file) {
-			debugfs_remove(entries->process_filter_file);
-			entries->process_filter_file = NULL;
-		}
 		debugfs_remove(entries->dir);
 		entries->dir = NULL;
 	}
@@ -110,12 +105,6 @@ static int init_fault_attr_entries(struc
 		goto fail;
 	entries->space_file = file;
 
-	file = debugfs_create_bool("process-filter", mode, dir,
-				   &attr->process_filter);
-	if (!file)
-		goto fail;
-	entries->process_filter_file = file;
-
 	return 0;
 fail:
 	cleanup_fault_attr_entries(entries);
#13 lore raw

Re: [patch 7/8] process filtering for fault-injection capabilities

In reply to Akinobu Mita

Add functionality to the process_filter variable: A negative argument injects failures for only for pid==-process_filter, thereby permitting per-process failures from boot time.

Restore process-filter arg to kernel command line.

Reintroduce process_filter_file addition to debugfs, factored out earlier in series.

Add printk, called upon each failure injection.

---
 Documentation/fault-injection/fault-injection.txt |    9 +++++++
 include/linux/fault-inject.h                      |    1 
 lib/fault-inject-debugfs.c                        |   11 ++++++++
 lib/fault-inject.c                                |   27 ++++++++++++++++------
 4 files changed, 40 insertions(+), 8 deletions(-)
include/linux/fault-inject.h +1 -4
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
@@ -15,11 +15,8 @@ struct fault_attr {
 	unsigned long interval;
 	atomic_t times;
 	atomic_t space;
-
 	unsigned long count;
-
-	/* A value of '0' means process filter is disabled. */
-	u32 process_filter;
+	atomic_t process_filter;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
lib/fault-inject.c +20 -7
Index: linux-2.6.17/lib/fault-inject.c
===================================================================
@@ -14,11 +14,12 @@ int setup_fault_attr(struct fault_attr *
 	unsigned long interval;
 	int times;
 	int space;
+	int process_filter;
 
-	/* "<interval>,<probability>,<space>,<times>" */
-	if (sscanf(str, "%lu,%lu,%d,%d",
-			&interval, &probability, &space, &times) < 4) {
-		printk(KERN_WARNING "SHOULD_FAIL: failed to parse arguments\n");
+	/* "<interval>,<probability>,<process-filter>,<space>,<times>" */
+	if (sscanf(str, "%lu,%lu,%d,%d,%d",
+			&interval, &probability, &process_filter, &space, &times) < 4) {
+		printk(KERN_WARNING "FAULT_INJECTION: failed to parse arguments\n");
 		return 0;
 	}
 
@@ -26,6 +27,7 @@ int setup_fault_attr(struct fault_attr *
 	attr->interval = interval;
 	atomic_set(&attr->times, times);
 	atomic_set(&attr->space, space);
+	atomic_set(&attr->process_filter, process_filter);
 
 	return 1;
 }
@@ -53,10 +55,20 @@ void should_fail_srandom(unsigned long e
 static int fail_process(struct fault_attr *attr, struct task_struct *task)
 {
 	/* process filter is disabled */
-	if (!attr->process_filter)
+	if (atomic_read(&attr->process_filter)==0)
 		return 1;
 
-	return !in_interrupt() && task->make_it_fail;
+	/* controlled by /proc/<pid>/make-it-fail, but boolean not set */
+	if (atomic_read(&attr->process_filter)>0 && !task->make_it_fail)
+		return 0;
+
+	/* enabled for single pid, but no match with pid */
+	if (atomic_read(&attr->process_filter)<0 &&
+	    -atomic_read(&attr->process_filter)!=task->pid)
+		return 0;
+
+	/* XX  Are we sure we never want calls from interrupt context to fail? */
+	return !in_interrupt();
 }
 
 /*
@@ -89,7 +101,8 @@ int should_fail(struct fault_attr *attr,
 	return 0;
 
 fail:
-
+	printk(KERN_NOTICE "FAULT_INJECTION: forcing a failure, pid==%d\n",
+	       current->pid);
 	if (atomic_read(&max_failures(attr)) != -1)
 		atomic_dec_not_zero(&max_failures(attr));
 
Documentation/fault-injection/fault-injection.txt +9 -0
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
@@ -52,6 +52,15 @@ configuration of fault-injection capabil
 	on each call to should_fail(,size).  Failure injection is
 	suppressed until "space" reaches zero.
 
+- /debug/*/process-filter:
+
+	filter failures by pid.
+	A value of '0' disables filtering by process.
+	Any positive value limits failures to only processes indicated by
+	/proc/<pid>/make-it-fail==1.
+	A negative value means that failures are enabled for
+	pid==-process_filter irrespective of /proc/<pid>/make-it-fail.
+
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
lib/fault-inject-debugfs.c +11 -0
Index: linux-2.6.17/lib/fault-inject-debugfs.c
===================================================================
@@ -8,6 +8,7 @@ struct fault_attr_entries {
 	struct dentry *interval_file;
 	struct dentry *times_file;
 	struct dentry *space_file;
+	struct dentry *process_filter_file;
 };
 
 static void debugfs_ul_set(void *data, u64 val)
@@ -66,6 +67,10 @@ static void cleanup_fault_attr_entries(s
 			debugfs_remove(entries->space_file);
 			entries->space_file = NULL;
 		}
+		if (entries->process_filter_file) {
+			debugfs_remove(entries->process_filter_file);
+			entries->process_filter_file = NULL;
+		}
 		debugfs_remove(entries->dir);
 		entries->dir = NULL;
 	}
@@ -105,6 +110,12 @@ static int init_fault_attr_entries(struc
 		goto fail;
 	entries->space_file = file;
 
+	file = debugfs_create_atomic_t("process-filter", mode, dir,
+				       &attr->process_filter);
+	if (!file)
+		goto fail;
+	entries->process_filter_file = file;
+
 	return 0;
 fail:
 	cleanup_fault_attr_entries(entries);
#16 lore raw

Re: [patch 7/8] process filtering for fault-injection capabilities

In reply to Akinobu Mita

Temporary fix-up so that next patch:
"[patch 8/8] stacktrace filtering for fault-injection capabilities" applies cleanly.

---
 include/linux/fault-inject.h |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
include/linux/fault-inject.h +3 -1
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
@@ -16,7 +16,9 @@ struct fault_attr {
 	atomic_t times;
 	atomic_t space;
 	unsigned long count;
-	atomic_t process_filter;
+
+	/* A value of '0' means process filter is disabled. */
+	u32 process_filter;
 };
 
 #define DEFINE_FAULT_ATTR(name) \
#18 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

In reply to Akinobu Mita

Undo temporary fix-up for clean application of patch "[patch 8/8] stacktrace filtering for fault-injection capabilities".

Fix bug in !(CONFIG_STACK_UNWIND || CONFIG_STACKTRACE) case, based on code inspection only. Anyone with a non-i386, -x86_64, -s390 willing to test this?

Reintroduce stacktrace-specific documentation, factored out earlier.

---
 Documentation/fault-injection/fault-injection.txt |   13 +++++++++++++
 include/linux/fault-inject.h                      |    2 +-
 lib/fault-inject.c                                |    2 +-
 3 files changed, 15 insertions(+), 2 deletions(-)
lib/fault-inject.c +1 -1
Index: linux-2.6.17/lib/fault-inject.c
===================================================================
@@ -136,7 +136,7 @@ static int fail_stacktrace(struct fault_
 
 #else
 
-#define fail_stacktrace(attr)	(0)
+#define fail_stacktrace(attr)	(1)
 
 #endif
 
Documentation/fault-injection/fault-injection.txt +13 -0
Index: linux-2.6.17/Documentation/fault-injection/fault-injection.txt
===================================================================
@@ -61,6 +61,19 @@ configuration of fault-injection capabil
 	A negative value means that failures are enabled for
 	pid==-process_filter irrespective of /proc/<pid>/make-it-fail.
 
+- /debug/*/stacktrace-depth:
+
+	specifies the maximum stacktrace depth walked during search
+	for a caller within [address-start,address-end).  A value of 0
+	disables the stacktrace filter.
+
+- /debug/*/address-start:
+- /debug/*/address-end:
+
+	specifies the range of virtual addresses tested during
+	stacktrace walking.  Failure is injected only if some caller
+	in the walked stacktrace lies within this range.
+
 o Boot option
 
 In order to inject faults while debugfs is not available (early boot time),
include/linux/fault-inject.h +1 -1
Index: linux-2.6.17/include/linux/fault-inject.h
===================================================================
@@ -16,7 +16,7 @@ struct fault_attr {
 	atomic_t times;
 	atomic_t space;
 	unsigned long count;
-	u32 process_filter;
+	atomic_t process_filter;
 	unsigned long stacktrace_depth;
 	unsigned long address_start;
 	unsigned long address_end;
#20 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

In reply to Akinobu Mita

Push fault-injection entries earlier in the list, so that they appear nested under DEBUG_KERNEL in menuconfig/xconfig.

---
 lib/Kconfig.debug |   70 +++++++++++++++++++++++++++---------------------------
 1 file changed, 35 insertions(+), 35 deletions(-)
lib/Kconfig.debug +35 -35
Index: linux-2.6.17/lib/Kconfig.debug
===================================================================
@@ -294,6 +294,41 @@ config DEBUG_INFO
 
 	  If unsure, say N.
 
+config FAULT_INJECTION
+	bool
+	depends on DEBUG_KERNEL
+	select STACKTRACE
+	select FRAME_POINTER
+
+config FAILSLAB
+	bool "fault-injection capability for kmalloc"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capability for kmalloc.
+
+config FAIL_PAGE_ALLOC
+	bool "fault-injection capability for alloc_pages()"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capability for alloc_pages().
+
+config FAIL_MAKE_REQUEST
+	bool "fault-injection capability for disk IO"
+	depends on DEBUG_KERNEL
+	select FAULT_INJECTION
+	help
+	  This option provides fault-injection capability to disk IO.
+
+config FAULT_INJECTION_DEBUGFS
+	tristate "runtime configuration for fault-injection capabilities"
+	depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
+	select DEBUG_FS
+	help
+	  This option provides kernel module that provides runtime
+	  configuration interface by debugfs.
+
 config DEBUG_FS
 	bool "Debug Filesystem"
 	depends on SYSFS
@@ -368,38 +403,3 @@ config RCU_TORTURE_TEST
 	  at boot time (you probably don't).
 	  Say M if you want the RCU torture tests to build as a module.
 	  Say N if you are unsure.
-
-config FAULT_INJECTION
-	bool
-	select STACKTRACE
-	select FRAME_POINTER
-
-config FAILSLAB
-	bool "fault-injection capabilitiy for kmalloc"
-	depends on DEBUG_KERNEL
-	select FAULT_INJECTION
-	help
-	  This option provides fault-injection capabilitiy for kmalloc.
-
-config FAIL_PAGE_ALLOC
-	bool "fault-injection capabilitiy for alloc_pages()"
-	depends on DEBUG_KERNEL
-	select FAULT_INJECTION
-	help
-	  This option provides fault-injection capabilitiy for alloc_pages().
-
-config FAIL_MAKE_REQUEST
-	bool "fault-injection capabilitiy for disk IO"
-	depends on DEBUG_KERNEL
-	select FAULT_INJECTION 
-	help
-	  This option provides fault-injection capabilitiy to disk IO.
-
-config FAULT_INJECTION_DEBUGFS 
-	tristate "runtime configuration for fault-injection capabilities"
-	depends on DEBUG_KERNEL && SYSFS && FAULT_INJECTION
-	select DEBUG_FS
-	help
-	  This option provides kernel module that provides runtime
-	  configuration interface by debugfs.
-
#3 lore raw

Re: [patch 1/8] documentation and scripts

In reply to Don Mullis
Quoted 6 lines · Don Mullis

Add reference to similar functionality in drivers/md/faulty.c .

Clarify debugfs descriptions.

Add an entry for fault-injection to
Documentation/kernel-parameters.txt .

Thank you for fixing my poor English and good description. I merged this patch.

#6 lore raw

Re: [patch 2/8] fault-injection capabilities infrastructure

In reply to Don Mullis
Quoted 7 lines · Don Mullis

Replace individual structure element comments with reference to Documentation/fault-injection/fault-injection.txt

Init "interval" to 1 (smallest useful value). Init "times" to 1 rather than -1 (infinity), for fewer accidental system lockups.

This patch also applied with small coding style fix.

include/linux/fault-inject.h line 23

Quoted 3 lines

#define DEFINE_FAULT_ATTR(name) \

include/linux/fault-inject.h +1 -1
-	struct fault_attr name = { .times = ATOMIC_INIT(-1), }
+	struct fault_attr name = { .interval=1, .times = ATOMIC_INIT(1), }
	struct fault_attr name = { .interval = 1, .times = ATOMIC_INIT(1), }
#14 lore raw

Re: [patch 7/8] process filtering for fault-injection capabilities

In reply to Don Mullis
Quoted 4 lines · Don Mullis

Add functionality to the process_filter variable: A negative argument injects failures for only for pid==-process_filter, thereby permitting per-process failures from boot time.

Is it better to add new filter for this purpose? Because someone may want to filter by tgid instead of pid.

- positive value is for task->pid
- nevative value is for task->tgid

Quoted 2 lines

Add printk, called upon each failure injection.

This printk() will be very useful. But it is better to make configurable, and the pid is not reliable in interrupt context.

#19 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

In reply to Don Mullis
Quoted 3 lines · Don Mullis

Fix bug in !(CONFIG_STACK_UNWIND || CONFIG_STACKTRACE) case, based on code inspection only. Anyone with a non-i386, -x86_64, -s390 willing to test this?

Yes, it didn't work completely without this patch on other architectures. Thank you.

#21 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

In reply to Don Mullis
Quoted 2 lines · Don Mullis

Push fault-injection entries earlier in the list, so that they appear nested under DEBUG_KERNEL in menuconfig/xconfig.

Disabling the option Kernel debugging can hide all config options realated to fault-injection without this patch.

Do I misunderstand something?

#22 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

Quoted 9 lines · Akinobu Mita
Quoted 2 lines · Don Mullis

Push fault-injection entries earlier in the list, so that they appear nested under DEBUG_KERNEL in menuconfig/xconfig.

Disabling the option Kernel debugging can hide all config options realated to fault-injection without this patch.

Do I misunderstand something?

There's no problem with hiding per se, but rather with the indentation level. It's most natural for the user to have dependent options indented under their "parent". For an example, in "menuconfig" try setting "Compile the kernel with frame unwind information"; notice that "Stack unwind support" appears immediately underneath it, indented. The indentation reminds the user why "Stack unwind support" has appeared.

Note that several of the pre-existing, non-fault-injection options under "Kernel debugging", are also broken in this way.

#15 lore raw

Re: [patch 7/8] process filtering for fault-injection capabilities

In reply to Akinobu Mita
Quoted 11 lines · Akinobu Mita
Quoted 4 lines · Don Mullis

Add functionality to the process_filter variable: A negative argument injects failures for only for pid==-process_filter, thereby permitting per-process failures from boot time.

Is it better to add new filter for this purpose? Because someone may want to filter by tgid instead of pid.

- positive value is for task->pid
- nevative value is for task->tgid

Your idea sounds good to me.

Quoted 5 lines
Quoted 2 lines

Add printk, called upon each failure injection.

This printk() will be very useful. But it is better to make configurable, and the pid is not reliable in interrupt context.

Okay. We do need some output now, for testing.

#23 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

In reply to Don Mullis
Quoted 8 lines · Don Mullis

There's no problem with hiding per se, but rather with the indentation level. It's most natural for the user to have dependent options indented under their "parent". For an example, in "menuconfig" try setting "Compile the kernel with frame unwind information"; notice that "Stack unwind support" appears immediately underneath it, indented. The indentation reminds the user why "Stack unwind support" has appeared.

I see.

Quoted 2 lines

Note that several of the pre-existing, non-fault-injection options under "Kernel debugging", are also broken in this way.

Perhaps you can move UNWIND_INFO, STACK_UNWIND, and DEBUG_FS entries ealier in the list. It improves improve appearance for other DEBUG_KERNEL dependent config options like (DEBUG_VM, FRAME_POINTER, ...).

#24 lore raw

Re: [patch 8/8] stacktrace filtering for fault-injection capabilities

Quoted 4 lines · Akinobu Mita

Perhaps you can move UNWIND_INFO, STACK_UNWIND, and DEBUG_FS entries ealier in the list. It improves improve appearance for other DEBUG_KERNEL dependent config options like (DEBUG_VM, FRAME_POINTER, ...).

Okay, I will prepare such a janitorial patch.