Skip to contentSkip to conversation

fault-injection: safer defaults, trivial optimization, cleanup

PATCH In review Don Mullis 3 participants 9 messages Nov 28, 2006 – Nov 29, 2006

Subsystems (5)
lore
Share & tools
lore

Don Mullis

author
#0 lore raw

PATCH 1/2 fault-injection: safer defaults, trivial optimization, cleanup

Set /debug/fail*/* defaults supposed most likely to please a new user. Clamp /debug/fail*/stacktrace-depth to MAX_STACK_TRACE_DEPTH.

In should_fail(), move stack-unwinding test past cheaper tests (performance gain not quantified). Simplify logic; eliminate goto.
Use bool/true/false consistently.

Correct and disambiguate documentation.

---
 Documentation/fault-injection/failmodule.sh       |    4 -
 Documentation/fault-injection/fault-injection.txt |   39 +++++------
 include/linux/fault-inject.h                      |    3 
 lib/fault-inject.c                                |   74 +++++++++++++---------
 mm/page_alloc.c                                   |    2 
 mm/slab.c                                         |    1 
 6 files changed, 71 insertions(+), 52 deletions(-)
mm/slab.c +1 -0
Index: linux-2.6.18/mm/slab.c
===================================================================
@@ -3111,6 +3111,7 @@ static struct failslab_attr {
 
 } failslab = {
 	.attr = FAULT_ATTR_INITIALIZER,
+	.ignore_gfp_wait = 1,
 };
 
 static int __init setup_failslab(char *str)
include/linux/fault-inject.h +2 -1
Index: linux-2.6.18/include/linux/fault-inject.h
===================================================================
@@ -52,12 +52,13 @@ struct fault_attr {
 		.times = ATOMIC_INIT(1),			\
 		.require_end = ULONG_MAX,			\
 		.stacktrace_depth = 32,				\
+		.verbose = 2,					\
 	}
 
 #define DECLARE_FAULT_ATTR(name) struct fault_attr name = FAULT_ATTR_INITIALIZER
 int setup_fault_attr(struct fault_attr *attr, char *str);
 void should_fail_srandom(unsigned long entropy);
-int should_fail(struct fault_attr *attr, ssize_t size);
+bool should_fail(struct fault_attr *attr, ssize_t size);
 
 #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS
 
lib/fault-inject.c +44 -30

173 lines not on this page · raw message

mm/page_alloc.c +2 -0
Index: linux-2.6.18/mm/page_alloc.c
===================================================================
@@ -929,6 +929,8 @@ static struct fail_page_alloc_attr {
 
 } fail_page_alloc = {
 	.attr = FAULT_ATTR_INITIALIZER,
+	.ignore_gfp_wait = 1,
+	.ignore_gfp_highmem = 1,
 };
 
 static int __init setup_fail_page_alloc(char *str)
Documentation/fault-injection/failmodule.sh +2 -2
Index: linux-2.6.18/Documentation/fault-injection/failmodule.sh
===================================================================
@@ -26,6 +26,6 @@ fi
 # Disable any fault injection
 echo 0 > /debug/$1/stacktrace-depth
 
-echo `cat /sys/module/$2/sections/.text` > /debug/$1/address-start
-echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/address-end
+echo `cat /sys/module/$2/sections/.text` > /debug/$1/require-start
+echo `cat /sys/module/$2/sections/.exit.text` > /debug/$1/require-end
 echo $STACKTRACE_DEPTH > /debug/$1/stacktrace-depth
Documentation/fault-injection/fault-injection.txt +20 -19
Index: linux-2.6.18/Documentation/fault-injection/fault-injection.txt
===================================================================
@@ -29,16 +29,16 @@ o debugfs entries
 fault-inject-debugfs kernel module provides some debugfs entries for runtime
 configuration of fault-injection capabilities.
 
-- /debug/*/probability:
+- /debug/fail*/probability:
 
 	likelihood of failure injection, in percent.
 	Format: <percent>
 
 	Note that one-failure-per-handred is a very high error rate
 	for some testcases. Please set probably=100 and configure
-	/debug/*/interval for such testcases.
+	/debug/fail*/interval for such testcases.
 
-- /debug/*/interval:
+- /debug/fail*/interval:
 
 	specifies the interval between failures, for calls to
 	should_fail() that pass all the other tests.
@@ -46,18 +46,18 @@ configuration of fault-injection capabil
 	Note that if you enable this, by setting interval>1, you will
 	probably want to set probability=100.
 
-- /debug/*/times:
+- /debug/fail*/times:
 
 	specifies how many times failures may happen at most.
 	A value of -1 means "no limit".
 
-- /debug/*/space:
+- /debug/fail*/space:
 
 	specifies an initial resource "budget", decremented by "size"
 	on each call to should_fail(,size).  Failure injection is
 	suppressed until "space" reaches zero.
 
-- /debug/*/verbose
+- /debug/fail*/verbose
 
 	Format: { 0 | 1 | 2 }
 	specifies the verbosity of the messages when failure is injected.
@@ -66,17 +66,17 @@ configuration of fault-injection capabil
 	it is useful to debug the problems revealed by fault injection
 	capabilities.
 
-- /debug/*/task-filter:
+- /debug/fail*/task-filter:
 
-	Format: { 0 | 1 }
-	A value of '0' disables filtering by process (default).
+	Format: { 'Y' | 'N' }
+	A value of 'N' disables filtering by process (default).
 	Any positive value limits failures to only processes indicated by
 	/proc/<pid>/make-it-fail==1.
 
-- /debug/*/require-start:
-- /debug/*/require-end:
-- /debug/*/reject-start:
-- /debug/*/reject-end:
+- /debug/fail*/require-start:
+- /debug/fail*/require-end:
+- /debug/fail*/reject-start:
+- /debug/fail*/reject-end:
 
 	specifies the range of virtual addresses tested during
 	stacktrace walking.  Failure is injected only if some caller
@@ -85,22 +85,23 @@ configuration of fault-injection capabil
 	Default required range is [0,ULONG_MAX) (whole of virtual address space).
 	Default rejected range is [0,0).
 
-- /debug/*/stacktrace-depth:
+- /debug/fail*/stacktrace-depth:
 
 	specifies the maximum stacktrace depth walked during search
-	for a caller within [address-start,address-end).
+	for a caller within [require-start,require-end) OR
+	[reject-start,reject-end).
 
 - /debug/fail_page_alloc/ignore-gfp-highmem:
 
-	Format: { 0 | 1 }
-	default is 0, setting it to '1' won't inject failures into
+	Format: { 'Y' | 'N' }
+	default is 'N', setting it to 'Y' won't inject failures into
 	highmem/user allocations.
 
 - /debug/failslab/ignore-gfp-wait:
 - /debug/fail_page_alloc/ignore-gfp-wait:
 
-	Format: { 0 | 1 }
-	default is 0, setting it to '1' will inject failures
+	Format: { 'Y' | 'N' }
+	default is 'N', setting it to 'Y' will inject failures
 	only into non-sleep allocations (GFP_ATOMIC allocations).
 
 o Boot option
author
#1 lore raw

PATCH 2/2 fault-injection: lightweight code-coverage maximizer

Allow all non-unique call stacks, as judged by pushed sequence of EIPs, to be to be ignored as failure candidates.

Upon keying in
echo 1 >probability
echo 3 >verbose
echo -1 >times
a few dozen stacks are printk'ed, then system responsiveness recovers to normal. Similarly, starting a non-trivial program will print a few stacks before responsiveness recovers.

Intent is to make code-coverage-maximizing test lightweight, perhaps light enough to remain enabled during the course of the developer's interactive testing of new code.

Enabled by default. (/debug/fail*/stacktrace-depth > 0)

---
 Documentation/fault-injection/fault-injection.txt |    7 +
 include/linux/fault-inject.h                      |    4 
 lib/fault-inject.c                                |  123 +++++++++++++++++++---
 3 files changed, 120 insertions(+), 14 deletions(-)
include/linux/fault-inject.h +4 -0
Index: linux-2.6.18/include/linux/fault-inject.h
===================================================================
@@ -23,6 +23,8 @@ struct fault_attr {
 	unsigned long require_end;
 	unsigned long reject_start;
 	unsigned long reject_end;
+	unsigned long uniquestack_depth;
+	atomic_t uniquestack_hash_table[256];
 
 	unsigned long count;
 
@@ -42,6 +44,7 @@ struct fault_attr {
 		struct dentry *require_end_file;
 		struct dentry *reject_start_file;
 		struct dentry *reject_end_file;
+		struct dentry *uniquestack_depth_file;
 	} dentries;
 
 #endif
@@ -53,6 +56,7 @@ struct fault_attr {
 		.require_end = ULONG_MAX,			\
 		.stacktrace_depth = 32,				\
 		.verbose = 2,					\
+		.uniquestack_depth = 32,			\
 	}
 
 #define DECLARE_FAULT_ATTR(name) struct fault_attr name = FAULT_ATTR_INITIALIZER
lib/fault-inject.c +109 -14

208 lines not on this page · raw message

Documentation/fault-injection/fault-injection.txt +7 -0
Index: linux-2.6.18/Documentation/fault-injection/fault-injection.txt
===================================================================
@@ -91,6 +91,13 @@ configuration of fault-injection capabil
 	for a caller within [require-start,require-end) OR
 	[reject-start,reject-end).
 
+- /debug/fail*/uniquestack-depth:
+
+	specifies the stacktrace depth walked during calculation of a
+	hash of caller's EIPs.  Hash is compared against a record of
+	those already seen since boot; if duplicate, no failure is injected.
+	'0' disables the test.
+
 - /debug/fail_page_alloc/ignore-gfp-highmem:
 
 	Format: { 'Y' | 'N' }
maintainer
#2 lore raw

Re: [PATCH 2/2 -mm] fault-injection: lightweight code-coverage maximizer

Quoted 10 lines · Don Mullis

Allow all non-unique call stacks, as judged by pushed sequence of EIPs, to be to be ignored as failure candidates.

Upon keying in
echo 1 >probability
echo 3 >verbose
echo -1 >times
a few dozen stacks are printk'ed, then system responsiveness recovers to normal. Similarly, starting a non-trivial program will print a few stacks before responsiveness recovers.

What kind of test did you do?

Quoted 5 lines

Intent is to make code-coverage-maximizing test lightweight, perhaps light enough to remain enabled during the course of the developer's interactive testing of new code.

Enabled by default. (/debug/fail*/stacktrace-depth > 0)

This doesn't maximize code coverage. It makes fault-injector reject any failures which have same stacktrace before.

So it should not be default.

Quoted 12 lines
lib/fault-inject.c +12 -0
+static bool fail_uniquestack(struct fault_attr *attr)
+{
+	u32 oldhash;
+	u32 newhash;
+	uint offset = 0;
+
+	newhash = unique_stack_p(attr);
+
+	for ( oldhash = newhash; oldhash != 0; offset++) {
+		oldhash = atomic_xchg(
+			&attr->uniquestack_hash_table[
+				(newhash+offset)%ARRAY_SIZE(attr->uniquestack_hash_table)],

lib/fault-inject.c static inline bool fail_stacktrace(struc · line 212

Quoted 10 lines
lib/fault-inject.c +10 -0
+			oldhash);
+		if (oldhash == newhash)
+			return false;
+		if (offset >= ARRAY_SIZE(attr->uniquestack_hash_table)) {
+			printk(KERN_NOTICE
+			       "FAULT_INJECTION: table overflow -- "
+			       "fault injection disabled\n");
+			return false;
+		}
+	}

Updating array in this way is not safe (SMP or interrupt).

author
#3 lore raw

Re: [PATCH 2/2 -mm] fault-injection: lightweight code-coverage maximizer

Quoted 10 lines · Akinobu Mita
Quoted 7 lines · Don Mullis

Upon keying in
echo 1 >probability
echo 3 >verbose
echo -1 >times
a few dozen stacks are printk'ed, then system responsiveness recovers to normal. Similarly, starting a non-trivial program will print a few stacks before responsiveness recovers.

What kind of test did you do?

First, waiting a few seconds for the standard FC-6 daemons to wake up. Then, Xemacs and Firefox. Not tested on SMP.

Quoted 2 lines

This doesn't maximize code coverage. It makes fault-injector reject any failures which have same stacktrace before.

Since the volume of (repeated) dumps is greatly reduced, interval/probability can be set more aggressively without crippling interaction. This increases the number of error recovery paths covered per unit of wall clock time.

Quoted 1 line

Updating array in this way is not safe (SMP or interrupt).

You're right. Patch forthcoming.

maintainer
#4 lore raw

Re: [PATCH 2/2 -mm] fault-injection: lightweight code-coverage maximizer

Quoted 2 lines · Don Mullis

First, waiting a few seconds for the standard FC-6 daemons to wake up. Then, Xemacs and Firefox. Not tested on SMP.

Is it failslab or fail_page_alloc ?

Quoted 8 lines
Quoted 2 lines

This doesn't maximize code coverage. It makes fault-injector reject any failures which have same stacktrace before.

Since the volume of (repeated) dumps is greatly reduced, interval/probability can be set more aggressively without crippling interaction. This increases the number of error recovery paths covered per unit of wall clock time.

It seems artificial. Injecting failures into slab or page allocator causes vastly greater range of errors and it should be. I feel what you really want is new fault capability.

Fault injection is designed be extensible. It's not only for failslab, fail_page_alloc, and fail_make_request.

If we want to inject errors into try_something() and have own tuning or setting, we just need to extend fault attribute and define own judging function,

struct fail_try_something_attr {

	struct gorgeous_tuning tuning;
	struct fail_attr attr;

} = fail_try_something = {
.attr = FAULT_ATTR_INITIALIZER,
};

static int should_fail_try_something(void *data)

{
	if (tuning_did_clever_decision(&fail_try_something.tuning, data))
		return 0;

	return should_fail(&fail_try_something.attr);
}

Then insert it into try_something()

int try_something(void *data)

{
	if (should_fail_try_something(data))
		return 0;
...
	return 1;
}

Common debugfs entries for fault capabilities will be complicated soon by pushing new entries for every fault case or pattern.

author
#5 lore raw

Re: [PATCH 2/2 -mm] fault-injection: lightweight code-coverage maximizer

Quoted 5 lines · Akinobu Mita
Quoted 2 lines · Don Mullis

First, waiting a few seconds for the standard FC-6 daemons to wake up. Then, Xemacs and Firefox. Not tested on SMP.

Is it failslab or fail_page_alloc ?

Usually failslab, as it exposes unique stacks more quickly.

Quoted 11 lines
Quoted 8 lines
Quoted 2 lines

This doesn't maximize code coverage. It makes fault-injector reject any failures which have same stacktrace before.

Since the volume of (repeated) dumps is greatly reduced, interval/probability can be set more aggressively without crippling interaction. This increases the number of error recovery paths covered per unit of wall clock time.

It seems artificial. Injecting failures into slab or page allocator causes vastly greater range of errors and it should be. I feel what you really want is new fault capability.

When conducting an expensive test, one would naturally prefer not to repeat it on cases that are nearly the same, and therefore unlikely to expose a bug, at least until all the "more distinctive" cases have been hit. Which cases actually do contain a bug is of course not knowable a priori, and "distinctiveness" is subjective.

The claim of this patch is that uniqueness of call stack is a better proxy for likelihood to contain a bug than mere number of calls to should_fail() -- which can be thought of as the null proxy.

Quoted 2 lines

Fault injection is designed be extensible. It's not only for failslab, fail_page_alloc, and fail_make_request.

Sure.

Quoted 2 lines

Common debugfs entries for fault capabilities will be complicated soon by pushing new entries for every fault case or pattern.

True. "space" seems useful only for storage allocation calls. Should it be dropped from the common set of debugfs entries?

#6 lore raw
In reply to Don Mullis
Quoted 8 lines · On Mon, 27 Nov 2006 23:44:26 -0800 Don Mullis <[email protected]> wrote:

Set /debug/fail*/* defaults supposed most likely to please a new user. Clamp /debug/fail*/stacktrace-depth to MAX_STACK_TRACE_DEPTH.

In should_fail(), move stack-unwinding test past cheaper tests (performance gain not quantified). Simplify logic; eliminate goto.
Use bool/true/false consistently.

Correct and disambiguate documentation.

We'd prefer one-patch-per-concept, please. This all sounds like about six patches.

We _could_ merge this patch as-is, but it means that when this stuff finally hits mainline it would go in as a nice sequence of logical patches, followed by a random thing which is splattered all over all the preceding patches.

author
#7 lore raw
Quoted 2 lines · Andrew Morton

We'd prefer one-patch-per-concept, please. This all sounds like about six patches.

Understood.

Quoted 4 lines

We _could_ merge this patch as-is, but it means that when this stuff finally hits mainline it would go in as a nice sequence of logical patches, followed by a random thing which is splattered all over all the preceding patches.

Does this argue for a respin of the original patches, folding in content from this one, rather than splitting it into an additional six to be appended to the series?

#8 lore raw
Quoted 7 lines · On Tue, 28 Nov 2006 14:50:45 -0800 Don Mullis <[email protected]> wrote:
Quoted 2 lines · Andrew Morton

We'd prefer one-patch-per-concept, please. This all sounds like about six patches.

Understood.

Quoted 8 lines
Quoted 4 lines

We _could_ merge this patch as-is, but it means that when this stuff finally hits mainline it would go in as a nice sequence of logical patches, followed by a random thing which is splattered all over all the preceding patches.

Does this argue for a respin of the original patches, folding in content from this one, rather than splitting it into an additional six to be appended to the series?

If the fixes are one-patch-per-concept, and if the original patch series is one-patch-per-concept (it is) then I can usually insert the fixups in the right place, later fold each into its appropriate base patch and everything lands in git squeaky-clean.