Skip to contentSkip to conversation

fault-injection capabilities (v5)

PATCH In review Akinobu Mita 2 participants 4 messages Oct 12, 2006 – Oct 13, 2006

lore
Share & tools
lore
#0 lore raw

cover fault-injection capabilities (v5)

Fault-injection capabilities patch set version 5. Please read the mail for the patch 1/7 for details

Changes from v3 to v5 (v4 was not delivered to linux-kernel list)

- do dump_stack() on injecting failures (if verbose option enabled)

- updated to the latest kernel version
- add debugfs entries automatically at boot time
(fault-inject-debugfs.ko has been removed)
- various bugfixes and cleanups

#1 lore raw
Quoted 1 line · On Thu, 12 Oct 2006 16:43:05 +0900 Akinobu Mita <[email protected]> wrote:

Fault-injection capabilities patch set version 5.

It all looks quite nice, thanks. Couple of things...

You've presumably run a kernel with these various things enabled. What happens? Does the kernel run really slowly? Does userspace collapse in a heap? Does it oops and die?

Also, one place where this infrastructure could be of benefit is in device
drivers: simulate a bad sector on the disk, a pulled cable, a timeout reading from a status register, etc. If that works well and is useful then I can see us encouraging driver developers to wire up fault-injection in the major drivers.

Hence it would be useful at some stage to go in and to actually do all this for a particular driver. As an example implementation for others to emulate and as a test for the fault-injection infrastructure itself - we may discover that new capabilities are needed as this work is done.

I wouldn't say this is an urgent thing to be doing, but it is a logical next step..

#2 lore raw
Quoted 3 lines · Andrew Morton

You've presumably run a kernel with these various things enabled. What happens? Does the kernel run really slowly? Does userspace collapse in a heap? Does it oops and die?

I don't feel much slowness with STACKTRACE & FRAME_POINTER and enabling stacktrace filter. But with enabling STACK_UNWIND I feel big latency on X. (There are two type of implementation of stacktrace filter in it [1] using STACKTRACE with FRAME_POINTER, and [2] STACK_UNWIND)

I don't know why there is quite difference between simple STACKTRACE and STACK_UNWIND. I'm about to try to use rb tree rather than linked list in unwind.

In order to prevent from breaking other userspace programs and to inject failures into only a specific code or process, process filter and stacktrace filter are available. Without using them the system would be almost unusable.

Now I'm stuck on the script in fault-injection.txt with random 700 modules. This script just tries to load/unload for all available kernel modules. It usually get several oopses or CPU soft lockup now. It seems that relatively large number of them involved around driver model (drivers/base/*). (I hope recent large number of error handle fixes especially by Jeff Garzik fix them)

Quoted 6 lines

Also, one place where this infrastructure could be of benefit is in device
drivers: simulate a bad sector on the disk, a pulled cable, a timeout reading from a status register, etc. If that works well and is useful then I can see us encouraging driver developers to wire up fault-injection in the major drivers.

Quoted 7 lines

Hence it would be useful at some stage to go in and to actually do all this for a particular driver. As an example implementation for others to emulate and as a test for the fault-injection infrastructure itself - we may discover that new capabilities are needed as this work is done.

I wouldn't say this is an urgent thing to be doing, but it is a logical next step..

Yes. I'm learning from md/faulty and scsi-debug module what they are doing and how to integrate such kind of features in general form.

#3 lore raw
Quoted 6 lines · On Sat, 14 Oct 2006 02:46:24 +0900 Akinobu Mita <[email protected]> wrote:
Quoted 3 lines · Andrew Morton

You've presumably run a kernel with these various things enabled. What happens? Does the kernel run really slowly? Does userspace collapse in a heap? Does it oops and die?

Quoted 8 lines

I don't feel much slowness with STACKTRACE & FRAME_POINTER and enabling stacktrace filter. But with enabling STACK_UNWIND I feel big latency on X. (There are two type of implementation of stacktrace filter in it [1] using STACKTRACE with FRAME_POINTER, and [2] STACK_UNWIND)

I don't know why there is quite difference between simple STACKTRACE and STACK_UNWIND. I'm about to try to use rb tree rather than linked list in unwind.

umm, we've hit this before, recently - iirc it was making lockdep run really slowly.

The new unwinding code is apparently really inefficient in some situations. It wasn't expected that it would be called at a high frequency, except people _do_ want to do that.

I forget the details, but I can cc people who have better memory.

Quoted 10 lines

In order to prevent from breaking other userspace programs and to inject failures into only a specific code or process, process filter and stacktrace filter are available. Without using them the system would be almost unusable.

Now I'm stuck on the script in fault-injection.txt with random 700 modules. This script just tries to load/unload for all available kernel modules. It usually get several oopses or CPU soft lockup now. It seems that relatively large number of them involved around driver model (drivers/base/*).

I'm shocked ;)

Quoted 2 lines

(I hope recent large number of error handle fixes
especially by Jeff Garzik fix them)

We're getting there, but there's a lot more to do.

Quoted 11 lines
Quoted 11 lines

Also, one place where this infrastructure could be of benefit is in device
drivers: simulate a bad sector on the disk, a pulled cable, a timeout reading from a status register, etc. If that works well and is useful then I can see us encouraging driver developers to wire up fault-injection in the major drivers.

Hence it would be useful at some stage to go in and to actually do all this for a particular driver. As an example implementation for others to emulate and as a test for the fault-injection infrastructure itself - we may discover that new capabilities are needed as this work is done.

Quoted 5 lines
Quoted 2 lines

I wouldn't say this is an urgent thing to be doing, but it is a logical next step..

Yes. I'm learning from md/faulty and scsi-debug module what they are doing and how to integrate such kind of features in general form.

Neato, thanks. Please don't slow down!

It's the GFP_ATOMIC allocations which we care about, really. Things like GFP_KERNEL actually don't fail in the current implementation (unless the caller got oom-killed), and all this check-for-allocation-failures work we're doing is mainly for completeness, and in anticipation of changes in the page allocator strategy.