fault-injection capabilities (v5)
cover fault-injection capabilities (v5)
Fault-injection capabilities patch set version 5. Please read the mail for the patch 1/7 for details
Changes from v3 to v5 (v4 was not delivered to linux-kernel list)
- do dump_stack() on injecting failures (if verbose option enabled)
- updated to the latest kernel version
- add debugfs entries automatically at boot time
(fault-inject-debugfs.ko has been removed)
- various bugfixes and cleanups
Quoted 1 line · On Thu, 12 Oct 2006 16:43:05 +0900 Akinobu Mita <[email protected]> wrote:
Fault-injection capabilities patch set version 5.
It all looks quite nice, thanks. Couple of things...
You've presumably run a kernel with these various things enabled. What happens? Does the kernel run really slowly? Does userspace collapse in a heap? Does it oops and die?
Also, one place where this infrastructure could be of benefit is in device
drivers: simulate a bad sector on the disk, a pulled cable, a timeout reading from a status register, etc. If that works well and is useful then I can see us encouraging driver developers to wire up fault-injection in the major drivers.
Hence it would be useful at some stage to go in and to actually do all this for a particular driver. As an example implementation for others to emulate and as a test for the fault-injection infrastructure itself - we may discover that new capabilities are needed as this work is done.
I wouldn't say this is an urgent thing to be doing, but it is a logical next step..
Quoted 3 lines · Andrew Morton
You've presumably run a kernel with these various things enabled. What happens? Does the kernel run really slowly? Does userspace collapse in a heap? Does it oops and die?
I don't feel much slowness with STACKTRACE & FRAME_POINTER and enabling stacktrace filter. But with enabling STACK_UNWIND I feel big latency on X. (There are two type of implementation of stacktrace filter in it [1] using STACKTRACE with FRAME_POINTER, and [2] STACK_UNWIND)
I don't know why there is quite difference between simple STACKTRACE and STACK_UNWIND. I'm about to try to use rb tree rather than linked list in unwind.
In order to prevent from breaking other userspace programs and to inject failures into only a specific code or process, process filter and stacktrace filter are available. Without using them the system would be almost unusable.
Now I'm stuck on the script in fault-injection.txt with random 700 modules. This script just tries to load/unload for all available kernel modules. It usually get several oopses or CPU soft lockup now. It seems that relatively large number of them involved around driver model (drivers/base/*). (I hope recent large number of error handle fixes especially by Jeff Garzik fix them)
Quoted 6 lines
Also, one place where this infrastructure could be of benefit is in device
drivers: simulate a bad sector on the disk, a pulled cable, a timeout reading from a status register, etc. If that works well and is useful then I can see us encouraging driver developers to wire up fault-injection in the major drivers.
Quoted 7 lines
Hence it would be useful at some stage to go in and to actually do all this for a particular driver. As an example implementation for others to emulate and as a test for the fault-injection infrastructure itself - we may discover that new capabilities are needed as this work is done.
I wouldn't say this is an urgent thing to be doing, but it is a logical next step..
Yes. I'm learning from md/faulty and scsi-debug module what they are doing and how to integrate such kind of features in general form.
Quoted 6 lines · On Sat, 14 Oct 2006 02:46:24 +0900 Akinobu Mita <[email protected]> wrote:
Quoted 3 lines · Andrew Morton
You've presumably run a kernel with these various things enabled. What happens? Does the kernel run really slowly? Does userspace collapse in a heap? Does it oops and die?
Quoted 8 lines
I don't feel much slowness with STACKTRACE &
FRAME_POINTERand enabling stacktrace filter. But with enablingSTACK_UNWINDI feel big latency on X. (There are two type of implementation of stacktrace filter in it [1] using STACKTRACE withFRAME_POINTER, and [2]STACK_UNWIND)I don't know why there is quite difference between simple STACKTRACE and
STACK_UNWIND. I'm about to try to use rb tree rather than linked list in unwind.
umm, we've hit this before, recently - iirc it was making lockdep run really slowly.
The new unwinding code is apparently really inefficient in some situations. It wasn't expected that it would be called at a high frequency, except people _do_ want to do that.
I forget the details, but I can cc people who have better memory.
Quoted 10 lines
In order to prevent from breaking other userspace programs and to inject failures into only a specific code or process, process filter and stacktrace filter are available. Without using them the system would be almost unusable.
Now I'm stuck on the script in fault-injection.txt with random 700 modules. This script just tries to load/unload for all available kernel modules. It usually get several oopses or CPU soft lockup now. It seems that relatively large number of them involved around driver model (
drivers/base/*).
I'm shocked ;)
Quoted 2 lines
(I hope recent large number of error handle fixes
especially by Jeff Garzik fix them)
We're getting there, but there's a lot more to do.
Quoted 11 lines
Quoted 11 lines
Also, one place where this infrastructure could be of benefit is in device
drivers: simulate a bad sector on the disk, a pulled cable, a timeout reading from a status register, etc. If that works well and is useful then I can see us encouraging driver developers to wire up fault-injection in the major drivers.Hence it would be useful at some stage to go in and to actually do all this for a particular driver. As an example implementation for others to emulate and as a test for the fault-injection infrastructure itself - we may discover that new capabilities are needed as this work is done.
Quoted 5 lines
Quoted 2 lines
I wouldn't say this is an urgent thing to be doing, but it is a logical next step..
Yes. I'm learning from md/faulty and scsi-debug module what they are doing and how to integrate such kind of features in general form.
Neato, thanks. Please don't slow down!
It's the GFP_ATOMIC allocations which we care about, really. Things like GFP_KERNEL actually don't fail in the current implementation (unless the caller got oom-killed), and all this check-for-allocation-failures work we're doing is mainly for completeness, and in anticipation of changes in the page allocator strategy.