PROBLEM: fatal I/O errors on IDE (ATA) device during heavy disc usage (2.6.0-test11 kernel)
[1.] Fatal I/O errors on IDE (ATA) device during heavy disc usage.
[2.] A few days ago I ran into a reproducable problem when I do a specific, disc-intense operation. In my system I use three harddisks, all connected to the built-in U-ATA controller.
A few days ago I started attempting inserting a fair amount of records into a MySQL 3.23.58 database (standard Red Hat 7.3 updated build), but after some time it will result in I/O errors, especially on the partition the MySQL-databases are physically stored on. The system will not freeze up, but it has become unusable then - no way to recover but doing a reboot. So far (a couple of times) I've only noticed the I/O errors to be indicating I/O errors for /dev/hdg, which has the slave disc attached. This occurs on that disc even though /dev/hde is identical (hardware-wise) and should have an equal amount of I/O, because of the mirrored partitions being mirrored between these two discs.
I first noticed this problem last weekend, running a 2.6.0-test9 kernel, but the problem still occurs under a 2.6.0-test11 kernel. I had been running this system under 2.6.0-test9 for some time without problems (but under relatively low I/O).
[3.] Keywords: IDE, ATA, I/O error, raid
[4.] Kernel versions used:
Linux version 2.6.0-test9 ([email protected]) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)) #1 Thu Nov 13 21:53:51 CET 2003 Linux version 2.6.0-test11 ([email protected]) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-113)) #1 Thu Dec 11 14:07:13 CET 2003
[5.] The kernel doesn't panic nor oops, but the errors reported are (in random order here, since I don't know which comes first):
raid1: hdg2: redirecting sector 4945832 to another mirrorend_request: I/O error, /dev/hdg, sector 5877602
EXT3-fs error (device md2) in start_transaction: Journal has aborted
All messages repeat. After a reboot and the fscks, the system will rebuild it's raid arrays...
[6.] I'm unable to produce a shell script which triggers the problem. I've attached the .config file for the kernel I use. I've experienced this problem with kernels both having multi-mode for IDE/ATA-2 discs enabled and disabled.
[7.1] ver_linux output:
Linux airraid.toptracker.com 2.6.0-test11 #1 Thu Dec 11 14:07:13 CET 2003 i686 unknown
Gnu C 2.96 Gnu make 3.79.1 util-linux 2.11n mount 2.11n module-init-tools 0.9.12 e2fsprogs 1.27 reiserfsprogs 3.x.0j quota-tools 3.06. nfs-utils 0.3.3 Linux C Library 2.2.5 Dynamic linker (ldd) 2.2.5 Procps 2.0.7 Net-tools 1.60 Console-tools 0.3.3 Sh-utils 2.0.11 Modules Loaded e1000 3c59x
[7.2] /proc/cpuinfo:
processor : 0vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) Processor
stepping : 2
cpu MHz : 807.372
cache size : 256 KBfdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips : 1589.24
[7.3] /proc/modules:
e1000 74912 0 - Live 0xf0aa6000
3c59x 32648 0 - Live 0xf0a84000
[7.4] /proc/ioports:
0000-001f : dma1
0020-0021 : pic1
0040-005f : timer
0060-006f : keyboard
0080-008f : dma page reg
00a0-00a1 : pic2
00c0-00df : dma2
00f0-00ff : fpu
02f8-02ff : serial
03c0-03df : vga+
03f8-03ff : serial
0cf8-0cff : PCI conf1
8400-843f : 0000:00:11.0
8400-8407 : ide2
8408-840f : ide3
8410-843f : PDC20265
8800-8803 : 0000:00:11.0
8802-8802 : ide3
9000-9007 : 0000:00:11.0
9000-9007 : ide3
9400-9403 : 0000:00:11.0
9402-9402 : ide2
9800-9807 : 0000:00:11.0
9800-9807 : ide2
a000-a07f : 0000:00:0d.0
a000-a07f : 0000:00:0d.0
a400-a43f : 0000:00:0a.0
a400-a43f : e1000
d000-d01f : 0000:00:04.3
d000-d01f : uhci_hcd
d400-d41f : 0000:00:04.2
d400-d41f : uhci_hcd
d800-d80f : 0000:00:04.1
d800-d807 : ide0
d808-d80f : ide1
e200-e27f : 0000:00:04.4
e400-e4ff : 0000:00:04.4
e400-e47f : pnp 00:12
e800-e80f : 0000:00:04.4
e800-e807 : viapro-smbus
/proc/iomem:
00000000-0009efff : System RAM
0009f000-0009ffff : reserved
000a0000-000bffff : Video RAM area
000cc000-000ce7ff : Extension ROM
000f0000-000fffff : System ROM
00100000-2ffebfff : System RAM
00100000-0033dab1 : Kernel code
0033dab2-003dd53f : Kernel data
2ffec000-2ffeefff : ACPI Tables
2ffef000-2fffefff : reserved
2ffff000-2fffffff : ACPI Non-volatile Storage
de800000-de81ffff : 0000:00:11.0
df000000-df00007f : 0000:00:0d.0
df800000-df81ffff : 0000:00:0a.0
df800000-df81ffff : e1000
e0000000-e001ffff : 0000:00:0a.0
e0000000-e001ffff : e1000
e2000000-e2ffffff : 0000:00:0c.0
e4000000-e7ffffff : 0000:00:00.0
ffff0000-ffffffff : reserved
[7.5] lspci -vvv:
00:00.0 Host bridge: VIA Technologies, Inc. VT8363/8365 [KT133/KM133] (rev 02)
Subsystem: Asustek Computer, Inc. A7V Mainboard
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort+ >SERR- <PERR+
Latency: 8
Region 0: Memory at e4000000 (32-bit, prefetchable) [size=64M]
Capabilities: [a0] AGP version 2.0
Status: RQ=31 SBA+ 64bit- FW+ Rate=x1,x2
Command: RQ=0 SBA- AGP- 64bit- FW- Rate=<none>
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:01.0 PCI bridge: VIA Technologies, Inc. VT8363/8365 [KT133/KM133 AGP] (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000e000-0000dfff
Memory behind bridge: e0f00000-e0efffff Prefetchable memory behind bridge: e4000000-e3ffffff
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:04.0 ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super South] (rev 22)
Subsystem: Asustek Computer, Inc. A7V Mainboard
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping+ SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 0
00:04.1 IDE interface: VIA Technologies, Inc. VT82C586B PIPC Bus Master IDE (rev 10) (prog-if 8a [Master SecP PriP])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Region 4: I/O ports at d800 [size=16]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:04.2 USB Controller: VIA Technologies, Inc. USB (rev 10) (prog-if 00 [UHCI])
Subsystem: VIA Technologies, Inc. (Wrong ID) USB Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 08
Interrupt: pin D routed to IRQ 5
Region 4: I/O ports at d400 [size=32]
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:04.3 USB Controller: VIA Technologies, Inc. USB (rev 10) (prog-if 00 [UHCI])
Subsystem: VIA Technologies, Inc. (Wrong ID) USB Controller
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, cache line size 08
Interrupt: pin D routed to IRQ 5
Region 4: I/O ports at d000 [size=32]
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:04.4 Bridge: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI] (rev 30)
Subsystem: Asustek Computer, Inc. A7V Mainboard
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin ? routed to IRQ 9
Capabilities: [68] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:0a.0 Ethernet controller: Intel Corp. 82540EM Gigabit Ethernet Controller (rev 02)
Subsystem: Intel Corp. PRO/1000 MT Desktop Adapter
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (63750ns min), cache line size 08
Interrupt: pin A routed to IRQ 10
Region 0: Memory at e0000000 (32-bit, non-prefetchable) [size=128K] Region 1: Memory at df800000 (32-bit, non-prefetchable) [size=128K] Region 2: I/O ports at a400 [size=64]
Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mAPME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [e4] PCI-X non-bridge device.
Command: DPERE- ERO+ RBC=0 OST=0
Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-,
DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-
Capabilities: [f0] Message Signalled Interrupts: 64bit+
Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
00:0c.0 VGA compatible controller: Cirrus Logic GD 5430/40 [Alpine] (rev 47) (prog-if 00 [VGA])
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Region 0: Memory at e2000000 (32-bit, prefetchable) [size=16M]
Expansion ROM at e1000000 [disabled] [size=16M]00:0d.0 Ethernet controller: 3Com Corporation 3c905B 100BaseTX [Cyclone] (rev 30)
Subsystem: 3Com Corporation 3C905B Fast Etherlink XL 10/100
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (2500ns min, 2500ns max), cache line size 08
Interrupt: pin A routed to IRQ 5
Region 0: I/O ports at a000 [size=128]
Region 1: Memory at df000000 (32-bit, non-prefetchable) [size=128] Expansion ROM at <unassigned> [disabled] [size=128K]
Capabilities: [dc] Power Management version 1
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mAPME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:11.0 Unknown mass storage controller: Promise Technology, Inc. 20265 (rev 02)
Subsystem: Promise Technology, Inc. Ultra100
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium
Quoted 1 line
TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Interrupt: pin A routed to IRQ 11
Region 0: I/O ports at 9800 [size=8]
Region 1: I/O ports at 9400 [size=4]
Region 2: I/O ports at 9000 [size=8]
Region 3: I/O ports at 8800 [size=4]
Region 4: I/O ports at 8400 [size=64]
Region 5: Memory at de800000 (32-bit, non-prefetchable) [size=128K]
Expansion ROM at <unassigned> [disabled] [size=64K]
Capabilities: [58] Power Management version 1
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
[7.6] No SCSI devices, nor SCSI support compiled into the kernel.
[7.7] /etc/sysconfig/harddisks relevant info:
USE_DMA=1
MULTIPLE_IO=16
EIDE_32BIT=1
LOOKAHEAD=1
EXTRA_PARAMS='-S180 -W1 -u1'
I've played around with disabling sleep on the two master discs, disabling DMA on all discs, changing 32bit support to 32bit with sync, all to no effect.
mount:/dev/hde1 on / type ext3 (rw)
none on /proc type proc (rw)/dev/md1 on /usr type ext3 (rw,noatime)/dev/md2 on /var type ext3 (rw,noatime)/dev/md0 on /misc type reiserfs (rw,noatime)/dev/hdh6 on /misc2 type reiserfs (rw,noatime)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/etc/raidtab:
raiddev /dev/md0 raid-level 1 nr-raid-disks 2 persistent-superblock 1 chunk-size 8
device /dev/hde6 raid-disk 0 device /dev/hdg6 raid-disk 1
raiddev /dev/md1 raid-level 1 nr-raid-disks 2 persistent-superblock 1 chunk-size 8
device /dev/hde2 raid-disk 0 device /dev/hdg2 raid-disk 1
raiddev /dev/md2 raid-level 1 nr-raid-disks 2 persistent-superblock 1 chunk-size 8
device /dev/hde5 raid-disk 0 device /dev/hdg5 raid-disk 1
raiddev /dev/md3 raid-level 1 nr-raid-disks 2 persistent-superblock 1 chunk-size 8
device /dev/hdg1 raid-disk 1 device /dev/hde1 raid-disk 0 failed-disk 0
raiddev /dev/md5 raid-level 0 nr-raid-disks 2 persistent-superblock 0 chunk-size 8
device /dev/hde3 raid-disk 0 device /dev/hdg3 raid-disk 1
/dev/hda to /dev/hdd are unused (no devices attached), but the controller is detected during bootup. During boot I always see errors for hda and hdb (dmesg output snippet):
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xxVP_IDE: IDE controller at PCI slot 0000:00:04.1VP_IDE: chipset revision 16VP_IDE: not 100% native mode: will probe irqs later
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xxVP_IDE: VIA vt82c686a (rev 22) IDE UDMA66 controller on pci0000:00:04.1
ide0: BM-DMA at 0xd800-0xd807, BIOS settings: hda:pio, hdb:pio
ide1: BM-DMA at 0xd808-0xd80f, BIOS settings: hdc:pio, hdd:piohda: IRQ probe failed (0xfffffd3a)
hda: IRQ probe failed (0xfffffdba)
hda: no response (status = 0x0a), resetting drive
hda: IRQ probe failed (0xfffffdba)
hda: no response (status = 0x0a)
hdb: IRQ probe failed (0xfffffdba)
hdb: IRQ probe failed (0xfffffdba)
hdb: no response (status = 0x0a), resetting drive
hdb: IRQ probe failed (0xfffffdba)
hdb: no response (status = 0x0a)
PDC20265: IDE controller at PCI slot 0000:00:11.0
PDC20265: chipset revision 2
PDC20265: 100% native mode on irq 11
PDC20265: (U)DMA Burst Bit ENABLED Primary PCI Mode Secondary PCI Mode.
ide2: BM-DMA at 0x8400-0x8407, BIOS settings: hde:DMA, hdf:pio
ide3: BM-DMA at 0x8408-0x840f, BIOS settings: hdg:DMA, hdh:DMAhde: IC35L120AVV207-1, ATA DISK drive
ide2 at 0x9800-0x9807,0x9402 on irq 11
hdg: IC35L120AVV207-1, ATA DISK drive
hdh: IC35L120AVVA07-0, ATA DISK drive
ide3 at 0x9000-0x9007,0x8802 on irq 11
pnp: the driver 'ide' has been registered
hda: IRQ probe failed (0xfffff5ba)
hda: no response (status = 0x0a), resetting drive
hda: no response (status = 0xa1)
hdb: no response (status = 0xa1), resetting drive
hdb: no response (status = 0xa1)
hde: max request size: 1024KiB
hde: 241254720 sectors (123522 MB) w/7965KiB Cache, CHS=16383/255/63, UDMA(100)
hde: hde1 hde2 hde3 hde4 < hde5 hde6 >
hdg: max request size: 1024KiB
hdg: 241254720 sectors (123522 MB) w/7965KiB Cache, CHS=16383/255/63, UDMA(100)
hdg: hdg1 hdg2 hdg3 hdg4 < hdg5 hdg6 >
hdh: max request size: 128KiB
hdh: 241254720 sectors (123522 MB) w/1863KiB Cache, CHS=65535/16/63, UDMA(100)
hdh: hdh1 hdh2 hdh3 hdh4 < hdh5 hdh6 >
I have no idea if this is a kernel bug, a hardware problem or some over-enthousiasticly tuned harddisk option, but I wouldn't know which option would trigger this problem. I have not been overclocked the hardware (even though the 800MHz cpu seems to be set to 807MHz).
Regards,
Arjan
PS. I am not subscribed to the linux-kernel mailinglist, so if you wish to respond, please send me a CC.
CONFIG_X86=yCONFIG_MMU=yCONFIG_UID16=yCONFIG_GENERIC_ISA_DMA=yCONFIG_EXPERIMENTAL=yCONFIG_CLEAN_COMPILE=yCONFIG_STANDALONE=yCONFIG_BROKEN_ON_SMP=yCONFIG_SWAP=yCONFIG_SYSVIPC=yCONFIG_SYSCTL=yCONFIG_KALLSYMS=yCONFIG_FUTEX=yCONFIG_EPOLL=yCONFIG_IOSCHED_NOOP=yCONFIG_IOSCHED_AS=yCONFIG_IOSCHED_DEADLINE=yCONFIG_MODULES=yCONFIG_MODULE_UNLOAD=yCONFIG_OBSOLETE_MODPARM=yCONFIG_KMOD=yCONFIG_X86_PC=yCONFIG_MK7=yCONFIG_X86_CMPXCHG=yCONFIG_X86_XADD=yCONFIG_RWSEM_XCHGADD_ALGORITHM=yCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INVLPG=yCONFIG_X86_BSWAP=yCONFIG_X86_POPAD_OK=yCONFIG_X86_GOOD_APIC=yCONFIG_X86_INTEL_USERCOPY=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_X86_USE_3DNOW=yCONFIG_X86_UP_APIC=yCONFIG_X86_UP_IOAPIC=yCONFIG_X86_LOCAL_APIC=yCONFIG_X86_IO_APIC=yCONFIG_X86_TSC=yCONFIG_X86_MCE=yCONFIG_X86_MCE_NONFATAL=yCONFIG_HIGHMEM4G=yCONFIG_HIGHMEM=yCONFIG_MTRR=yCONFIG_PM=yCONFIG_ACPI=yCONFIG_ACPI_BOOT=yCONFIG_ACPI_INTERPRETER=yCONFIG_ACPI_FAN=yCONFIG_ACPI_PROCESSOR=yCONFIG_ACPI_THERMAL=yCONFIG_ACPI_BUS=yCONFIG_ACPI_EC=yCONFIG_ACPI_POWER=yCONFIG_ACPI_PCI=yCONFIG_ACPI_SYSTEM=yCONFIG_CPU_FREQ=yCONFIG_CPU_FREQ_PROC_INTF=yCONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE=yCONFIG_CPU_FREQ_GOV_PERFORMANCE=yCONFIG_CPU_FREQ_GOV_USERSPACE=yCONFIG_CPU_FREQ_24_API=yCONFIG_CPU_FREQ_TABLE=yCONFIG_PCI=yCONFIG_PCI_GOANY=yCONFIG_PCI_BIOS=yCONFIG_PCI_DIRECT=yCONFIG_PCI_LEGACY_PROC=yCONFIG_PCI_NAMES=yCONFIG_BINFMT_ELF=yCONFIG_BINFMT_AOUT=mCONFIG_BINFMT_MISC=mCONFIG_PNP=yCONFIG_PNP_DEBUG=yCONFIG_PNPBIOS=yCONFIG_BLK_DEV_FD=yCONFIG_BLK_DEV_LOOP=mCONFIG_IDE=yCONFIG_BLK_DEV_IDE=yCONFIG_BLK_DEV_IDEDISK=yCONFIG_IDEDISK_MULTI_MODE=yCONFIG_BLK_DEV_IDECD=yCONFIG_IDE_TASKFILE_IO=yCONFIG_BLK_DEV_IDEPNP=yCONFIG_BLK_DEV_IDEPCI=yCONFIG_IDEPCI_SHARE_IRQ=yCONFIG_BLK_DEV_GENERIC=yCONFIG_BLK_DEV_IDEDMA_PCI=yCONFIG_IDEDMA_PCI_AUTO=yCONFIG_BLK_DEV_ADMA=yCONFIG_BLK_DEV_PIIX=yCONFIG_BLK_DEV_PDC202XX_OLD=yCONFIG_BLK_DEV_PDC202XX_NEW=yCONFIG_PDC202XX_FORCE=yCONFIG_BLK_DEV_VIA82CXXX=yCONFIG_BLK_DEV_IDEDMA=yCONFIG_IDEDMA_AUTO=yCONFIG_MD=yCONFIG_BLK_DEV_MD=yCONFIG_MD_LINEAR=mCONFIG_MD_RAID0=yCONFIG_MD_RAID1=yCONFIG_MD_RAID5=yCONFIG_NET=yCONFIG_PACKET=yCONFIG_PACKET_MMAP=yCONFIG_UNIX=yCONFIG_INET=yCONFIG_IP_MULTICAST=yCONFIG_NET_IPIP=mCONFIG_NET_IPGRE=mCONFIG_INET_IPCOMP=yCONFIG_NETFILTER=yCONFIG_IP_NF_IPTABLES=yCONFIG_IP_NF_MATCH_LIMIT=yCONFIG_IP_NF_MATCH_MAC=yCONFIG_IP_NF_MATCH_PKTTYPE=yCONFIG_IP_NF_MATCH_MARK=yCONFIG_IP_NF_MATCH_MULTIPORT=yCONFIG_IP_NF_MATCH_TOS=yCONFIG_IP_NF_MATCH_ECN=yCONFIG_IP_NF_MATCH_DSCP=yCONFIG_IP_NF_MATCH_AH_ESP=yCONFIG_IP_NF_MATCH_LENGTH=yCONFIG_IP_NF_MATCH_TTL=yCONFIG_IP_NF_MATCH_TCPMSS=yCONFIG_IP_NF_FILTER=yCONFIG_IP_NF_TARGET_REJECT=yCONFIG_IP_NF_MANGLE=yCONFIG_IP_NF_TARGET_LOG=yCONFIG_IP_NF_TARGET_ULOG=yCONFIG_IP_NF_TARGET_TCPMSS=yCONFIG_IP_NF_ARPTABLES=yCONFIG_IP_NF_ARPFILTER=yCONFIG_XFRM=yCONFIG_IPV6_SCTP__=yCONFIG_NETDEVICES=yCONFIG_DUMMY=mCONFIG_NET_ETHERNET=yCONFIG_NET_VENDOR_3COM=yCONFIG_VORTEX=mCONFIG_NET_PCI=yCONFIG_E1000=mCONFIG_INPUT=yCONFIG_INPUT_MOUSEDEV=yCONFIG_INPUT_MOUSEDEV_PSAUX=yCONFIG_SOUND_GAMEPORT=yCONFIG_SERIO=yCONFIG_SERIO_I8042=yCONFIG_INPUT_KEYBOARD=yCONFIG_KEYBOARD_ATKBD=yCONFIG_INPUT_MISC=yCONFIG_INPUT_PCSPKR=yCONFIG_VT=yCONFIG_VT_CONSOLE=yCONFIG_HW_CONSOLE=yCONFIG_SERIAL_8250=yCONFIG_SERIAL_CORE=yCONFIG_UNIX98_PTYS=yCONFIG_I2C=yCONFIG_I2C_CHARDEV=yCONFIG_I2C_VIAPRO=yCONFIG_I2C_SENSOR=yCONFIG_SENSORS_VIA686A=yCONFIG_HW_RANDOM=yCONFIG_VGA_CONSOLE=yCONFIG_DUMMY_CONSOLE=yCONFIG_USB=yCONFIG_USB_DEVICEFS=yCONFIG_USB_UHCI_HCD=yCONFIG_USB_PRINTER=yCONFIG_EXT2_FS=yCONFIG_EXT3_FS=yCONFIG_EXT3_FS_XATTR=yCONFIG_JBD=yCONFIG_FS_MBCACHE=yCONFIG_REISERFS_FS=yCONFIG_REISERFS_PROC_INFO=yCONFIG_XFS_FS=mCONFIG_MINIX_FS=mCONFIG_AUTOFS_FS=mCONFIG_AUTOFS4_FS=mCONFIG_ISO9660_FS=yCONFIG_JOLIET=yCONFIG_ZISOFS=yCONFIG_ZISOFS_FS=yCONFIG_UDF_FS=mCONFIG_FAT_FS=mCONFIG_MSDOS_FS=mCONFIG_VFAT_FS=mCONFIG_NTFS_FS=mCONFIG_PROC_FS=yCONFIG_PROC_KCORE=yCONFIG_DEVPTS_FS=yCONFIG_TMPFS=yCONFIG_RAMFS=yCONFIG_CRAMFS=mCONFIG_UFS_FS=mCONFIG_NFS_FS=mCONFIG_NFS_V3=yCONFIG_NFS_V4=yCONFIG_NFSD=mCONFIG_NFSD_V3=yCONFIG_NFSD_TCP=yCONFIG_LOCKD=mCONFIG_LOCKD_V4=yCONFIG_EXPORTFS=mCONFIG_SUNRPC=mCONFIG_SUNRPC_GSS=mCONFIG_SMB_FS=mCONFIG_CIFS=mCONFIG_MSDOS_PARTITION=yCONFIG_SMB_NLS=yCONFIG_NLS=yCONFIG_NLS_CODEPAGE_437=yCONFIG_NLS_ISO8859_1=yCONFIG_DEBUG_KERNEL=yCONFIG_DEBUG_SLAB=yCONFIG_DEBUG_SPINLOCK=yCONFIG_X86_EXTRA_IRQS=yCONFIG_X86_FIND_SMP_CONFIG=yCONFIG_X86_MPPARSE=yCONFIG_CRYPTO=yCONFIG_CRYPTO_DEFLATE=yCONFIG_ZLIB_INFLATE=yCONFIG_ZLIB_DEFLATE=yCONFIG_X86_BIOS_REBOOT=yCONFIG_PC=y