2.4.23pre6aa3 scsi oops
Hi,
Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was completely wedged so I ended up transcribing the oops from the vga console. No typos I think.
As an aside: we're running 2.4.19pre10aa4 on a few high-load fileservers with 2.5G RAM and have been running into the same problem mentioned a month or so ago (Subject: "Strange dcache memory pressure when highmem enabled"). I had been suspicious of the dcache for quote a while so that thread rang true. I guessed that a later 2.4aa kernel would be a good approach. Anyone had success with a more recent one? 2.4.19pre10aa4 has been rock-solid, and if it weren't for the dcache performance hit (and the ever increasing server load) I'd rather stick with it.
-Jamie Clark
ksymoops 2.4.5 on i686 2.4.23-pre6aa3. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.23-pre6aa3/ (default) -m /boot/System.map (specified)
Unable to handle kernel paging request at virtual address 0f0f0f47 c01abda5 *pde = 00000000 Oops: 0000 2.4.23-pre6aa3 #2 SMP Mon Nov 3 12:03:35 SGT 2003 CPU: 0 EIP: 0010:[<c01abda5>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010096
eax: 00000f0f ebx: c1000020 ecx: d93fa241 edx: afafbe1e
esi: 0f0f0f0f edi: c03a7590 ebp: 00000014 esp: c02e1e28
ds: 0018 es: 00018 ss: 0018
Process swapper (pid: 0, stackpage=c02e1000)
Stack: 0f0f0f0f eb171800 c01d476b d93fa241 0f0f0f0f eb171800 f797ca18 eb171920
f7ad8000 c03a7400 eb1718c8 c01d28ab eb171800 f794b618 00000246 00000000
00000008 f797c9c0 eb1718c8 c02b2580 f794b600 00000000 c01d1ef8 f794b618
Call Trace: [<c01d476b>] (36) [<c01d28ab>] (44) [<c01d1ef8>] (24)
[<c01d21b0>] (36) [<c01d247f>] (40) [<c01cb681>] (28) [<c01ee58a>] (44)
[<c01cbab8>] (20) [<c01cb8f7>] (32) [<c011e41d>] (40) [<c0108b2b>] (04)
[<c01052c0>] (08) [<c01052c0>] (20) [<c01052c0>] (16) [<c01052c0>] (24)
[<c01052ec>] (12) [<c0105342>] (12) [<c0105000>] (04) [<c0105048>] (24)
Code: 8b 46 38 8b 4e 34 29 d8 69 c0 ab aa aa aa c1 f8 04 c1 e0 0cQuoted 9 lines
Quoted 9 lines
EIP;
c01abda5<blk_seg_merge_ok+31/64> <=====eax;
00000f0fBefore first symbol
ebx;c1000020<_end+c615ac/2cc158c>
ecx;d93fa241<END_OF_CODE+1598972e/????>
edx;afafbe1eBefore first symbol
esi;0f0f0f0fBefore first symbol
edi;c03a7590<_end+8b1c/2cc158c>
esp;c02e1e28<init_task_union+1e28/2000>
Trace; c01d476b <scsi_init_io_vc+cb/1b4>
Trace; c01d28ab <scsi_request_fn+1e7/3a8>
Trace; c01d1ef8 <scsi_queue_next_request+6c/168>
Trace; c01d21b0 <__scsi_end_request+1bc/1c8>
Trace; c01d247f <scsi_io_completion+1e7/3ec>
Trace; c01cb681 <scsi_done+d/a8>
Trace; c01ee58a <rw_intr+226/234>
Trace; c01cbab8 <scsi_finish_command+ac/b4>
Trace; c01cb8f7 <scsi_softirq+1db/26c>
Trace; c011e41d <do_softirq+dd/174>
Trace; c0108b2b <do_IRQ+db/ec>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052c0 <default_idle+0/34>
Trace; c01052ec <default_idle+2c/34>
Trace; c0105342 <cpu_idle+2e/44>
Trace; c0105000 <_stext+0/0>
Trace; c0105048 <rest_init+48/4c>
Code; c01abda5 <blk_seg_merge_ok+31/64>
00000000 <_EIP>:
Code; c01abda5 <blk_seg_merge_ok+31/64> <=====
0: 8b 46 38 mov 0x38(%esi),%eax <=====
Code; c01abda8 <blk_seg_merge_ok+34/64>
3: 8b 4e 34 mov 0x34(%esi),%ecx
Code; c01abdab <blk_seg_merge_ok+37/64>
6: 29 d8 sub %ebx,%eax
Code; c01abdad <blk_seg_merge_ok+39/64>
8: 69 c0 ab aa aa aa imul $0xaaaaaaab,%eax,%eax
Code; c01abdb3 <blk_seg_merge_ok+3f/64>
e: c1 f8 04 sar $0x4,%eax
Code; c01abdb6 <blk_seg_merge_ok+42/64>
11: c1 e0 0c shl $0xc,%eax
<0>Kernel panic: Aieee, killing interrupt handler!
Linux version 2.4.23-pre6aa3 (root@vmclu2) (gcc version 2.95.4 20011002 (Debian prerelease)) #2 SMP Mon Nov 3 12:03:35 SGT 2003 BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003fff0000 (usable)
BIOS-e820: 000000003fff0000 - 000000003ffff000 (ACPI data)
BIOS-e820: 000000003ffff000 - 0000000040000000 (ACPI NVS)
BIOS-e820: 00000000fec00000 - 00000000fec02000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
127MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000ff780
hm, page 000ff000 reserved twice.
hm, page 00100000 reserved twice.
hm, page 000f0000 reserved twice.
hm, page 000f1000 reserved twice.
On node 0 totalpages: 262128
zone(0): 4096 pages.
zone(1): 225280 pages.
zone(2): 32752 pages.
ACPI disabled because your bios is from 2000 and too old
You can enable it with acpi=force
ACPI: RSDP (v000 AMI ) @ 0x000ff8f0
ACPI: RSDT (v001 RCC RCCNILE 0x00000001 MSFT 0x01000000) @ 0x3fff0000
ACPI: FADT (v001 RCC RCCNILE 0x00000001 MSFT 0x01000000) @ 0x3fff0030
ACPI: MADT (v001 RCC RCCNILE 0x00000001 MSFT 0x01000000) @ 0x3fff00b0
ACPI: DSDT (v001 RCC CNB20HE 0x00000100 MSFT 0x0100000b) @ 0x00000000
ACPI: Local APIC address 0xfee00000
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 Pentium(tm) Pro APIC version 17
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1 Pentium(tm) Pro APIC version 17
ACPI: LAPIC (acpi_id[0x02] lapic_id[0x00] disabled)
ACPI: LAPIC (acpi_id[0x03] lapic_id[0x00] disabled)
Using ACPI for processor (LAPIC) configuration information
Intel MultiProcessor Specification v1.4
Virtual Wire compatibility mode.
OEM ID: AMI Product ID: CNB20HE APIC at: 0xFEE00000
I/O APIC #4 Version 17 at 0xFEC00000.
I/O APIC #5 Version 17 at 0xFEC01000.
Enabling APIC mode: Flat. Using 2 I/O APICs
Processors: 2
Building zonelist for node : 0
Kernel command line: ro root=/dev/sda6
Initializing CPU#0
Detected 999.581 MHz processor.
Console: colour VGA+ 80x25
Calibrating delay loop... 1992.29 BogoMIPS
Memory: 1024736k/1048512k available (1323k kernel code, 15068k reserved, 596k data, 96k init, 131008k highmem) Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Inode cache hash table entries: 65536 (order: 7, 524288 bytes) Mount cache hash table entries: 512 (order: 0, 4096 bytes) Buffer cache hash table entries: 65536 (order: 6, 262144 bytes) Page-cache hash table entries: 262144 (order: 8, 1048576 bytes) CPU: L1 I cache: 16K, L1 D cache: 16K CPU: L2 cache: 256K CPU: After generic, caps: 0383fbff 00000000 00000000 00000000 CPU: Common caps: 0383fbff 00000000 00000000 00000000
Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Checking 'hlt' instruction... OK.
POSIX conformance testing by UNIFIX
mtrr: v1.40 (20010327) Richard Gooch ([email protected]) mtrr: detected mtrr type: Intel CPU: L1 I cache: 16K, L1 D cache: 16K CPU: L2 cache: 256K CPU: After generic, caps: 0383fbff 00000000 00000000 00000000 CPU: Common caps: 0383fbff 00000000 00000000 00000000 CPU0: Intel Pentium III (Coppermine) stepping 06 per-CPU timeslice cutoff: 731.73 usecs.
task migration cache decay timeout: 10 msecs.
enabled ExtINT on CPU#0
ESR value before enabling vector: 00000004
ESR value after enabling vector: 00000000
Booting processor 1/1 eip 2000
Initializing CPU#1
masked ExtINT on CPU#1
ESR value before enabling vector: 00000000
ESR value after enabling vector: 00000000
Calibrating delay loop... 1998.84 BogoMIPS
CPU: L1 I cache: 16K, L1 D cache: 16K CPU: L2 cache: 256K CPU: After generic, caps: 0383fbff 00000000 00000000 00000000 CPU: Common caps: 0383fbff 00000000 00000000 00000000 CPU1: Intel Pentium III (Coppermine) stepping 06
Total of 2 processors activated (3991.14 BogoMIPS).
ENABLING IO-APIC IRQs
Setting 4 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 4 ... ok.
Setting 5 in the phys_id_present_map
...changing IO-APIC physical APIC ID to 5 ... ok.
init IO_APIC IRQs
IO-APIC (apicid-pin) 4-0, 4-5, 4-7, 4-9, 4-11, 5-0, 5-1, 5-2, 5-3, 5-4, 5-5, 5-7, 5-8, 5-9, 5-13 not connected.
..TIMER: vector=0x31 pin1=2 pin2=0
..MP-BIOS bug: 8254 timer not connected to IO-APIC
...trying to set up timer (IRQ0) through the 8259A ...
..... (found pin 0) ...works.
number of MP IRQ sources: 18.
number of IO-APIC #4 registers: 16.
number of IO-APIC #5 registers: 16.
testing the IO APIC.......................
IO APIC #4......
.... register #00: 04000000
....... : physical APIC id: 04 ....... : Delivery Type: 0 ....... : LTS : 0 .... register #01: 000F0011 ....... : max redirection entries: 000F ....... : PRQ implemented: 0 ....... : IO APIC version: 0011
.... register #02: 00000000
....... : arbitration: 00
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
00 003 03 0 0 0 0 0 1 1 31 01 003 03 0 0 0 0 0 1 1 39 02 000 00 1 0 0 0 0 0 0 00 03 003 03 0 0 0 0 0 1 1 41 04 003 03 0 0 0 0 0 1 1 49 05 000 00 1 0 0 0 0 0 0 00 06 003 03 0 0 0 0 0 1 1 51 07 000 00 1 0 0 0 0 0 0 00 08 003 03 0 0 0 0 0 1 1 59 09 000 00 1 0 0 0 0 0 0 00 0a 003 03 1 1 0 1 0 1 1 61 0b 000 00 1 0 0 0 0 0 0 00 0c 003 03 0 0 0 0 0 1 1 69 0d 003 03 0 0 0 0 0 1 1 71 0e 003 03 0 0 0 0 0 1 1 79 0f 003 03 0 0 0 0 0 1 1 81
IO APIC #5......
.... register #00: 05000000
....... : physical APIC id: 05 ....... : Delivery Type: 0 ....... : LTS : 0 .... register #01: 000F0011 ....... : max redirection entries: 000F ....... : PRQ implemented: 0 ....... : IO APIC version: 0011
.... register #02: 01000000
....... : arbitration: 01
.... IRQ redirection table:
NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect:
00 000 00 1 0 0 0 0 0 0 00 01 000 00 1 0 0 0 0 0 0 00 02 000 00 1 0 0 0 0 0 0 00 03 000 00 1 0 0 0 0 0 0 00 04 000 00 1 0 0 0 0 0 0 00 05 000 00 1 0 0 0 0 0 0 00 06 003 03 1 1 0 1 0 1 1 89 07 000 00 1 0 0 0 0 0 0 00 08 000 00 1 0 0 0 0 0 0 00 09 000 00 1 0 0 0 0 0 0 00 0a 003 03 1 1 0 1 0 1 1 91 0b 003 03 1 1 0 1 0 1 1 99 0c 003 03 1 1 0 1 0 1 1 A1 0d 000 00 1 0 0 0 0 0 0 00 0e 003 03 1 1 0 1 0 1 1 A9 0f 003 03 1 1 0 1 0 1 1 B1
IRQ to pin mappings:
IRQ0 -> 0:0
IRQ1 -> 0:1
IRQ3 -> 0:3
IRQ4 -> 0:4
IRQ6 -> 0:6
IRQ8 -> 0:8
IRQ10 -> 0:10
IRQ12 -> 0:12
IRQ13 -> 0:13
IRQ14 -> 0:14
IRQ15 -> 0:15
IRQ22 -> 1:6
IRQ26 -> 1:10
IRQ27 -> 1:11
IRQ28 -> 1:12
IRQ30 -> 1:14
IRQ31 -> 1:15
.................................... done.
Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 999.5537 MHz.
..... host bus clock speed is 133.2732 MHz.
cpu: 0, clocks: 1332732, slice: 444244 CPU0<T0:1332720,T1:888464,D:12,S:444244,C:1332732>
cpu: 1, clocks: 1332732, slice: 444244 CPU1<T0:1332720,T1:444224,D:8,S:444244,C:1332732>
checking TSC synchronization across CPUs: passed.migration_task 0 on cpu=0migration_task 1 on cpu=1
PCI: PCI BIOS revision 2.10 entry at 0xfdbb1, last bus=2
PCI: Using configuration type 1
PCI: Probing PCI hardware
PCI: Probing PCI hardware (bus 00)
PCI: Discovered primary peer bus 02 [IRQ]
PCI: Using IRQ router ServerWorks [1166/0200] at 00:0f.0PCI->APIC IRQ transform: (B0,I4,P0) -> 28PCI->APIC IRQ transform: (B0,I5,P0) -> 26PCI->APIC IRQ transform: (B0,I5,P1) -> 27PCI->APIC IRQ transform: (B0,I6,P0) -> 31PCI->APIC IRQ transform: (B0,I15,P0) -> 10PCI->APIC IRQ transform: (B1,I0,P0) -> 30PCI->APIC IRQ transform: (B2,I1,P0) -> 22
PCI: Cannot allocate resource region 1 of device 00:00.0
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Initializing RT netlink socket
Starting kswapd
bigpage subsystem: allocated 0 bigpages (=0MB).
allocated 32 pages and 32 bhs reserved for the highmem bounces
VFS: Disk quotas vdquot_6.5.1aio_setup: num_physpages = 65532aio_setup: sizeof(struct page) = 48
Journalled Block Device driver loaded
pty: 256 Unix98 ptys configured
Serial driver version 5.05c (2001-07-08) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled ttyS00 at 0x03f8 (irq = 4) is a 16550A
ttyS01 at 0x02f8 (irq = 3) is a 16550A
Real Time Clock Driver v1.10e
Floppy drive(s): fd0 is 1.44M
FDC 0 is a National Semiconductor PC87306
Uniform Multi-Platform E-IDE driver Revision: 7.00beta4-2.4
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx SvrWks OSB4: IDE controller at PCI slot 00:0f.1
SvrWks OSB4: chipset revision 0
SvrWks OSB4: not 100% native mode: will probe irqs later
ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:piohda: MATSHITA CR-177, ATAPI CD/DVD-ROM drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
hda: attached ide-cdrom driver.
hda: ATAPI 24X CD-ROM drive, 128kB Cache, (U)DMA Uniform CD-ROM driver Revision: 3.12
SCSI subsystem driver Revision: 1.00
scsi0 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
<Adaptec aic7899 Ultra160 SCSI adapter>
aic7899: Ultra160 Wide Channel A, SCSI Id=7, 32/253 SCBs
scsi1 : Adaptec AIC7XXX EISA/VLB/PCI SCSI HBA DRIVER, Rev 6.2.36
<Adaptec aic7899 Ultra160 SCSI adapter>
aic7899: Ultra160 Wide Channel B, SCSI Id=7, 32/253 SCBs
blk: queue f7939418, I/O limit 4095Mb (mask 0xffffffff)
(scsi0:A:0): 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
Vendor: SEAGATE Model: ST318404LC Rev: 0006
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f7939218, I/O limit 4095Mb (mask 0xffffffff)
scsi0:A:0:0: Tagged Queuing enabled. Depth 253
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
SCSI device sda: 35843670 512-byte hdwr sectors (18352 MB)
Partition check:
sda: sda1 sda2 < sda5 sda6 sda7 sda8 >
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP, IGMP
IP: routing cache hash table of 8192 buckets, 64Kbytes
TCP: Hash tables configured (established 262144 bind 65536)
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
EXT3-fs: INFO: recovery required on readonly filesystem.
EXT3-fs: write access will be enabled during recovery.
kjournald starting. Commit interval 5 seconds
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 96k freed
Adding Swap: 1052216k swap-space (priority -1)
Adding Swap: 1052216k swap-space (priority -2)
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,6), internal journal
eepro100.c:v1.09j-t 9/29/99 Donald Becker http://www.scyld.com/network/eepro100.htmleepro100.c: $Revision: 1.36 $ 2000/11/17 Modified by Andrey V. Savochkin <[email protected]> and others
eth0: OEM i82557/i82558 10/100 Ethernet, 00:30:48:11:4A:32, IRQ 31.
Board assembly 000000-000, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).
eth1: OEM i82557/i82558 10/100 Ethernet, 00:30:48:11:4A:33, IRQ 28.
Board assembly 000000-000, Physical connectors present: RJ45
Primary interface chip i82555 PHY #1.
General self-test: passed.
Serial sub-system self-test: passed.
Internal registers self-test: passed.
ROM checksum self-test: passed (0x04f4518b).qla2x00_set_info starts at address = c3780060
qla2x00: Found VID=1077 DID=2300 SSVID=1077 SSDID=9
scsi(2): Found a QLA2300 @ bus 2, device 0x1, irq 22, iobase 0xe800
scsi(2): Allocated 4096 SRB(s).
scsi(2): Configure NVRAM parameters...
scsi(2): 32 Bit PCI Addressing Enabled.
scsi(2): Verifying loaded RISC code...
scsi(2): Verifying chip...
scsi(2): Waiting for LIP to complete...
scsi(2): LIP reset occurred.
scsi(2): LIP occurred.
scsi(2): LOOP UP detected.
scsi(2): Topology - (Loop), Host Loop address 0x1
scsi2 : QLogic QLA2300 PCI to Fibre Channel Host Adapter: bus 2 device 1 irq 22
Firmware version: 3.01.18, Driver version 6.05.60
blk: queue f794b818, I/O limit 4095Mb (mask 0xffffffff)
Vendor: CNSi Model: JFS224 Rev: L425
Type: Processor ANSI SCSI revision: 03
blk: queue f794b618, I/O limit 4095Mb (mask 0xffffffff)
Vendor: CNSi Model: JFS224 Rev: L425
Type: Direct-Access ANSI SCSI revision: 03
blk: queue f794b418, I/O limit 4095Mb (mask 0xffffffff) scsi(2:0:0:0): Enabled tagged queuing, queue depth 16. scsi(2:0:0:1): Enabled tagged queuing, queue depth 16. Attached scsi generic sg1 at scsi2, channel 0, id 0, lun 0, type 3 Attached scsi disk sdb at scsi2, channel 0, id 0, lun 1 SCSI device sdb: 70970368 512-byte hdwr sectors (36337 MB)
sdb: sdb1 sdb2
LVM version 1.0.7(28/03/2003) module loadedusb.c: registered new driver usbdevfsusb.c: registered new driver hubusb-ohci.c: USB OHCI at membase 0xf890e000, IRQ 10 usb-ohci.c: usb-00:0f.2, ServerWorks OSB4/CSB5 OHCI USB Controller usb.c: new USB bus registered, assigned bus number 1 hub.c: USB hub foundhub.c: 4 ports detectedusb.c: registered new driver serialusbserial.c: USB Serial Driver core v1.4usbserial.c: USB Serial support registered for PL-2303 pl2303.c: Prolific PL2303 USB to serial adaptor driver v0.10 kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,8), internal journal
EXT3-fs: mounted filesystem with ordered data mode. kjournald starting. Commit interval 5 seconds
EXT3 FS 2.4-0.9.19, 19 August 2002 on sd(8,1), internal journal
EXT3-fs: mounted filesystem with ordered data mode.ip_tables: (C) 2000-2002 Netfilter core team
Installing knfsd (copyright (C) 1996 [email protected]).
#
# Automatically generated by make menuconfig: don't edit #CONFIG_X86=y
# CONFIG_SBUS is not setCONFIG_UID16=y
#
# Code maturity level options
#CONFIG_EXPERIMENTAL=y
#
# Loadable module support
#CONFIG_MODULES=yCONFIG_MODVERSIONS=yCONFIG_KMOD=y
#
# Processor type and features
#
# CONFIG_M386 is not set
# CONFIG_M486 is not set
# CONFIG_M586 is not set
# CONFIG_M586TSC is not set
# CONFIG_M586MMX is not set
# CONFIG_M686 is not setCONFIG_MPENTIUMIII=y
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
# CONFIG_MK8 is not set
# CONFIG_MELAN is not set
# CONFIG_MCRUSOE is not set
# CONFIG_MWINCHIPC6 is not set
# CONFIG_MWINCHIP2 is not set
# CONFIG_MWINCHIP3D is not set
# CONFIG_MCYRIXIII is not set
# CONFIG_MVIAC3_2 is not setCONFIG_X86_WP_WORKS_OK=yCONFIG_X86_INVLPG=yCONFIG_X86_CMPXCHG=yCONFIG_X86_XADD=yCONFIG_X86_BSWAP=yCONFIG_X86_POPAD_OK=yCONFIG_X86_CMPXCHG8=yCONFIG_X86_L1_CACHE_SHIFT=5CONFIG_X86_HAS_TSC=yCONFIG_X86_GOOD_APIC=yCONFIG_X86_PGE=yCONFIG_X86_USE_PPRO_CHECKSUM=yCONFIG_X86_F00F_WORKS_OK=y
# CONFIG_X86_MCE is not set
# CONFIG_TOSHIBA is not set
# CONFIG_I8K is not set
# CONFIG_THINKPAD is not set
# CONFIG_MICROCODE is not set
# CONFIG_X86_MSR is not set
# CONFIG_X86_CPUID is not set
# CONFIG_NOHIGHMEM is not setCONFIG_HIGHMEM4G=y
# CONFIG_HIGHMEM64G is not setCONFIG_HIGHMEM=yCONFIG_FORCE_MAX_ZONEORDER=11CONFIG_1GB=y
# CONFIG_2GB is not set
# CONFIG_3GB is not set
# CONFIG_05GB is not setCONFIG_HIGHIO=y
# CONFIG_MATH_EMULATION is not setCONFIG_MTRR=yCONFIG_SMP=yCONFIG_NR_CPUS=32
# CONFIG_X86_NUMA is not set
# CONFIG_X86_TSC_DISABLE is not setCONFIG_X86_TSC=yCONFIG_HAVE_DEC_LOCK=y
#
# General setup
#CONFIG_NET=yCONFIG_X86_IO_APIC=yCONFIG_X86_LOCAL_APIC=yCONFIG_PCI=y
# CONFIG_PCI_GOBIOS is not set
# CONFIG_PCI_GODIRECT is not setCONFIG_PCI_GOANY=yCONFIG_PCI_BIOS=yCONFIG_PCI_DIRECT=yCONFIG_ISA=yCONFIG_PCI_NAMES=y
# CONFIG_EISA is not set
# CONFIG_MCA is not setCONFIG_HOTPLUG=y
#
# PCMCIA/CardBus support
#
# CONFIG_PCMCIA is not set
#
# PCI Hotplug Support
#
# CONFIG_HOTPLUG_PCI is not set
# CONFIG_HOTPLUG_PCI_COMPAQ is not set
# CONFIG_HOTPLUG_PCI_COMPAQ_NVRAM is not set
# CONFIG_HOTPLUG_PCI_IBM is not setCONFIG_SYSVIPC=yCONFIG_BSD_PROCESS_ACCT=yCONFIG_SYSCTL=yCONFIG_MAX_USER_RT_PRIO=100CONFIG_MAX_RT_PRIO=0CONFIG_KCORE_ELF=y
# CONFIG_KCORE_AOUT is not set
# CONFIG_BINFMT_AOUT is not setCONFIG_BINFMT_ELF=yCONFIG_BINFMT_MISC=y
# CONFIG_PM is not set
# CONFIG_APM is not set
#
# ACPI Support
#
# CONFIG_ACPI is not setCONFIG_ACPI_BOOT=y
#
# Memory Technology Devices (MTD)
#
# CONFIG_MTD is not set
#
# Parallel port support
#
# CONFIG_PARPORT is not set
#
# Plug and Play configuration
#
# CONFIG_PNP is not set
# CONFIG_ISAPNP is not set
#
# Block devices
#CONFIG_BLK_DEV_FD=y
# CONFIG_BLK_DEV_XD is not set
# CONFIG_PARIDE is not set
# CONFIG_BLK_CPQ_DA is not set
# CONFIG_BLK_CPQ_CISS_DA is not set
# CONFIG_CISS_SCSI_TAPE is not set
# CONFIG_CISS_MONITOR_THREAD is not set
# CONFIG_BLK_DEV_DAC960 is not set
# CONFIG_BLK_DEV_UMEM is not setCONFIG_BLK_DEV_LOOP=mCONFIG_BLK_DEV_NBD=mCONFIG_BLK_DEV_RAM=mCONFIG_BLK_DEV_RAM_SIZE=4096
# CONFIG_BLK_DEV_INITRD is not set
# CONFIG_BLK_STATS is not set
#
# Multi-device support (RAID and LVM)
#CONFIG_MD=yCONFIG_BLK_DEV_MD=mCONFIG_MD_LINEAR=mCONFIG_MD_RAID0=mCONFIG_MD_RAID1=mCONFIG_MD_RAID5=mCONFIG_MD_MULTIPATH=mCONFIG_BLK_DEV_LVM=m
# CONFIG_BLK_DEV_DM is not set
#
# Networking options
#CONFIG_PACKET=yCONFIG_PACKET_MMAP=yCONFIG_NETLINK_DEV=mCONFIG_NETFILTER=y
# CONFIG_NETFILTER_DEBUG is not set
# CONFIG_FILTER is not setCONFIG_UNIX=yCONFIG_INET=y
# CONFIG_TUX is not setCONFIG_IP_MULTICAST=yCONFIG_IP_ADVANCED_ROUTER=yCONFIG_IP_MULTIPLE_TABLES=yCONFIG_IP_ROUTE_FWMARK=yCONFIG_IP_ROUTE_NAT=y
# CONFIG_IP_ROUTE_MULTIPATH is not setCONFIG_IP_ROUTE_TOS=y
# CONFIG_IP_ROUTE_VERBOSE is not set
# CONFIG_IP_PNP is not set
# CONFIG_NET_IPIP is not set
# CONFIG_NET_IPGRE is not set
# CONFIG_IP_MROUTE is not set
# CONFIG_ARPD is not set
# CONFIG_INET_ECN is not set
# CONFIG_SYN_COOKIES is not set
#
# IP: Netfilter Configuration
#CONFIG_IP_NF_CONNTRACK=m
# CONFIG_IP_NF_FTP is not set
# CONFIG_IP_NF_AMANDA is not set
# CONFIG_IP_NF_TFTP is not set
# CONFIG_IP_NF_IRC is not set
# CONFIG_IP_NF_QUEUE is not setCONFIG_IP_NF_IPTABLES=mCONFIG_IP_NF_MATCH_LIMIT=mCONFIG_IP_NF_MATCH_MAC=m
# CONFIG_IP_NF_MATCH_PKTTYPE is not setCONFIG_IP_NF_MATCH_MARK=mCONFIG_IP_NF_MATCH_MULTIPORT=mCONFIG_IP_NF_MATCH_TOS=m
# CONFIG_IP_NF_MATCH_RECENT is not set
# CONFIG_IP_NF_MATCH_ECN is not set
# CONFIG_IP_NF_MATCH_DSCP is not set
# CONFIG_IP_NF_MATCH_AH_ESP is not setCONFIG_IP_NF_MATCH_LENGTH=mCONFIG_IP_NF_MATCH_TTL=mCONFIG_IP_NF_MATCH_TCPMSS=m
# CONFIG_IP_NF_MATCH_HELPER is not setCONFIG_IP_NF_MATCH_STATE=m
# CONFIG_IP_NF_MATCH_CONNTRACK is not set
# CONFIG_IP_NF_MATCH_UNCLEAN is not set
# CONFIG_IP_NF_MATCH_OWNER is not setCONFIG_IP_NF_FILTER=mCONFIG_IP_NF_TARGET_REJECT=mCONFIG_IP_NF_TARGET_MIRROR=mCONFIG_IP_NF_NAT=mCONFIG_IP_NF_NAT_NEEDED=yCONFIG_IP_NF_TARGET_MASQUERADE=mCONFIG_IP_NF_TARGET_REDIRECT=mCONFIG_IP_NF_NAT_LOCAL=y
# CONFIG_IP_NF_NAT_SNMP_BASIC is not setCONFIG_IP_NF_MANGLE=mCONFIG_IP_NF_TARGET_TOS=m
# CONFIG_IP_NF_TARGET_ECN is not set
# CONFIG_IP_NF_TARGET_DSCP is not setCONFIG_IP_NF_TARGET_MARK=mCONFIG_IP_NF_TARGET_LOG=m
# CONFIG_IP_NF_TARGET_ULOG is not setCONFIG_IP_NF_TARGET_TCPMSS=m
# CONFIG_IP_NF_ARPTABLES is not set
# CONFIG_IP_NF_COMPAT_IPCHAINS is not set
# CONFIG_IP_NF_COMPAT_IPFWADM is not set
#
# IP: Virtual Server Configuration
#
# CONFIG_IP_VS is not set
# CONFIG_IPV6 is not set
# CONFIG_KHTTPD is not set
#
# SCTP Configuration (EXPERIMENTAL)
#CONFIG_IPV6_SCTP__=y
# CONFIG_IP_SCTP is not set
# CONFIG_ATM is not set
# CONFIG_VLAN_8021Q is not set
# CONFIG_IPX is not setCONFIG_ATALK=m
#
# Appletalk devices
#
# CONFIG_DEV_APPLETALK is not set
# CONFIG_DECNET is not set
# CONFIG_BRIDGE is not set
# CONFIG_X25 is not set
# CONFIG_LAPB is not set
# CONFIG_LLC is not set
# CONFIG_NET_DIVERT is not set
# CONFIG_ECONET is not set
# CONFIG_WAN_ROUTER is not set
# CONFIG_NET_FASTROUTE is not set
# CONFIG_NET_HW_FLOWCONTROL is not set
#
# QoS and/or fair queueing
#
# CONFIG_NET_SCHED is not set
#
# Network testing
#
# CONFIG_NET_PKTGEN is not set
#
# Telephony Support
#
# CONFIG_PHONE is not set
# CONFIG_PHONE_IXJ is not set
# CONFIG_PHONE_IXJ_PCMCIA is not set
#
# ATA/IDE/MFM/RLL support
#CONFIG_IDE=y
#
# IDE, ATA and ATAPI Block devices
#CONFIG_BLK_DEV_IDE=y
# CONFIG_BLK_DEV_HD_IDE is not set
# CONFIG_BLK_DEV_HD is not setCONFIG_BLK_DEV_IDEDISK=yCONFIG_IDEDISK_MULTI_MODE=y
# CONFIG_IDEDISK_STROKE is not set
# CONFIG_BLK_DEV_IDECS is not setCONFIG_BLK_DEV_IDECD=y
# CONFIG_BLK_DEV_IDETAPE is not set
# CONFIG_BLK_DEV_IDEFLOPPY is not set
# CONFIG_BLK_DEV_IDESCSI is not set
# CONFIG_IDE_TASK_IOCTL is not set
# CONFIG_BLK_DEV_CMD640 is not set
# CONFIG_BLK_DEV_CMD640_ENHANCED is not set
# CONFIG_BLK_DEV_ISAPNP is not setCONFIG_BLK_DEV_IDEPCI=y
# CONFIG_BLK_DEV_GENERIC is not setCONFIG_IDEPCI_SHARE_IRQ=yCONFIG_BLK_DEV_IDEDMA_PCI=y
# CONFIG_BLK_DEV_OFFBOARD is not set
# CONFIG_BLK_DEV_IDEDMA_FORCED is not setCONFIG_IDEDMA_PCI_AUTO=y
# CONFIG_IDEDMA_ONLYDISK is not setCONFIG_BLK_DEV_IDEDMA=y
# CONFIG_IDEDMA_PCI_WIP is not set
# CONFIG_BLK_DEV_ADMA100 is not set
# CONFIG_BLK_DEV_AEC62XX is not set
# CONFIG_BLK_DEV_ALI15X3 is not set
# CONFIG_WDC_ALI15X3 is not set
# CONFIG_BLK_DEV_AMD74XX is not set
# CONFIG_AMD74XX_OVERRIDE is not set
# CONFIG_BLK_DEV_CMD64X is not set
# CONFIG_BLK_DEV_TRIFLEX is not set
# CONFIG_BLK_DEV_CY82C693 is not set
# CONFIG_BLK_DEV_CS5530 is not set
# CONFIG_BLK_DEV_HPT34X is not set
# CONFIG_HPT34X_AUTODMA is not set
# CONFIG_BLK_DEV_HPT366 is not setCONFIG_BLK_DEV_PIIX=y
# CONFIG_BLK_DEV_NS87415 is not set
# CONFIG_BLK_DEV_OPTI621 is not set
# CONFIG_BLK_DEV_PDC202XX_OLD is not set
# CONFIG_PDC202XX_BURST is not set
# CONFIG_BLK_DEV_PDC202XX_NEW is not set
# CONFIG_BLK_DEV_RZ1000 is not set
# CONFIG_BLK_DEV_SC1200 is not setCONFIG_BLK_DEV_SVWKS=y
# CONFIG_BLK_DEV_SIIMAGE is not set
# CONFIG_BLK_DEV_SIS5513 is not set
# CONFIG_BLK_DEV_SLC90E66 is not set
# CONFIG_BLK_DEV_TRM290 is not set
# CONFIG_BLK_DEV_VIA82CXXX is not set
# CONFIG_IDE_CHIPSETS is not setCONFIG_IDEDMA_AUTO=y
# CONFIG_IDEDMA_IVB is not set
# CONFIG_DMA_NONPCI is not setCONFIG_BLK_DEV_IDE_MODES=y
# CONFIG_BLK_DEV_ATARAID is not set
# CONFIG_BLK_DEV_ATARAID_PDC is not set
# CONFIG_BLK_DEV_ATARAID_HPT is not set
# CONFIG_BLK_DEV_ATARAID_SII is not set
#
# SCSI support
#CONFIG_SCSI=yCONFIG_BLK_DEV_SD=yCONFIG_SD_EXTRA_DEVS=40
# CONFIG_CHR_DEV_ST is not set
# CONFIG_CHR_DEV_OSST is not set
# CONFIG_BLK_DEV_SR is not setCONFIG_CHR_DEV_SG=yCONFIG_SCSI_DEBUG_QUEUES=yCONFIG_SCSI_MULTI_LUN=yCONFIG_SCSI_CONSTANTS=yCONFIG_SCSI_LOGGING=y
#
# SCSI low-level drivers
#
# CONFIG_BLK_DEV_3W_XXXX_RAID is not set
# CONFIG_SCSI_7000FASST is not set
# CONFIG_SCSI_ACARD is not set
# CONFIG_SCSI_AHA152X is not set
# CONFIG_SCSI_AHA1542 is not set
# CONFIG_SCSI_AHA1740 is not set
# CONFIG_SCSI_AACRAID is not setCONFIG_SCSI_AIC7XXX=yCONFIG_AIC7XXX_CMDS_PER_DEVICE=253CONFIG_AIC7XXX_RESET_DELAY_MS=15000
# CONFIG_AIC7XXX_PROBE_EISA_VL is not set
# CONFIG_AIC7XXX_BUILD_FIRMWARE is not set
# CONFIG_AIC7XXX_DEBUG_ENABLE is not setCONFIG_AIC7XXX_DEBUG_MASK=0
# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
# CONFIG_SCSI_AIC79XX is not set
# CONFIG_SCSI_DPT_I2O is not set
# CONFIG_SCSI_ADVANSYS is not set
# CONFIG_SCSI_IN2000 is not set
# CONFIG_SCSI_AM53C974 is not set
# CONFIG_SCSI_MEGARAID is not set
# CONFIG_SCSI_BUSLOGIC is not set
# CONFIG_SCSI_CPQFCTS is not set
# CONFIG_SCSI_DMX3191D is not set
# CONFIG_SCSI_DTC3280 is not set
# CONFIG_SCSI_EATA is not set
# CONFIG_SCSI_EATA_DMA is not set
# CONFIG_SCSI_EATA_PIO is not set
# CONFIG_SCSI_FUTURE_DOMAIN is not set
# CONFIG_SCSI_GDTH is not set
# CONFIG_SCSI_GENERIC_NCR5380 is not set
# CONFIG_SCSI_IPS is not set
# CONFIG_SCSI_INITIO is not set
# CONFIG_SCSI_INIA100 is not set
# CONFIG_SCSI_NCR53C406A is not set
# CONFIG_SCSI_NCR53C7xx is not set
# CONFIG_SCSI_SYM53C8XX_2 is not set
# CONFIG_SCSI_NCR53C8XX is not set
# CONFIG_SCSI_SYM53C8XX is not set
# CONFIG_SCSI_PAS16 is not set
# CONFIG_SCSI_PCI2000 is not set
# CONFIG_SCSI_PCI2220I is not set
# CONFIG_SCSI_PSI240I is not set
# CONFIG_SCSI_QLOGIC_FAS is not set
# CONFIG_SCSI_QLOGIC_ISP is not set
# CONFIG_SCSI_QLOGIC_FC is not set
# CONFIG_SCSI_QLOGIC_1280 is not setCONFIG_SCSI_QLOGIC_QLA2XXX=y
# CONFIG_SCSI_QLOGIC_QLA2XXX_QLA2100 is not set
# CONFIG_SCSI_QLOGIC_QLA2XXX_QLA2200 is not setCONFIG_SCSI_QLOGIC_QLA2XXX_QLA2300=m
# CONFIG_SCSI_SEAGATE is not set
# CONFIG_SCSI_SIM710 is not set
# CONFIG_SCSI_SYM53C416 is not set
# CONFIG_SCSI_DC390T is not set
# CONFIG_SCSI_T128 is not set
# CONFIG_SCSI_U14_34F is not set
# CONFIG_SCSI_ULTRASTOR is not set
# CONFIG_SCSI_NSP32 is not set
# CONFIG_SCSI_DEBUG is not set
#
# Fusion MPT device support
#
# CONFIG_FUSION is not set
# CONFIG_FUSION_BOOT is not set
# CONFIG_FUSION_ISENSE is not set
# CONFIG_FUSION_CTL is not set
# CONFIG_FUSION_LAN is not set
#
# IEEE 1394 (FireWire) support (EXPERIMENTAL)
#
# CONFIG_IEEE1394 is not set
#
# I2O device support
#
# CONFIG_I2O is not set
# CONFIG_I2O_PCI is not set
# CONFIG_I2O_BLOCK is not set
# CONFIG_I2O_LAN is not set
# CONFIG_I2O_SCSI is not set
# CONFIG_I2O_PROC is not set
#
# Network device support
#CONFIG_NETDEVICES=y
#
# ARCnet devices
#
# CONFIG_ARCNET is not setCONFIG_DUMMY=m
# CONFIG_BONDING is not set
# CONFIG_EQUALIZER is not set
# CONFIG_TUN is not set
# CONFIG_ETHERTAP is not set
#
# Ethernet (10 or 100Mbit)
#CONFIG_NET_ETHERNET=y
# CONFIG_SUNLANCE is not set
# CONFIG_HAPPYMEAL is not set
# CONFIG_SUNBMAC is not set
# CONFIG_SUNQE is not set
# CONFIG_SUNGEM is not set
# CONFIG_NET_VENDOR_3COM is not set
# CONFIG_LANCE is not set
# CONFIG_NET_VENDOR_SMC is not set
# CONFIG_NET_VENDOR_RACAL is not set
# CONFIG_AT1700 is not set
# CONFIG_DEPCA is not set
# CONFIG_HP100 is not set
# CONFIG_NET_ISA is not setCONFIG_NET_PCI=y
# CONFIG_PCNET32 is not set
# CONFIG_AMD8111_ETH is not set
# CONFIG_ADAPTEC_STARFIRE is not set
# CONFIG_AC3200 is not set
# CONFIG_APRICOT is not set
# CONFIG_B44 is not set
# CONFIG_CS89x0 is not set
# CONFIG_TULIP is not set
# CONFIG_DE4X5 is not set
# CONFIG_DGRS is not set
# CONFIG_DM9102 is not setCONFIG_EEPRO100=m
# CONFIG_EEPRO100_PIO is not set
# CONFIG_E100 is not set
# CONFIG_LNE390 is not set
# CONFIG_FEALNX is not set
# CONFIG_NATSEMI is not set
# CONFIG_NE2K_PCI is not set
# CONFIG_NE3210 is not set
# CONFIG_ES3210 is not set
# CONFIG_8139CP is not set
# CONFIG_8139TOO is not set
# CONFIG_8139TOO_PIO is not set
# CONFIG_8139TOO_TUNE_TWISTER is not set
# CONFIG_8139TOO_8129 is not set
# CONFIG_8139_OLD_RX_RESET is not set
# CONFIG_SIS900 is not set
# CONFIG_EPIC100 is not set
# CONFIG_SUNDANCE is not set
# CONFIG_SUNDANCE_MMIO is not set
# CONFIG_TLAN is not set
# CONFIG_VIA_RHINE is not set
# CONFIG_VIA_RHINE_MMIO is not set
# CONFIG_WINBOND_840 is not set
# CONFIG_NET_POCKET is not set
#
# Ethernet (1000 Mbit)
#
# CONFIG_ACENIC is not set
# CONFIG_DL2K is not set
# CONFIG_E1000 is not set
# CONFIG_MYRI_SBUS is not set
# CONFIG_NS83820 is not set
# CONFIG_HAMACHI is not set
# CONFIG_YELLOWFIN is not set
# CONFIG_R8169 is not set
# CONFIG_SK98LIN is not set
# CONFIG_TIGON3 is not set
# CONFIG_FDDI is not set
# CONFIG_NETCONSOLE is not set
# CONFIG_HIPPI is not set
# CONFIG_PLIP is not set
# CONFIG_PPP is not set
# CONFIG_SLIP is not set
#
# Wireless LAN (non-hamradio)
#
# CONFIG_NET_RADIO is not set
#
# Token Ring devices
#
# CONFIG_TR is not set
# CONFIG_NET_FC is not set
# CONFIG_RCPCI is not set
# CONFIG_SHAPER is not set
#
# Wan interfaces
#
# CONFIG_WAN is not set
#
# Amateur Radio support
#
# CONFIG_HAMRADIO is not set
#
# IrDA (infrared) support
#
# CONFIG_IRDA is not set
#
# ISDN subsystem
#
# CONFIG_ISDN is not set
#
# Old CD-ROM drivers (not SCSI, not IDE)
#
# CONFIG_CD_NO_IDESCSI is not set
#
# Input core support
#
# CONFIG_INPUT is not set
# CONFIG_INPUT_KEYBDEV is not set
# CONFIG_INPUT_MOUSEDEV is not set
# CONFIG_INPUT_JOYDEV is not set
# CONFIG_INPUT_EVDEV is not set
#
# Character devices
#CONFIG_VT=yCONFIG_VT_CONSOLE=yCONFIG_SERIAL=yCONFIG_SERIAL_CONSOLE=y
# CONFIG_SERIAL_EXTENDED is not set
# CONFIG_SERIAL_NONSTANDARD is not setCONFIG_UNIX98_PTYS=yCONFIG_UNIX98_PTY_COUNT=256
#
# I2C support
#
# CONFIG_I2C is not set
#
# Mice
#
# CONFIG_BUSMOUSE is not setCONFIG_MOUSE=yCONFIG_PSMOUSE=y
# CONFIG_82C710_MOUSE is not set
# CONFIG_PC110_PAD is not set
# CONFIG_MK712_MOUSE is not set
#
# Joysticks
#
# CONFIG_INPUT_GAMEPORT is not set
# CONFIG_QIC02_TAPE is not set
# CONFIG_IPMI_HANDLER is not set
# CONFIG_IPMI_PANIC_EVENT is not set
# CONFIG_IPMI_DEVICE_INTERFACE is not set
# CONFIG_IPMI_KCS is not set
# CONFIG_IPMI_WATCHDOG is not set
#
# Watchdog Cards
#
# CONFIG_WATCHDOG is not set
# CONFIG_SCx200_GPIO is not set
# CONFIG_AMD_RNG is not set
# CONFIG_INTEL_RNG is not set
# CONFIG_HW_RANDOM is not set
# CONFIG_AMD_PM768 is not set
# CONFIG_NVRAM is not setCONFIG_RTC=y
# CONFIG_DTLK is not set
# CONFIG_R3964 is not set
# CONFIG_APPLICOM is not set
# CONFIG_SONYPI is not set
#
# Ftape, the floppy tape device driver
#
# CONFIG_FTAPE is not set
# CONFIG_AGP is not set
# CONFIG_ECC is not set
#
# Direct Rendering Manager (XFree86 DRI support)
#
# CONFIG_DRM is not set
# CONFIG_MWAVE is not set
#
# Multimedia devices
#
# CONFIG_VIDEO_DEV is not set
#
# File systems
#CONFIG_QUOTA=y
# CONFIG_QFMT_V2 is not set
# CONFIG_AUTOFS_FS is not setCONFIG_AUTOFS4_FS=y
# CONFIG_REISERFS_FS is not set
# CONFIG_REISERFS_CHECK is not set
# CONFIG_REISERFS_PROC_INFO is not set
# CONFIG_ADFS_FS is not set
# CONFIG_ADFS_FS_RW is not set
# CONFIG_AFFS_FS is not set
# CONFIG_HFS_FS is not set
# CONFIG_HFSPLUS_FS is not set
# CONFIG_BEFS_FS is not set
# CONFIG_BEFS_DEBUG is not set
# CONFIG_BFS_FS is not setCONFIG_EXT3_FS=yCONFIG_JBD=y
# CONFIG_JBD_DEBUG is not set
# CONFIG_FAT_FS is not set
# CONFIG_MSDOS_FS is not set
# CONFIG_UMSDOS_FS is not set
# CONFIG_VFAT_FS is not set
# CONFIG_EFS_FS is not set
# CONFIG_JFFS_FS is not set
# CONFIG_JFFS2_FS is not set
# CONFIG_CRAMFS is not setCONFIG_TMPFS=yCONFIG_RAMFS=yCONFIG_ISO9660_FS=y
# CONFIG_JOLIET is not set
# CONFIG_ZISOFS is not set
# CONFIG_JFS_FS is not set
# CONFIG_JFS_DEBUG is not set
# CONFIG_JFS_STATISTICS is not set
# CONFIG_MINIX_FS is not set
# CONFIG_VXFS_FS is not set
# CONFIG_NTFS_FS is not set
# CONFIG_NTFS_RW is not set
# CONFIG_HPFS_FS is not setCONFIG_PROC_FS=y
# CONFIG_DEVFS_FS is not set
# CONFIG_DEVFS_MOUNT is not set
# CONFIG_DEVFS_DEBUG is not setCONFIG_DEVPTS_FS=y
# CONFIG_QNX4FS_FS is not set
# CONFIG_QNX4FS_RW is not set
# CONFIG_ROMFS_FS is not setCONFIG_EXT2_FS=y
# CONFIG_SYSV_FS is not set
# CONFIG_UDF_FS is not set
# CONFIG_UDF_RW is not set
# CONFIG_UFS_FS is not set
# CONFIG_UFS_FS_WRITE is not set
# CONFIG_XFS_FS is not set
# CONFIG_XFS_POSIX_ACL is not set
# CONFIG_XFS_RT is not set
# CONFIG_XFS_QUOTA is not set
# CONFIG_XFS_DMAPI is not set
# CONFIG_XFS_DEBUG is not set
# CONFIG_PAGEBUF_DEBUG is not set
#
# Network File Systems
#
# CONFIG_CODA_FS is not set
# CONFIG_INTERMEZZO_FS is not setCONFIG_NFS_FS=m
# CONFIG_NFS_V3 is not set
# CONFIG_NFS_DIRECTIO is not set
# CONFIG_ROOT_NFS is not setCONFIG_NFSD=m
# CONFIG_NFSD_V3 is not set
# CONFIG_NFSD_TCP is not setCONFIG_SUNRPC=mCONFIG_LOCKD=mCONFIG_SMB_FS=m
# CONFIG_SMB_NLS_DEFAULT is not set
# CONFIG_NCP_FS is not set
# CONFIG_NCPFS_PACKET_SIGNING is not set
# CONFIG_NCPFS_IOCTL_LOCKING is not set
# CONFIG_NCPFS_STRONG is not set
# CONFIG_NCPFS_NFS_NS is not set
# CONFIG_NCPFS_OS2_NS is not set
# CONFIG_NCPFS_SMALLDOS is not set
# CONFIG_NCPFS_NLS is not set
# CONFIG_NCPFS_EXTRAS is not set
# CONFIG_ZISOFS_FS is not set
#
# Partition Types
#CONFIG_PARTITION_ADVANCED=y
# CONFIG_ACORN_PARTITION is not set
# CONFIG_OSF_PARTITION is not set
# CONFIG_AMIGA_PARTITION is not set
# CONFIG_ATARI_PARTITION is not set
# CONFIG_MAC_PARTITION is not setCONFIG_MSDOS_PARTITION=y
# CONFIG_BSD_DISKLABEL is not set
# CONFIG_MINIX_SUBPARTITION is not set
# CONFIG_SOLARIS_X86_PARTITION is not set
# CONFIG_UNIXWARE_DISKLABEL is not set
# CONFIG_LDM_PARTITION is not set
# CONFIG_SGI_PARTITION is not set
# CONFIG_ULTRIX_PARTITION is not set
# CONFIG_SUN_PARTITION is not set
# CONFIG_EFI_PARTITION is not setCONFIG_SMB_NLS=yCONFIG_NLS=y
#
# Native Language Support
#CONFIG_NLS_DEFAULT="iso8859-1"
# CONFIG_NLS_CODEPAGE_437 is not set
# CONFIG_NLS_CODEPAGE_737 is not set
# CONFIG_NLS_CODEPAGE_775 is not set
# CONFIG_NLS_CODEPAGE_850 is not set
# CONFIG_NLS_CODEPAGE_852 is not set
# CONFIG_NLS_CODEPAGE_855 is not set
# CONFIG_NLS_CODEPAGE_857 is not set
# CONFIG_NLS_CODEPAGE_860 is not set
# CONFIG_NLS_CODEPAGE_861 is not set
# CONFIG_NLS_CODEPAGE_862 is not set
# CONFIG_NLS_CODEPAGE_863 is not set
# CONFIG_NLS_CODEPAGE_864 is not set
# CONFIG_NLS_CODEPAGE_865 is not set
# CONFIG_NLS_CODEPAGE_866 is not set
# CONFIG_NLS_CODEPAGE_869 is not set
# CONFIG_NLS_CODEPAGE_936 is not set
# CONFIG_NLS_CODEPAGE_950 is not set
# CONFIG_NLS_CODEPAGE_932 is not set
# CONFIG_NLS_CODEPAGE_949 is not set
# CONFIG_NLS_CODEPAGE_874 is not set
# CONFIG_NLS_ISO8859_8 is not set
# CONFIG_NLS_CODEPAGE_1250 is not set
# CONFIG_NLS_CODEPAGE_1251 is not set
# CONFIG_NLS_ISO8859_1 is not set
# CONFIG_NLS_ISO8859_2 is not set
# CONFIG_NLS_ISO8859_3 is not set
# CONFIG_NLS_ISO8859_4 is not set
# CONFIG_NLS_ISO8859_5 is not set
# CONFIG_NLS_ISO8859_6 is not set
# CONFIG_NLS_ISO8859_7 is not set
# CONFIG_NLS_ISO8859_9 is not set
# CONFIG_NLS_ISO8859_13 is not set
# CONFIG_NLS_ISO8859_14 is not set
# CONFIG_NLS_ISO8859_15 is not set
# CONFIG_NLS_KOI8_R is not set
# CONFIG_NLS_KOI8_U is not set
# CONFIG_NLS_UTF8 is not set
#
# Console drivers
#CONFIG_VGA_CONSOLE=yCONFIG_VIDEO_SELECT=y
# CONFIG_MDA_CONSOLE is not set
#
# Frame-buffer support
#
# CONFIG_FB is not set
#
# Sound
#
# CONFIG_SOUND is not set
#
# USB support
#CONFIG_USB=m
# CONFIG_USB_DEBUG is not setCONFIG_USB_DEVICEFS=y
# CONFIG_USB_BANDWIDTH is not set
# CONFIG_USB_EHCI_HCD is not set
# CONFIG_USB_UHCI is not set
# CONFIG_USB_UHCI_ALT is not setCONFIG_USB_OHCI=m
# CONFIG_USB_AUDIO is not set
# CONFIG_USB_EMI26 is not set
# CONFIG_USB_BLUETOOTH is not set
# CONFIG_USB_MIDI is not set
# CONFIG_USB_STORAGE is not set
# CONFIG_USB_STORAGE_DEBUG is not set
# CONFIG_USB_STORAGE_DATAFAB is not set
# CONFIG_USB_STORAGE_FREECOM is not set
# CONFIG_USB_STORAGE_ISD200 is not set
# CONFIG_USB_STORAGE_DPCM is not set
# CONFIG_USB_STORAGE_HP8200e is not set
# CONFIG_USB_STORAGE_SDDR09 is not set
# CONFIG_USB_STORAGE_SDDR55 is not set
# CONFIG_USB_STORAGE_JUMPSHOT is not set
# CONFIG_USB_ACM is not set
# CONFIG_USB_PRINTER is not set
# CONFIG_USB_HID is not set
# CONFIG_USB_HIDINPUT is not set
# CONFIG_USB_HIDDEV is not set
# CONFIG_USB_KBD is not set
# CONFIG_USB_MOUSE is not set
# CONFIG_USB_AIPTEK is not set
# CONFIG_USB_WACOM is not set
# CONFIG_USB_KBTAB is not set
# CONFIG_USB_POWERMATE is not set
# CONFIG_USB_DC2XX is not set
# CONFIG_USB_MDC800 is not set
# CONFIG_USB_SCANNER is not set
# CONFIG_USB_MICROTEK is not set
# CONFIG_USB_HPUSBSCSI is not set
# CONFIG_USB_PEGASUS is not set
# CONFIG_USB_RTL8150 is not set
# CONFIG_USB_KAWETH is not set
# CONFIG_USB_CATC is not set
# CONFIG_USB_AX8817X is not set
# CONFIG_USB_CDCETHER is not set
# CONFIG_USB_USBNET is not set
# CONFIG_USB_USS720 is not set
#
# USB Serial Converter support
#CONFIG_USB_SERIAL=m
# CONFIG_USB_SERIAL_DEBUG is not set
# CONFIG_USB_SERIAL_GENERIC is not set
# CONFIG_USB_SERIAL_BELKIN is not set
# CONFIG_USB_SERIAL_WHITEHEAT is not set
# CONFIG_USB_SERIAL_DIGI_ACCELEPORT is not set
# CONFIG_USB_SERIAL_EMPEG is not set
# CONFIG_USB_SERIAL_FTDI_SIO is not set
# CONFIG_USB_SERIAL_VISOR is not set
# CONFIG_USB_SERIAL_IPAQ is not set
# CONFIG_USB_SERIAL_IR is not set
# CONFIG_USB_SERIAL_EDGEPORT is not set
# CONFIG_USB_SERIAL_EDGEPORT_TI is not set
# CONFIG_USB_SERIAL_KEYSPAN_PDA is not set
# CONFIG_USB_SERIAL_KEYSPAN is not set
# CONFIG_USB_SERIAL_MCT_U232 is not set
# CONFIG_USB_SERIAL_KLSI is not set
# CONFIG_USB_SERIAL_KOBIL_SCT is not setCONFIG_USB_SERIAL_PL2303=m
# CONFIG_USB_SERIAL_CYBERJACK is not set
# CONFIG_USB_SERIAL_XIRCOM is not set
# CONFIG_USB_SERIAL_OMNINET is not set
# CONFIG_USB_RIO500 is not set
# CONFIG_USB_AUERSWALD is not set
# CONFIG_USB_TIGL is not set
# CONFIG_USB_BRLVGER is not set
# CONFIG_USB_LCD is not set
#
# Bluetooth support
#
# CONFIG_BLUEZ is not set
#
# Kernel hacking
#CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_HZ is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_HIGHMEM is not set
# CONFIG_DEBUG_SLAB is not set
# CONFIG_DEBUG_IOVIRT is not setCONFIG_MAGIC_SYSRQ=y
# CONFIG_KMSGDUMP is not set
# CONFIG_DEBUG_SPINLOCK is not set
# CONFIG_FRAME_POINTER is not set
# CONFIG_HIGHMEM_EMULATION is not set
# CONFIG_X86_REMOTE_DEBUG is not set
# CONFIG_KERNEL_DEBUGGING is not setCONFIG_LOG_BUF_SHIFT=0
#
# Cryptographic options
#
# CONFIG_CRYPTO is not set
#
# Library routines
#
# CONFIG_CRC32 is not set
# CONFIG_ZLIB_INFLATE is not set
# CONFIG_ZLIB_DEFLATE is not set
# CONFIG_FW_LOADER is not set
# CONFIG_QSORT is not set
Quoted 6 lines · Jamie Clark
Hi,
Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was completely wedged so I ended up transcribing the oops from the vga console. No typos I think.
I need to release an update that has a chance to fix it. Jens identified problem in his last_merge scsi patch so I will back it out.
you can try to backout it by yourself in the meantime, it's called \*elevator-merge-fast-path\* . or you can disable it with this patch:
drivers/block/elevator.c renamed from drivers/block/elevator.c.~1~ +1 -0
@@ -77,6 +77,7 @@ inline int bh_rq_in_between(struct buffestatic int rq_mergeable (struct request *req,struct buffer_head *bh,request_queue_t *q,int rw,int count,int max_sectors) { +return 0 ;if (q->head_active && !q->plugged) {struct request *next; next =blkdev_entry_next_request (&q->queue_head);
Hope this helps.
I made the quick fix (disabling rq_mergeable) and started the load test. Will let it run for a week or so.
Thanks for the help!
-Jamie
FYI an observation from my last test: the read latency seems to be much improved and more consistent under this kernel (2.4.23pre6aa3, before the oops and before this fix). The maximum latency seemed steady over the whole test without any of the longish pauses that showed up under 2.4.19. Quite a difference.
Quoted 12 lines · Andrea Arcangeli
Quoted 4 lines · Jamie Clark
Hi,
Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was completely wedged so I ended up transcribing the oops from the vga console. No typos I think.
I need to release an update that has a chance to fix it. Jens identified problem in his
last_mergescsi patch so I will back it out.you can try to backout it by yourself in the meantime, it's called \*elevator-merge-fast-path\* . or you can disable it with this patch:
Quoted 3 lines · Jamie Clark
the oops and before this fix). The maximum latency seemed steady over the whole test without any of the longish pauses that showed up under 2.4.19. Quite a difference.
this is the effect of the elevator-lowlatency also merged in mainline recently. Good to hear, thanks.
2.4.23pre6aa3 - possible ext3 deadlock
In reply to Jamie ClarkFollowup to last test with 2.4.23pre6aa3. same config as before (ext3 /ql2300 / bonnie++) I forgot to mention that I have quotas enabled on the partition under test.
There have been a few 2.4 / quota / ext3 deadlock-type mails on the list recently so I'm sot sure if this is related. I can see an atime update in the call trace of one of the bonnies but no dquot_ calls anywhere. I haven't applied Jan's patch as it came out before I could reliably reproduce this problem.
In the meantime I'll restart testing with noatime - which is how I'd usually mount such a fs anyhow.
The box seems to deadlock after 3-6 days of filesystem exercise. I have repeated this 3 times. The last run was 6 days and and then I see all disk I/O has ceased. The system remains pingable and sshd connects at the TCP layer but nothing happens after that. The root filesystem is ext3 but is a completely separate device (motherboard scsi) to the filesystems running bonnie (external FC RAID). It seems that all filesystem activity ceases.
I have run the alt-sysrq-t ouput through ksymoops and attached. Hope this is useful to someone who knows.
-jamie
Quoted 20 lines · Jamie Clark
I made the quick fix (disabling
rq_mergeable) and started the load test. Will let it run for a week or so.Thanks for the help!
-JamieQuoted 11 lines · Andrea Arcangeli
Quoted 4 lines · Jamie Clark
Hi,
Consistent oops with 2.4.23pre6aa3 after 3-4 hours running bonnie on ext3 fs through qla2300 HBA. (w SMP, HIGHMEM) The test machine was completely wedged so I ended up transcribing the oops from the vga console. No typos I think.
I need to release an update that has a chance to fix it. Jens identified problem in his
last_mergescsi patch so I will back it out.you can try to backout it by yourself in the meantime, it's called \*elevator-merge-fast-path\* . or you can disable it with this patch:
ksymoops 2.4.5 on i686 2.4.23-pre6aa3. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.23-pre6aa3/ (default) -m /boot/System.map (specified)
init R F7907F14 0 1 0 12973 (NOTLB)
Using defaults from ksymoops -t elf32-i386 -a i386
Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (40) [<c0149454>] (44) [<c0107143>] (60) migration_CPU S C1C1E000 0 2 1 3 (L-TLB) Call Trace: [<c011768f>] (16) [<c0117568>] (36) [<c01056d0>] (12) migration_CPU S F790E346 0 3 1 4 2 (L-TLB) Call Trace: [<c011768f>] (16) [<c0117568>] (36) [<c01056d0>] (12) keventd R 00000000 0 4 1 5 3 (L-TLB) Call Trace: [<c0127045>] (64) [<c01056d0>] (12) ksoftirqd_CPU S F7908000 0 5 1 6 4 (L-TLB) Call Trace: [<c011e96f>] (20) [<c01056d0>] (12) ksoftirqd_CPU S C1C1E000 0 6 1 7 5 (L-TLB) Call Trace: [<c011e96f>] (20) [<c01056d0>] (12) kswapd R C1C1C000 5332 7 1 8 6 (L-TLB) Call Trace: [<c0137ca9>] (36) [<c01056d0>] (12) bdflush S 00000286 0 8 1 9 7 (L-TLB) Call Trace: [<c0116351>] (36) [<c01459e3>] (16) [<c01056d0>] (12) kupdated R C1C19FB0 5224 9 1 10 8 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c0145aaa>] (36) [<c01056d0>] (12) ahc_dv_0 S F75C3F3C 12 10 1 11 9 (L-TLB) Call Trace: [<c0105e67>] (40) [<c0105f4b>] (12) [<c01dc79e>] (24) [<c01056d0>] (12) ahc_dv_1 S F7524D1C 6188 11 1 12 10 (L-TLB) Call Trace: [<c0105e67>] (40) [<c0105f4b>] (12) [<c01dc79e>] (24) [<c01056d0>] (12) scsi_eh_0 S C1C3BFDC 3304 12 1 13 11 (L-TLB) Call Trace: [<c0105e67>] (40) [<c0105f4b>] (12) [<c01d0d48>] (56) [<c01056d0>] (12) scsi_eh_1 S C1C39FDC 6176 13 1 14 12 (L-TLB) Call Trace: [<c0105e67>] (40) [<c0105f4b>] (12) [<c01d0d48>] (56) [<c01056d0>] (12) kjournald R 00000286 0 14 1 57 13 (L-TLB) Call Trace: [<c0116351>] (36) [<c017de49>] (36) [<c017dca0>] (12) [<c01056d0>] (12) qla2300_dpc2 S C2665FDC 2384 57 1 58 14 (L-TLB) Call Trace: [<c0105e67>] (40) [<c0105f4b>] (12) [<c2797943>] (120) [<c01056d0>] (12) scsi_eh_2 S C2663FDC 2384 58 1 61 57 (L-TLB) Call Trace: [<c0105e67>] (40) [<c0105f4b>] (12) [<c01d0d48>] (56) [<c01056d0>] (12) khubd S 00000001 2384 61 1 99 58 (L-TLB) Call Trace: [<c26048bb>] (28) [<c260ddac>] (04) [<c260ddac>] (08) [<c01056d0>] (12) kjournald R 00000286 2384 99 1 100 61 (L-TLB) Call Trace: [<c0116351>] (36) [<c017de49>] (36) [<c017dca0>] (12) [<c01056d0>] (12) kjournald S 00000286 52 100 1 137 99 (L-TLB) Call Trace: [<c0116351>] (36) [<c017de49>] (36) [<c017dca0>] (12) [<c01056d0>] (12) portmap S 7FFFFFFF 0 137 1 208 100 (NOTLB) Call Trace: [<c012283c>] (24) [<c015267a>] (16) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60) syslogd R 7FFFFFFF 0 208 1 211 137 (NOTLB) Call Trace: [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) klogd R F7AA6000 52 211 1 219 208 (NOTLB) Call Trace: [<c01198bc>] (52) [<c016af45>] (16) [<c0140be7>] (36) [<c0107143>] (60) rpc.statd S 7FFFFFFF 0 219 1 225 211 (NOTLB) Call Trace: [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) gpm S F7A4BF14 52 225 1 230 219 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) inetd S 7FFFFFFF 0 230 1 249 225 (NOTLB) Call Trace: [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) nfsd R C23EBF54 0 241 1 253 242 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (44) [<f7a9a520>] (12) [<c01056d0>] (12) nfsd R C23C9F54 0 242 1 241 243 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12) nfsd R C23A9F54 52 243 1 242 244 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12) nfsd R C238FF54 0 244 1 243 245 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12) nfsd R C236FF54 52 245 1 244 246 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12) nfsd R C234DF54 0 246 1 245 247 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12) nfsd R C232DF54 52 247 1 246 248 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (56) [<c01056d0>] (12) nfsd R C2313F54 52 248 1 247 249 (L-TLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c3228e99>] (72) [<f7a9029c>] (48) [<f7a9a520>] (08) [<c01056d0>] (12) lockd S 7FFFFFFF 2384 249 1 250 248 230 (L-TLB) Call Trace: [<c012283c>] (20) [<c3225244>] (20) [<c3228e99>] (72) [<f7ab1cb3>] (36) [<c01056d0>] (12) rpciod S 00000001 52 250 249 (L-TLB) Call Trace: [<c3225410>] (12) [<c322f74c>] (12) [<c322f74c>] (12) [<c322f744>] (04) [<c322f744>] (08) [<c01056d0>] (04) [<c322f74c>] (08) rpc.mountd S 7FFFFFFF 52 253 1 259 241 (NOTLB) Call Trace: [<c012283c>] (24) [<c015267a>] (16) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60) sshd S 7FFFFFFF 52 259 1 18848 263 253 (NOTLB) Call Trace: [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) xfs R C2287F14 0 263 1 268 259 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (44) [<c011dd7d>] (40) [<c0107143>] (60) atd R F7A31F60 52 268 1 271 263 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c01229ec>] (68) [<c0107143>] (60) cron R C2281F60 0 271 1 19062 275 268 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c01229ec>] (68) [<c0107143>] (60) apache R C227DF14 52 275 1 18141 281 271 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) getty S 7FFFFFFF 5028 281 1 282 275 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) getty S 7FFFFFFF 0 282 1 283 281 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) getty S 7FFFFFFF 0 283 1 284 282 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) getty S 7FFFFFFF 52 284 1 285 283 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) getty S 7FFFFFFF 52 285 1 286 284 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) getty S 7FFFFFFF 52 286 1 884 285 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) kjournald D F7A9D294 4796 884 1 895 286 (L-TLB) Call Trace: [<c0105da8>] (36) [<c0105f40>] (16) [<c017c0b4>] (244) [<c0115f53>] (08) [<c0115f5f>] (76) [<c017de26>] (40) [<c017dca0>] (12) [<c01056d0>] (12) screen S 7FFFFFFF 2384 895 1 896 902 884 (NOTLB) Call Trace: [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) rlbench.sh S 00000000 0 896 895 898 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) rlbench R C93DFF60 2384 897 896 898 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c01229ec>] (68) [<c0107143>] (60) tee S C936E000 4888 898 896 897 (NOTLB) Call Trace: [<c014afeb>] (60) [<c014b0c4>] (36) [<c0140be7>] (36) [<c0107143>] (60) screen S 7FFFFFFF 5524 902 1 903 907 895 (NOTLB) Call Trace: [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) bash S 00000000 2384 903 902 9385 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) screen S 7FFFFFFF 0 907 1 908 9945 902 (NOTLB) Call Trace: [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) bash S 00000000 0 908 907 9387 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) bonnie++ D F7A9D294 0 9385 903 9400 (NOTLB) Call Trace: [<c0105da8>] (36) [<c0105f40>] (16) [<c017adfa>] (24) [<c0178ffd>] (24) [<c0173d48>] (24) [<c0156cbe>] (20) [<c01584aa>] (12) [<c012e882>] (52) [<c012ec64>] (16) [<c012eac0>] (24) [<c016f007>] (36) [<c0140be7>] (36) [<c0107143>] (60) bonnie++ R C17FE560 0 9387 908 9394 (NOTLB) bonnie++ R D7933F30 0 9394 9387 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60) bonnie++ R E157FF30 0 9400 9385 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60) screen S 7FFFFFFF 0 9945 1 9946 12973 907 (NOTLB) Call Trace: [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) bash S 00000000 0 9946 9945 9947 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) bonnie++ D F7A9D294 0 9947 9946 9963 (NOTLB) Call Trace: [<c0105da8>] (36) [<c0105f40>] (16) [<c017adfa>] (24) [<c0178ffd>] (24) [<c0173d48>] (24) [<c0156cbe>] (20) [<c01584aa>] (12) [<c012e882>] (52) [<c012ec64>] (16) [<c012eac0>] (60) [<c0140be7>] (36) [<c0107143>] (60) bonnie++ R DF49FF30 0 9963 9947 (NOTLB) Call Trace: [<c01228b8>] (28) [<c012281c>] (12) [<c01526b0>] (32) [<c0152884>] (64) [<c0151bdc>] (20) [<c0107143>] (60) lpd S 7FFFFFFF 0 12973 1 9945 (NOTLB) Call Trace: [<c012283c>] (12) [<c01fe467>] (28) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (44) [<c01ff657>] (40) [<c0107143>] (60) apache S 7FFFFFFF 52 18137 275 18138 (NOTLB) Call Trace: [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60) apache S 7FFFFFFF 52 18138 275 18139 18137 (NOTLB) Call Trace: [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60) apache S 7FFFFFFF 52 18139 275 18140 18138 (NOTLB) Call Trace: [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60) apache S 7FFFFFFF 52 18140 275 18141 18139 (NOTLB) Call Trace: [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60) apache S 7FFFFFFF 52 18141 275 18140 (NOTLB) Call Trace: [<c012283c>] (40) [<c021f5ed>] (68) [<c021f77a>] (32) [<c0239a44>] (36) [<c01fec56>] (52) [<c0113d58>] (36) [<c012c729>] (52) [<c0124543>] (28) [<c01ff66c>] (44) [<c0107143>] (60) sshd R 7FFFFFFF 0 18848 259 18850 (NOTLB) Call Trace: [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) bash S 7FFFFFFF 0 18850 18848 (NOTLB) Call Trace: [<c012283c>] (36) [<c0196084>] (04) [<c01960cf>] (88) [<c0191c3a>] (32) [<c0140be7>] (36) [<c0107143>] (60) cron S E291A000 0 19062 271 19063 (NOTLB) Call Trace: [<c014afeb>] (60) [<c014b0c4>] (36) [<c0140be7>] (36) [<c0107143>] (60) sh S 00000000 0 19063 19062 19064 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) run-parts S 7FFFFFFF 0 19064 19063 19091 (NOTLB) Call Trace: [<c012283c>] (40) [<c0151f0c>] (40) [<c0151bdc>] (20) [<c015238a>] (84) [<c0107143>] (60) find S 00000000 0 19091 19064 19092 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) updatedb S 00000000 0 19092 19091 19109 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) updatedb S 00000000 0 19106 19092 19107 19108 (NOTLB) Call Trace: [<c011d6d6>] (60) [<c0107143>] (60) find R F735CAC0 2384 19107 19106 (NOTLB) Call Trace: [<c0141e75>] (40) [<c01715e2>] (20) [<c016eca6>] (84) [<c01493d4>] (52) [<c01510f4>] (12) [<c01516e0>] (24) [<c015182e>] (08) [<c01516e0>] (32) [<c015087b>] (24) [<c0107143>] (60) sort R current 0 19108 19092 19109 19106 (NOTLB) frcode S E20EA000 16 19109 19092 19108 (NOTLB) Call Trace: [<c014afeb>] (60) [<c014b0c4>] (36) [<c0140be7>] (36) [<c0107143>] (60)
Warning (Oops_read): Code line not seen, dumping what data is available
Proc; init
Quoted 1 line
Quoted 1 line
EIP;
f7907f14<END_OF_CODE+17ce15/????> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0149454 <sys_stat64+64/70>
Trace; c0107143 <system_call+33/38>
Proc; migration_CPU
Quoted 1 line
Quoted 1 line
EIP;
c1c1e000<_end+187f58c/1dad58c> <=====
Trace; c011768f <migration_thread+127/340>
Trace; c0117568 <migration_thread+0/340>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; migration_CPU
Quoted 1 line
Quoted 1 line
EIP;
f790e346<END_OF_CODE+183247/????> <=====
Trace; c011768f <migration_thread+127/340>
Trace; c0117568 <migration_thread+0/340>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; keventd
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c0127045 <context_thread+115/1cc>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; ksoftirqd_CPU
Quoted 1 line
Quoted 1 line
EIP;
f7908000<END_OF_CODE+17cf01/????> <=====
Trace; c011e96f <ksoftirqd+93/c8>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; ksoftirqd_CPU
Quoted 1 line
Quoted 1 line
EIP;
c1c1e000<_end+187f58c/1dad58c> <=====
Trace; c011e96f <ksoftirqd+93/c8>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; kswapd
Quoted 1 line
Quoted 1 line
EIP;
c1c1c000<_end+187d58c/1dad58c> <=====
Trace; c0137ca9 <kswapd+75/ae>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; bdflush
Quoted 1 line
Quoted 1 line
EIP; 00000286 Before first symbol <=====
Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c01459e3 <bdflush+cb/d0>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; kupdated
Quoted 1 line
Quoted 1 line
EIP;
c1c19fb0<_end+187b53c/1dad58c> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0145aaa <kupdate+c2/188>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; ahc_dv_0
Quoted 1 line
Quoted 1 line
EIP;
f75c3f3c<[sunrpc].bss.end+34273429/343f34ed> <=====
Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01dc79e <.text.lock.aic7xxx_osm+bb/21d>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; ahc_dv_1
Quoted 1 line
Quoted 1 line
EIP;
f7524d1c<[sunrpc].bss.end+341d4209/343f34ed> <=====
Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01dc79e <.text.lock.aic7xxx_osm+bb/21d>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; scsi_eh_0
Quoted 1 line
Quoted 1 line
EIP;
c1c3bfdc<_end+189d568/1dad58c> <=====
Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01d0d48 <.text.lock.scsi_error+dd/f5>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; scsi_eh_1
Quoted 1 line
Quoted 1 line
EIP;
c1c39fdc<_end+189b568/1dad58c> <=====
Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01d0d48 <.text.lock.scsi_error+dd/f5>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; kjournald
Quoted 1 line
Quoted 1 line
EIP; 00000286 Before first symbol <=====
Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c017de49 <kjournald+199/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; qla2300_dpc2
Quoted 1 line
Quoted 1 line
EIP;
c2665fdc<[mii].rodata.end+2dc7dd/bf7801> <=====
Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c2797943 <[mii].rodata.end+40e144/bf7801>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; scsi_eh_2
Quoted 1 line
Quoted 1 line
EIP;
c2663fdc<[mii].rodata.end+2da7dd/bf7801> <=====
Trace; c0105e67 <__down_interruptible+5f/e4>
Trace; c0105f4b <__down_failed_interruptible+7/c>
Trace; c01d0d48 <.text.lock.scsi_error+dd/f5>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; khubd
Quoted 1 line
Quoted 1 line
EIP; 00000001 Before first symbol <=====
Trace; c26048bb <[mii].rodata.end+27b0bc/bf7801>
Trace; c260ddac <[mii].rodata.end+2845ad/bf7801>
Trace; c260ddac <[mii].rodata.end+2845ad/bf7801>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; kjournald
Quoted 1 line
Quoted 1 line
EIP; 00000286 Before first symbol <=====
Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c017de49 <kjournald+199/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; kjournald
Quoted 1 line
Quoted 1 line
EIP; 00000286 Before first symbol <=====
Trace; c0116351 <interruptible_sleep_on+51/80>
Trace; c017de49 <kjournald+199/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; portmap
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c015267a <do_poll+86/dc>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc; syslogd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; klogd
Quoted 1 line
Quoted 1 line
EIP;
f7aa6000<END_OF_CODE+31af01/????> <=====
Trace; c01198bc <do_syslog+ec/3e4>
Trace; c016af45 <kmsg_read+11/18>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; rpc.statd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; gpm
Quoted 1 line
Quoted 1 line
EIP;
f7a4bf14<END_OF_CODE+2c0e15/????> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; inetd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c23ebf54<[mii].rodata.end+62755/bf7801> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; f7a9a520 <END_OF_CODE+30f421/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c23c9f54<[mii].rodata.end+40755/bf7801> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c23a9f54<[mii].rodata.end+20755/bf7801> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c238ff54<[mii].rodata.end+6755/bf7801> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c236ff54<[qla2300].bss.end+39655/41701> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c234df54<[qla2300].bss.end+17655/41701> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c232df54<[qla2300]fw2300tp_code01+126e4/1804e> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; nfsd
Quoted 1 line
Quoted 1 line
EIP;
c2313f54<[qla2300]qla2x00_cfg_build_path_tree+1dc/2a4> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7a9029c <END_OF_CODE+30519d/????>
Trace; f7a9a520 <END_OF_CODE+30f421/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; lockd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c3225244 <[iptable_filter].data.end+2a3b45/3be901> Trace; c3228e99 <[iptable_filter].data.end+2a779a/3be901> Trace; f7ab1cb3 <END_OF_CODE+326bb4/????>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; rpciod
Quoted 1 line
Quoted 1 line
EIP; 00000001 Before first symbol <=====
Trace; c3225410 <[iptable_filter].data.end+2a3d11/3be901> Trace; c322f74c <[iptable_filter].data.end+2ae04d/3be901> Trace; c322f74c <[iptable_filter].data.end+2ae04d/3be901> Trace; c322f744 <[iptable_filter].data.end+2ae045/3be901> Trace; c322f744 <[iptable_filter].data.end+2ae045/3be901> Trace; c01056d0 <arch_kernel_thread+28/38> Trace; c322f74c <[iptable_filter].data.end+2ae04d/3be901> Proc; rpc.mountd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c015267a <do_poll+86/dc>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc; sshd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; xfs
Quoted 1 line
Quoted 1 line
EIP;
c2287f14<[lvm-mod].bss.end+fd355/175441> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c011dd7d <sys_gettimeofday+25/54>
Trace; c0107143 <system_call+33/38>
Proc; atd
Quoted 1 line
Quoted 1 line
EIP;
f7a31f60<END_OF_CODE+2a6e61/????> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01229ec <sys_nanosleep+104/209>
Trace; c0107143 <system_call+33/38>
Proc; cron
Quoted 1 line
Quoted 1 line
EIP;
c2281f60<[lvm-mod].bss.end+f73a1/175441> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01229ec <sys_nanosleep+104/209>
Trace; c0107143 <system_call+33/38>
Proc; apache
Quoted 1 line
Quoted 1 line
EIP;
c227df14<[lvm-mod].bss.end+f3355/175441> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; getty
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; getty
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; getty
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; getty
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; getty
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; getty
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; kjournald
Quoted 1 line
Quoted 1 line
EIP;
f7a9d294<END_OF_CODE+312195/????> <=====
Trace; c0105da8 <__down+5c/bc>
Trace; c0105f40 <__down_failed+8/c>
Trace; c017c0b4 <.text.lock.commit+5/141>
Trace; c0115f53 <do_schedule+36f/3c4>
Trace; c0115f5f <do_schedule+37b/3c4>
Trace; c017de26 <kjournald+176/250>
Trace; c017dca0 <commit_timeout+0/c>
Trace; c01056d0 <arch_kernel_thread+28/38>
Proc; screen
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; rlbench.sh
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; rlbench
Quoted 1 line
Quoted 1 line
EIP;
c93dff60<[sunrpc].bss.end+608f44d/343f34ed> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01229ec <sys_nanosleep+104/209>
Trace; c0107143 <system_call+33/38>
Proc; tee
Quoted 1 line
Quoted 1 line
EIP;
c936e000<[sunrpc].bss.end+601d4ed/343f34ed> <=====
Trace; c014afeb <pipe_wait+5b/84>
Trace; c014b0c4 <pipe_read+b0/1fc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; screen
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; bash
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; screen
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; bash
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; bonnie++
Quoted 1 line
Quoted 1 line
EIP;
f7a9d294<END_OF_CODE+312195/????> <=====
Trace; c0105da8 <__down+5c/bc>
Trace; c0105f40 <__down_failed+8/c>
Trace; c017adfa <.text.lock.transaction+5/26b>
Trace; c0178ffd <journal_start+95/c4>
Trace; c0173d48 <ext3_dirty_inode+74/108>
Trace; c0156cbe <__mark_inode_dirty+32/b0>
Trace; c01584aa <update_atime+4e/54>
Trace; c012e882 <__do_generic_file_read+4e2/4f0>
Trace; c012ec64 <generic_file_read+90/1d8>
Trace; c012eac0 <file_read_actor+0/114>
Trace; c016f007 <ext3_file_write+23/bc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; bonnie++
Quoted 1 line
Quoted 1 line
EIP;
c17fe560<_end+145faec/1dad58c> <=====
Proc; bonnie++
Quoted 1 line
Quoted 1 line
EIP;
d7933f30<[sunrpc].bss.end+145e341d/343f34ed> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc; bonnie++
Quoted 1 line
Quoted 1 line
EIP;
e157ff30<[sunrpc].bss.end+1e22f41d/343f34ed> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc; screen
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; bash
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; bonnie++
Quoted 1 line
Quoted 1 line
EIP;
f7a9d294<END_OF_CODE+312195/????> <=====
Trace; c0105da8 <__down+5c/bc>
Trace; c0105f40 <__down_failed+8/c>
Trace; c017adfa <.text.lock.transaction+5/26b>
Trace; c0178ffd <journal_start+95/c4>
Trace; c0173d48 <ext3_dirty_inode+74/108>
Trace; c0156cbe <__mark_inode_dirty+32/b0>
Trace; c01584aa <update_atime+4e/54>
Trace; c012e882 <__do_generic_file_read+4e2/4f0>
Trace; c012ec64 <generic_file_read+90/1d8>
Trace; c012eac0 <file_read_actor+0/114>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; bonnie++
Quoted 1 line
Quoted 1 line
EIP;
df49ff30<[sunrpc].bss.end+1c14f41d/343f34ed> <=====
Trace; c01228b8 <schedule_timeout+90/b0>
Trace; c012281c <process_timeout+0/c>
Trace; c01526b0 <do_poll+bc/dc>
Trace; c0152884 <sys_poll+1b4/2a3>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c0107143 <system_call+33/38>
Proc; lpd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c01fe467 <sock_poll+23/28>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c01ff657 <sys_socketcall+9b/1fc>
Trace; c0107143 <system_call+33/38>
Proc; apache
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc; apache
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc; apache
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc; apache
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc; apache
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c021f5ed <wait_for_connect+105/1d8>
Trace; c021f77a <tcp_accept+ba/234>
Trace; c0239a44 <inet_accept+2c/1b8>
Trace; c01fec56 <sys_accept+66/fc>
Trace; c0113d58 <do_page_fault+0/5fb>
Trace; c012c729 <unmap_fixup+165/174>
Trace; c0124543 <sys_rt_sigaction+67/80>
Trace; c01ff66c <sys_socketcall+b0/1fc>
Trace; c0107143 <system_call+33/38>
Proc; sshd
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; bash
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0196084 <read_chan+3b4/7e8>
Trace; c01960cf <read_chan+3ff/7e8>
Trace; c0191c3a <tty_read+da/128>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; cron
Quoted 1 line
Quoted 1 line
EIP;
e291a000<[sunrpc].bss.end+1f5c94ed/343f34ed> <=====
Trace; c014afeb <pipe_wait+5b/84>
Trace; c014b0c4 <pipe_read+b0/1fc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
Proc; sh
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; run-parts
Quoted 1 line
Quoted 1 line
EIP;
7fffffffBefore first symbol <=====
Trace; c012283c <schedule_timeout+14/b0>
Trace; c0151f0c <do_select+1c8/204>
Trace; c0151bdc <__pollwait+0/c0>
Trace; c015238a <sys_select+41a/5fc>
Trace; c0107143 <system_call+33/38>
Proc; find
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; updatedb
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; updatedb
Quoted 1 line
Quoted 1 line
EIP; 00000000 Before first symbol
Trace; c011d6d6 <sys_wait4+386/3bc>
Trace; c0107143 <system_call+33/38>
Proc; find
Quoted 1 line
Quoted 1 line
EIP;
f735cac0<[sunrpc].bss.end+3400bfad/343f34ed> <=====
Trace; c0141e75 <__wait_on_buffer+69/8c>
Trace; c01715e2 <ext3_bread+56/7c>
Trace; c016eca6 <ext3_readdir+96/390>
Trace; c01493d4 <cp_new_stat64+134/150>
Trace; c01510f4 <vfs_readdir+94/e0>
Trace; c01516e0 <filldir64+0/104>
Trace; c015182e <sys_getdents64+4a/96>
Trace; c01516e0 <filldir64+0/104>
Trace; c015087b <sys_fcntl64+7f/88>
Trace; c0107143 <system_call+33/38>
Proc; sort
Quoted 1 line
Quoted 1 line
EIP;
0000000cBefore first symbol <=====
Proc; frcode
Quoted 1 line
Quoted 1 line
EIP;
e20ea000<[sunrpc].bss.end+1ed994ed/343f34ed> <=====
Trace; c014afeb <pipe_wait+5b/84>
Trace; c014b0c4 <pipe_read+b0/1fc>
Trace; c0140be7 <sys_read+8f/100>
Trace; c0107143 <system_call+33/38>
1 warning issued. Results may not be reliable.
Re: 2.4.23pre6aa3 - possible ext3 deadlock
Hi,
Quoted 9 lines
Followup to last test with 2.4.23pre6aa3. same config as before (ext3 /ql2300 / bonnie++) I forgot to mention that I have quotas enabled on the partition under test.
There have been a few 2.4 / quota / ext3 deadlock-type mails on the list recently so I'm sot sure if this is related. I can see an atime update in the call trace of one of the bonnies but no
dquot_calls anywhere. I haven't applied Jan's patch as it came out before I could reliably reproduce this problem.
Yes, actually there's still a deadlock there for ext3+quotas (patch is
already written but not yet included in mainline I think) but as I've looked through the trace it doesn't seem to be the case. About a week ago I saw a report on deadlock of 2.4.22 with just plain ext3 without quotas - so can you try to reproduce the problem with quotas not compiled into the kernel (or just turning them off should be enough)?
Quoted 11 lines
In the meantime I'll restart testing with noatime - which is how I'd usually mount such a fs anyhow.
The box seems to deadlock after 3-6 days of filesystem exercise. I have repeated this 3 times. The last run was 6 days and and then I see all disk I/O has ceased. The system remains pingable and sshd connects at the TCP layer but nothing happens after that. The root filesystem is ext3 but is a completely separate device (motherboard scsi) to the filesystems running bonnie (external FC RAID). It seems that all filesystem activity ceases.
Quoted 2 lines
I have run the alt-sysrq-t ouput through ksymoops and attached. Hope this is useful to someone who knows.
Thanks for report
Honza
Quoted 2 lines · Jamie Clark
I made the quick fix (disabling
rq_mergeable) and started the load test. Will let it run for a week or so.
does your later recent email means it deadlocked again even with this disabled?
Could you try again with 2.4.23aa1 again just in case?
Quoted 5 lines
FYI an observation from my last test: the read latency seems to be much improved and more consistent under this kernel (2.4.23pre6aa3, before the oops and before this fix). The maximum latency seemed steady over the whole test without any of the longish pauses that showed up under 2.4.19. Quite a difference.
nice to hear! thanks.
Quoted 11 lines · Andrea Arcangeli
Quoted 4 lines · Jamie Clark
I made the quick fix (disabling
rq_mergeable) and started the load test. Will let it run for a week or so.does your later recent email means it deadlocked again even with this disabled?
Yes, although it appeared to be a different deadlock: ext3 perhaps.
Quoted 3 lines
Could you try again with 2.4.23aa1 again just in case?
I started another test on the same kernel with quotas disabled as Jan suggested but I had to abort the test for unrelated reason (needed spare hw). The interesting case for me is WITH quotas.
Will try 2.4.23aa1 as soon as I can re-assemble the test unit.
-Jamie
2.4.23aa1 ext3 oops
In reply to Andrea ArcangeliOK, no deadlock yet with 2.4.23aa1 however it oopsed under ext3_file_write() in __mark_inode_dirty().
Just to recap: this test is dual PIII, running several bonnie++ loads on an ext3+noatime+quota filesystem mounted off
From the oops the fault happens on the last instruction of:
movl $0,8(%ebx)
movl $0,4(%edx)
movl 100(%edi),%eax
movl %edx,4(%eax) <-- herewhich appears to be this code in inode.c [line 221+]
if (!(inode->i_state & (I_LOCK|I_FREEING|I_CLEAR)) &&
!list_empty(&inode->i_hash)) {
list_del(&inode->i_list);
list_add(&inode->i_list, &sb->s_dirty);After a quick browse of the assembler output the zeroing would appear to be part of the list_del inline, and edi seems to equate to &sb. If I have read that correctly then the oops happens at the beginning of the list_add() inline and eax is the head of the s_dirty list - pointing into oblivion.
__mark_inode_dirty() does not appear to take sb_lock before adding to the s_dirty list. Could that
be the culprit? I'm completely unfamiliar with linux kernel so I might be way off here.
-Jamie
Quoted 23 lines · Andrea Arcangeli
Quoted 4 lines · Jamie Clark
I made the quick fix (disabling
rq_mergeable) and started the load test. Will let it run for a week or so.does your later recent email means it deadlocked again even with this disabled?
Could you try again with 2.4.23aa1 again just in case?
Quoted 7 lines
FYI an observation from my last test: the read latency seems to be much improved and more consistent under this kernel (2.4.23pre6aa3, before the oops and before this fix). The maximum latency seemed steady over the whole test without any of the longish pauses that showed up under 2.4.19. Quite a difference.
Quoted 3 lines
nice to hear! thanks.
ksymoops 2.4.5 on i686 2.4.23aa1. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.23aa1/ (default)
-m /boot/System.map-2.4.23aa1 (default)
Warning: You did not tell me where to find symbol information. I will assume that the log matches the kernel and modules that are running right now and I'll use the default options above for symbol resolution. If the current kernel and/or modules do not match the log, you can get more accurate output by telling me the kernel version and where to find map, modules, ksyms etc. ksymoops -h explains the options.
Unable to handle kernel paging request at virtual address d7ffbc0c c0156eaf *pde = 17c001e3 Oops: 0002 2.4.23aa1 #2 SMP Sat Dec 6 10:58:56 SGT 2003 CPU: 0 EIP: 0010:[<c0156eaf>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010287
eax: d7ffbc08 ebx: e2ae6a00 ecx: c02a11f8 edx: e2ae6a08
esi: 00000001 edi: f769e000 ebp: 00002000 esp: f2799ef8
ds: 0018 es: 0018 ss: 0018
Process bonnie++ (pid: 330, stackpage=f2799000)
Stack: e2ae6a00 e2ae6a00 f5110420 c0130973 e2ae6a00 00000001 e2ae6a6c e2ae6a00
f5110420 00002000 e2ae6a00 ffffffeb f785d660 c014bf57 e2ae6a00 00000000
00000000 e2ae6a00 e2ae6ac4 00000000 00000000 00000000 f5110420 c013108f
Call Trace: [<c0130973>] (40) [<c014bf57>] (40) [<c013108f>] (36) [<c016f377>] (36) [<c0140e57>] (36) [<c0107133>] (60)
Code: 89 50 04 89 43 08 8d 47 64 89 42 04 89 57 64 c6 05 10 12 2aQuoted 9 lines
Quoted 9 lines
EIP;
c0156eaf<__mark_inode_dirty+93/b0> <=====eax;
d7ffbc08<END_OF_CODE+1462abd5/????>
ebx;e2ae6a00<END_OF_CODE+1f1159cd/????>
ecx;c02a11f8<inode_in_use+0/8>
edx;e2ae6a08<END_OF_CODE+1f1159d5/????>
edi;f769e000<END_OF_CODE+33cccfcd/????>
ebp; 00002000 Before first symbol
esp;f2799ef8<END_OF_CODE+2edc8ec5/????>
Trace; c0130973 <generic_file_write_nolock+d3/4c8> Trace; c014bf57 <permission+7b/84>
Trace; c013108f <generic_file_write+13f/158> Trace; c016f377 <ext3_file_write+23/bc> Trace; c0140e57 <sys_write+8f/100>
Trace; c0107133 <system_call+33/38>
Code; c0156eaf <__mark_inode_dirty+93/b0>
00000000 <_EIP>:
Code; c0156eaf <__mark_inode_dirty+93/b0> <=====
0: 89 50 04 mov %edx,0x4(%eax) <=====
Code; c0156eb2 <__mark_inode_dirty+96/b0>
3: 89 43 08 mov %eax,0x8(%ebx)
Code; c0156eb5 <__mark_inode_dirty+99/b0>
6: 8d 47 64 lea 0x64(%edi),%eax
Code; c0156eb8 <__mark_inode_dirty+9c/b0>
9: 89 42 04 mov %eax,0x4(%edx)
Code; c0156ebb <__mark_inode_dirty+9f/b0>
c: 89 57 64 mov %edx,0x64(%edi)
Code; c0156ebe <__mark_inode_dirty+a2/b0>
f: c6 05 10 12 2a 00 00 movb $0x0,0x2a1210
1 warning issued. Results may not be reliable.
Re: 2.4.23aa1 ext3 oops
Jamie,
Did you try the patch I suggested for you to revert ?
Quoted 64 lines · Jamie Clark
OK, no deadlock yet with 2.4.23aa1 however it oopsed under
ext3_file_write()in__mark_inode_dirty().Just to recap: this test is dual PIII, running several bonnie++ loads on an ext3+noatime+quota filesystem
mounted offFrom the oops the fault happens on the last instruction of:
movl $0,8(%ebx) movl $0,4(%edx) movl 100(%edi),%eax movl %edx,4(%eax) <-- herewhich appears to be this code in
inode.c[line 221+]if (!(inode->i_state & (I_LOCK|I_FREEING|I_CLEAR)) && !list_empty(&inode->i_hash)) { list_del(&inode->i_list); list_add(&inode->i_list, &sb->s_dirty);After a quick browse of the assembler output the zeroing would appear to be part of the
list_delinline, and edi seems to equate to &sb. If I have read that correctly then the
oops happens at the beginning of
thelist_add()inline and eax is the head of thes_dirtylist - pointing into oblivion.
__mark_inode_dirty()does not appear to takesb_lockbefore adding to thes_dirtylist. Could that
be the culprit? I'm completely unfamiliar with linux kernel so I might
be way off here.-Jamie
Quoted 26 lines · Andrea Arcangeli
Quoted 4 lines · Jamie Clark
I made the quick fix (disabling
rq_mergeable) and started the load test. Will let it run for a week or so.does your later recent email means it deadlocked again even with this disabled?
Could you try again with 2.4.23aa1 again just in case?
Quoted 7 lines
FYI an observation from my last test: the read latency seems to be much improved and more consistent under this kernel (2.4.23pre6aa3, before the oops and before this fix). The maximum latency seemed steady over the whole test without any of the longish pauses that showed up under 2.4.19. Quite a difference.
nice to hear! thanks.
Re: 2.4.23aa1 ext3 oops
Hi Marcelo,
Sorry - I had started another test run after the previous crash (to collect more forensics) and have been waiting for the oops. The last one took 4 or 5 days. Just now I see I have another crash in the same place so I will apply the reversions you suggested and start again.
My apologies - this is slow work. I'm attempting to find a new kernel to keep our busy fileserver happy and I've had a spell of bad luck for the past month or two. I'll be happy when I get two weeks of uptime with a few bonnies running.
Quoted 76 lines · Marcelo Tosatti
Jamie,
Did you try the patch I suggested for you to revert ?
Quoted 65 lines · Jamie Clark
OK, no deadlock yet with 2.4.23aa1 however it oopsed under
ext3_file_write()in__mark_inode_dirty().Just to recap: this test is dual PIII, running several bonnie++ loads on an ext3+noatime+quota filesystem mounted off
From the oops the fault happens on the last instruction of:
movl $0,8(%ebx) movl $0,4(%edx) movl 100(%edi),%eax movl %edx,4(%eax) <-- herewhich appears to be this code in
inode.c[line 221+]if (!(inode->i_state & (I_LOCK|I_FREEING|I_CLEAR)) && !list_empty(&inode->i_hash)) { list_del(&inode->i_list); list_add(&inode->i_list, &sb->s_dirty);After a quick browse of the assembler output the zeroing would appear to be part of the
list_delinline, and edi seems to equate to &sb. If I have read that correctly then the oops happens at the beginning of thelist_add()inline and eax is the head of thes_dirtylist - pointing into oblivion.
__mark_inode_dirty()does not appear to takesb_lockbefore adding to thes_dirtylist. Could that
be the culprit? I'm completely unfamiliar with linux kernel so I might be way off here.-Jamie
Quoted 34 lines · Andrea Arcangeli
Quoted 6 lines · Jamie Clark
I made the quick fix (disabling
rq_mergeable) and started the load test. Will let it run for a week or so.does your later recent email means it deadlocked again even with this disabled?
Could you try again with 2.4.23aa1 again just in case?
Quoted 9 lines
FYI an observation from my last test: the read latency seems to be much improved and more consistent under this kernel (2.4.23pre6aa3, before the oops and before this fix). The maximum latency seemed steady over the whole test without any of the longish pauses that showed up under 2.4.19. Quite a difference.
nice to hear! thanks.
Re: 2.4.23aa1 ext3 oops
In reply to Jamie ClarkQuoted 2 lines · Jamie Clark
After a quick browse of the assembler output the zeroing would appear to be part of the
list_delinline, and edi seems to equate to &sb.
Seems reasonable. It does look like something's stomped on sb->s_dirty.
Quoted 2 lines
__mark_inode_dirty()does not appear to takesb_lockbefore adding to thes_dirtylist. Could that be the culprit?
I don't think so; it's holding the inode_lock which should be sufficient. Besides -- in practice all updates to the 4-byte pointer sb->s_dirty.next are going to be atomic, and there's no reason _ever_ for it to be set to d7ffbc08. It's hard to see how a simple locking problem is going to cause such a thing.
How repeatable is this? Can you turn on slab poisoning?
Signature · 1 line
dwmw2
Re: 2.4.23aa1 ext3 oops
Quoted 13 lines · David Woodhouse
Quoted 3 lines · Jamie Clark
After a quick browse of the assembler output the zeroing would appear to be part of the
list_delinline, and edi seems to equate to &sb.Seems reasonable. It does look like something's stomped on
sb->s_dirty.Quoted 3 lines
__mark_inode_dirty()does not appear to takesb_lockbefore adding to thes_dirtylist. Could that be the culprit?
Quoted 7 lines
I don't think so; it's holding the
inode_lockwhich should be sufficient. Besides -- in practice all updates to the 4-byte pointersb->s_dirty.nextare going to be atomic, and there's no reason _ever_ for it to be set tod7ffbc08. It's hard to see how a simple locking problem is going to cause such a thing.
True. I confess I didn't think too hard after narrowing down where it tripped up.
Quoted 2 lines
How repeatable is this?
The first oops ocurred after 4 or 5 days. My second run crashed in the first night, this time in filemap.c: precheck_file_write(). This oops seemed to be at or near the first dereference of inode, before the f_flags test.
/* FIXME: this is for backwards compatibility with 2.4 */
if (!S_ISBLK(inode->i_mode) && (file->f_flags & O_APPEND))
*ppos = pos = inode->i_size;Quoted 1 line
Quoted 1 line
EIP;
c01306fb<precheck_file_write+53/1f8> <=====
Trace; c01308f8 <generic_file_write_nolock+58/4c8> Trace; c013108f <generic_file_write+13f/158> Trace; c016f3a7 <ext3_file_write+23/bc> Trace; c0140e57 <sys_write+8f/100>
Trace; c0107133 <system_call+33/38>
Quoted 3 lines
Can you turn on slab poisoning?
Is CONFIG_DEBUG_SLAB all that I need?
I'm currently running 2.4.23aa1 with the inode.c patch reversions as Marcelo first suggested. When (IF) that crashes, or if I can get the test running on another SMP box I will revert to 2.4.23 + the qla2300 driver that I need for the fibre-channel array.
-Jamie
Re: 2.4.23aa1 ext3 oops
Quoted 8 lines · Jamie Clark
The first oops ocurred after 4 or 5 days. My second run crashed in the first night, this time in
filemap.c:precheck_file_write(). This oops seemed to be at or near the first dereference of inode, before thef_flagstest./* FIXME: this is for backwards compatibility with 2.4 */ if (!S_ISBLK(inode->i_mode) && (file->f_flags & O_APPEND)) *ppos = pos = inode->i_size;
Odd. And what's 'inode' then? What compiler are you using?
Quoted 1 line
Is
CONFIG_DEBUG_SLABall that I need?
Yes, I believe so.
Signature · 1 line
dwmw2
Re: 2.4.23aa1 ext3 oops
Quoted 3 lines · David Woodhouse
Odd. And what's 'inode' then? What compiler are you using?
Debian woody gcc 2.95.4
The code emitted at the start of precheck_file_write is:
precheck_file_write:
pushl %ebp
pushl %edi
pushl %esi
pushl %ebx
movl 24(%esp),%ebp
movl 32(%esp),%edx
movl $-8192,%eax[...]
I guessed inode to be ebp. the oops text copied from serial console (I lost the first line):
c01306fb
*pde = 00000000
Oops: 0002 2.4.23aa1 #12 SMP Thu Dec 11 11:25:47 SGT 2003
CPU: 1
EIP: 0010:[<c01306fb>] Not tainted
EFLAGS: 00010206
eax: 00008000 ebx: 12ba4000 ecx: ffffffff edx: f3a8ff48
esi: 00000000 edi: 00000000 ebp: c3526200 esp: f3a8feec
ds: 0018 es: 0018 ss: 0018
Process bonnie++ (pid: 316, stackpage=f3a8f000)
Stack: e7d715c0 c3526200 e7d715a0 00002000 c01308f8 e7d715a0 c3526200 f3a8ff60
f3a8ff48 c352626c c3526200 e7d715a0 00002000 f3a8ff44 0061a931
00001000
00000000 00001000 00001000 00000000 c3526200 c35262c4 00000000
12ba4000
Call Trace: [<c01308f8>] (88) [<c013108f>] (36) [<c016f3a7>] (36)
[<c0140e57>] (36) [<c0107133>] (60)
Code: 00 60 74 15 8b 7c 24 14 f6 47 19 04 74 0b 8b 5d 44 8b 75 48I'm now a little less confident about that last crash because I was a bit hasty in getting another run started. I have since trampled over the supporting evidence (kernel, map, build output).