totally random "VFS: Cannot open root device"

Hi!

(Please CC me your answers as I am not subscribed.)

I have a problem with 2.6.14.3 kernel (but this probably isn't too version-specific). I have a kernel which succesfully boots on totally random basis (cca 70% is success). My root partition resides on a SATA disc connected to a controller on Intel 6300ESB ICH southbridge (mb Intel se7320vp2). There is a reiserfs 3 filesystem on my root partition. Without any changes to configuration (os or bios or whatever) I sometimes get:

VFS: Cannot open root device "801" or unknown block(8,1)

Could this be some timeout issue, or indication of crappy hw? I've tried this about 10 times (immediately ctrl+alt+del on successfull boot or reset button on aforementioned panic) and I saw no regularities in this misbehaviour.

I sincerely appreciate any advice anyone can give.

Thanks,

r.

What SATA controler are you using in your box? What does lspci report? It sounds to me like the driver you are using is not playing well with your SATA controler or drive. It could be related to crappy HW or a driver issue.

When you system boots the drive says online and behaves ok? You just see device trouble when you boot up correct?

Is it possible to capture the boot messages from a failed boot (serial console, netconsole)? It can tell alot about why no root fs was found. Most likley no disk were found by the SATA driver.

Thanks,

Keith

[CC'ing linux-ide]

Hello, 0602. :-)

Can you please post dmesg of a successful booting? That will tell us which SATA controller/disks you are using. Also, the boot log of a failed boot will be very helpful - the best way to get this is via serial console. If you don't have access to serial console, taking note
/ picture of the part where SATA detection fails will do too.

Also, when the machine boots successfully, does it work without generating disk related kernel logs? Just perform any IO-heavy operations - cp'ing directories which contain large files, tar/untarring... - and see if the kernel complains about anyting.

tejun

Note that I experienced the same problem yesterday evening with a 2.6.15-rc3 on HP Proliant DL140, which has an ICH-5 based controller and a Maxtor drive

Successfull boot shows :
...
libata version 1.12 loaded.
ata_piix version 1.04
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 17
PCI: Setting latency timer of device 0000:00:1f.2 to 64
ata1: SATA max UDMA/133 cmd 0x1F0 ctl 0x3F6 bmdma 0x1470 irq 14
ata1: dev 0 cfg 49:2f00 82:7869 83:7d09 84:4633 85:7869 86:3c01 87:4623 88:203f
ata1: dev 0 ATA, max UDMA/100, 156301488 sectors: lba48
ata1: dev 0 configured for UDMA/100
scsi0 : ata_piix
Vendor: ATA Model: Maxtor 6L080M0 Rev: BANC
Type: Direct-Access ANSI SCSI revision: 05
ata2: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x1478 irq 15
ata2: SATA port has no device.
scsi1 : ata_piix
st: Version 20050830, fixed bufsize 32768, s/g segs 256
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 < sda5 sda6 sda7 sda8 >
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi generic sg0 at scsi0, channel 0, id 0, lun 0, type 0
...

A failed boot shows "ata1: SATA port has no device". I'll try to set-up a serial console to log the boot messages.

This behaviour can be reproduced on four different DL140's, so it's fairly safe to say it's not a cabling/hardware problem.

Jean-François "Jef" Stenuit
Network & Security manager
Keytrade Bank SA

I've seen this a few times.
If you were using MSI interrupts, I would blame that, but you aren't AFAICT, so try booting with the "irqpoll" kernel boot option to see if the kernel can reroute the (seemingly) lost interrupts. Maybe we can deduce something from that. If it works, please post the /proc/interrupts file.

~Randy

Hi Randy,

Booting with irqpoll does not help. I guess we'll need to dig a little further ;-)

Jean-François "Jef" Stenuit
Network & Security manager
Keytrade Bank SA

As promised, here is the complete bootlog of a failed, then a successfull boot :

GNU GRUB version 0.95 (622K lower / 1046976K upper memory)

+-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line. Debian GNU/Linux, kernel 2.6.15-rc3-dl140-10 Debian GNU/Linux, kernel 2.6.15-rc3-dl140-10 (recovery mode) Debian GNU/Linux, kernel 2.6.15-rc3-dl140-9 Debian GNU/Linux, kernel 2.6.15-rc3-dl140-9 (recovery mode) The highlighted entry will be booted automatically in 5 seconds. The highlighted entry will be booted automatically in 4 seconds. The highlighted entry will be booted automatically in 3 seconds. The highlighted entry will be booted automatically in 2 seconds. The highlighted entry will be booted automatically in 1 seconds. Booting 'Debian GNU/Linux, kernel 2.6.15-rc3-dl140-10 '

root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz-2.6.15-rc3-dl140-10 root=/dev/sda1 ro console=ttyS0,9600n8
[Linux-bzImage, setup=0x1c00, size=0x1a817b]
savedefault
boot
Linux version 2.6.15-rc3-dl140-10 (jfs@dl140) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Thu Dec 1 11:44:28 CET 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009b800 (usable)
BIOS-e820: 000000000009b800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003ff70000 (usable)
BIOS-e820: 000000003ff70000 - 000000003ff7a000 (ACPI data)
BIOS-e820: 000000003ff7a000 - 000000003ff80000 (ACPI NVS)
BIOS-e820: 000000003ff80000 - 0000000040000000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ff800000 - 00000000ffc00000 (reserved)
BIOS-e820: 00000000fffffc00 - 0000000100000000 (reserved)
127MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000f6790
DMI present.
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1 15:4 APIC version 20
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x03] address[0xfec84000] gsi_base[24])
IOAPIC[1]: apic_id 3, version 32, address 0xfec84000, GSI 24-47
ACPI: IOAPIC (id[0x04] address[0xfec84400] gsi_base[48])
IOAPIC[2]: apic_id 4, version 32, address 0xfec84400, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Enabling APIC mode: Flat. Using 3 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 50000000 (gap: 40000000:a0000000)
Built 1 zonelists
Kernel command line: root=/dev/sda1 ro console=ttyS0,9600n8
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 2800.863 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 1034136k/1048000k available (2098k kernel code, 12972k reserved, 1222k data, 204k init, 130496k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5608.53 BogoMIPS (lpj=11217073)
Mount-cache hash table entries: 512
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Xeon(TM) CPU 2.80GHz stepping 01
Booting processor 1/1 eip 2000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5600.62 BogoMIPS (lpj=11201255)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU1: Intel(R) Xeon(TM) CPU 2.80GHz stepping 01
Total of 2 processors activated (11209.16 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfd83a, last bus=7
PCI: Using MMCONFIG
ACPI: Subsystem revision 20050902
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI quirk: region 1000-107f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 1180-11bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 *5 6 7 10 11 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs *3 4 5 6 7 10 11 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 6 7 10 *11 14 15)
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
TC classifier action (bugs to [email protected] cc [email protected])
PCI: Bridge: 0000:00:02.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:04.0
IO window: disabled.
MEM window: dd100000-dd1fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:05.0
IO window: disabled.
MEM window: dd200000-dd2fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:04:00.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:04:00.2
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:06.0
IO window: disabled.
MEM window: dd300000-dd3fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: 2000-2fff
MEM window: dd400000-deffffff
PREFETCH window: 50000000-500fffff
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:04.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:05.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI 16 (level, low) -> IRQ 16
Simple Boot Flag at 0x3b set to 0x1
highmem bounce pool size: 64 pages
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ACPI: Processor [CPU0] (supports 8 throttling states)
ACPI: Processor [CPU1] (supports 8 throttling states)
Real Time Clock Driver v1.12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 6.1.16-k2-NAPI
Copyright (c) 1999-2005 Intel Corporation.
Ethernet Channel Bonding Driver: v2.6.5 (November 4, 2005)
bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details. tg3.c:v3.43 (Oct 24, 2005)
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
eth0: Tigon3 [partno(BCM95721) rev 4101 PHY(5750)] (PCI Express) 10/100/1000BaseT Ethernet 00:14:c2:54:b4:b6
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[1] TSOcap[1] eth0: dma_rwctrl[76180000]
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 16 (level, low) -> IRQ 16
eth1: Tigon3 [partno(BCM95721) rev 4101 PHY(5750)] (PCI Express) 10/100/1000BaseT Ethernet 00:14:c2:54:b4:b7
eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] eth1: dma_rwctrl[76180000]
Equalizer2002: Simon Janes ([email protected]) and David S. Miller ([email protected])
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 17
ata1: SATA max UDMA/133 cmd 0x1F0 ctl 0x3F6 bmdma 0x1470 irq 14
ata1: SATA port has no device.
scsi0 : ata_piix
ata2: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x1478 irq 15
ata2: SATA port has no device.
scsi1 : ata_piix
mice: PS/2 mouse device common for all mice
Mirror/redirect action on
netem: version 1.1
u32 classifier
Perfomance counters on
input device check on
Actions configured
NET: Registered protocol family 2
input: AT Translated Set 2 keyboard as /class/input/input0
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
ip_conntrack version 2.4 (8187 buckets, 65496 max) - 212 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
logips2pp: Detected unknown logitech mouse model 1
input: PS/2 Logitech Mouse as /class/input/input1
ipt_recent v0.3.1: Stephen Frost <[email protected]>. http://snowman.net/projects/ipt_recent/
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
Starting balanced_irq
Using IPI Shortcut mode
VFS: Cannot open root device "sda1" or unknown-block(0,0)
Please append a correct "root=" boot option

Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

GNU GRUB version 0.95 (622K lower / 1046976K upper memory)

+-------------------------------------------------------------------------+||||||||||||||||||||||||+-------------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line. Debian GNU/Linux, kernel 2.6.15-rc3-dl140-10 Debian GNU/Linux, kernel 2.6.15-rc3-dl140-10 (recovery mode) Debian GNU/Linux, kernel 2.6.15-rc3-dl140-9 Debian GNU/Linux, kernel 2.6.15-rc3-dl140-9 (recovery mode) The highlighted entry will be booted automatically in 5 seconds. The highlighted entry will be booted automatically in 4 seconds. The highlighted entry will be booted automatically in 3 seconds. The highlighted entry will be booted automatically in 2 seconds. The highlighted entry will be booted automatically in 1 seconds. Booting 'Debian GNU/Linux, kernel 2.6.15-rc3-dl140-10 '

root (hd0,0)
Filesystem type is ext2fs, partition type 0x83
kernel /boot/vmlinuz-2.6.15-rc3-dl140-10 root=/dev/sda1 ro console=ttyS0,9600n8
[Linux-bzImage, setup=0x1c00, size=0x1a817b]
savedefault
boot
Linux version 2.6.15-rc3-dl140-10 (jfs@dl140) (gcc version 3.3.5 (Debian 1:3.3.5-13)) #1 SMP Thu Dec 1 11:44:28 CET 2005
BIOS-provided physical RAM map:
BIOS-e820: 0000000000000000 - 000000000009b800 (usable)
BIOS-e820: 000000000009b800 - 00000000000a0000 (reserved)
BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
BIOS-e820: 0000000000100000 - 000000003ff70000 (usable)
BIOS-e820: 000000003ff70000 - 000000003ff7a000 (ACPI data)
BIOS-e820: 000000003ff7a000 - 000000003ff80000 (ACPI NVS)
BIOS-e820: 000000003ff80000 - 0000000040000000 (reserved)
BIOS-e820: 00000000e0000000 - 00000000f0000000 (reserved)
BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
BIOS-e820: 00000000ff800000 - 00000000ffc00000 (reserved)
BIOS-e820: 00000000fffffc00 - 0000000100000000 (reserved)
127MB HIGHMEM available.
896MB LOWMEM available.
found SMP MP-table at 000f6790
DMI present.
ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Processor #0 15:4 APIC version 20
ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Processor #1 15:4 APIC version 20
ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
ACPI: IOAPIC (id[0x02] address[0xfec00000] gsi_base[0])
IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
ACPI: IOAPIC (id[0x03] address[0xfec84000] gsi_base[24])
IOAPIC[1]: apic_id 3, version 32, address 0xfec84000, GSI 24-47
ACPI: IOAPIC (id[0x04] address[0xfec84400] gsi_base[48])
IOAPIC[2]: apic_id 4, version 32, address 0xfec84400, GSI 48-71
ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
Enabling APIC mode: Flat. Using 3 I/O APICs
Using ACPI (MADT) for SMP configuration information
Allocating PCI resources starting at 50000000 (gap: 40000000:a0000000)
Built 1 zonelists
Kernel command line: root=/dev/sda1 ro console=ttyS0,9600n8
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 65536 bytes)
Detected 2800.704 MHz processor.
Using tsc for high-res timesource
Console: colour VGA+ 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Memory: 1034136k/1048000k available (2098k kernel code, 12972k reserved, 1222k data, 204k init, 130496k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5608.50 BogoMIPS (lpj=11217008)
Mount-cache hash table entries: 512
monitor/mwait feature present.
using mwait in idle threads.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Checking 'hlt' instruction... OK.
CPU0: Intel(R) Xeon(TM) CPU 2.80GHz stepping 01
Booting processor 1/1 eip 2000
Initializing CPU#1
Calibrating delay using timer specific routine.. 5600.65 BogoMIPS (lpj=11201307)
monitor/mwait feature present.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 0
CPU1: Intel(R) Xeon(TM) CPU 2.80GHz stepping 01
Total of 2 processors activated (11209.15 BogoMIPS).
ENABLING IO-APIC IRQs
..TIMER: vector=0x31 apic1=0 pin1=2 apic2=-1 pin2=-1
checking TSC synchronization across 2 CPUs: passed.
Brought up 2 CPUs
NET: Registered protocol family 16
ACPI: bus type pci registered
PCI: PCI BIOS revision 2.10 entry at 0xfd83a, last bus=7
PCI: Using MMCONFIG
ACPI: Subsystem revision 20050902
ACPI: Interpreter enabled
ACPI: Using IOAPIC for interrupt routing
ACPI: PCI Root Bridge [PCI0] (0000:00)
PCI quirk: region 1000-107f claimed by ICH4 ACPI/GPIO/TCO
PCI quirk: region 1180-11bf claimed by ICH4 GPIO
PCI: Ignoring BAR0-3 of IDE controller 0000:00:1f.2
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: PXH quirk detected, disabling MSI for SHPC device
PCI: Transparent bridge - 0000:00:1e.0
ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 *5 6 7 10 11 14 15)
ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *10 11 14 15)
ACPI: PCI Interrupt Link [LNKD] (IRQs *3 4 5 6 7 10 11 14 15)
ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKF] (IRQs 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 10 11 14 15) *0, disabled.
ACPI: PCI Interrupt Link [LNKH] (IRQs 4 5 6 7 10 *11 14 15)
SCSI subsystem initialized
PCI: Using ACPI for IRQ routing
PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report
TC classifier action (bugs to [email protected] cc [email protected])
PCI: Bridge: 0000:00:02.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:04.0
IO window: disabled.
MEM window: dd100000-dd1fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:05.0
IO window: disabled.
MEM window: dd200000-dd2fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:04:00.0
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:04:00.2
IO window: disabled.
MEM window: disabled.
PREFETCH window: disabled.
PCI: Bridge: 0000:00:06.0
IO window: disabled.
MEM window: dd300000-dd3fffff
PREFETCH window: disabled.
PCI: Bridge: 0000:00:1e.0
IO window: 2000-2fff
MEM window: dd400000-deffffff
PREFETCH window: 50000000-500fffff
ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:04.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:05.0[A] -> GSI 16 (level, low) -> IRQ 16
ACPI: PCI Interrupt 0000:00:06.0[A] -> GSI 16 (level, low) -> IRQ 16
Simple Boot Flag at 0x3b set to 0x1
highmem bounce pool size: 64 pages
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
ACPI: Power Button (FF) [PWRF]
ACPI: Power Button (CM) [PWRB]
ACPI: Processor [CPU0] (supports 8 throttling states)
ACPI: Processor [CPU1] (supports 8 throttling states)
Real Time Clock Driver v1.12
serio: i8042 AUX port at 0x60,0x64 irq 12
serio: i8042 KBD port at 0x60,0x64 irq 1
Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
RAMDISK driver initialized: 16 RAM disks of 8192K size 1024 blocksize
loop: loaded (max 8 devices)
Intel(R) PRO/1000 Network Driver - version 6.1.16-k2-NAPI
Copyright (c) 1999-2005 Intel Corporation.
Ethernet Channel Bonding Driver: v2.6.5 (November 4, 2005)
bonding: Warning: either miimon or arp_interval and arp_ip_target module parameters must be specified, otherwise bonding will not detect link failures! see bonding.txt for details. tg3.c:v3.43 (Oct 24, 2005)
ACPI: PCI Interrupt 0000:02:00.0[A] -> GSI 16 (level, low) -> IRQ 16
eth0: Tigon3 [partno(BCM95721) rev 4101 PHY(5750)] (PCI Express) 10/100/1000BaseT Ethernet 00:14:c2:54:b4:b6
eth0: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[1] TSOcap[1] eth0: dma_rwctrl[76180000]
ACPI: PCI Interrupt 0000:03:00.0[A] -> GSI 16 (level, low) -> IRQ 16
eth1: Tigon3 [partno(BCM95721) rev 4101 PHY(5750)] (PCI Express) 10/100/1000BaseT Ethernet 00:14:c2:54:b4:b7
eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] eth1: dma_rwctrl[76180000]
Equalizer2002: Simon Janes ([email protected]) and David S. Miller ([email protected])
tun: Universal TUN/TAP device driver, 1.6
tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
ACPI: PCI Interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 17
ata1: SATA max UDMA/133 cmd 0x1F0 ctl 0x3F6 bmdma 0x1470 irq 14
ata1: dev 0 ATA-7, max UDMA/100, 156301488 sectors: LBA48
ata1: dev 0 configured for UDMA/100
scsi0 : ata_piix
Vendor: ATA Model: Maxtor 6L080M0 Rev: BANC
Type: Direct-Access ANSI SCSI revision: 05
ata2: SATA max UDMA/133 cmd 0x170 ctl 0x376 bmdma 0x1478 irq 15
ata2: SATA port has no device.
scsi1 : ata_piix
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
SCSI device sda: drive cache: write through
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB)
SCSI device sda: drive cache: write through
sda: sda1 sda2 < sda5 sda6 sda7 sda8 sda9 >
sd 0:0:0:0: Attached scsi disk sda
mice: PS/2 mouse device common for all mice
Mirror/redirect action on
netem: version 1.1
u32 classifier
Perfomance counters on
input device check on
Actions configured
NET: Registered protocol family 2
input: AT Translated Set 2 keyboard as /class/input/input0
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
logips2pp: Detected unknown logitech mouse model 1
input: PS/2 Logitech Mouse as /class/input/input1
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
ip_conntrack version 2.4 (8187 buckets, 65496 max) - 212 bytes per conntrack
ip_tables: (C) 2000-2002 Netfilter core team
ipt_recent v0.3.1: Stephen Frost <[email protected]>. http://snowman.net/projects/ipt_recent/
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
802.1Q VLAN Support v1.8 Ben Greear <[email protected]>
All bugs added by David S. Miller <[email protected]>
Starting balanced_irq
Using IPI Shortcut mode
kjournald starting. Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 204k freed
INIT: version 2.86 booting
Activating swap.
Adding 2714944k swap on /dev/sda7. Priority:-1 extents:1 across:2714944k
Checking root file system...
fsck 1.37 (21-Mar-2005)
/: clean, 6566/136544 files, 441EXT3 FS on sda1, 68/273073 blocksinternal journal

System time was Thu Dec 1 10:55:29 UTC 2005.
Setting the System Clock using the Hardware Clock as reference... System Clock set. System local time is now Thu Dec 1 10:55:31 UTC 2005. Cleaning up ifupdown...done.
Checking all file systems...
fsck 1.37 (21-Mar-2005)
/home: clean, 22860/8372224 files, 366885/16737714 blocks /tmp: clean, 11/192768 files, 32456/393217 blocks
/usr: clean, 18050/kjournald starting. Commit interval 5 seconds 610432 files, 86EXT3 FS on sda9, internal journal 948/1220932 blocEXT3-fs: mounted filesystem with ordered data mode. ks
/var: clean,kjournald starting. Commit interval 5 seconds 1295/366528 filEXT3 FS on sda8, internal journal
es, 40433/732957EXT3-fs: mounted filesystem with ordered data mode. blocks
Setting kernel variables ...
... done.kjournald starting. Commit interval 5 seconds

Mounting localEXT3 FS on sda5, internal journal filesystems...EXT3-fs: mounted filesystem with ordered data mode.

kjournald starting. Commit interval 5 seconds EXT3 FS on sda6, internal journal
EXT3-fs: mounted filesystem with ordered data mode.
tg3: eth0: Link is up at 100 Mbps, full duplex.
tg3: eth0: Flow control is off for TX and off for RX.
INIT: Entering runlevel: 2
Starting system log daemon: syslogd.
Starting kernel log daemon: klogd.
Starting MTA: exim4.
Starting internet superserver: inetd. Starting SMP IRQ Balancer: irqbalance. Starting OpenBSD Secure Shell server: sshd. Starting deferred execution scheduler: atd.

Starting periodic command scheduler: cron.

Jean-François "Jef" Stenuit
Network & Security manager
Keytrade Bank SA

Hi,

(Please CC me your answers as I am not subscribed.)

Tejun Heo wrote:
[...]

I've placed the dmesg and the picture of panic here: http://26143.eq.cz/

There are no signs of anything wrong in logs, all my IO tests passed ok.

Regards,

r.

On 11/30/05, [email protected] <[email protected]> wrote:

Best guess is driver initilization troubles. The screen shot only dosen't really show anything besides it is broken. A full boot log from the failed boot would be nice. There were some recent SATA change could you try the current 2.6.15-rc3 as well?

Thanks,
Keith

Hi Keith,

I have been able to reproduce the bug (randomly - needing a few reboots to make it happen) on a 2.6.15-rc3 kernel.

System is an HP DL140G2 with following SATA controller (from lspci -vv) :

0000:00:1f.2 IDE interface: Intel Corp. 82801EB (ICH5) Serial ATA 150 Storage Controller (rev 02) (prog-if 8a [Master SecP PriP])
Subsystem: Hewlett-Packard Company: Unknown device 3208
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Interrupt: pin A routed to IRQ 17
Region 0: I/O ports at <unassigned>
Region 1: I/O ports at <unassigned>
Region 2: I/O ports at <unassigned>
Region 3: I/O ports at <unassigned>
Region 4: I/O ports at 1470 [size=16]

Jean-François "Jef" Stenuit
Network & Security manager
Keytrade Bank SA

Hi,

Can you try the following patch? It adds a module parameter override_PCS. If your piix module is built into the kernel, adding the following to the kernel paramter shoudl work.

ata_piix.override_PCS=n

Where, n is one of

0: normal behavior
1: ignore PCS and always try to probe ports
2: overwrite PCS such that all ENABLED bits are 1

First, try with override_PCS=0 and make not of the printed combined and orig_mask values on success and failure cases. Then, try with override_PCS=1, if it also fails, try with override_PCS=2.

You can tell if an override option works or not by comparing the combined and orig_mask value to the failure case. If the machine boots successfully with the same combined/orig_mask values as the failure case, it means that the override option works.

BTW, all these are based on the previous message in which you said that when boot failed the kernel printed the following log message.

"ata1: SATA port has no device."

Right?

index 887b2b9..639da80 100644
@@ -340,6 +340,9 @@ static void piix_pata_phy_reset(struct a
 	ata_bus_reset(ap);
 }
 
+static int override_PCS;
+module_param(override_PCS, int, 0700);
+
 /**
  *	piix_sata_probe - Probe PCI device for present SATA devices
  *	@ap: Port associated with the PCI device we wish to probe

@@ -367,6 +370,17 @@ static int piix_sata_probe (struct ata_p
 
 	pci_read_config_byte(pdev, ICH5_PCS, &pcs);
 	orig_mask = (int) pcs & 0xff;
+	printk("ata%u: combined=%d orig_mask=0x%x\n", ap->id, combined, orig_mask);
+
+	switch (override_PCS) {
+	case 1:
+		printk("ata%u: ignoring PCS\n", ap->id);
+		return 1;
+	case 2:
+		printk("ata%u: turning on PCS ENABLED bits\n", ap->id);
+		pci_write_config_byte(pdev, ICH5_PCS, 0x0f);
+		return 1;
+	}
 
 	/* TODO: this is vaguely wrong for ICH6 combined mode,
 	 * where only two of the four SATA ports are mapped

Hi Tejun,

Thanks for taking the time to check.

Output of your trace with ata_piix.override_PCS=0
1st boot : success : combined=0 orig_mask=0x11
2nd boot : success : combined=0 orig_mask=0x11
3rd boot : failure : combined=0 orig_mask=0x0
4th boot : failure : combined=0 orig_mask=0x0
5th boot : failure : combined=0 orig_mask=0x0
6th boot : failure : combined=0 orig_mask=0x0
7th boot : success : combined=0 orig_mask=0x11
Output of your trace with ata_piix.override_PCS=1
1st boot : success : combined=0 orig_mask=0x0
2nd boot : success : combined=0 orig_mask=0x0
3rd boot : success : combined=0 orig_mask=0x0
4th boot : success : combined=0 orig_mask=0x0
5th boot : success : combined=0 orig_mask=0x0
6th boot : success : combined=0 orig_mask=0x0
7th boot : success : combined=0 orig_mask=0x0
8th boot : success : combined=0 orig_mask=0x0

Looks like you have found a fix/workaround for this bug (but it still does not give the reason why it's failing).

Jean-François "Jef" Stenuit
Network & Security manager
Keytrade Bank SA

It probably is a BIOS issue. The weird thing though is that the port works fine with its corresponding ENABLED bit cleared. Anyways, if it works by ignoring the ENABLED bit, ignoring should just be fine.

0602, can you verify this workaround works on your machine too?

Jeff (Hi!), if 0602 also confirms that this workaround works, I'll submit a patch to make ata_piix ignore PCS values on ICH5's. How does that sound to you?

tejun

Hi,

Tejun Heo wrote:
[...]

[..]

Yes, I made about 20 subsequent reboots with ata_piix.override_PCS=1 and the system always booted without a hiccup.

Regards,

r.

I am being dragged into this thread with little background info. Here's some data points that may be relevant:

* until very recently, ata_piix's definitions for the ENABLED and PRESENT bits was reversed.

* the PRESENT bit reflects a device present status just like the SStatus phy register does. This implies that one must wait, before assuming that the PRESENT bit's absence truly indicates absence of a device.

* the device may be in various power management states. It may be wise to issue COMRESET by
- clear ENABLED bit
- set ENABLED bit
- wait for device to appear
- if no device appeared, clear ENABLED bit

In sum, think about the underlying SATA phy registers, and how they logically map to the PCS bits.

Jeff

Hi, Jeff.

I saw that in the log but it doesn't seem to be the reason for this as PCS reports 0x00 on failure cases. None of ENABLED/PRESENTS bits is set.

Interestingly, it seems that those problemetic ICH5's seem to work happily on ports where ENABLED bits are cleared. Turning off ENABLED bits on my ICH7 certainly disables the ports. It almost seems that the ENABLED bits are read incorrectly even though they are set correctly.

I'm a little bit scared about turning on or off those bits. As probing a disabled/non-present port doesn't cause any problem (we're doing it already for combined cases) and simply ignoring the ENABLED bits does cure the symptom. IMHO, it's best to just ignore those mysteriously zero but nevertheless working bits. I'll submit a patch to ignore ENABLED bits on ICH5's. If you don't like it, please NACK it; then, I'll try to cook something up which dances with the ENABLED bits.

Thanks.

tejun

Some ICH5's report 0 for PCS even when they have active working ports. This patch makes piix_sata_probe() ignore 0 PCS value on ICH5's and probe all ports in such cases. And while at it, remove bogus assignment to mask.

--

Hi, Jean-Francois (sorry, cannot type that character) and 0602.

Can you guys please verify that this patch works? This patch is supposed to do the same thing as my previous patch with override_PCS=1.

index 887b2b9..482995b 100644
@@ -58,6 +58,7 @@ enum {
 	ICH5_PCS		= 0x92,	/* port control and status */
 	PIIX_SCC		= 0x0A, /* sub-class code register */
 
+	PIIX_FLAG_UNRELIABLE_PCS= (1 << 27), /* PCS bits are unreliable */
 	PIIX_FLAG_AHCI		= (1 << 28), /* AHCI possible */
 	PIIX_FLAG_CHECKINTR	= (1 << 29), /* make sure PCI INTx enabled */
 	PIIX_FLAG_COMBINED	= (1 << 30), /* combined mode possible */

@@ -223,7 +224,8 @@ static struct ata_port_info piix_port_in
 	{
 		.sht		= &piix_sht,
 		.host_flags	= ATA_FLAG_SATA | ATA_FLAG_SRST |
-				  PIIX_FLAG_COMBINED | PIIX_FLAG_CHECKINTR,
+				  PIIX_FLAG_COMBINED | PIIX_FLAG_CHECKINTR |
+				  PIIX_FLAG_UNRELIABLE_PCS,
 		.pio_mask	= 0x1f,	/* pio0-4 */
 		.mwdma_mask	= 0x07, /* mwdma0-2 */
 		.udma_mask	= 0x7f,	/* udma0-6 */

@@ -362,12 +364,15 @@ static int piix_sata_probe (struct ata_p
 	int orig_mask, mask, i;
 	u8 pcs;
 
-	mask = (PIIX_PORT_PRESENT << ap->hard_port_no) |
-	       (PIIX_PORT_ENABLED << ap->hard_port_no);
-
 	pci_read_config_byte(pdev, ICH5_PCS, &pcs);
 	orig_mask = (int) pcs & 0xff;
 
+	if (orig_mask == 0 && ap->flags & PIIX_FLAG_UNRELIABLE_PCS) {
+		printk(KERN_INFO "ata%u: PIIX PCS reports 0x00, ignoring PCS\n",
+		       ap->id);
+		return 1;
+	}
+
 	/* TODO: this is vaguely wrong for ICH6 combined mode,
 	 * where only two of the four SATA ports are mapped
 	 * onto a single ATA channel.  It is also vaguely inaccurate

Tejun Heo wrote:
[...]

[...]

Works for me (10 successful subsequent reboots).

Regards,

r.

Hi Tejun,

Back to work this morning. I applied this patch to 2.6.15-rc5 to test my problematic HP DL140G2.

Result of the tests :
Reboot 1 : successfull, patch code used Reboot 2 : successfull, patch code not used Reboot 3 : successfull, patch code used Reboot 4 : successfull, patch code not used Reboot 5 : successfull, patch code used Cold boot 1 : successfull, patch code used Cold boot 2 : successfull, patch code not used Cold boot 3 : successfull, patch code not used Cold boot 4 : successfull, patch code used Cold boot 5 : successfull, patch code used

Seems the patch is working.

Jean-François "Jef" Stenuit
Network & Security manager
Keytrade Bank SA

Thanks, Jean-François. Thanks, 0602.

Jeff, the patch seems to work. Can you ack or nack it?

tejun

I'm not too sure about this one either, but we have two independent consistent bug reports and PCS handling in ata_piix isn't that necessary or strict to begin with, so....

How about making it a driver option? ata_piix.ignore_PCS maybe?

tejun

I'm OK with that.

Jeff