Problems with SATA/AHCI with 'nosmp' boot option

DISCUSSION Reuben Farrelly 3 participants 7 messages Sep 21, 2005 – Nov 26, 2005 libata

Share & tools

Reuben Farrelly

author

2005-09-21 09:10 UTC #0 lore raw

Hi,

While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.

However when I did this, I could no longer boot the machine, because the ahci driver reported timeouts when probing the drives attached to the SATA ports, like this:

Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later

    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio

ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)

etc, all ata ports fail to respond (at all).

Then of course the machine panics later on because it can't find a root to mount other critical things like /dev on since the physically attached drives aren't seen.

The problem seemed was very reproduceable, I attempted this a couple of times
to make sure that it was consistent - and yes it was. However, if I build a kernel without SMP, it runs fine, so it appears to be just a problem with the boot option rather than non-SMP as such.

Normally the boot would look like this:

ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 193
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 193
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 193
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 193
ata1: dev 0 cfg 49:2f00 82:346b 83:7d01 84:4003 85:3469 86:3c01 87:4003 88:007f
ata1: dev 0 ATA-6, max UDMA/133, 156301488 sectors: LBA48
ata1: dev 0 configured for UDMA/133
scsi0 : ahci
ata2: dev 0 cfg 49:2f00 82:346b 83:7d01 84:4003 85:3469 86:3c01 87:4003 88:007f
ata2: dev 0 ATA-6, max UDMA/133, 156301488 sectors: LBA48
ata2: dev 0 configured for UDMA/133
scsi1 : ahci
ata3: no device found (phy stat 00000000)
scsi2 : ahci
ata4: no device found (phy stat 00000000)
scsi3 : ahci

  Vendor: ATA       Model: ST380817AS        Rev: 3.42
  Type:   Direct-Access                      ANSI SCSI revision: 05
  Vendor: ATA       Model: ST380817AS        Rev: 3.42
  Type:   Direct-Access                      ANSI SCSI revision: 05

SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB) SCSI device sda: drive cache: write back
SCSI device sda: 156301488 512-byte hdwr sectors (80026 MB) SCSI device sda: drive cache: write back
sda: sda1 sda2 sda3 sda4 < sda5 sda6 sda7 sda8 sda9 sda10 >
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

Has anyone else seen this (probably somewhat obscure) problem? It's hardly a showstopper but still, would have saved me a bunch of messing around rebuilding if it worked :) Looking back in time looks like the 'nosmp' has caused problems before, so seems possible that this is not an IDE specific problem, although the rest of the boot up looked OK.

The machine has an Intel 925XCV board ( http://support.intel.com/support/motherboards/desktop/d925xcv/index.htm ), and this was with 2.6.14-rc1-mm1.

Kernel config is at http://www.reub.net/kernel/

reuben

Jeff Garzik

2005-10-04 10:31 UTC #1 lore raw

Quoted 15 lines · Reuben Farrelly

Hi,
While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.
However when I did this, I could no longer boot the machine, because the ahci driver reported timeouts when probing the drives attached to the SATA ports, like this:
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio

Quoted 12 lines

ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)

Sounds like AHCI or MSI is broken.

Jeff

Randy.Dunlap

2005-10-05 02:04 UTC #2 lore raw

Quoted 34 lines · On Tue, 04 Oct 2005 06:31:42 -0400 Jeff Garzik wrote:

Quoted 19 lines · Reuben Farrelly
Hi,
While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.
However when I did this, I could no longer boot the machine, because the ahci
driver reported timeouts when probing the drives attached to the SATA ports, like this:
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
Quoted 13 lines
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode
ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)

Quoted 1 line

Sounds like AHCI or MSI is broken.

Reuben, is AHCI trying to use MSI interrupts in this case? (reported in /proc/interrupts)

If so, I've seen problems with Linux and MSI interrupts when "nomsp" is used too. Can you disable CONFIG_PCI_MSI or is this a vendor kernel with it already enabled?

I posted a patch about 1 week ago that disables MSI interrupts if "nomsp" or some similar options are used. It's in Andrew's -mm patch set afaik. or I just put it at
http://www.xenotime.net/linux/patches/msi_disable2.patch
if you care to try it.

---
~Randy
You can't do anything without having to do something else first. -- Belefant's Law

Reuben Farrelly

author

2005-10-05 13:27 UTC #3 lore raw

Hi Randy,

Quoted 30 lines · Randy.Dunlap

Quoted 28 lines · On Tue, 04 Oct 2005 06:31:42 -0400 Jeff Garzik wrote:
Quoted 27 lines · Reuben Farrelly
Hi,
While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.
However when I did this, I could no longer boot the machine, because the ahci driver reported timeouts when probing the drives attached to the SATA ports, like this:
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)

Quoted 8 lines

Quoted 1 line
Sounds like AHCI or MSI is broken.
Reuben, is AHCI trying to use MSI interrupts in this case? (reported in /proc/interrupts)
If so, I've seen problems with Linux and MSI interrupts when "nomsp" is used too. Can you disable CONFIG_PCI_MSI or is this a vendor kernel with it already enabled?

It's a self-built kernel, however it has PCI express support, so I've been building with PCI-E and MSI-X support.

With the box up now, and not even attempting to use 'nosmp', nothing at all is using MSI - everything including is using either level or edge triggered. Not sure why but AHCI is not even showing up - should it be listed as well as libata? [both are built in, not modular]

[root@tornado ~]# cat /proc/interrupts
CPU0 CPU1

  0:     143311          0    IO-APIC-edge  timer
  4:        400          0    IO-APIC-edge  serial
  8:          1          0    IO-APIC-edge  rtc
  9:          0          0   IO-APIC-level  acpi
 16:        132          0   IO-APIC-level  uhci_hcd:usb5
 17:          0          0   IO-APIC-level  sky2
 18:          4          0   IO-APIC-level  uhci_hcd:usb4
 19:      35332          0   IO-APIC-level  uhci_hcd:usb3, libata, eth0
 20:          3          0   IO-APIC-level  ehci_hcd:usb1, uhci_hcd:usb2
 21:      23666          0   IO-APIC-level  aic7xxx
NMI:          0          0
LOC:     143226     143220
ERR:          0
MIS:          0

[root@tornado ~]

If you still think it would be handy to test with 'nosmp' and a non-MSI
enabled kernel then I'll give it a go..

As both drives are SATA, and aren't detected with 'nosmp', I guess I can't really look at /proc/interrupts as it never boots up that far.

Quoted 5 lines

I posted a patch about 1 week ago that disables MSI interrupts if "nomsp" or some similar options are used. It's in Andrew's -mm patch set afaik. or I just put it at
http://www.xenotime.net/linux/patches/msi_disable2.patch
if you care to try it.

Cool, I'm running -mm anyway so I guess it's all in there. However my board has PCI-E support, which according to Documentation/MSI-HOWTO.txt talks about MSI-X being "a required feature for PCI Express devices". If that is the case then the kbuild system probably needs a dependency to check for it (it lets me choose no MSI while retaining PCI-E).

It's not a particularly big problem for me, but it's always nice if the options which are handy for tracking down other problems, work. And of course, it could be hiding a bug that shows up in other circumstances too, I guess.

Reuben

Reuben Farrelly

author

2005-11-18 04:53 UTC #4 lore raw

In reply to Jeff Garzik

Hi,

Quoted 29 lines · Jeff Garzik

Quoted 15 lines · Reuben Farrelly
Hi,
While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.
However when I did this, I could no longer boot the machine, because the ahci driver reported timeouts when probing the drives attached to the SATA ports, like this:
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
Quoted 12 lines
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)

Quoted 3 lines

Sounds like AHCI or MSI is broken.
Jeff

I am still seeing this on 2.6.15-rc1-mm1. I don't think we got any further than to conclude it _might_ be AHCI or MSI.

What other information is required for me to be able to help someone look closer and/or resolve this bug?

I have information about my system up at http://www.reub.net/kernel/ including some system specifications and my .config. (It's a lot easier to put it there than attaching to every email)

Thanks,
Reuben

Randy.Dunlap

2005-11-18 07:06 UTC #5 lore raw

Quoted 45 lines · On Fri, 18 Nov 2005 17:53:10 +1300 Reuben Farrelly wrote:

Hi,
Quoted 35 lines · Jeff Garzik
Quoted 20 lines · Reuben Farrelly
Hi,
While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.
However when I did this, I could no longer boot the machine, because the ahci
driver reported timeouts when probing the drives attached to the SATA ports, like this:
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
Quoted 13 lines
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode
ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)
Quoted 3 lines
Sounds like AHCI or MSI is broken.
Jeff
I am still seeing this on 2.6.15-rc1-mm1. I don't think we got any further than to conclude it _might_ be AHCI or MSI.

Quoted 6 lines

What other information is required for me to be able to help someone look closer and/or resolve this bug?
I have information about my system up at http://www.reub.net/kernel/ including some system specifications and my .config. (It's a lot easier to put it there than attaching to every email)

Hi,

Jeff sent a message to lkml yesterday that he is seeing some bad results with MSI interrupts:
http://lkml.org/lkml/2005/11/17/4

Hopefully that will get some results, although not yet AFAIK.

---
~Randy

Reuben Farrelly

author

2005-11-26 09:09 UTC #6 lore raw

Hi again Randy and others,

Quoted 44 lines · Randy.Dunlap

Quoted 31 lines · On Fri, 18 Nov 2005 17:53:10 +1300 Reuben Farrelly wrote:
Hi,
Quoted 28 lines · Jeff Garzik
Quoted 27 lines · Reuben Farrelly
Hi,
While trying to gather more info about a problem with the latest sky2.c Gig ethernet driver hanging, I thought I'd boot my SMP built kernel (on an SMP/HT machine) with the 'nosmp' boot option.
However when I did this, I could no longer boot the machine, because the ahci driver reported timeouts when probing the drives attached to the SATA ports, like this:
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
ICH6: IDE controller at PCI slot 0000:00:1f.1
ACPI: PCI Interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 73
ICH6: chipset revision 3
ICH6: not 100% native mode: will probe irqs later
    ide0: BM-DMA at 0xffa0-0xffa7, BIOS settings: hda:pio, hdb:pio
    ide1: BM-DMA at 0xffa8-0xffaf, BIOS settings: hdc:pio, hdd:pio
ACPI: PCI Interrupt 0000:00:1f.2[B] -> GSI 19 (level, low) -> IRQ 81 ahci(0000:00:1f.2) AHCI 0001.0000 32 slots 4 ports 1.5 Gbps 0xf impl SATA mode ahci(0000:00:1f.2) flags: 64bit ncq led slum part
ata1: SATA max UDMA/133 cmd 0xF8806D00 ctl 0x0 bmdma 0x0 irq 81
ata2: SATA max UDMA/133 cmd 0xF8806D80 ctl 0x0 bmdma 0x0 irq 81
ata3: SATA max UDMA/133 cmd 0xF8806E00 ctl 0x0 bmdma 0x0 irq 81
ata4: SATA max UDMA/133 cmd 0xF8806E80 ctl 0x0 bmdma 0x0 irq 81 ata1 is slow to respond, please be patient
ata1 failed to respond (30 secs)
scsi0 : ahci
ata2 is slow to respond, please be patient
ata2 failed to respond (30 secs)
Quoted 8 lines
Quoted 3 lines
Sounds like AHCI or MSI is broken.
Jeff
I am still seeing this on 2.6.15-rc1-mm1. I don't think we got any further than to conclude it _might_ be AHCI or MSI.
What other information is required for me to be able to help someone look closer and/or resolve this bug?
I have information about my system up at http://www.reub.net/kernel/ including some system specifications and my .config. (It's a lot easier to put it there than attaching to every email)
Hi,

Quoted 8 lines

Jeff sent a message to lkml yesterday that he is seeing some bad results with MSI interrupts:
http://lkml.org/lkml/2005/11/17/4
Hopefully that will get some results, although not yet AFAIK.
---
~Randy

I've just compiled up a 2.6.15-rc2-mm1 but without PCI-X and also without MSI support, and I'm seeing the same problem of AHCI timing out when booting with the nosmp option.

I guess this looks more like an AHCI issue rather than MSI?

I have put up an updated boot dump on http://www.reub.net/files/kernel/

Reuben