Skip to contentSkip to conversation

PCI init issues

DISCUSSION jamal 5 participants 10 messages Mar 2, 2003 – Mar 4, 2003

lore
Share & tools
lore

jamal

author
#0 lore raw

Seems like Linux 2.4.x has PCI initialization issues when you start having funky PCI bus setups. Now before you point a finger at the BIOs please read on...
In other words Linux needs a lot of help from the bios and if it doesnt get it things get messed up. When the bios initializes _all_ the mp table or acpi stuff things work great.
For example Linux doesnt seem to be very smart about things like second level pci bridges...

Example:
------
]0;root@localhost:~[root@localhost root]# lspci -tv -[00]-+-00.0 Intel Corp. e7500 DRAM Controller

      +-00.1  Intel Corp. e7500 DRAM Controller Error Reporting
      +-02.0-[01-04]--+-1c.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               +-1d.0-[02]--
      |               +-1e.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               \-1f.0-[03-04]----01.0-[04]--+-04.0  Digital

Equipment Corporation DECchip 21142/43
| +-05.0 Digital
Equipment Corporation DECchip 21142/43
| +-06.0 Digital
Equipment Corporation DECchip 21142/43
| \-07.0 Digital
Equipment Corporation DECchip 21142/43

      +-03.0-[05-07]--+-1c.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               +-1d.0-[06]--+-01.0  Intel Corp.: Unknown device

100f
| | +-02.0 Intel Corp.: Unknown device
1010
| | \-02.1 Intel Corp.: Unknown device
1010

      |               +-1e.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
      |               \-1f.0-[07]--
      +-1d.0  Intel Corp. 82801CA/CAM USB (Hub
      +-1d.1  Intel Corp. 82801CA/CAM USB (Hub
      +-1d.2  Intel Corp. 82801CA/CAM USB (Hub
      +-1e.0-[08]--+-01.0  ATI Technologies Inc Rage XL
      |            +-02.0  Intel Corp. 82557/8/9 [Ethernet Pro 100]
      |            \-03.0  Intel Corp. 82557/8/9 [Ethernet Pro 100]
      +-1f.0  Intel Corp. 82801CA ISA Bridge (LPC)
      +-1f.1  Intel Corp. 82801CA IDE U100
      \-1f.3  Intel Corp. 82801CA/CAM SMBus

-------------

See those DECchip 21142/43 devices? They are about 3 layers down from the root and theyll never work properly unless the BIOS is capable of initializing everything. The first of the 4 devices will receive interupts correctly, the rest dont.

-------
]0;root@localhost:~[root@localhost root]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3

  0:     441198     441826     442361     439749    IO-APIC-edge  timer
  1:          1          0          0          1    IO-APIC-edge  keyboard
  2:          0          0          0          0          XT-PIC  cascade
  4:         75         78         82         52    IO-APIC-edge  serial
  8:          1          0          0          0    IO-APIC-edge  rtc
 14:       2000       1787       1960       1535    IO-APIC-edge  ide0
 15:       1860       1750       2021       1603    IO-APIC-edge  ide1
 17:       3396       3440       3400       3415   IO-APIC-level  eth0
 18:          6          2          2          3   IO-APIC-level  eth1
 24:          5          3          5          3   IO-APIC-level  eth5,
eth6, eth7, eth8
 96:        426        424        425        424   IO-APIC-level  eth2
100:        427        424        424        425   IO-APIC-level  eth3
101:        425        424        425        423   IO-APIC-level  eth4
NMI:          0          0          0          0
LOC:    1762876    1762850    1762971    1762970
ERR:          0
MIS:          0

--------------

Interupt routing as can be seen above is really messed for that device.

The workaround is to move the board so the the DECchip 21142/43 devices are at a depth not to exceed 3.

Seems this has been an ongoing problem on Linux.
http://www.tux.org/hypermail/linux-tulip/2002-Aug/0000.html

The above is the same motherboard as mentioned in the thread above except the two processors have hyperthreading capability.

Q1: How do we resolve this other than moving around boards?
Q2: Should we really be dependent on bios this bad?
According to that thread things dont work on the BSDs either but work fine on WinXP (shudder). Does this mean winxp doesnt depend on BIOS? Should we really be dependent on the BIOS this much?
Shouldnt things like second level pci bridge be part of Linux discovery?

cheers,
jamal

#1 lore raw
Quoted 1 line · jamal

Interupt routing as can be seen above is really messed for that device.

Indeed. Quad tulip cards usually have irq pin A of each chip routed to pins A-D on the connector, so they cannot have the same irq unless all four irq pins are shared in the parent slot, which is unlikely in this case.

Quoted 1 line

Q1: How do we resolve this other than moving around boards?

Here's [completely untested] patch that attempts to validate BIOS irq assignments. This may break "good" irqs for other devices, but it's interesting to see if it changes something.

Quoted 1 line

Q2: Should we really be dependent on bios this bad?

We certainly shouldn't wrt PCI IO and MMIO allocations, but irq routing appears to be so horribly overcomplexed on x86 that dependence on BIOS seems almost unavoidable here...
Other architectures don't have such kind of problems, BTW.

Ivan.

arch/i386/kernel/pci-irq.c +10 -6
@@ -730,7 +730,7 @@ void __init pcibios_fixup_irqs(void)
 		 */
 		if (io_apic_assign_pci_irqs)
 		{
-			int irq;
+			int irq, irq1 = -1;
 
 			if (pin) {
 				pin--;		/* interrupt pins are numbered starting from 1 */
@@ -741,15 +741,19 @@ void __init pcibios_fixup_irqs(void)
 	 * parent slot, and pin number. The SMP code detects such bridged
 	 * busses itself so we should get into this branch reliably.
 	 */
-				if (irq < 0 && dev->bus->parent) { /* go back to the bridge */
+				if (dev->bus->parent) { /* go back to the bridge */
 					struct pci_dev * bridge = dev->bus->self;
 
 					pin = (pin + PCI_SLOT(dev->devfn)) % 4;
-					irq = IO_APIC_get_PCI_irq_vector(bridge->bus->number, 
+					irq1 = IO_APIC_get_PCI_irq_vector(bridge->bus->number, 
 							PCI_SLOT(bridge->devfn), pin);
-					if (irq >= 0)
-						printk(KERN_WARNING "PCI: using PPB(B%d,I%d,P%d) to get irq %d\n", 
-							bridge->bus->number, PCI_SLOT(bridge->devfn), pin, irq);
+				}
+				if (irq1 >= 0 && irq1 != irq) {
+					/* IRQ listed in MP-table doesn't match one for parent slot.
+					   Use that we've found instead. */
+					irq = irq1;
+					printk(KERN_WARNING "PCI: using PPB(B%d,I%d,P%d) to get irq %d\n", 
+						bridge->bus->number, PCI_SLOT(bridge->devfn), pin, irq);
 				}
 				if (irq >= 0) {
 					printk(KERN_INFO "PCI->APIC IRQ transform: (B%d,I%d,P%d) -> %d\n",
#2 lore raw
Quoted 6 lines · Ivan Kokshaysky
Quoted 1 line

Q2: Should we really be dependent on bios this bad?

We certainly shouldn't wrt PCI IO and MMIO allocations, but irq routing appears to be so horribly overcomplexed on x86 that dependence on BIOS seems almost unavoidable here...
Other architectures don't have such kind of problems, BTW.

Actually, for issues like that one Jamal is pointing out, I believe there is something we can do.

What varies so much from motherboard to motherboard on x86 is how the interrupts are wired up. So for example, this tells us that PCI slot X maps to x86 IRQ Y. If we know that, we can figure out where his deeply bridged tulips will send IRQs.

If the mp tables don't give exactly this kind of information, then Ivan is right :(

Anyone know if FreeBSD fares better in situations like this? What do they do if so?

author
#4 lore raw
In reply to Ivan Kokshaysky

Ivan,

Patch was applied; no luck at all. Below is dmesg output with debug turned on in the pci-irq.c file.

--------

Mar  3 18:01:39 localhost kernel: PCI: PCI BIOS revision 2.10 entry at
0xfd875last bus=8
Mar  3 18:01:39 localhost kernel: PCI: Using configuration type 1
Mar  3 18:01:39 localhost kernel: PCI: Probing PCI hardware
Mar  3 18:01:39 localhost kernel: PCI: Probing PCI hardware
Mar  3 18:01:39 localhost kernel: Transparent bridge - Intel Corp.
82801BA/CA/ PCI Bridge
Mar  3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 10
[IRQ]
Mar  3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 11
[IRQ]
Mar  3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 12
[IRQ]
Mar  3 18:01:39 localhost kernel: PCI: Using IRQ router PIIX [8086/2480]
at 00f.0
Mar  3 18:01:39 localhost kernel: PCI: IRQ fixup
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P0) ->
16
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P1) ->
19
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P2) ->
18
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B2,I1,P1) ->
48
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B3,I1,P1) ->
24
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B3,I1,P2) ->
25
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I4,P0) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I5,P1) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I6,P2) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I7,P3) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I1,P1) ->
16
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I2,P2) ->
17
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I3,P3) ->

18
---------------

BIOS irq assignments i suppose are validated.

I have a feeling the reason it works in windows is because of a functional ACPI.

What next? ;->

cheers,
jamal

author
#3 lore raw
In reply to David S. Miller
Quoted 1 line · David S. Miller

Anyone know if FreeBSD fares better in situations like this?

All BSDs apparently have this problem. Ive heard rumors of windows XP working just fine with the same setup on these mboards. Unfortunately up north we have igloos so i cant verify these rumors ;->

cheers,
jamal

#5 lore raw
In reply to jamal
Quoted 8 lines · jamal
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I4,P0) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I5,P1) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I6,P2) ->
96
Mar  3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I7,P3) ->

96

...

Quoted 1 line

BIOS irq assignments i suppose are validated.

No, this means that mp tables are broken in a way I didn't expect... Also, it's unclear whether or not the IOxAPIC routing is broken as well. Probably full dmesg output (especially things related to IO_APICs setup and irq<->pin mapping) could shed some light on this.

Quoted 2 lines

I have a feeling the reason it works in windows is because of a functional ACPI.

Perhaps.

Ivan.

#6 lore raw
In reply to Ivan Kokshaysky
Quoted 5 lines · Ivan Kokshaysky
Quoted 1 line · jamal

Interupt routing as can be seen above is really messed for that device.

Indeed. Quad tulip cards usually have irq pin A of each chip routed to pins A-D on the connector, so they cannot have the same irq unless

Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts are sent on the INTA output of the board.

The tulip.c driver has worked around the BIOS bug on x86 machines for years. It should really be handled in the generic PCI quirks code, but has not yet been correctly handled in the kernel's code.

Signature · 4 lines
Donald Becker				[email protected]
Scyld Computing Corporation		http://www.scyld.com
410 Severn Ave. Suite 210		Scyld Beowulf cluster system
Annapolis MD 21403			410-990-9993
#7 lore raw
Quoted 3 lines · Donald Becker

Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts are sent on the INTA output of the board.

This can be true for older cards, but post-1998 hardware must follow the spec.
PCI-to-PCI Bridge Architecture Specification, Rev 1.1, Dec 18, 1998, pp 113-114:

"... The interrupt binding defined in this table is mandatory for
expansion boards utilizing a bridge.

[table skipped]

Device 0 on a secondary bus will have its INTA# line connected to the INTA# line of the connector. Device 1 will have its INTA# line connected to INTB# of the connector. This sequence continues and then wraps around once INTD# has been assigned."

I know for a fact that at least D-Link card mentioned in some reports utilizes all four INT# lines, because I have one.

Ivan.

#8 lore raw

Hello!

Quoted 2 lines

I know for a fact that at least D-Link card mentioned in some reports utilizes all four INT# lines, because I have one.

I confirm, my Znyx card (pci subsys 110d:0029) is wired correctly to INTA/B/C/D and works nicely with plain flat motherboards.

Look at Jamal's mail: the cards work well when attached to primary bus. They stop to work when connected via one more bridge (which is common for server motherboards).

Alexey

#9 lore raw
In reply to Ivan Kokshaysky
Quoted 8 lines · Ivan Kokshaysky
Quoted 3 lines · Donald Becker

Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts are sent on the INTA output of the board.

This can be true for older cards, but post-1998 hardware must follow the spec.
PCI-to-PCI Bridge Architecture Specification, Rev 1.1, Dec 18, 1998,

The reality is that most quad Tulip boards were designed before 1999, and therefore act as I described.

Most of the experience is with quad Ethernet adapters, as there are few other common PCI boards with bus bridges.

Quoted 2 lines

I know for a fact that at least D-Link card mentioned in some reports utilizes all four INT# lines, because I have one.

The D-Link board is

Quoted 4 lines

Device 0 on a secondary bus will have its INTA# line connected to the INTA# line of the connector. Device 1 will have its INTA# line connected to INTB# of the connector. This sequence continues and then wraps around once INTD# has been assigned."

This seems to be what most x86 BIOSes assume (almost all BIOSes use the Intel reference code for PCI setup), even from 1995-era machines. But again, this does not match how _most_ of the quad boards are wired. Curiously, non-x86 machines usually work fine without the work-around, meaning that they have a different interpretation.

Signature · 4 lines
Donald Becker				[email protected]
Scyld Computing Corporation		http://www.scyld.com
410 Severn Ave. Suite 210		Scyld Beowulf cluster system
Annapolis MD 21403			410-990-9993