PCI init issues
Seems like Linux 2.4.x has PCI initialization issues when you start having funky PCI bus setups. Now before you point a finger at the BIOs please read on...
In other words Linux needs a lot of help from the bios and if it doesnt get it things get messed up. When the bios initializes _all_ the mp table or acpi stuff things work great.
For example Linux doesnt seem to be very smart about things like second level pci bridges...
Example:
------
]0;root@localhost:~[root@localhost root]# lspci -tv -[00]-+-00.0 Intel Corp. e7500 DRAM Controller
+-00.1 Intel Corp. e7500 DRAM Controller Error Reporting
+-02.0-[01-04]--+-1c.0 Intel Corp. 82870P2 P64H2 I/OxAPIC
| +-1d.0-[02]--
| +-1e.0 Intel Corp. 82870P2 P64H2 I/OxAPIC
| \-1f.0-[03-04]----01.0-[04]--+-04.0 DigitalEquipment Corporation DECchip 21142/43
| +-05.0 Digital
Equipment Corporation DECchip 21142/43
| +-06.0 Digital
Equipment Corporation DECchip 21142/43
| \-07.0 Digital
Equipment Corporation DECchip 21142/43
+-03.0-[05-07]--+-1c.0 Intel Corp. 82870P2 P64H2 I/OxAPIC
| +-1d.0-[06]--+-01.0 Intel Corp.: Unknown device100f
| | +-02.0 Intel Corp.: Unknown device
1010
| | \-02.1 Intel Corp.: Unknown device
1010
| +-1e.0 Intel Corp. 82870P2 P64H2 I/OxAPIC
| \-1f.0-[07]--
+-1d.0 Intel Corp. 82801CA/CAM USB (Hub
+-1d.1 Intel Corp. 82801CA/CAM USB (Hub
+-1d.2 Intel Corp. 82801CA/CAM USB (Hub
+-1e.0-[08]--+-01.0 ATI Technologies Inc Rage XL
| +-02.0 Intel Corp. 82557/8/9 [Ethernet Pro 100]
| \-03.0 Intel Corp. 82557/8/9 [Ethernet Pro 100]
+-1f.0 Intel Corp. 82801CA ISA Bridge (LPC)
+-1f.1 Intel Corp. 82801CA IDE U100
\-1f.3 Intel Corp. 82801CA/CAM SMBus-------------
See those DECchip 21142/43 devices? They are about 3 layers down from the root and theyll never work properly unless the BIOS is capable of initializing everything. The first of the 4 devices will receive interupts correctly, the rest dont.
-------
]0;root@localhost:~[root@localhost root]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 441198 441826 442361 439749 IO-APIC-edge timer 1: 1 0 0 1 IO-APIC-edge keyboard 2: 0 0 0 0 XT-PIC cascade 4: 75 78 82 52 IO-APIC-edge serial 8: 1 0 0 0 IO-APIC-edge rtc 14: 2000 1787 1960 1535 IO-APIC-edge ide0 15: 1860 1750 2021 1603 IO-APIC-edge ide1 17: 3396 3440 3400 3415 IO-APIC-level eth0 18: 6 2 2 3 IO-APIC-level eth1 24: 5 3 5 3 IO-APIC-level eth5, eth6, eth7, eth8 96: 426 424 425 424 IO-APIC-level eth2 100: 427 424 424 425 IO-APIC-level eth3 101: 425 424 425 423 IO-APIC-level eth4 NMI: 0 0 0 0 LOC: 1762876 1762850 1762971 1762970 ERR: 0 MIS: 0
--------------
Interupt routing as can be seen above is really messed for that device.
The workaround is to move the board so the the DECchip 21142/43 devices are at a depth not to exceed 3.
Seems this has been an ongoing problem on Linux.
http://www.tux.org/hypermail/linux-tulip/2002-Aug/0000.html
The above is the same motherboard as mentioned in the thread above except the two processors have hyperthreading capability.
Q1: How do we resolve this other than moving around boards?
Q2: Should we really be dependent on bios this bad?
According to that thread things dont work on the BSDs either but work fine on WinXP (shudder). Does this mean winxp doesnt depend on BIOS? Should we really be dependent on the BIOS this much?
Shouldnt things like second level pci bridge be part of Linux discovery?
cheers,
jamal
Quoted 1 line · jamal
Interupt routing as can be seen above is really messed for that device.
Indeed. Quad tulip cards usually have irq pin A of each chip routed to pins A-D on the connector, so they cannot have the same irq unless all four irq pins are shared in the parent slot, which is unlikely in this case.
Quoted 1 line
Q1: How do we resolve this other than moving around boards?
Here's [completely untested] patch that attempts to validate BIOS irq assignments. This may break "good" irqs for other devices, but it's interesting to see if it changes something.
Quoted 1 line
Q2: Should we really be dependent on bios this bad?
We certainly shouldn't wrt PCI IO and MMIO allocations, but irq routing appears to be so horribly overcomplexed on x86 that dependence on BIOS seems almost unavoidable here...
Other architectures don't have such kind of problems, BTW.
Ivan.
arch/i386/kernel/pci-irq.c +10 -6
@@ -730,7 +730,7 @@ void __init pcibios_fixup_irqs(void)*/ if (io_apic_assign_pci_irqs) { -int irq; +int irq, irq1 =-1 ;if (pin) { pin--;/* interrupt pins are numbered starting from 1 */
@@ -741,15 +741,19 @@ void __init pcibios_fixup_irqs(void)* parent slot, and pin number. The SMP code detects such bridged * busses itself so we should get into this branch reliably. */ -if (irq <0 && dev->bus->parent) {/* go back to the bridge */ +if (dev->bus->parent) {/* go back to the bridge */ struct pci_dev * bridge = dev->bus->self; pin = (pin +PCI_SLOT (dev->devfn)) %4 ; - irq =IO_APIC_get_PCI_irq_vector (bridge->bus->number, + irq1 =IO_APIC_get_PCI_irq_vector (bridge->bus->number,PCI_SLOT (bridge->devfn), pin); -if (irq >=0 ) -printk (KERN_WARNING "PCI: using PPB(B%d,I%d,P%d) to get irq %d\n" , - bridge->bus->number,PCI_SLOT (bridge->devfn), pin, irq); + } +if (irq1 >=0 && irq1 != irq) { +/* IRQ listed in MP-table doesn't match one for parent slot. +Use that we've found instead. */ + irq = irq1; +printk (KERN_WARNING "PCI: using PPB(B%d,I%d,P%d) to get irq %d\n" , + bridge->bus->number,PCI_SLOT (bridge->devfn), pin, irq); }if (irq >=0 ) { printk(KERN_INFO "PCI->APIC IRQ transform: (B%d,I%d,P%d) -> %d\n",
Quoted 6 lines · Ivan Kokshaysky
Quoted 1 line
Q2: Should we really be dependent on bios this bad?
We certainly shouldn't wrt PCI IO and MMIO allocations, but irq routing appears to be so horribly overcomplexed on x86 that dependence on BIOS seems almost unavoidable here...
Other architectures don't have such kind of problems, BTW.
Actually, for issues like that one Jamal is pointing out, I believe there is something we can do.
What varies so much from motherboard to motherboard on x86 is how the interrupts are wired up. So for example, this tells us that PCI slot X maps to x86 IRQ Y. If we know that, we can figure out where his deeply bridged tulips will send IRQs.
If the mp tables don't give exactly this kind of information, then Ivan is right :(
Anyone know if FreeBSD fares better in situations like this? What do they do if so?
Quoted 1 line · David S. Miller
Anyone know if FreeBSD fares better in situations like this?
All BSDs apparently have this problem. Ive heard rumors of windows XP working just fine with the same setup on these mboards. Unfortunately up north we have igloos so i cant verify these rumors ;->
cheers,
jamal
Ivan,
Patch was applied; no luck at all. Below is dmesg output with debug turned on in the pci-irq.c file.
--------
Mar 3 18:01:39 localhost kernel: PCI: PCI BIOS revision 2.10 entry at 0xfd875last bus=8 Mar 3 18:01:39 localhost kernel: PCI: Using configuration type 1 Mar 3 18:01:39 localhost kernel: PCI: Probing PCI hardware Mar 3 18:01:39 localhost kernel: PCI: Probing PCI hardware Mar 3 18:01:39 localhost kernel: Transparent bridge - Intel Corp. 82801BA/CA/ PCI Bridge Mar 3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 10 [IRQ] Mar 3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 11 [IRQ] Mar 3 18:01:39 localhost kernel: PCI: Discovered primary peer bus 12 [IRQ] Mar 3 18:01:39 localhost kernel: PCI: Using IRQ router PIIX [8086/2480] at 00f.0 Mar 3 18:01:39 localhost kernel: PCI: IRQ fixup Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P0) -> 16 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P1) -> 19 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B0,I29,P2) -> 18 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B2,I1,P1) -> 48 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B3,I1,P1) -> 24 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B3,I1,P2) -> 25 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I4,P0) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I5,P1) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I6,P2) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I7,P3) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I1,P1) -> 16 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I2,P2) -> 17 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B8,I3,P3) ->
18
---------------
BIOS irq assignments i suppose are validated.
I have a feeling the reason it works in windows is because of a functional ACPI.
What next? ;->
cheers,
jamal
Quoted 8 lines · jamal
Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I4,P0) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I5,P1) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I6,P2) -> 96 Mar 3 18:01:39 localhost kernel: PCI->APIC IRQ transform: (B6,I7,P3) ->96
...
Quoted 1 line
BIOS irq assignments i suppose are validated.
No, this means that mp tables are broken in a way I didn't expect... Also, it's unclear whether or not the IOxAPIC routing is broken as well. Probably full dmesg output (especially things related to IO_APICs setup and irq<->pin mapping) could shed some light on this.
Quoted 2 lines
I have a feeling the reason it works in windows is because of a functional ACPI.
Perhaps.
Ivan.
Quoted 5 lines · Ivan Kokshaysky
Quoted 1 line · jamal
Interupt routing as can be seen above is really messed for that device.
Indeed. Quad tulip cards usually have irq pin A of each chip routed to pins A-D on the connector, so they cannot have the same irq unless
Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts are sent on the INTA output of the board.
The tulip.c driver has worked around the BIOS bug on x86 machines for years. It should really be handled in the generic PCI quirks code, but has not yet been correctly handled in the kernel's code.
Signature · 4 lines
Donald Becker [email protected] Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993
Quoted 3 lines · Donald Becker
Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts are sent on the INTA output of the board.
This can be true for older cards, but post-1998 hardware must follow the spec.
PCI-to-PCI Bridge Architecture Specification, Rev 1.1, Dec 18, 1998, pp 113-114:
"... The interrupt binding defined in this table is mandatory for
expansion boards utilizing a bridge.
[table skipped]
Device 0 on a secondary bus will have its INTA# line connected to the INTA# line of the connector. Device 1 will have its INTA# line connected to INTB# of the connector. This sequence continues and then wraps around once INTD# has been assigned."
I know for a fact that at least D-Link card mentioned in some reports utilizes all four INT# lines, because I have one.
Ivan.
Hello!
Quoted 2 lines
I know for a fact that at least D-Link card mentioned in some reports utilizes all four INT# lines, because I have one.
I confirm, my Znyx card (pci subsys 110d:0029) is wired correctly to INTA/B/C/D and works nicely with plain flat motherboards.
Look at Jamal's mail: the cards work well when attached to primary bus. They stop to work when connected via one more bridge (which is common for server motherboards).
Alexey
Quoted 8 lines · Ivan Kokshaysky
Quoted 3 lines · Donald Becker
Incorrect.
Most quad Tulip boards have the bus bridge wired so that all interrupts are sent on the INTA output of the board.This can be true for older cards, but post-1998 hardware must follow the spec.
PCI-to-PCI Bridge Architecture Specification, Rev 1.1, Dec 18, 1998,
The reality is that most quad Tulip boards were designed before 1999, and therefore act as I described.
Most of the experience is with quad Ethernet adapters, as there are few other common PCI boards with bus bridges.
Quoted 2 lines
I know for a fact that at least D-Link card mentioned in some reports utilizes all four INT# lines, because I have one.
The D-Link board is
Quoted 4 lines
Device 0 on a secondary bus will have its INTA# line connected to the INTA# line of the connector. Device 1 will have its INTA# line connected to INTB# of the connector. This sequence continues and then wraps around once INTD# has been assigned."
This seems to be what most x86 BIOSes assume (almost all BIOSes use the Intel reference code for PCI setup), even from 1995-era machines. But again, this does not match how _most_ of the quad boards are wired. Curiously, non-x86 machines usually work fine without the work-around, meaning that they have a different interpretation.
Signature · 4 lines
Donald Becker [email protected] Scyld Computing Corporation http://www.scyld.com 410 Severn Ave. Suite 210 Scyld Beowulf cluster system Annapolis MD 21403 410-990-9993