PCI-PCI bridge scanning broken on 460EX

Hi,

I'm running linux-2.6.33-rc2 on Canyonlands board. When PLX 6254 transparent PCI-PCI
bridge is plugged into PCI slot the kernel simply resets the board without printing anything
to console. Without PLX bridge kernel boots fine.

I've tracked down the problem to the following code in pci_scan_bridge() in drivers/pci/probe.c:

if (pcibios_assign_all_busses() || broken)
/* Temporarily disable forwarding of the
configuration cycles on all bridges in this bus segment to avoid possible conflicts in the second pass between two bridges programmed with overlapping bus ranges. */
pci_write_config_dword(dev, PCI_PRIMARY_BUS,
buses & ~0xffffff);

If test for broken is removed, kernel boots fine, detects the bridge, but does not detect the device behind the bridge. The same device plugged directly into PCI slot is detected correctly.

To remind you, tests for broken were added by commit a1c19894b786f10c76ac40e93c6b5d70c9b946d2,
and were intended to solve device detection problem behind PCI-E switches, as discussed in this thread:
http://lists.ozlabs.org/pipermail/linuxppc-dev/2008-October/063939.html

PCI: Probing PCI hardware
pci_bus 0000:00: scanning bus
pci 0000:00:06.0: found [3388:0020] class 000604 header type 01 pci 0000:00:06.0: supports D1 D2
pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot pci 0000:00:06.0: PME# disabled
pci_bus 0000:00: fixups for bus
pci 0000:00:06.0: scanning behind bridge, config 000000, pass 0 pci 0000:00:06.0: bus configuration invalid, reconfiguring

pci 0000:00:06.0: scanning behind bridge, config 000000, pass 1

pci_bus 0000:01: scanning bus

pci 0000:01:06.0: found [3388:0020] class 000604 header type 01

pci 0000:01:06.0: supports D1 D2
pci 0000:01:06.0: PME# supported from D0 D1 D2 D3hot pci 0000:01:06.0: PME# disabled
pci_bus 0000:01: fixups for bus
pci 0000:00:06.0: PCI bridge to [bus 01-ff]

pci 0000:00:06.0:   bridge window [io  0x0000-0x0fff]
pci 0000:00:06.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0000:00:06.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]

pci 0000:01:06.0: scanning behind bridge, config ff0100, pass 0

pci 0000:01:06.0: bus configuration invalid, reconfiguring

pci 0000:01:06.0: scanning behind bridge, config ff0100, pass 1 pci_bus 0000:01: bus scan returning with max=01 pci_bus 0000:00: bus scan returning with max=01

Quoted 6 lines · Felix Radensky

Hi,

I'm running linux-2.6.33-rc2 on Canyonlands board. When PLX 6254 transparent PCI-PCI bridge is plugged into PCI slot the kernel simply resets the board without printing anything to console. Without PLX bridge kernel boots fine.

Sorry for the late reply...

Quoted 13 lines

I've tracked down the problem to the following code in pci_scan_bridge() in drivers/pci/probe.c:

if (pcibios_assign_all_busses() || broken)
/* Temporarily disable forwarding of the
configuration cycles on all bridges in this bus segment to avoid possible conflicts in the second pass between two bridges programmed with overlapping bus ranges. */
pci_write_config_dword(dev, PCI_PRIMARY_BUS,
buses & ~0xffffff);

Quoted 4 lines

If test for broken is removed, kernel boots fine, detects the bridge, but does not detect the device behind the bridge. The same device plugged directly into PCI slot is detected correctly.

So we would have a similar mismatch between the initial setup and the kernel... However, I don't quite see yet why the kernel trying to fix it up breaks things, that will need a bit more debugging here...

Can you give it a quick try with adding something like :

ppc_pci_add_flags(PPC_PCI_REASSIGN_ALL_BUS);

Near the end of ppc4xx_pci.c ? It looks like another case of reset not actually resetting bridges (are we not properly doing a fundamental reset ? Stefan what's your take there ?)

The above will cause busses to be re-assigned which is risky because it will allow the kernel to assign numbers beyond the limits of what ppc4xx_pci.c supports (see my comments in the thread you quotes).

The good thing is that we now have a working fixmap infrastructure, so we could/should just move ppc4xx_pci.c to use that, and just always re-assign busses.

Quoted 6 lines

To remind you, tests for broken were added by commit a1c19894b786f10c76ac40e93c6b5d70c9b946d2, and were intended to solve device detection problem behind PCI-E switches, as discussed in this thread:
http://lists.ozlabs.org/pipermail/linuxppc-dev/2008-October/063939.html

Quoted 10 lines

PCI: Probing PCI hardware
pci_bus 0000:00: scanning bus
pci 0000:00:06.0: found [3388:0020] class 000604 header type 01 pci 0000:00:06.0: supports D1 D2
pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot pci 0000:00:06.0: PME# disabled
pci_bus 0000:00: fixups for bus
pci 0000:00:06.0: scanning behind bridge, config 000000, pass 0 pci 0000:00:06.0: bus configuration invalid, reconfiguring

Ok so we hit a P2P bridge whose primary, secondary and subordinate bus numbers are all 0, which is clearly unconfigured. I think this is the root complex bridge

Quoted 2 lines

pci 0000:00:06.0: scanning behind bridge, config 000000, pass 1

Now this is when the bus should be reconfigured (pass 1). Sadly the code doesn't print much debug.

Also from that point, it should renumber things and work...

Quoted 2 lines

pci_bus 0000:01: scanning bus

Which it does to some extent. It assigned bus number 1 to it afaik so we now start looking below the RC bridge:

Quoted 2 lines

pci 0000:01:06.0: found [3388:0020] class 000604 header type 01

Hrm... class PCI bridge, vendor 3388 device 0020, is that your PLX ? It's not the right vendor ID but maybe that's configurable by our OEM or something...

Quoted 11 lines

pci 0000:01:06.0: supports D1 D2
pci 0000:01:06.0: PME# supported from D0 D1 D2 D3hot pci 0000:01:06.0: PME# disabled
pci_bus 0000:01: fixups for bus
pci 0000:00:06.0: PCI bridge to [bus 01-ff]

pci 0000:00:06.0:   bridge window [io  0x0000-0x0fff]
pci 0000:00:06.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0000:00:06.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]

pci 0000:01:06.0: scanning behind bridge, config ff0100, pass 0

Allright, that's where it gets interesting. It tries to scan behind the bridge. It gets something it doesn't like. IE, it gets a secondary bus number of 1 (what the heck ? I wonder what your firmware does) which Linux is not happy about and decides to renumber it.

Quoted 3 lines

pci 0000:01:06.0: bus configuration invalid, reconfiguring

Now, that's where Linux should have written 000000 to the register, which is what you commented out.

Quoted 4 lines

pci 0000:01:06.0: scanning behind bridge, config ff0100, pass 1 pci_bus 0000:01: bus scan returning with max=01 pci_bus 0000:00: bus scan returning with max=01

Because of that commenting out, it doesn't see the config as 000000 and thus doesn't re-assign a bus number in pass 1, so from there you can't see what's behind the bus.

So we have two things here:

- It seems like the writing of 000000 to the register in pass 0 is
causing your crash. Can you verify that ? IE. Can you verify that it's indeed crashing on this specific statement:

pci_write_config_dword(dev, PCI_PRIMARY_BUS,

                               buses & ~0xffffff);

When writing to the bridge, and that this seems to be causing a hard reboot of the system ?

It might be useful to ask AMCC how that is possible in HW, ie what kind of signal can be causing that. IE, even if the bridge is causing a PCIe error, that should not cause a reboot ... right ?

- You can test a quick hack workaround which consists of changing:

	/* Check if setup is sensible at all */

(quoted diff) +0 -2

-	if (!pass &&
-	if (1 &&

	    ((buses & 0xff) != bus->number || ((buses >> 8) & 0xff) <= bus->number)) {
		dev_dbg(&dev->dev, "bus configuration invalid, reconfiguring\n");
		broken = 1;
	}

In -addition- to your commenting out of the broken test. This will cause the second pass to go through the re-assign code path despite the fact that you have not written 000000 to the bus numbers.

Hi, Ben

Adding Feng Kan from AMCC to CC.

Quoted 11 lines · Benjamin Herrenschmidt

Quoted 6 lines · Felix Radensky

Hi,

I'm running linux-2.6.33-rc2 on Canyonlands board. When PLX 6254 transparent PCI-PCI bridge is plugged into PCI slot the kernel simply resets the board without printing anything to console. Without PLX bridge kernel boots fine.

Sorry for the late reply...

No need to apologize, I appreciate you help very much.

Quoted 76 lines

Quoted 13 lines

I've tracked down the problem to the following code in pci_scan_bridge() in drivers/pci/probe.c:

if (pcibios_assign_all_busses() || broken)
/* Temporarily disable forwarding of the
configuration cycles on all bridges in this bus segment to avoid possible conflicts in the second pass between two bridges programmed with overlapping bus ranges. */
pci_write_config_dword(dev, PCI_PRIMARY_BUS,
buses & ~0xffffff);

Quoted 4 lines

If test for broken is removed, kernel boots fine, detects the bridge, but does not detect the device behind the bridge. The same device plugged directly into PCI slot is detected correctly.

So we would have a similar mismatch between the initial setup and the kernel... However, I don't quite see yet why the kernel trying to fix it up breaks things, that will need a bit more debugging here...

Can you give it a quick try with adding something like :

ppc_pci_add_flags(PPC_PCI_REASSIGN_ALL_BUS);

Near the end of ppc4xx_pci.c ? It looks like another case of reset not actually resetting bridges (are we not properly doing a fundamental reset ? Stefan what's your take there ?)

The above will cause busses to be re-assigned which is risky because it will allow the kernel to assign numbers beyond the limits of what ppc4xx_pci.c supports (see my comments in the thread you quotes).

The good thing is that we now have a working fixmap infrastructure, so we could/should just move ppc4xx_pci.c to use that, and just always re-assign busses.

Quoted 6 lines

To remind you, tests for broken were added by commit a1c19894b786f10c76ac40e93c6b5d70c9b946d2, and were intended to solve device detection problem behind PCI-E switches, as discussed in this thread:
http://lists.ozlabs.org/pipermail/linuxppc-dev/2008-October/063939.html

Quoted 10 lines

PCI: Probing PCI hardware
pci_bus 0000:00: scanning bus
pci 0000:00:06.0: found [3388:0020] class 000604 header type 01 pci 0000:00:06.0: supports D1 D2
pci 0000:00:06.0: PME# supported from D0 D1 D2 D3hot pci 0000:00:06.0: PME# disabled
pci_bus 0000:00: fixups for bus
pci 0000:00:06.0: scanning behind bridge, config 000000, pass 0 pci 0000:00:06.0: bus configuration invalid, reconfiguring

Ok so we hit a P2P bridge whose primary, secondary and subordinate bus numbers are all 0, which is clearly unconfigured. I think this is the root complex bridge

Quoted 2 lines

pci 0000:00:06.0: scanning behind bridge, config 000000, pass 1

Now this is when the bus should be reconfigured (pass 1). Sadly the code doesn't print much debug.

Also from that point, it should renumber things and work...

Quoted 2 lines

pci_bus 0000:01: scanning bus

Which it does to some extent. It assigned bus number 1 to it afaik so we now start looking below the RC bridge:

Quoted 7 lines

Quoted 2 lines

pci 0000:01:06.0: found [3388:0020] class 000604 header type 01

Hrm... class PCI bridge, vendor 3388 device 0020, is that your PLX ? It's not the right vendor ID but maybe that's configurable by our OEM or something...

The bridge and its evaluation board were manufactured by HiNT, later purchased by PLX. 3388:0020 is HiNT HB6 Universal PCI-PCI bridge in transparent mode.

Quoted 12 lines

Quoted 11 lines

pci 0000:01:06.0: supports D1 D2
pci 0000:01:06.0: PME# supported from D0 D1 D2 D3hot pci 0000:01:06.0: PME# disabled
pci_bus 0000:01: fixups for bus
pci 0000:00:06.0: PCI bridge to [bus 01-ff]

pci 0000:00:06.0:   bridge window [io  0x0000-0x0fff]
pci 0000:00:06.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0000:00:06.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]

pci 0000:01:06.0: scanning behind bridge, config ff0100, pass 0

Quoted 5 lines

Allright, that's where it gets interesting. It tries to scan behind the bridge. It gets something it doesn't like. IE, it gets a secondary bus number of 1 (what the heck ? I wonder what your firmware does) which Linux is not happy about and decides to renumber it.

U-boot has problems with this bridge as well, so I had to completely disable PCI support in u-boot to get linux running.

Quoted 23 lines

Quoted 3 lines

pci 0000:01:06.0: bus configuration invalid, reconfiguring

Now, that's where Linux should have written 000000 to the register, which is what you commented out.

Quoted 4 lines

pci 0000:01:06.0: scanning behind bridge, config ff0100, pass 1 pci_bus 0000:01: bus scan returning with max=01 pci_bus 0000:00: bus scan returning with max=01

Because of that commenting out, it doesn't see the config as 000000 and thus doesn't re-assign a bus number in pass 1, so from there you can't see what's behind the bus.

So we have two things here:

- It seems like the writing of 000000 to the register in pass 0 is
causing your crash. Can you verify that ? IE. Can you verify that it's indeed crashing on this specific statement:

Quoted 6 lines

pci_write_config_dword(dev, PCI_PRIMARY_BUS,

                               buses & ~0xffffff);

When writing to the bridge, and that this seems to be causing a hard reboot of the system ?

Yes, this particular statement causes hard reboot. With original broken tests restored and writing to bridge commented out the system boots. If writing to bridge happens I get hard reset.

Quoted 4 lines

It might be useful to ask AMCC how that is possible in HW, ie what kind of signal can be causing that. IE, even if the bridge is causing a PCIe error, that should not cause a reboot ... right ?

Feng, can you please comment on this ?

Quoted 12 lines

- You can test a quick hack workaround which consists of changing:

    /* Check if setup is sensible at all */

(quoted diff) +0 -2

-    if (!pass &&
-    if (1 &&
        ((buses & 0xff) != bus->number || ((buses >> 8) & 0xff) <= 

bus->number)) {
        dev_dbg(&dev->dev, "bus configuration invalid, 
reconfiguring\n");
        broken = 1;
    }

Quoted 4 lines

In -addition- to your commenting out of the broken test. This will cause the second pass to go through the re-assign code path despite the fact that you have not written 000000 to the bus numbers.

With this change and commented out broken test I still get hard reset.

I didn't try adding ppc_pci_add_flags(PPC_PCI_REASSIGN_ALL_BUS) If you still want me to try this, please let me know. Should I leave broken tests enabled in that case ?

Thanks a lot for your help.

Felix.

I now have a custom board with 460EX and the same PLX bridge, running 2.6.23-rc3
Things look better here, as u-boot is now able to properly detect PLX and device behind
it, but kernel still has problems. First, I'm still getting hard reset on

pci_write_config_dword(dev, PCI_PRIMARY_BUS,

                               buses & ~0xffffff);

If this line is removed, PLX is detected twice, see below. I also get hard reset
if pass test is modified as you requested and broken test removed.

Any ideas how to fix this ? I was suspecting PLX evaluation board, but PLX on our custom board seems to be OK, so it looks like kernel needs fixing.

Quoted 4 lines · Felix Radensky

I now have a custom board with 460EX and the same PLX bridge, running 2.6.23-rc3 Things look better here, as u-boot is now able to properly detect PLX and device behind it, but kernel still has problems. First, I'm still getting hard reset on

Quoted 9 lines

pci_write_config_dword(dev, PCI_PRIMARY_BUS,

                               buses & ~0xffffff);

If this line is removed, PLX is detected twice, see below. I also get hard reset if pass test is modified as you requested and broken test removed.

Any ideas how to fix this ? I was suspecting PLX evaluation board, but PLX on our custom board seems to be OK, so it looks like kernel needs fixing.

I have no idea no. It looks like something is wrong with the PLX bridge but again, I don't know why that would cause the 460EX to hard reset like that, unless some of the PCIe error handling of the 460 has been configured to cause such a reset on some kind of errors (which it shouldn't at least not in host mode).

Can you try instead of writing all the bus number related registers in one single dword write above, writing them byte by byte ? Which one is causing the reset ? Does it reset whatever the value you write there is ?

It looks like something is causing a hard reset as soon as you try to configure the PLX bridge and without configuring it properly I fail to see how you'll get things working.

OK, I'll try writing byte by byte. The funny thing is the u-boot also writes the
same value to PCI_PRIMARY_BUS register and it doesn't cause reset.

=20OK,=20I'll=20try=20writing=20byte=20by=20byte.=20The=20funny=20thing=

=20writes=20the=20same=20value=20to=20PCI_PRIMARY_BUS=20register=20and=

=20reset.

Hello Felix,

I had a problem similar to this on the 440GX, the PCI code was not sending type 1 transactions when scanning behind bridges. Perhaps you could try this:

Index: linux/arch/powerpc/sysdev/ppc4xx_pci.c
===================================================================

arch/powerpc/sysdev/ppc4xx_pci.c (revision 27) renamed from arch/powerpc/sysdev/ppc4xx_pci.c (revision 26) +1 -1

@@ -569,7 +569,7 @@
        hose->last_busno = bus_range ? bus_range[1] : 0xff;
 
        /* Setup config space */
-       setup_indirect_pci(hose, rsrc_cfg.start, rsrc_cfg.start + 0x4,
 0);
+       setup_indirect_pci(hose, rsrc_cfg.start, rsrc_cfg.start + 0x4,
 PPC_INDIRECT_TYPE_SET_CFG_TYPE);
 

        /* Disable all windows */
        writel(0, reg + PCIX0_POM0SA);

With kind regards / Met vriendelijke groet,

Stef van Os

Prodrive B.V.

Hi Stef,

Quoted 26 lines · Stef van Os

Hello Felix,

I had a problem similar to this on the 440GX, the PCI code was not sending type 1 transactions when scanning behind bridges. Perhaps you could try this:

Index: linux/arch/powerpc/sysdev/ppc4xx_pci.c
===================================================================

arch/powerpc/sysdev/ppc4xx_pci.c (revision 27) renamed from arch/powerpc/sysdev/ppc4xx_pci.c (revision 26) +1 -1

@@ -569,7 +569,7 @@
        hose->last_busno = bus_range ? bus_range[1] : 0xff;
 
        /* Setup config space */
-       setup_indirect_pci(hose, rsrc_cfg.start, rsrc_cfg.start + 0x4,
 0);
+       setup_indirect_pci(hose, rsrc_cfg.start, rsrc_cfg.start + 0x4,
 PPC_INDIRECT_TYPE_SET_CFG_TYPE);
 

        /* Disable all windows */
        writel(0, reg + PCIX0_POM0SA);

With kind regards / Met vriendelijke groet,

Quoted 5 lines

Stef van Os

Prodrive B.V.

I think you patch is a valid one, and should be applied, but unfortunately it doesn't fix by problem. BTW, in u-boot transaction type bit is set correctly.

It seems I was wrong. I've manually applied the patch at the wrong place. After patching the correct function
I'm not getting hard resets any more, which is a great improvement ! Thanks a lot, I really appreciate your help !

Unfortunately not all problems are gone. PLX is now identified correctly, but device behind it is not detected, although u-boot detects it correctly. See below.

pci 0000:01:02.0: scanning behind bridge, config 010100, pass 0 pci 0000:01:02.0: bus configuration invalid, reconfiguring pci 0000:01:02.0: scanning behind bridge, config 010100, pass 1 pci_bus 0000:01: bus scan returning with max=01
pci 0000:00:02.0: scanning behind bridge, config 010100, pass 1 pci_bus 0000:00: bus scan returning with max=01

pci 0000:00:02.0: disabling bridge window [mem 0xd80000000-0xd8c0fffff] to [bus 01-01] (unused)
pci 0000:00:02.0: PCI bridge to [bus 01-01]

pci 0000:00:02.0:   bridge window [io  disabled]
pci 0000:00:02.0:   bridge window [mem disabled]
pci 0000:00:02.0:   bridge window [mem pref disabled]

pci_bus 0000:00: resource 0 [io 0x0000-0xffff]
pci_bus 0000:00: resource 1 [mem 0xd80000000-0xdffffffff] pci_bus 0000:01: resource 1 [??? 57982058496-58184433663 flags 0x0]

Any ideas what could be wrong now ?

Thanks a lot.

Felix.

Quoted 3 lines

It seems I was wrong. I've manually applied the patch at the wrong place. After patching the correct function I'm not getting hard resets any more, which is a great improvement ! Thanks a lot, I really appreciate your help !

This is somewhat funny... I wonder how it would have managed to find anything behind the root complex P2P bridge with broken type 1 cycles... very very strange.

Quoted 4 lines

Unfortunately not all problems are gone. PLX is now identified correctly, but device behind it is not detected, although u-boot detects it correctly. See below.

You have removed all your changes to that code right ?

Also the log still looks weird:

Quoted 7 lines

pci 0000:01:02.0: scanning behind bridge, config 010100, pass 0 pci 0000:01:02.0: bus configuration invalid, reconfiguring pci 0000:01:02.0: scanning behind bridge, config 010100, pass 1 pci_bus 0000:01: bus scan returning with max=01
pci 0000:00:02.0: scanning behind bridge, config 010100, pass 1 pci_bus 0000:00: bus scan returning with max=01

Unless you left some experimental changes in, the above isn't right, the "config" value should have changed due to the write of ~ffffff to it

Maybe because the bus behind root P2P bridge is bus 0, and type 1 cycles are
needed for bus numbers greater than 0. That's what 460EX manual says.

You are correct, the log is from older version, without Stef's fix. I don't have access to a
system with devices behind PLX, and the guy who did the testing used wrong kernel.
I'll make sure he uses the correct one and get back to you. Maybe everything works after all :)
I'm really sorry for confusion.

Quoted 4 lines · Felix Radensky

Maybe because the bus behind root P2P bridge is bus 0, and type 1 cycles are
needed for bus numbers greater than 0. That's what 460EX manual says.

Well, no... the bus behind the root P2P is bus 1 ... the root P2P itself is on bus 0... but then, it's some trick in the way they implemented it I suppose.

Quoted 5 lines

You are correct, the log is from older version, without Stef's fix. I don't have access to a system with devices behind PLX, and the guy who did the testing used wrong kernel. I'll make sure he uses the correct one and get back to you. Maybe everything works after all :) I'm really sorry for confusion.

No worries :-) Feel free to send a proper patch to fix that problem upstream too !

=20On=20Tue,=202010-01-12=20at=2000:48=20+0200,=20Felix=20Radensky=20wro=

=20=20=20

Quoted 1 line

=20Maybe=20because=20the=20bus=20behind=20root=20P2P=20bridge=20is=20bu=

Quoted 1 line

=20cycles=20are=20needed=20for=20bus=20numbers=20greater=20than=200.=20=

Quoted 2 lines

=20manual=20says.
=20=20=20=20=20

=20Well,=20no...=20the=20bus=20behind=20the=20root=20P2P=20is=20bus=201=

=20itself=20is=20on=20bus=200...=20but=20then,=20it's=20some=20trick=20i=

=20implemented=20it=20I=20suppose.

=20=20=20

Quoted 1 line

=20You=20are=20correct,=20the=20log=20is=20from=20older=20version,=20wi=

Quoted 1 line

=20don't=20have=20access=20to=20a=20system=20with=20devices=20behind=20=

Quoted 2 lines

=20who=20did=20the=20testing=20used=20wrong=20kernel. =20I'll=20make=20sure=20he=20uses=20the=20correct=20one=20and=20get=20b=

Quoted 1 line

=20everything=20works=20after=20all=20:)=20I'm=20really=20sorry=20for=

Quoted 1 line

=20=20=20=20=20

=20No=20worries=20:-)=20Feel=20free=20to=20send=20a=20proper=20patch=20t=

=20upstream=20too=20!

=20=20=20