pci: mvebu: disable prefetchable memory support in PCI-to-PCI bridge

The Marvell PCIe driver uses an emulated PCI-to-PCI bridge to be able to dynamically set up MBus address decoding windows for PCI I/O and memory regions depending on the PCI devices enumerated by Linux.

However, this emulated PCI-to-PCI bridge logic makes the Linux PCI core believe that prefetchable memory regions are supported (because the registers are read/write), while in fact no adress decoding window is ever created for such regions. Since the Marvell MBus address decoding windows do not distinguish memory regions and prefetchable memory regions, this patch takes a simple approach: change the PCI-to-PCI bridge emulation to let the Linux PCI core know that we don't support prefetchable memory regions.

To achieve this, we simply make the prefetchable memory base a read-only register that always returns 0. Reading/writing all the other prefetchable memory related registers has no effect.

This problem was originally reported by Finn Hoffmann <[email protected]>, who couldn't get a RTL8111/8168B PCI NIC working on the NSA310 Kirkwood platform after updating to 3.11-rc. The problem was that the PCI-to-PCI bridge emulation was making the Linux PCI core believe that we support prefetchable memory, so the Linux PCI core was only filling the prefetchable memory base and limit registers, which does not lead to a MBus window being created. The below patch has been confirmed by Finn Hoffmann to fix his problem on Kirkwood, and has otherwise been successfully tested on the Armada XP GP platform with a e1000e PCIe NIC and a Marvell SATA PCIe card.

Signed-off-by Thomas Petazzoni <[email protected]>

Reported-by Finn Hoffmann <[email protected]>

Signed-off-by Thomas Petazzoni <[email protected]>

---
This patch fixes a regression introduced in 3.11-rc, so it should be applied to 3.11.

---
 drivers/pci/host/pci-mvebu.c | 27 +--------------------------
 1 file changed, 1 insertion(+), 26 deletions(-)

drivers/pci/host/pci-mvebu.c +1 -27

index 13a633b..7bf3926 100644
@@ -86,10 +86,6 @@ struct mvebu_sw_pci_bridge {
        u16 secondary_status;
        u16 membase;
        u16 memlimit;
-       u16 prefmembase;
-       u16 prefmemlimit;
-       u32 prefbaseupper;
-       u32 preflimitupper;
        u16 iobaseupper;
        u16 iolimitupper;
        u8 cappointer;

@@ -419,15 +415,7 @@ static int mvebu_sw_pci_bridge_read(struct mvebu_pcie_port *port,
                break;
 
        case PCI_PREF_MEMORY_BASE:
-               *value = (bridge->prefmemlimit << 16 | bridge->prefmembase);
-               break;
-
-       case PCI_PREF_BASE_UPPER32:
-               *value = bridge->prefbaseupper;
-               break;
-
-       case PCI_PREF_LIMIT_UPPER32:
-               *value = bridge->preflimitupper;
+               *value = 0;
                break;
 
        case PCI_IO_BASE_UPPER16:

@@ -501,19 +489,6 @@ static int mvebu_sw_pci_bridge_write(struct mvebu_pcie_port *port,
                mvebu_pcie_handle_membase_change(port);
                break;
 
-       case PCI_PREF_MEMORY_BASE:
-               bridge->prefmembase = value & 0xffff;
-               bridge->prefmemlimit = value >> 16;
-               break;
-
-       case PCI_PREF_BASE_UPPER32:
-               bridge->prefbaseupper = value;
-               break;
-
-       case PCI_PREF_LIMIT_UPPER32:
-               bridge->preflimitupper = value;
-               break;
-
        case PCI_IO_BASE_UPPER16:
                bridge->iobaseupper = value & 0xffff;
                bridge->iolimitupper = value >> 16;
--

1.8.1.2

Quoted 28 lines · Thomas Petazzoni

The Marvell PCIe driver uses an emulated PCI-to-PCI bridge to be able to dynamically set up MBus address decoding windows for PCI I/O and memory regions depending on the PCI devices enumerated by Linux.

However, this emulated PCI-to-PCI bridge logic makes the Linux PCI core believe that prefetchable memory regions are supported (because the registers are read/write), while in fact no adress decoding window is ever created for such regions. Since the Marvell MBus address decoding windows do not distinguish memory regions and prefetchable memory regions, this patch takes a simple approach: change the PCI-to-PCI bridge emulation to let the Linux PCI core know that we don't support prefetchable memory regions.

To achieve this, we simply make the prefetchable memory base a read-only register that always returns 0. Reading/writing all the other prefetchable memory related registers has no effect.

This problem was originally reported by Finn Hoffmann <[email protected]>, who couldn't get a RTL8111/8168B PCI NIC working on the NSA310 Kirkwood platform after updating to 3.11-rc. The problem was that the PCI-to-PCI bridge emulation was making the Linux PCI core believe that we support prefetchable memory, so the Linux PCI core was only filling the prefetchable memory base and limit registers, which does not lead to a MBus window being created. The below patch has been confirmed by Finn Hoffmann to fix his problem on Kirkwood, and has otherwise been successfully tested on the Armada XP GP platform with a e1000e PCIe NIC and a Marvell SATA PCIe card.

Quoted 3 lines

Signed-off-by Thomas Petazzoni <[email protected]>

Reported-by Finn Hoffmann <[email protected]>

Signed-off-by Thomas Petazzoni <[email protected]>

Acked-by Bjorn Helgaas <[email protected]>

Please merge along with your other pci-mvebu.c changes.

Dear Bjorn Helgaas,

Quoted 36 lines · Bjorn Helgaas

Quoted 28 lines · Thomas Petazzoni

The Marvell PCIe driver uses an emulated PCI-to-PCI bridge to be able to dynamically set up MBus address decoding windows for PCI I/O and memory regions depending on the PCI devices enumerated by Linux.

However, this emulated PCI-to-PCI bridge logic makes the Linux PCI core believe that prefetchable memory regions are supported (because the registers are read/write), while in fact no adress decoding window is ever created for such regions. Since the Marvell MBus address decoding windows do not distinguish memory regions and prefetchable memory regions, this patch takes a simple approach: change the PCI-to-PCI bridge emulation to let the Linux PCI core know that we don't support prefetchable memory regions.

To achieve this, we simply make the prefetchable memory base a read-only register that always returns 0. Reading/writing all the other prefetchable memory related registers has no effect.

This problem was originally reported by Finn Hoffmann <[email protected]>, who couldn't get a RTL8111/8168B PCI NIC working on the NSA310 Kirkwood platform after updating to 3.11-rc. The problem was that the PCI-to-PCI bridge emulation was making the Linux PCI core believe that we support prefetchable memory, so the Linux PCI core was only filling the prefetchable memory base and limit registers, which does not lead to a MBus window being created. The below patch has been confirmed by Finn Hoffmann to fix his problem on Kirkwood, and has otherwise been successfully tested on the Armada XP GP platform with a e1000e PCIe NIC and a Marvell SATA PCIe card.

Quoted 3 lines

Signed-off-by Thomas Petazzoni <[email protected]>

Reported-by Finn Hoffmann <[email protected]>

Signed-off-by Thomas Petazzoni <[email protected]>

Acked-by Bjorn Helgaas <[email protected]>

Quoted 1 line

Please merge along with your other pci-mvebu.c changes.

I don't think we have other pci-mvebu.c lined up for 3.11. All the MSI stuff, mvebu-mbus DT stuff and so on is for 3.12.

So I guess you could take that one through your pci tree. Jason, can you confirm?

The Marvell PCIe driver uses an emulated PCI-to-PCI bridge to be able to dynamically set up MBus address decoding windows for PCI I/O and memory regions depending on the PCI devices enumerated by Linux.

However, this emulated PCI-to-PCI bridge logic makes the Linux PCI core believe that prefetchable memory regions are supported (because the registers are read/write), while in fact no adress decoding window is ever created for such regions. Since the Marvell MBus address decoding windows do not distinguish memory regions and prefetchable memory regions, this patch takes a simple approach: change the PCI-to-PCI bridge emulation to let the Linux PCI core know that we don't support prefetchable memory regions.

To achieve this, we simply make the prefetchable memory base a read-only register that always returns 0. Reading/writing all the other prefetchable memory related registers has no effect.

Quoted 3 lines

To achieve this, we simply make the prefetchable memory base a read-only register that always returns 0. Reading/writing all the other prefetchable memory related registers has no effect.

Looks good to me. It is very good that you were able to merge the downstream prefetchable BARs into the normal MMIO window.

Though looking at the original thread I really wonder if something else is wrong here as well. The ethernet should not have only prefetchable BARs.

For instance, I found this link:

https://bugzilla.redhat.com/show_bug.cgi?id=448712

Which shows a more resonable arrangement:

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 02)
Region 0: I/O ports at e800 [size=256]
Region 2: Memory at dffff000 (64-bit, non-prefetchable) [size=4K]
Region 4: Memory at deff0000 (64-bit, prefetchable) [size=64K]
Expansion ROM at dffc0000 [disabled] [size=128K]

vs Finn's:

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev)

         Subsystem: Realtek Semiconductor Co., Ltd. TEG-ECTX Gigabit PCI-E Adapter [Trendnet]
         Flags: bus master, fast devsel, latency 0, IRQ 9
         I/O ports at 10000 [size=256]
         Memory at e0014000 (64-bit, prefetchable) [size=4K]
         Memory at e0010000 (64-bit, prefetchable) [size=16K]