Skip to contentSkip to conversation

Marvell PCIe driver improvements

PATCH Superseded Thomas Petazzoni 3 participants 8 messages May 23, 2013 PCI

lore
Share & tools
lore
#0 lore raw

v2 cover Marvell PCIe driver improvements

Bjorn, Jason, Andrew, Gregory,

Here is an updated version of the PCIe improvements.

Changes since v1:

* Fix some typos in the comments of PATCH 2, noticed by Bjorn Helgaas.

* Rework the two last patches (first one was fixing the capability list, the other one the emulation of the devsel bits in the status register), after the comments of Bjorn. In fact, it turns out that the status register is composed of read-only bits for which we want to emulate a value of zero, and the other bits are write-1-to-clear bits that expose error conditions that we do not handle in the emulated bridge. So, in summary, emulating a 0 status register on reads, and ignoring writes to the status register is the simplest solution. If we later on implement a capability list, we can use a more clever emulation of the status register, but for the moment, this simple solution is enough. Therefore, the two last patches are now one single, simpler patch.

I'd like those improvements to be kept separated from the original PCIe driver itself: while the PCIe driver has been around and reviewed for a long time, those improvements are newer. And I clearly do not want the PCIe driver to miss 3.11 because of any problem that could be found in those additional improvements.

The main improvement being brought here is that the PCI-to-PCI bridge logic is fixed/extended to properly support physical PCIe bridges that are connected on a PCIe interface of a Marvell board. Without this improvement, only the devices connected directly to the PCIe interfaces of the board are properly enumerated. Any device that would sit beyond a physical bridge is not visible.

Bjorn, with your Acked-by, could the Marvell maintainers include those patches in their branch, merged through arm-soc? They already have the Marvell PCIe driver itself, so I believe it makes sense to merge those improvements through the same path.

Jason, those patches have been prepared on top of my marvell-pcie-v10 branch, I hope that's ok for you. If you want me to rebase them on some other branch in which you have integrated the PCIe driver, don't hesitate to tell me to do so. That said, since those patches are only touching the driver itself and no other file, they should not cause any conflict with other changes.

Thanks,

Thomas

Thomas Petazzoni (3):
pci: mvebu: no longer fake the slot location of downstream devices
pci: mvebu: allow the enumeration of devices beyond physical bridges
pci: mvebu: fix the emulation of the status register

 drivers/pci/host/pci-mvebu.c |   55 +++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)
Signature · 1 line
1.7.9.5
#1 lore raw

PATCH v2 1/3 pci: mvebu: no longer fake the slot location of downstream devices

By default, the Marvell hardware, for each PCIe interface, exhibits the following devices:

* On slot 0, a "Marvell Memory controller", identical on all PCIe interfaces, and which isn't useful when the Marvell SoC is the PCIe root complex (i.e, the normal case when we run Linux on the Marvell SoC).

* On slot 1, the real PCIe card connected into the PCIe slot of the board.

So, what the Marvell PCIe driver was doing in its PCI-to-PCI bridge emulation is that when the Linux PCI core was trying to access the device in slot 0, we were in fact forwarding the configuration transaction to the device in slot 1. For all other slots, we were telling the Linux PCI core that there was no device connected.

However, new versions of bootloaders from Marvell change the default PCIe configuration, and make the real device appear in slot 0, and the "Marvell Memory controller" in slot 1.

Therefore, this commit modifies the Marvell PCIe driver to adjust the PCIe hardware configuration to make sure that this behavior (real device in slot 0, "Marvell Memory controller" in slot 1) is the one we'll see regardless of what the bootloader has done. It allows to remove the little hack that was forwarding configuration transactions on slot 0 to slot 1, which is nice.

---
 drivers/pci/host/pci-mvebu.c |   19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)
drivers/pci/host/pci-mvebu.c +15 -4
index ad1c46b..0bc21b0 100644
@@ -51,6 +51,7 @@
 #define  PCIE_CTRL_X1_MODE		0x0001
 #define PCIE_STAT_OFF		0x1a04
 #define  PCIE_STAT_BUS                  0xff00
+#define  PCIE_STAT_DEV                  0x1f0000
 #define  PCIE_STAT_LINK_DOWN		BIT(0)
 #define PCIE_DEBUG_CTRL         0x1a60
 #define  PCIE_DEBUG_SOFT_RESET		BIT(20)
@@ -148,6 +149,16 @@ static void mvebu_pcie_set_local_bus_nr(struct mvebu_pcie_port *port, int nr)
 	writel(stat, port->base + PCIE_STAT_OFF);
 }
 
+static void mvebu_pcie_set_local_dev_nr(struct mvebu_pcie_port *port, int nr)
+{
+	u32 stat;
+
+	stat = readl(port->base + PCIE_STAT_OFF);
+	stat &= ~PCIE_STAT_DEV;
+	stat |= nr << 16;
+	writel(stat, port->base + PCIE_STAT_OFF);
+}
+
 /*
  * Setup PCIE BARs and Address Decode Wins:
  * BAR[0,2] -> disabled, BAR[1] -> covers all DRAM banks
@@ -572,8 +583,7 @@ static int mvebu_pcie_wr_conf(struct pci_bus *bus, u32 devfn,
 
 	/* Access the real PCIe interface */
 	spin_lock_irqsave(&port->conf_lock, flags);
-	ret = mvebu_pcie_hw_wr_conf(port, bus,
-				    PCI_DEVFN(1, PCI_FUNC(devfn)),
+	ret = mvebu_pcie_hw_wr_conf(port, bus, devfn,
 				    where, size, val);
 	spin_unlock_irqrestore(&port->conf_lock, flags);
 
@@ -606,8 +616,7 @@ static int mvebu_pcie_rd_conf(struct pci_bus *bus, u32 devfn, int where,
 
 	/* Access the real PCIe interface */
 	spin_lock_irqsave(&port->conf_lock, flags);
-	ret = mvebu_pcie_hw_rd_conf(port, bus,
-				    PCI_DEVFN(1, PCI_FUNC(devfn)),
+	ret = mvebu_pcie_hw_rd_conf(port, bus, devfn,
 				    where, size, val);
 	spin_unlock_irqrestore(&port->conf_lock, flags);
 
@@ -817,6 +826,8 @@ static int __init mvebu_pcie_probe(struct platform_device *pdev)
 			continue;
 		}
 
+		mvebu_pcie_set_local_dev_nr(port, 1);
+
 		if (mvebu_pcie_link_up(port)) {
 			port->haslink = 1;
 			dev_info(&pdev->dev, "PCIe%d.%d: link up\n",
Signature · 1 line
1.7.9.5
#2 lore raw

PATCH v2 2/3 pci: mvebu: allow the enumeration of devices beyond physical bridges

In reply to Thomas Petazzoni

Until now, the Marvell PCIe driver was only allowing the enumeration of the devices in the secondary bus of the emulated PCI-to-PCI bridge. This works fine when a PCIe device is directly connected into a PCIe slot of the Marvell board.

However, when the device connected in the PCIe slot is a physical PCIe bridge, beyond which a real PCIe device is connected, it no longer worked, as the driver was preventing the Linux PCI core from seeing such devices.

This commit fixes that by ensuring that configuration transactions on subordinate busses are properly forwarded on the right PCIe interface.

Thanks to this patch, a PCIe card beyond a PCIe bridge, itself beyond the emulated PCI-to-PCI bridge is properly detected, with the following layout:

-[0000:00]-+-01.0-[01]----00.0

           +-09.0-[02-07]----00.0-[03-07]--+-01.0-[04]--
           |                               +-05.0-[05]--
           |                               +-07.0-[06]--
           |                               \-09.0-[07]----00.0
           \-0a.0-[08]----00.0

Where the PCIe interface that sits beyond the emulated PCI-to-PCI bridge at 09.0 allows to access the secondary bus 02, on which there is a PCIe bridge that allows to access the 3 to 7 busses, that are subordinates to this bridge. And on one of this bus (bus 7), there is one real PCIe device connected.

---
 drivers/pci/host/pci-mvebu.c |   31 ++++++++++++++++++++++++++++---
 1 file changed, 28 insertions(+), 3 deletions(-)
drivers/pci/host/pci-mvebu.c +28 -3
index 0bc21b0..a3c4638 100644
@@ -554,7 +554,8 @@ mvebu_pcie_find_port(struct mvebu_pcie *pcie, struct pci_bus *bus,
 		if (bus->number == 0 && port->devfn == devfn)
 			return port;
 		if (bus->number != 0 &&
-		    port->bridge.secondary_bus == bus->number)
+		    bus->number >= port->bridge.secondary_bus &&
+		    bus->number <= port->bridge.subordinate_bus)
 			return port;
 	}
 
@@ -578,7 +579,18 @@ static int mvebu_pcie_wr_conf(struct pci_bus *bus, u32 devfn,
 	if (bus->number == 0)
 		return mvebu_sw_pci_bridge_write(port, where, size, val);
 
-	if (!port->haslink || PCI_SLOT(devfn) != 0)
+	if (!port->haslink)
+		return PCIBIOS_DEVICE_NOT_FOUND;
+
+	/*
+	 * On the secondary bus, we don't want to expose any other
+	 * device than the device physically connected in the PCIe
+	 * slot, visible in slot 0. In slot 1, there's a special
+	 * Marvell device that only makes sense when the Armada is
+	 * used as a PCIe endpoint.
+	 */
+	if (bus->number == port->bridge.secondary_bus &&
+	    PCI_SLOT(devfn) != 0)
 		return PCIBIOS_DEVICE_NOT_FOUND;
 
 	/* Access the real PCIe interface */
@@ -609,7 +621,20 @@ static int mvebu_pcie_rd_conf(struct pci_bus *bus, u32 devfn, int where,
 	if (bus->number == 0)
 		return mvebu_sw_pci_bridge_read(port, where, size, val);
 
-	if (!port->haslink || PCI_SLOT(devfn) != 0) {
+	if (!port->haslink) {
+		*val = 0xffffffff;
+		return PCIBIOS_DEVICE_NOT_FOUND;
+	}
+
+	/*
+	 * On the secondary bus, we don't want to expose any other
+	 * device than the device physically connected in the PCIe
+	 * slot, visible in slot 0. In slot 1, there's a special
+	 * Marvell device that only makes sense when the Armada is
+	 * used as a PCIe endpoint.
+	 */
+	if (bus->number == port->bridge.secondary_bus &&
+	    PCI_SLOT(devfn) != 0) {
 		*val = 0xffffffff;
 		return PCIBIOS_DEVICE_NOT_FOUND;
 	}
Signature · 1 line
1.7.9.5
#3 lore raw

PATCH v2 3/3 pci: mvebu: fix the emulation of the status register

In reply to Thomas Petazzoni

The status register of the PCI configuration space of PCI-to-PCI bridges contain some read-only bits, and so write-1-to-clear bits. So, the Linux PCI core sometimes writes 0xffff to this status register, and in the current PCI-to-PCI bridge emulation code of the Marvell driver, we do take all those 1s being written. Even the read-only bits are being overwritten.

For now, all the read-only bits should be emulated to have the zero value.

The other bits, that are write-1-to-clear bits are used to report various kind of errors, and are never set by the emulated bridge, so there is no need to support this write-1-to-clear bits mechanism.

As a conclusion, the easiest solution is to simply emulate this status register by returning zero when read, and ignore the writes to it.

This has two visible effects:

* The devsel is no longer 'unknown' in, i.e

Flags: bus master, 66MHz, user-definable features, ?? devsel, latency 0

becomes:

Flags: bus master, 66MHz, user-definable features, fast devsel, latency 0

in lspci -v.

This was caused by a value of 11b being read for devsel, which is an invalid value. This 11b value being read was due to a previous write of 0xffff into the status register.

* The capability list is no longer broken, because we indicate to the Linux PCI core that we don't have a Capabilities Pointer in the PCI configuration space of this bridge. The following message is therefore no longer visible in lspci -v:

Capabilities: [fc] <chain broken>

---
 drivers/pci/host/pci-mvebu.c |    5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)
drivers/pci/host/pci-mvebu.c +1 -4
index a3c4638..481e44a 100644
@@ -69,7 +69,6 @@ struct mvebu_sw_pci_bridge {
 	u16 vendor;
 	u16 device;
 	u16 command;
-	u16 status;
 	u16 class;
 	u8 interface;
 	u8 revision;
@@ -359,7 +358,6 @@ static void mvebu_sw_pci_bridge_init(struct mvebu_pcie_port *port)
 
 	memset(bridge, 0, sizeof(struct mvebu_sw_pci_bridge));
 
-	bridge->status = PCI_STATUS_CAP_LIST;
 	bridge->class = PCI_CLASS_BRIDGE_PCI;
 	bridge->vendor = PCI_VENDOR_ID_MARVELL;
 	bridge->device = MARVELL_EMULATED_PCI_PCI_BRIDGE_ID;
@@ -386,7 +384,7 @@ static int mvebu_sw_pci_bridge_read(struct mvebu_pcie_port *port,
 		break;
 
 	case PCI_COMMAND:
-		*value = bridge->status << 16 | bridge->command;
+		*value = bridge->command;
 		break;
 
 	case PCI_CLASS_REVISION:
@@ -479,7 +477,6 @@ static int mvebu_sw_pci_bridge_write(struct mvebu_pcie_port *port,
 	switch (where & ~3) {
 	case PCI_COMMAND:
 		bridge->command = value & 0xffff;
-		bridge->status = value >> 16;
 		break;
 
 	case PCI_BASE_ADDRESS_0 ... PCI_BASE_ADDRESS_1:
Signature · 1 line
1.7.9.5
maintainer
#4 lore raw

v2 cover Marvell PCIe driver improvements

In reply to Thomas Petazzoni
Quoted 51 lines · Thomas Petazzoni

Bjorn, Jason, Andrew, Gregory,

Here is an updated version of the PCIe improvements.

Changes since v1:

* Fix some typos in the comments of PATCH 2, noticed by Bjorn Helgaas.

* Rework the two last patches (first one was fixing the capability list, the other one the emulation of the devsel bits in the status register), after the comments of Bjorn. In fact, it turns out that the status register is composed of read-only bits for which we want to emulate a value of zero, and the other bits are write-1-to-clear bits that expose error conditions that we do not handle in the emulated bridge. So, in summary, emulating a 0 status register on reads, and ignoring writes to the status register is the simplest solution. If we later on implement a capability list, we can use a more clever emulation of the status register, but for the moment, this simple solution is enough. Therefore, the two last patches are now one single, simpler patch.

I'd like those improvements to be kept separated from the original PCIe driver itself: while the PCIe driver has been around and reviewed for a long time, those improvements are newer. And I clearly do not want the PCIe driver to miss 3.11 because of any problem that could be found in those additional improvements.

The main improvement being brought here is that the PCI-to-PCI bridge logic is fixed/extended to properly support physical PCIe bridges that are connected on a PCIe interface of a Marvell board. Without this improvement, only the devices connected directly to the PCIe interfaces of the board are properly enumerated. Any device that would sit beyond a physical bridge is not visible.

Bjorn, with your Acked-by, could the Marvell maintainers include those patches in their branch, merged through arm-soc? They already have the Marvell PCIe driver itself, so I believe it makes sense to merge those improvements through the same path.

Jason, those patches have been prepared on top of my marvell-pcie-v10 branch, I hope that's ok for you. If you want me to rebase them on some other branch in which you have integrated the PCIe driver, don't hesitate to tell me to do so. That said, since those patches are only touching the driver itself and no other file, they should not cause any conflict with other changes.

Thanks,

Thomas

Quoted 7 lines

Thomas Petazzoni (3):
pci: mvebu: no longer fake the slot location of downstream devices
pci: mvebu: allow the enumeration of devices beyond physical bridges
pci: mvebu: fix the emulation of the status register

 drivers/pci/host/pci-mvebu.c |   55 +++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

For all three:

Feel free to merge these through the arm-soc tree or whatever makes sense for you. Nobody else is working on this file, so there shouldn't be any conflicts.

Bjorn

#5 lore raw

v2 cover Marvell PCIe driver improvements

Quoted 61 lines · Bjorn Helgaas
Quoted 51 lines · Thomas Petazzoni

Bjorn, Jason, Andrew, Gregory,

Here is an updated version of the PCIe improvements.

Changes since v1:

* Fix some typos in the comments of PATCH 2, noticed by Bjorn Helgaas.

* Rework the two last patches (first one was fixing the capability list, the other one the emulation of the devsel bits in the status register), after the comments of Bjorn. In fact, it turns out that the status register is composed of read-only bits for which we want to emulate a value of zero, and the other bits are write-1-to-clear bits that expose error conditions that we do not handle in the emulated bridge. So, in summary, emulating a 0 status register on reads, and ignoring writes to the status register is the simplest solution. If we later on implement a capability list, we can use a more clever emulation of the status register, but for the moment, this simple solution is enough. Therefore, the two last patches are now one single, simpler patch.

I'd like those improvements to be kept separated from the original PCIe driver itself: while the PCIe driver has been around and reviewed for a long time, those improvements are newer. And I clearly do not want the PCIe driver to miss 3.11 because of any problem that could be found in those additional improvements.

The main improvement being brought here is that the PCI-to-PCI bridge logic is fixed/extended to properly support physical PCIe bridges that are connected on a PCIe interface of a Marvell board. Without this improvement, only the devices connected directly to the PCIe interfaces of the board are properly enumerated. Any device that would sit beyond a physical bridge is not visible.

Bjorn, with your Acked-by, could the Marvell maintainers include those patches in their branch, merged through arm-soc? They already have the Marvell PCIe driver itself, so I believe it makes sense to merge those improvements through the same path.

Jason, those patches have been prepared on top of my marvell-pcie-v10 branch, I hope that's ok for you. If you want me to rebase them on some other branch in which you have integrated the PCIe driver, don't hesitate to tell me to do so. That said, since those patches are only touching the driver itself and no other file, they should not cause any conflict with other changes.

Thanks,

Thomas

Quoted 7 lines

Thomas Petazzoni (3):
pci: mvebu: no longer fake the slot location of downstream devices
pci: mvebu: allow the enumeration of devices beyond physical bridges
pci: mvebu: fix the emulation of the status register

 drivers/pci/host/pci-mvebu.c |   55 +++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)
Quoted 7 lines

For all three:

Feel free to merge these through the arm-soc tree or whatever makes sense for you. Nobody else is working on this file, so there shouldn't be any conflicts.

Thanks, Bjorn!

Jason.

#6 lore raw

v2 cover Marvell PCIe driver improvements

In reply to Thomas Petazzoni
Quoted 51 lines · Thomas Petazzoni

Bjorn, Jason, Andrew, Gregory,

Here is an updated version of the PCIe improvements.

Changes since v1:

* Fix some typos in the comments of PATCH 2, noticed by Bjorn Helgaas.

* Rework the two last patches (first one was fixing the capability list, the other one the emulation of the devsel bits in the status register), after the comments of Bjorn. In fact, it turns out that the status register is composed of read-only bits for which we want to emulate a value of zero, and the other bits are write-1-to-clear bits that expose error conditions that we do not handle in the emulated bridge. So, in summary, emulating a 0 status register on reads, and ignoring writes to the status register is the simplest solution. If we later on implement a capability list, we can use a more clever emulation of the status register, but for the moment, this simple solution is enough. Therefore, the two last patches are now one single, simpler patch.

I'd like those improvements to be kept separated from the original PCIe driver itself: while the PCIe driver has been around and reviewed for a long time, those improvements are newer. And I clearly do not want the PCIe driver to miss 3.11 because of any problem that could be found in those additional improvements.

The main improvement being brought here is that the PCI-to-PCI bridge logic is fixed/extended to properly support physical PCIe bridges that are connected on a PCIe interface of a Marvell board. Without this improvement, only the devices connected directly to the PCIe interfaces of the board are properly enumerated. Any device that would sit beyond a physical bridge is not visible.

Bjorn, with your Acked-by, could the Marvell maintainers include those patches in their branch, merged through arm-soc? They already have the Marvell PCIe driver itself, so I believe it makes sense to merge those improvements through the same path.

Jason, those patches have been prepared on top of my marvell-pcie-v10 branch, I hope that's ok for you. If you want me to rebase them on some other branch in which you have integrated the PCIe driver, don't hesitate to tell me to do so. That said, since those patches are only touching the driver itself and no other file, they should not cause any conflict with other changes.

Thanks,

Thomas

Quoted 7 lines

Thomas Petazzoni (3):
pci: mvebu: no longer fake the slot location of downstream devices
pci: mvebu: allow the enumeration of devices beyond physical bridges
pci: mvebu: fix the emulation of the status register

 drivers/pci/host/pci-mvebu.c |   55 +++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

Whole series applied to mvebu/pcie_bridge with Bjorn's Ack. The whole pcie tree now looks like:

/-mvebu/of_pci /-mvebu/pcie
| |

+---+---+---+---+---+---+---+---+---+---+ mvebu/pcie_bridge
                        |
                        \---+---+---+---+ mvebu/pcie_kirkwood

(not to scale :) )

This is getting pretty hairy, do you intend on anything else for mvebu pcie for this merge window?

thx,

Jason.

#7 lore raw

v2 cover Marvell PCIe driver improvements

Dear Jason Cooper,

Quoted 6 lines · Jason Cooper
Quoted 2 lines
 drivers/pci/host/pci-mvebu.c |   55 +++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 11 deletions(-)

Whole series applied to mvebu/pcie_bridge with Bjorn's Ack. The whole pcie tree now looks like:

Quoted 6 lines

/-mvebu/of_pci /-mvebu/pcie
| |

+---+---+---+---+---+---+---+---+---+---+ mvebu/pcie_bridge
                        |
                        \---+---+---+---+ mvebu/pcie_kirkwood

(not to scale :) )

Nice diagram, I like it :-)

Quoted 2 lines

This is getting pretty hairy, do you intend on anything else for mvebu pcie for this merge window?

No, nothing significant. There might be some minor additional fixes that you can just stack into the mvebu/pcie_bridge branch, but I don't have any such fixes sitting anywhere in my repo at this point.

I now intend to work on the MSI support, but since it requires a quite significant work to properly expose the msi_chip from the IRQ controller driver, I don't really think it will be ready in time for 3.11.

Thanks a lot for the quick merges. I'm really eager to see this PCIe code that has been around for a long time finally merged.

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com