Skip to contentSkip to conversation

PCIe controller driver for Marvell Armada 3700

PATCH Superseded Thomas Petazzoni 6 participants 21 messages Jun 2, 2016 – Jun 10, 2016

Subsystems (4)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

cover PCIe controller driver for Marvell Armada 3700

Hello,

This small patch series adds a new driver for the PCIe controller found on Marvell Armada 3700 ARM64 SoCs. Like any other new driver submission, this patch series includes: a Device Tree binding documentation patch, a patch adding the driver itself, and a patch adding the Device Tree description of the hardware.

Thanks in advance for your reviews!

Thomas

Thomas Petazzoni (3):
dt-bindings: add DT binding for the Aardvark PCIe controller
PCI: host: new PCI host controller driver for Marvell Armada 3700
arm64: dts: marvell: PCIe support for Armada 3700

 .../devicetree/bindings/pci/aardvark-pci.txt       |   51 +
 MAINTAINERS                                        |    7 +
 arch/arm64/boot/dts/marvell/armada-3720-db.dts     |    5 +
 arch/arm64/boot/dts/marvell/armada-37xx.dtsi       |   23 +
 drivers/pci/host/Kconfig                           |    9 +
 drivers/pci/host/Makefile                          |    1 +
 drivers/pci/host/pci-aardvark.c                    | 1029 ++++++++++++++++++++
 7 files changed, 1125 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/aardvark-pci.txt
 create mode 100644 drivers/pci/host/pci-aardvark.c
Signature · 1 line
2.7.4
#1 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

This commit adds the documentation for the Device Tree binding used to describe the Aardvark PCIe controller, found on Marvell Armada 3700 ARM64 SoCs.

---
 .../devicetree/bindings/pci/aardvark-pci.txt       | 51 ++++++++++++++++++++++
 1 file changed, 51 insertions(+)
 create mode 100644 Documentation/devicetree/bindings/pci/aardvark-pci.txt
Documentation/devicetree/bindings/pci/aardvark-pci.txt new file +51 -0
new file mode 100644
index 0000000..3517234
@@ -0,0 +1,51 @@
+Aardvark PCIe controller
+
+This PCIe controller is used on the Marvell Armada 3700 ARM64 SoC.
+
+The Device Tree node describing an Aardvark PCIe controller must
+contain the following properties:
+
+ - compatible: Should be "marvell,armada-3700-pcie"
+ - reg: range of registers for the PCIe controller
+ - interrupts: the interrupt line of the PCIe controller
+ - #address-cells: set to <3>
+ - #size-cells: set to <2>
+ - device_type: set to "pci"
+ - ranges: ranges for the PCI memory and I/O regions
+ - #interrupt-cells: set to <1>
+ - interrupt-map-mask and interrupt-map: standard PCI properties to
+   define the mapping of the PCIe interface to interrupt numbers.
+ - bus-range: PCI bus numbers covered
+
+In addition, the Device Tree describing an Aardvark PCIe controller
+must include a sub-node that describes the legacy interrupt controller
+built into the PCIe controller. This sub-node must have the following
+properties:
+
+ - interrupt-controller
+ - #interrupt-cells: set to <1>
+
+Example:
+
+	pcie0: pcie at d0070000 {
+		compatible = "marvell,armada-3700-pcie";
+		device_type = "pci";
+		status = "disabled";
+		reg = <0 0xd0070000 0 0x20000>;
+		#address-cells = <3>;
+		#size-cells = <2>;
+		bus-range = <0x00 0xff>;
+		interrupts = <GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>;
+		#interrupt-cells = <1>;
+		ranges = <0x82000000 0 0xe8000000   0 0xe8000000 0 0x1000000 /* Port 0 MEM */
+			  0x81000000 0 0xe9000000   0 0xe9000000 0 0x10000>; /* Port 0 IO*/
+		interrupt-map-mask = <0 0 0 7>;
+		interrupt-map = <0 0 0 1 &pcie_intc 0>,
+				<0 0 0 2 &pcie_intc 1>,
+				<0 0 0 3 &pcie_intc 2>,
+				<0 0 0 4 &pcie_intc 3>;
+		pcie_intc: interrupt-controller {
+			interrupt-controller;
+			#interrupt-cells = <1>;
+		};
+	};
Signature · 1 line
2.7.4
#2 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Documentation/devicetree/bindings/pci/aardvark-pci.txt line 41

Quoted 3 lines · CEST Thomas Petazzoni
Documentation/devicetree/bindings/pci/aardvark-pci.txt new file +2 -0
+               ranges = <0x82000000 0 0xe8000000   0 0xe8000000 0 0x1000000 /* Port 0 MEM */
+                         0x81000000 0 0xe9000000   0 0xe9000000 0 0x10000>; /* Port 0 IO*/

Any reason for not having a 64-bit MEM prefetchable area in the example? Does the host not support that?

Arnd

#3 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Hello,

Thanks for your review!

Quoted 7 lines · Arnd Bergmann

Documentation/devicetree/bindings/pci/aardvark-pci.txt line 41

Quoted 3 lines · CEST Thomas Petazzoni
Documentation/devicetree/bindings/pci/aardvark-pci.txt new file +2 -0
+               ranges = <0x82000000 0 0xe8000000   0 0xe8000000 0 0x1000000 /* Port 0 MEM */
+                         0x81000000 0 0xe9000000   0 0xe9000000 0 0x10000>; /* Port 0 IO*/

Any reason for not having a 64-bit MEM prefetchable area in the example? Does the host not support that?

I'll have to admit I am not sure how to find this out from the datasheet. My datasheet says about the PCIe controller:

"""
64-bit PCIe address and system address space for outbound transactions """

So I guess this would indicate that a 64-bit MEM area is possible. However, since anyway the area used above is at 0xe8000000 for a length of 0x1000000, what would be the benefit of declaring this range as a 64-bit one ?

Regarding the prefetchable aspect, I couldn't find any reference in the datasheet. However, the original driver code explicitly errors out if there is no non-prefetchable memory area, so I guess prefetchable areas is not supported.

In of_bus_pci_get_flags(), both the 32-bit and 64-bit cases are handled in the same way, so is this distinction actually being used by the kernel?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#4 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Quoted 25 lines · CEST Thomas Petazzoni

Hello,

Thanks for your review!

Quoted 7 lines · Arnd Bergmann

Documentation/devicetree/bindings/pci/aardvark-pci.txt line 41

Quoted 3 lines · CEST Thomas Petazzoni
Documentation/devicetree/bindings/pci/aardvark-pci.txt new file +2 -0
+               ranges = <0x82000000 0 0xe8000000   0 0xe8000000 0 0x1000000 /* Port 0 MEM */
+                         0x81000000 0 0xe9000000   0 0xe9000000 0 0x10000>; /* Port 0 IO*/

Any reason for not having a 64-bit MEM prefetchable area in the example? Does the host not support that?

I'll have to admit I am not sure how to find this out from the datasheet. My datasheet says about the PCIe controller:

"""
64-bit PCIe address and system address space for outbound transactions """

So I guess this would indicate that a 64-bit MEM area is possible. However, since anyway the area used above is at 0xe8000000 for a length of 0x1000000, what would be the benefit of declaring this range as a 64-bit one ?

Quoted 8 lines

Regarding the prefetchable aspect, I couldn't find any reference in the datasheet. However, the original driver code explicitly errors out if there is no non-prefetchable memory area, so I guess prefetchable areas is not supported.

In of_bus_pci_get_flags(), both the 32-bit and 64-bit cases are handled in the same way, so is this distinction actually being used by the kernel?

Each device needs to have at least one non-prefetcheable range, which is why some drivers check for that. Non-prefetchable BARs always use 32-bit addressing, so 64-bit BARs are by definition prefetchable, but you can also have prefetchable registers in the first 4GB in some cases.

Some devices require large prefetchable BARs (hundreds of MB, or more), so if you have the option, you should enable that in the host, as the 16MB you have available for non-prefetchable devices is not going to be sufficient.

Arnd

maintainer
#5 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

In reply to Thomas Petazzoni
Quoted 21 lines
Documentation/devicetree/bindings/pci/aardvark-pci.txt new file +21 -0
+In addition, the Device Tree describing an Aardvark PCIe controller
+must include a sub-node that describes the legacy interrupt controller
+built into the PCIe controller. This sub-node must have the following
+properties:
+
+ - interrupt-controller
+ - #interrupt-cells: set to <1>
+
+Example:
+
+	pcie0: pcie at d0070000 {
+		compatible = "marvell,armada-3700-pcie";
+		device_type = "pci";
+		status = "disabled";
+		reg = <0 0xd0070000 0 0x20000>;
+		#address-cells = <3>;
+		#size-cells = <2>;
+		bus-range = <0x00 0xff>;
+		interrupts = <GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>;
+		#interrupt-cells = <1>;
+		ranges = <0x82000000 0 0xe8000000   0 0xe8000000 0 0x1000000 /* Port 0 MEM */

Documentation/devicetree/bindings/pci/aardvark-pci.txt line 50

Quoted 10 lines
Documentation/devicetree/bindings/pci/aardvark-pci.txt new file +10 -0
+			  0x81000000 0 0xe9000000   0 0xe9000000 0 0x10000>; /* Port 0 IO*/
+		interrupt-map-mask = <0 0 0 7>;
+		interrupt-map = <0 0 0 1 &pcie_intc 0>,
+				<0 0 0 2 &pcie_intc 1>,
+				<0 0 0 3 &pcie_intc 2>,
+				<0 0 0 4 &pcie_intc 3>;
+		pcie_intc: interrupt-controller {
+			interrupt-controller;
+			#interrupt-cells = <1>;
+		};

Hi Thomas

It is possible to list PCIe devices on the bus here as child nodes. I've done this when i needed a phandle to an intel ethernet controller on the PCIe bus, which i know is soldered onto the board.

I think your current implementation simply uses the first child node. It would be good to document that ordering is important. It must be the first child node, and any pcie devices children must come afterwards.

Andrew

#6 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Quoted 9 lines · CEST Andrew Lunn

It is possible to list PCIe devices on the bus here as child nodes. I've done this when i needed a phandle to an intel ethernet controller on the PCIe bus, which i know is soldered onto the board.

I think your current implementation simply uses the first child node. It would be good to document that ordering is important. It must be the first child node, and any pcie devices children must come afterwards.

I think some other PCI hosts just move the interrupt-controller and #interrupt-cells properties into the PCI host node itself, which avoids the ambiguity here.

Arnd

#7 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Hello,

Quoted 3 lines · Arnd Bergmann

I think some other PCI hosts just move the interrupt-controller and #interrupt-cells properties into the PCI host node itself, which avoids the ambiguity here.

Do you have an example of this? I have modeled my DT binding after the one used by the pci-dra7xx driver, which is in the same situation as I am in terms of interrupt handling.

But I can indeed try to make the top-level PCIe controller node the interrupt controller. Let me try that quickly.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#8 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Quoted 7 lines · CEST Thomas Petazzoni
Quoted 3 lines · Arnd Bergmann

I think some other PCI hosts just move the interrupt-controller and #interrupt-cells properties into the PCI host node itself, which avoids the ambiguity here.

Quoted 7 lines

Do you have an example of this? I have modeled my DT binding after the one used by the pci-dra7xx driver, which is in the same situation as I am in terms of interrupt handling.

But I can indeed try to make the top-level PCIe controller node the interrupt controller. Let me try that quickly.

Looking at the binding files, I see only this one:

Documentation/devicetree/bindings/pci/altera-pcie.txt

and a couple of others using a child node.

Arnd

#9 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

In reply to Arnd Bergmann

Hello,

Quoted 3 lines · Arnd Bergmann

I think some other PCI hosts just move the interrupt-controller and #interrupt-cells properties into the PCI host node itself, which avoids the ambiguity here.

I've tried that, and it works fine. Makes the code simpler, the binding simpler, so: adopted! Thanks for suggesting!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#10 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

In reply to Thomas Petazzoni
Quoted 5 lines · Thomas Petazzoni

This commit adds the documentation for the Device Tree binding used to describe the Aardvark PCIe controller, found on Marvell Armada 3700 ARM64 SoCs.

Hi Thomas,

If you post a v2 of these, can you align the subject lines of these binding, DT, and driver patches so it's easier to find the related pieces, e.g., add "Armada 3700" to this one or "Aardvark" to the others? It looks like maybe Aardvark could be used in several different systems, so maybe this patch should stay the same and the drivers/pci patch should mention Aardvark instead of only "Marvell Armada 3700"?

Bjorn

#11 lore raw

PATCH 1/3 dt-bindings: add DT binding for the Aardvark PCIe controller

Hello,

Quoted 7 lines · Bjorn Helgaas

If you post a v2 of these, can you align the subject lines of these binding, DT, and driver patches so it's easier to find the related pieces, e.g., add "Armada 3700" to this one or "Aardvark" to the others? It looks like maybe Aardvark could be used in several different systems, so maybe this patch should stay the same and the drivers/pci patch should mention Aardvark instead of only "Marvell Armada 3700"?

Sure thing. I was about to send a v2, so I've fixed up the commit titles so that they all include Aardvark.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#12 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

In reply to Thomas Petazzoni

This commit adds a new driver for a PCIe controller called Aardvark, which is used on the Marvell Armada 3700 ARM64 SoC.

This patch is based on work done by Hezi Shahmoon <[email protected]> and Marcin Wojtas <[email protected]>.

---
 MAINTAINERS                     |    7 +
 drivers/pci/host/Kconfig        |    9 +
 drivers/pci/host/Makefile       |    1 +
 drivers/pci/host/pci-aardvark.c | 1029 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 1046 insertions(+)
 create mode 100644 drivers/pci/host/pci-aardvark.c
MAINTAINERS +7 -0
index 7304d2e..373215c 100644
@@ -8740,6 +8740,13 @@ L:	linux-arm-kernel at lists.infradead.org (moderated for non-subscribers)
 S:	Maintained
 F:	drivers/pci/host/*mvebu*
 
+PCI DRIVER FOR AARDVARK (Marvell Armada 3700)
+M:	Thomas Petazzoni <[email protected]>
+L:	linux-pci at vger.kernel.org
+L:	linux-arm-kernel at lists.infradead.org (moderated for non-subscribers)
+S:	Maintained
+F:	drivers/pci/host/pci-aardvark.c
+
 PCI DRIVER FOR NVIDIA TEGRA
 M:	Thierry Reding <[email protected]>
 L:	linux-tegra at vger.kernel.org
drivers/pci/host/Kconfig +9 -0
index 5d2374e..5f085fd 100644
@@ -16,6 +16,15 @@ config PCI_MVEBU
 	depends on ARM
 	depends on OF
 
+config PCI_AARDVARK
+	bool "Aardvark PCIe controller"
+	depends on ARCH_MVEBU && ARM64
+	depends on OF
+	select PCI_MSI
+	help
+	 Add support for Aardvark 64bit PCIe Host Controller. This
+	 controller is part of the South Bridge of the Marvel Armada
+	 3700 SoC.
 
 config PCIE_XILINX_NWL
 	bool "NWL PCIe Core"
drivers/pci/host/Makefile +1 -0
index 9c8698e..66f45b6 100644
@@ -5,6 +5,7 @@ obj-$(CONFIG_PCI_EXYNOS) += pci-exynos.o
 obj-$(CONFIG_PCI_IMX6) += pci-imx6.o
 obj-$(CONFIG_PCI_HYPERV) += pci-hyperv.o
 obj-$(CONFIG_PCI_MVEBU) += pci-mvebu.o
+obj-$(CONFIG_PCI_AARDVARK) += pci-aardvark.o
 obj-$(CONFIG_PCI_TEGRA) += pci-tegra.o
 obj-$(CONFIG_PCI_RCAR_GEN2) += pci-rcar-gen2.o
 obj-$(CONFIG_PCIE_RCAR) += pcie-rcar.o
drivers/pci/host/pci-aardvark.c new file +1029 -0

1,029 lines not on this page · raw message

Signature · 1 line
2.7.4
#13 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

Quoted 6 lines · CEST Thomas Petazzoni
drivers/pci/host/pci-aardvark.c new file +6 -0
+static bool advk_pcie_link_up(struct advk_pcie *pcie)
+{
+	int timeout;
+	u32 ltssm_state;
+	u32 val;
+

drivers/pci/host/pci-aardvark.c line 257

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+	timeout = PCIE_LINKUP_TIMEOUT;
+	do {
+		val = advk_readl(pcie, CFG_REG);
+		ltssm_state = (val >> LTSSM_SHIFT) & LTSSM_MASK;
+		timeout--;
+	/* Use ltssm < LTSSM_L0 instead of ltssm != LTSSM_L0 */
+	} while (ltssm_state < LTSSM_L0 && timeout > 0);
+
+	return (timeout > 0);
+}

Maybe wait for some amount of time to have passed here (using time_before(jiffies, end)) rather than a number of retries, for robustness and determinism.

drivers/pci/host/pci-aardvark.c line 286

Quoted 3 lines
drivers/pci/host/pci-aardvark.c new file +3 -0
+	/* Point PCIe unit MBUS decode windows to DRAM space. */
+	for (i = 0; i < 8; i++)
+		advk_pcie_set_ob_win(pcie, i, 0, 0, 0, 0, 0, 0, 0);

Is this actually mbus based, or is the comment copied from some other Marvell driver?

Quoted 12 lines
drivers/pci/host/pci-aardvark.c new file +12 -0
+static int advk_pcie_setup_msi_irq(struct msi_controller *chip,
+				   struct pci_dev *pdev,
+				   struct msi_desc *desc)
+{
+	struct advk_pcie *pcie = pdev->bus->sysdata;
+	struct msi_msg msg;
+	int virq, hwirq;
+	phys_addr_t msi_msg_phys;
+
+	/* We support MSI, but not MSI-X */
+	if (desc->msi_attrib.is_msix)
+		return -EINVAL;

drivers/pci/host/pci-aardvark.c line 632

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+
+	hwirq = advk_pcie_alloc_msi(pcie);
+	if (hwirq < 0)
+		return hwirq;
+
+	virq = irq_create_mapping(pcie->msi_domain, hwirq);
+	if (!virq) {
+		advk_pcie_free_msi(pcie, hwirq);
+		return -EINVAL;
+	}

What is the version of the GIC in the Armada 3700? If you have GICv3 or GICv2m, could you use that instead of the built-in MSI logic?

We typically handle this using the msi-map or msi-parent properties pointing to either the gic or the PCI host, depending on which one you want to use, but either of them should work, and the GIC should be more efficient because you can distribute the interrupts of the PCI devices over all CPUs by workload, rather than having to multiplex all MSI through a single GIC interrupt.

Arnd

#14 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

Hello,

Quoted 6 lines · Arnd Bergmann
Quoted 6 lines
drivers/pci/host/pci-aardvark.c new file +6 -0
+	timeout = PCIE_LINKUP_TIMEOUT;
+	do {
+		val = advk_readl(pcie, CFG_REG);
+		ltssm_state = (val >> LTSSM_SHIFT) & LTSSM_MASK;
+		timeout--;
+	/* Use ltssm < LTSSM_L0 instead of ltssm != LTSSM_L0 */
Quoted 8 lines

drivers/pci/host/pci-aardvark.c line 257

Quoted 4 lines
drivers/pci/host/pci-aardvark.c new file +4 -0
+	} while (ltssm_state < LTSSM_L0 && timeout > 0);
+
+	return (timeout > 0);
+}  

Maybe wait for some amount of time to have passed here (using time_before(jiffies, end)) rather than a number of retries, for robustness and determinism.

Will. There was another similar loop in the driver which was doing a number of retries rather than a timeout-bounded loop, so I've fixed that one as well.

Quoted 6 lines

drivers/pci/host/pci-aardvark.c line 286

Quoted 3 lines
drivers/pci/host/pci-aardvark.c new file +3 -0
+	/* Point PCIe unit MBUS decode windows to DRAM space. */
+	for (i = 0; i < 8; i++)
+		advk_pcie_set_ob_win(pcie, i, 0, 0, 0, 0, 0, 0, 0);  

Is this actually mbus based, or is the comment copied from some other Marvell driver?

No, it's not MBUS based, the address decoding logic is very different from previous mvebu SoCs, I guess it's just a "familiar" name to say that it's configuring address decoding. I'll remove the comment.

Quoted 9 lines

What is the version of the GIC in the Armada 3700? If you have GICv3 or GICv2m, could you use that instead of the built-in MSI logic?

We typically handle this using the msi-map or msi-parent properties pointing to either the gic or the PCI host, depending on which one you want to use, but either of them should work, and the GIC should be more efficient because you can distribute the interrupts of the PCI devices over all CPUs by workload, rather than having to multiplex all MSI through a single GIC interrupt.

There is a GIC-500, but Marcin told me that attempts to use MSI with it have not been successful so far. There will be investigation on this topic in the future, but for the moment, we'd like to have the MSI functionality built into the PCIe driver supported. We can migrate later to GIC-500 powered MSIs once working.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#15 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

Quoted 10 lines · CEST Thomas Petazzoni
Quoted 9 lines

What is the version of the GIC in the Armada 3700? If you have GICv3 or GICv2m, could you use that instead of the built-in MSI logic?

We typically handle this using the msi-map or msi-parent properties pointing to either the gic or the PCI host, depending on which one you want to use, but either of them should work, and the GIC should be more efficient because you can distribute the interrupts of the PCI devices over all CPUs by workload, rather than having to multiplex all MSI through a single GIC interrupt.

Quoted 5 lines

There is a GIC-500, but Marcin told me that attempts to use MSI with it have not been successful so far. There will be investigation on this topic in the future, but for the moment, we'd like to have the MSI functionality built into the PCIe driver supported. We can migrate later to GIC-500 powered MSIs once working.

I think you should still add the msi-parent or msi-map properties then, just have them point at the PCI host node instead of the GIC-500, and evaluate that to get the built-in controller.

Arnd

#16 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

Hello,

Quoted 9 lines · Arnd Bergmann
Quoted 5 lines

There is a GIC-500, but Marcin told me that attempts to use MSI with it have not been successful so far. There will be investigation on this topic in the future, but for the moment, we'd like to have the MSI functionality built into the PCIe driver supported. We can migrate later to GIC-500 powered MSIs once working.

I think you should still add the msi-parent or msi-map properties then, just have them point at the PCI host node instead of the GIC-500, and evaluate that to get the built-in controller.

Sure, I'll have a look at doing that.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#17 lore raw

[PATCH 2/3] PCI: host: new PCI host controller driver for Marvell Armada 3700

In reply to Thomas Petazzoni
reproduce:

Hi,

[auto build test ERROR on pci/next]
[also build test ERROR on v4.7-rc1 next-20160603]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url:    https://github.com/0day-ci/linux/commits/Thomas-Petazzoni/PCIe-controller-driver-for-Marvell-Armada-3700/20160602-171515
base:   https://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git next
config: x86_64-lkp (attached as .config)
compiler: gcc-4.9 (Debian 4.9.3-14) 4.9.3
reproduce:
        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

>> drivers/of/Kconfig:4:error: recursive dependency detected!

For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
drivers/of/Kconfig:4: symbol OF is selected by X86_INTEL_CE
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:501: symbol X86_INTEL_CE depends on X86_IO_APIC
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:970: symbol X86_IO_APIC depends on X86_LOCAL_APIC
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:964: symbol X86_LOCAL_APIC depends on X86_UP_APIC
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:939: symbol X86_UP_APIC depends on PCI_MSI
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
drivers/pci/Kconfig:11: symbol PCI_MSI is selected by PCI_AARDVARK
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
drivers/pci/host/Kconfig:19: symbol PCI_AARDVARK depends on OF
--

>> drivers/of/Kconfig:4:error: recursive dependency detected!

For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
drivers/of/Kconfig:4: symbol OF is selected by X86_INTEL_CE
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:501: symbol X86_INTEL_CE depends on X86_IO_APIC
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:970: symbol X86_IO_APIC depends on X86_LOCAL_APIC
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:964: symbol X86_LOCAL_APIC depends on X86_UP_APIC
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
arch/x86/Kconfig:939: symbol X86_UP_APIC depends on PCI_MSI
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
drivers/pci/Kconfig:11: symbol PCI_MSI is selected by PCI_AARDVARK
For a resolution refer to Documentation/kbuild/kconfig-language.txt subsection "Kconfig recursive dependency limitations"
drivers/pci/host/Kconfig:19: symbol PCI_AARDVARK depends on OF
.config:429:warning: symbol value 'm' invalid for MICROCODE .config:2991:warning: symbol value 'm' invalid for FAULT_INJECTION

vim +4 drivers/of/Kconfig

5ab5fc7e Grant Likely     2010-07-05   1  config DTC
5ab5fc7e Grant Likely     2010-07-05   2  	bool
5ab5fc7e Grant Likely     2010-07-05   3  
0166dc11 Rob Herring      2015-05-28  @4  menuconfig OF
0166dc11 Rob Herring      2015-05-28   5  	bool "Device Tree and Open Firmware support"
0166dc11 Rob Herring      2015-05-28   6  	help
0166dc11 Rob Herring      2015-05-28   7  	  This option enables the device tree infrastructure.
0166dc11 Rob Herring      2015-05-28   8  	  It is automatically selected by platforms that need it or can
0166dc11 Rob Herring      2015-05-28   9  	  be enabled manually for unittests, overlays or
0166dc11 Rob Herring      2015-05-28  10  	  compile-coverage.

bcbefae2 Stephen Rothwell 2010-06-29 11
0166dc11 Rob Herring 2015-05-28 12 if OF

:::::: The code at line 4 was first introduced by commit
:::::: 0166dc11be911213e0b1b764488c671be4c48cf3 of: make CONFIG_OF user selectable

:::::: TO: Rob Herring <[email protected]> :::::: CC: Rob Herring <[email protected]>

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation

#18 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

In reply to Thomas Petazzoni

Hi Thomas,

Quoted 15 lines
drivers/pci/host/pci-aardvark.c new file +15 -0
+static const struct irq_domain_ops advk_pcie_msi_irq_ops = {
+       .map = advk_pcie_msi_map,
+};
+
+static void advk_pcie_irq_mask(struct irq_data *d)
+{
+       struct advk_pcie *pcie = d->domain->host_data;
+       irq_hw_number_t hwirq = irqd_to_hwirq(d);
+       u32 mask;
+
+       mask = advk_readl(pcie, PCIE_ISR0_MASK_REG);
+       mask |= PCIE_ISR0_INTX_ASSERT(hwirq) |
+               PCIE_ISR0_INTX_DEASSERT(hwirq);
+       advk_writel(pcie, mask, PCIE_ISR0_MASK_REG);
+}

drivers/pci/host/pci-aardvark.c line 694

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+
+static void advk_pcie_irq_unmask(struct irq_data *d)
+{
+       struct advk_pcie *pcie = d->domain->host_data;
+       irq_hw_number_t hwirq = irqd_to_hwirq(d);
+       u32 mask;
+
+       mask = advk_readl(pcie, PCIE_ISR0_MASK_REG);
+       mask &= ~(PCIE_ISR0_INTX_ASSERT(hwirq) |
+                 PCIE_ISR0_INTX_DEASSERT(hwirq));

Have you checked my latest improvements of interrupt handling in this controller? As DEASSERT is polled in advk_pcie_handle_int() I don't think it should be able to trigger root complex interrupt at all. Hence I think it shouldn't be unmasked.

Quoted 60 lines
drivers/pci/host/pci-aardvark.c new file +60 -0
+       advk_writel(pcie, mask, PCIE_ISR0_MASK_REG);
+}
+
+static int advk_pcie_irq_map(struct irq_domain *h,
+                            unsigned int virq, irq_hw_number_t hwirq)
+{
+       struct advk_pcie *pcie = h->host_data;
+
+       advk_pcie_irq_mask(irq_get_irq_data(virq));
+       irq_set_status_flags(virq, IRQ_LEVEL);
+       irq_set_chip_and_handler(virq, &pcie->irq_chip,
+                                handle_level_irq);
+       irq_set_chip_data(virq, pcie);
+
+       return 0;
+}
+
+static const struct irq_domain_ops advk_pcie_irq_domain_ops = {
+       .map = advk_pcie_irq_map,
+       .xlate = irq_domain_xlate_onecell,
+};
+
+static int advk_pcie_init_irq(struct advk_pcie *pcie)
+{
+       struct device *dev = &pcie->pdev->dev;
+       struct device_node *node = dev->of_node;
+       struct device_node *pcie_intc_node;
+       struct irq_chip *irq_chip, *msi_irq_chip;
+       struct msi_controller *msi;
+       phys_addr_t msi_msg_phys;
+
+       pcie_intc_node =  of_get_next_child(node, NULL);
+       if (!pcie_intc_node) {
+               dev_err(dev, "No PCIe Intc node found\n");
+               return PTR_ERR(pcie_intc_node);
+       }
+
+       irq_chip = &pcie->irq_chip;
+
+       irq_chip->name = devm_kasprintf(dev, GFP_KERNEL, "%s-irq",
+                                       dev_name(dev));
+       if (!irq_chip->name)
+               return -ENOMEM;
+       irq_chip->irq_mask = advk_pcie_irq_mask;
+       irq_chip->irq_mask_ack = advk_pcie_irq_mask;
+       irq_chip->irq_unmask = advk_pcie_irq_unmask;
+
+       pcie->irq_domain =
+               irq_domain_add_linear(pcie_intc_node, LEGACY_IRQ_NUM,
+                                     &advk_pcie_irq_domain_ops, pcie);
+       if (!pcie->irq_domain) {
+               dev_err(dev, "Failed to get a INTx IRQ domain\n");
+               return PTR_ERR(pcie->irq_domain);
+       }
+
+       msi_irq_chip = &pcie->msi_irq_chip;
+
+       msi_irq_chip->name = devm_kasprintf(dev, GFP_KERNEL, "%s-msi",
+                                                dev_name(dev));
+       if (!msi_irq_chip->name) {

drivers/pci/host/pci-aardvark.c line 764

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+               irq_domain_remove(pcie->irq_domain);
+               return -ENOMEM;
+       }
+
+       msi_irq_chip->irq_enable = pci_msi_unmask_irq;
+       msi_irq_chip->irq_disable = pci_msi_mask_irq;
+       msi_irq_chip->irq_mask = pci_msi_mask_irq;
+       msi_irq_chip->irq_unmask = pci_msi_unmask_irq;
+
+       msi = &pcie->msi;

Do we consciously resign from CONFIG_MSI dependency? I know it should always be enabled, but just making sure.

Quoted 64 lines
drivers/pci/host/pci-aardvark.c new file +64 -0
+
+       msi->setup_irq = advk_pcie_setup_msi_irq;
+       msi->teardown_irq = advk_pcie_teardown_msi_irq;
+       msi->of_node = node;
+
+       mutex_init(&pcie->msi_used_lock);
+
+       msi_msg_phys = virt_to_phys(&pcie->msi_msg);
+
+       advk_writel(pcie, lower_32_bits(msi_msg_phys),
+                   PCIE_MSI_ADDR_LOW_REG);
+       advk_writel(pcie, upper_32_bits(msi_msg_phys),
+                   PCIE_MSI_ADDR_HIGH_REG);
+
+       pcie->msi_domain =
+               irq_domain_add_linear(NULL, MSI_IRQ_NUM,
+                                     &advk_pcie_msi_irq_ops, pcie);
+       if (!pcie->msi_domain) {
+               irq_domain_remove(pcie->irq_domain);
+               return -ENOMEM;
+       }
+
+       return 0;
+}
+
+static void advk_pcie_handle_msi(struct advk_pcie *pcie)
+{
+       u32 msi_val, msi_mask, msi_status, msi_idx;
+       u16 msi_data;
+
+       msi_mask = advk_readl(pcie, PCIE_MSI_MASK_REG);
+       msi_val = advk_readl(pcie, PCIE_MSI_STATUS_REG);
+       msi_status = msi_val & ~msi_mask;
+
+       for (msi_idx = 0; msi_idx < MSI_IRQ_NUM; msi_idx++) {
+               if (!(BIT(msi_idx) & msi_status))
+                       continue;
+
+               advk_writel(pcie, BIT(msi_idx), PCIE_MSI_STATUS_REG);
+               msi_data = advk_readl(pcie, PCIE_MSI_PAYLOAD_REG) & 0xFF;
+               generic_handle_irq(msi_data);
+       }
+
+       advk_writel(pcie, PCIE_ISR0_MSI_INT_PENDING,
+                   PCIE_ISR0_REG);
+}
+
+static void advk_pcie_handle_int(struct advk_pcie *pcie)
+{
+       u32 val, mask, status;
+
+       val = advk_readl(pcie, PCIE_ISR0_REG);
+       mask = advk_readl(pcie, PCIE_ISR0_MASK_REG);
+       status = val & ((~mask) & PCIE_ISR0_ALL_MASK);
+
+       if (!status) {
+               advk_writel(pcie, val, PCIE_ISR0_REG);
+               return;
+       }
+
+       /* Process MSI interrupts */
+       if (status & PCIE_ISR0_MSI_INT_PENDING)
+               advk_pcie_handle_msi(pcie);
+

drivers/pci/host/pci-aardvark.c line 838

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+       /* Process legacy interrupts */
+       do {
+               int i;
+
+               /*
+                * As documented in the datasheet, we need to raise an
+                * interrupt for each ASSERT(i) bit that is set, and
+                * continue to deliver the interrupt until the
+                * DEASSERT(i) bit gets set
+                */

Below code will result in spurious interrupt flood (I can provide test details with SATA3.0 card). I know this part of specs very well, but expected EP operation flow should look like this:
a. EP interrupt
b. INTX_ASSERT signal
c. root complex interrupt
d. clear INTX_ASSERT bit
e. execute EP handler
f. INTX_DEASSERT signal
g. clear INTX_DEASSERT bit
h. exit interrupt

Now between d. and f., in case of any shortest delay we will force executing EP handler numerous of times, which IMO is not desired. All of them for e.g. the SATA3.0 card are registered as spurious interrupts, pretty quickly resulting in "switching to irq poll" failure after 100k of such happenings. Actually I think that after executing generic_handle_irq(virq); we should leave root complex isr asap, the polling for DEASSERT in such case is not resulting in any benefits. In my code I ignored DEASSERT signal existence - it neither triggers interrupt nor we don't poll for it. Under big stress nothing wrong happens. We rely only on interrupts triggered by INTX_ASSERT signal, so we care only for its status.

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+               for (i = 0; i < LEGACY_IRQ_NUM; i++) {
+                       if (!(status & PCIE_ISR0_INTX_ASSERT(i)))
+                               continue;
+
+                       do {
+                               int virq;
+
+                               advk_writel(pcie, PCIE_ISR0_INTX_ASSERT(i),
+                                           PCIE_ISR0_REG);
+                               virq = irq_find_mapping(pcie->irq_domain, i);

drivers/pci/host/pci-aardvark.c line 858

Quoted 10 lines
drivers/pci/host/pci-aardvark.c new file +10 -0
+                               generic_handle_irq(virq);
+                               status = advk_readl(pcie, PCIE_ISR0_REG);
+                       } while (!(status & PCIE_ISR0_INTX_DEASSERT(i)));
+
+                       advk_writel(pcie, PCIE_ISR0_INTX_DEASSERT(i),
+                                   PCIE_ISR0_REG);
+               }
+
+               status = advk_readl(pcie, PCIE_ISR0_REG);
+       } while (status & PCIE_ISR0_INTX_MASK);

How about simplifying code a bit and remove this do-while loop? IMO it's enough to do single iterating through INTA/B/C/D and exit - in such case we wouldn't spent too much time in single top half handler.

I'm looking forward to your feedback.

Best regards,
Marcin

#19 lore raw

PATCH 2/3 PCI: host: new PCI host controller driver for Marvell Armada 3700

Hello,

Thanks for your feedback!

Quoted 4 lines · Marcin Wojtas

Have you checked my latest improvements of interrupt handling in this controller? As DEASSERT is polled in advk_pcie_handle_int() I don't think it should be able to trigger root complex interrupt at all. Hence I think it shouldn't be unmasked.

Correct. Makes sense. Though you are suggesting below to no longer poll the DEASSERT signal in the interrupt handler, but you're instead saying to ignore it entirely.

Quoted 4 lines
Quoted 1 line

+ msi = &pcie->msi;

Do we consciously resign from CONFIG_MSI dependency? I know it should always be enabled, but just making sure.

Yes, I discussed this with Arnd, and we decided to simply make MSI support a hard dependency. You can always disable it by passing pci=nomsi on the kernel command line (tested to be working).

Quoted 12 lines

Below code will result in spurious interrupt flood (I can provide test details with SATA3.0 card). I know this part of specs very well, but expected EP operation flow should look like this:
a. EP interrupt
b. INTX_ASSERT signal
c. root complex interrupt
d. clear INTX_ASSERT bit
e. execute EP handler
f. INTX_DEASSERT signal
g. clear INTX_DEASSERT bit
h. exit interrupt

Quoted 11 lines

Now between d. and f., in case of any shortest delay we will force executing EP handler numerous of times, which IMO is not desired. All of them for e.g. the SATA3.0 card are registered as spurious interrupts, pretty quickly resulting in "switching to irq poll" failure after 100k of such happenings. Actually I think that after executing generic_handle_irq(virq); we should leave root complex isr asap, the polling for DEASSERT in such case is not resulting in any benefits. In my code I ignored DEASSERT signal existence - it neither triggers interrupt nor we don't poll for it. Under big stress nothing wrong happens. We rely only on interrupts triggered by INTX_ASSERT signal, so we care only for its status.

Thanks for the detailed explanation. So in practice, the code should be just:

	for (i = 0; i < LEGACY_IRQ_NUM; i++) {
		int virq;

		if (!(status & PCIE_ISR0_INTX_ASSERT(i)))
			continue;

		advk_writel(pcie, PCIE_ISR0_INTX_ASSERT(i), PCIE_ISR0_REG);
		virq = irq_find_mapping(pcie->irq_domain, i);
		generic_handle_irq(virq);
	}

and that's it.

Is that what you suggest?

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com
#20 lore raw

PATCH 3/3 arm64: dts: marvell: PCIe support for Armada 3700

In reply to Thomas Petazzoni

This commit adds the SoC-level description of the PCIe controller found on the Marvell Armada 3700, and also enables this PCIe controller on the development board for this SoC.

---
 arch/arm64/boot/dts/marvell/armada-3720-db.dts |  5 +++++
 arch/arm64/boot/dts/marvell/armada-37xx.dtsi   | 23 +++++++++++++++++++++++
 2 files changed, 28 insertions(+)
arch/arm64/boot/dts/marvell/armada-3720-db.dts +5 -0
index 86110a6..1372e9a6 100644
@@ -76,3 +76,8 @@
 &usb3 {
 	status = "okay";
 };
+
+/* CON17 (PCIe) / CON12 (mini-PCIe) */
+&pcie0 {
+	status = "okay";
+};
arch/arm64/boot/dts/marvell/armada-37xx.dtsi +23 -0
index 9e2efb8..01d3b20 100644
@@ -141,5 +141,28 @@
 				      <0x1d40000 0x40000>; /* GICR */
 			};
 		};
+
+		pcie0: pcie at d0070000 {
+			compatible = "marvell,armada-3700-pcie";
+			device_type = "pci";
+			status = "disabled";
+			reg = <0 0xd0070000 0 0x20000>;
+			#address-cells = <3>;
+			#size-cells = <2>;
+			bus-range = <0x00 0xff>;
+			interrupts = <GIC_SPI 29 IRQ_TYPE_LEVEL_HIGH>;
+			#interrupt-cells = <1>;
+			ranges = <0x82000000 0 0xe8000000   0 0xe8000000 0 0x1000000 /* Port 0 MEM */
+				  0x81000000 0 0xe9000000   0 0xe9000000 0 0x10000>; /* Port 0 IO*/
+			interrupt-map-mask = <0 0 0 7>;
+			interrupt-map = <0 0 0 1 &pcie_intc 0>,
+					<0 0 0 2 &pcie_intc 1>,
+					<0 0 0 3 &pcie_intc 2>,
+					<0 0 0 4 &pcie_intc 3>;
+			pcie_intc: interrupt-controller {
+				interrupt-controller;
+				#interrupt-cells = <1>;
+			};
+		};
 	};
 };
Signature · 1 line
2.7.4