Skip to contentSkip to conversation

Add iProc PCIe PAXC and MSI support

PATCH Superseded Ray Jui 3 participants 13 messages Nov 27, 2015 – Dec 2, 2015

Subsystems (4)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Ray Jui

author maintainer
#0 lore raw

v4 cover Add iProc PCIe PAXC and MSI support

This patch series adds support for the iProc PAXC interface and support for event queue based MSI, integrated in the iProc PCIe core

This patch series is based on Linux v4.4-rc1 and is avaliable here:
https://github.com/Broadcom/cygnus-linux/tree/iproc-msi-v4

Between v4 and v3, changes are low risk. Sanity tested below:
PAXB:
- Broadcom NS2 SVK board with Intel e1000e network card

Changes from v3:
- Detect the number of possible CPUs instead online CPUs in the driver. Note CPU notifier based implementation still needs to be added in the future, for proper support of MSI IRQ affinity when CPU is brought online/offline at runtime. Support in this dirver will be added when we support CPU hotplug in one of iProc family of SoCs so the changes can be tested
- Use dma_zalloc_coherent for event queue host memory allocation and zeoring

Changes from v2:
- Improved descriptions in the iProc MSI commit message
- Removed redundant host memory used for MSI address. The MSI posted writes never really hit the memory. Use iProc PCIe controller base address instead
- Fixed deadlock when MSI vectors are used up
- Enforced the number of MSI groups to always be multiple of the number of CPUs
- Improved the efficiency of MSI event processing by only updating the head pointer after finishing processing all outstanding events
- Fixed error handling code to make sure all configurations are rolled back
- Added code to zero the host memory used for event queues after allocation
- Removed redundant 'brcm,num-eq-region' and 'brcm,num-msi-msg-region' DT properties. Now determine the number of regions based on interface type
- Other misc. changes

Changes from v1:
- Fixed incorrect 1-to-1 mapping between MSI vector and GIC interrupt. Now the driver supports multiple MSI vectors per GIC interrupt
- Added MSI IRQ affinity support by distributing GIC interrupts across available CPU cores and dynamically steer MSI vectors to the target CPU
- replace readl/writel with readl_relaxed/writel_relaxed since all register accesses within the iProc MSI driver are to/from the same I/O block, i.e., the iProc PCIe core
- Removed all redundant irq_chip callback assignments
- Changed to use uncached host memory for both MSI posted writes and event queues
- Add functions to free resources in error/exit cases
- In pcie-iproc-platform.c, pass in interface type through OF device data
- Moved define for max number of interrupts from pcie-iproc.h to pcie-iproc-msi.c
- Other misc. changes

Ray Jui (5):
PCI: iproc: Update iProc PCIe device tree binding
PCI: iproc: Add PAXC interface support
PCI: iproc: Add iProc PCIe MSI device tree binding
PCI: iproc: Add iProc PCIe MSI support
ARM: dts: Enable MSI support for Broadcom Cygnus

 .../devicetree/bindings/pci/brcm,iproc-pcie.txt    |  40 +-
 arch/arm/boot/dts/bcm-cygnus.dtsi                  |  22 +
 drivers/pci/host/Kconfig                           |   9 +
 drivers/pci/host/Makefile                          |   1 +
 drivers/pci/host/pcie-iproc-bcma.c                 |   1 +
 drivers/pci/host/pcie-iproc-msi.c                  | 675 +++++++++++++++++++++
 drivers/pci/host/pcie-iproc-platform.c             |  25 +-
 drivers/pci/host/pcie-iproc.c                      | 228 +++++--
 drivers/pci/host/pcie-iproc.h                      |  42 +-
 9 files changed, 1000 insertions(+), 43 deletions(-)
 create mode 100644 drivers/pci/host/pcie-iproc-msi.c
Signature · 1 line
1.9.1
author maintainer
#1 lore raw

PATCH v4 1/5 PCI: iproc: Update iProc PCIe device tree binding

Add a new compatible string "brcm,iproc-pcie-paxc", for PAXC based iProc PCIe root complex. A PAXC based PCIe root complex is connected to emulated endpoint devices internal to the ASIC

---
 Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +4 -1
index 45c2a80..06eae0f 100644
@@ -1,7 +1,10 @@
 * Broadcom iProc PCIe controller with the platform bus interface
 
 Required properties:
-- compatible: Must be "brcm,iproc-pcie"
+- compatible: Must be "brcm,iproc-pcie" for PAXB, or "brcm,iproc-pcie-paxc"
+  for PAXC. PAXB based root complex is used for external endpoint devices.
+  PAXC based root complex is connected to emulated endpoint devices
+  internal to the ASIC
 - reg: base address and length of the PCIe controller I/O register space
 - #interrupt-cells: set to <1>
 - interrupt-map-mask and interrupt-map, standard PCI properties to define the
Signature · 1 line
1.9.1
author maintainer
#2 lore raw

PATCH v4 2/5 PCI: iproc: Add PAXC interface support

In reply to Ray Jui

Traditionally, all iProc PCIe root complexes use PAXB based wrapper, with an integrated on-chip Serdes to support external endpoint devices. On newer iProc platforms, a PAXC based wrapper is introduced, for connection with internally emulated PCIe endpoint devices in the ASIC

This patch adds support for PAXC based iProc PCIe root complex in the iProc PCIe core driver. This change fators out common logic between PAXB and PAXC, and use tables to store register offsets that are different between PAXB and PAXC. This allows the driver to be scaled to support subsequent PAXC revisions in the future

---
 drivers/pci/host/pcie-iproc-platform.c |  24 +++-
 drivers/pci/host/pcie-iproc.c          | 202 +++++++++++++++++++++++++++------
 drivers/pci/host/pcie-iproc.h          |  19 ++++
 3 files changed, 205 insertions(+), 40 deletions(-)
drivers/pci/host/pcie-iproc-platform.c +18 -6
index c9550dc..e8b32d8 100644
@@ -26,8 +26,21 @@
 
 #include "pcie-iproc.h"
 
+static const struct of_device_id iproc_pcie_of_match_table[] = {
+	{
+		.compatible = "brcm,iproc-pcie",
+		.data = (int *)IPROC_PCIE_PAXB,
+	}, {
+		.compatible = "brcm,iproc-pcie-paxc",
+		.data = (int *)IPROC_PCIE_PAXC,
+	},
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, iproc_pcie_of_match_table);
+
 static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 {
+	const struct of_device_id *of_id;
 	struct iproc_pcie *pcie;
 	struct device_node *np = pdev->dev.of_node;
 	struct resource reg;
@@ -35,11 +48,16 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 	LIST_HEAD(res);
 	int ret;
 
+	of_id = of_match_device(iproc_pcie_of_match_table, &pdev->dev);
+	if (!of_id)
+		return -EINVAL;
+
 	pcie = devm_kzalloc(&pdev->dev, sizeof(struct iproc_pcie), GFP_KERNEL);
 	if (!pcie)
 		return -ENOMEM;
 
 	pcie->dev = &pdev->dev;
+	pcie->type = (enum iproc_pcie_type)of_id->data;
 	platform_set_drvdata(pdev, pcie);
 
 	ret = of_address_to_resource(np, 0, &reg);
@@ -114,12 +132,6 @@ static int iproc_pcie_pltfm_remove(struct platform_device *pdev)
 	return iproc_pcie_remove(pcie);
 }
 
-static const struct of_device_id iproc_pcie_of_match_table[] = {
-	{ .compatible = "brcm,iproc-pcie", },
-	{ /* sentinel */ }
-};
-MODULE_DEVICE_TABLE(of, iproc_pcie_of_match_table);
-
 static struct platform_driver iproc_pcie_pltfm_driver = {
 	.driver = {
 		.name = "iproc-pcie",
drivers/pci/host/pcie-iproc.c +168 -34

295 lines not on this page · raw message

drivers/pci/host/pcie-iproc.h +19 -0
index d3dc940..051b651 100644
@@ -15,6 +15,20 @@
 #define _PCIE_IPROC_H
 
 /**
+ * iProc PCIe interface type
+ *
+ * PAXB is the wrapper used in root complex that can be connected to an
+ * external endpoint device
+ *
+ * PAXC is the wrapper used in root complex dedicated for internal emulated
+ * endpoint devices
+ */
+enum iproc_pcie_type {
+	IPROC_PCIE_PAXB = 0,
+	IPROC_PCIE_PAXC,
+};
+
+/**
  * iProc PCIe outbound mapping
  * @set_oarr_size: indicates the OARR size bit needs to be set
  * @axi_offset: offset from the AXI address to the internal address used by
@@ -29,7 +43,10 @@ struct iproc_pcie_ob {
 
 /**
  * iProc PCIe device
+ *
  * @dev: pointer to device data structure
+ * @type: iProc PCIe interface type
+ * @reg_offsets: register offsets
  * @base: PCIe host controller I/O register base
  * @sysdata: Per PCI controller data (ARM-specific)
  * @root_bus: pointer to root bus
@@ -41,6 +58,8 @@ struct iproc_pcie_ob {
  */
 struct iproc_pcie {
 	struct device *dev;
+	enum iproc_pcie_type type;
+	const u16 *reg_offsets;
 	void __iomem *base;
 #ifdef CONFIG_ARM
 	struct pci_sys_data sysdata;
Signature · 1 line
1.9.1
#3 lore raw

Re: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support

drivers/pci/host/pcie-iproc-platform.c line 39

Quoted 12 lines · On Friday 27 November 2015 09:37:45 Ray Jui wrote:
drivers/pci/host/pcie-iproc-platform.c +11 -0
+static const struct of_device_id iproc_pcie_of_match_table[] = {
+       {
+               .compatible = "brcm,iproc-pcie",
+               .data = (int *)IPROC_PCIE_PAXB,
+       }, {
+               .compatible = "brcm,iproc-pcie-paxc",
+               .data = (int *)IPROC_PCIE_PAXC,
+       },
+       { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, iproc_pcie_of_match_table);

You seem to only need the identifiers in order to set a single pointer, so just point to that array directly. Alternatively, do the more common thing and point to a structure of function pointers and have different implementations of the low-level access functions there.

Arnd

author maintainer
#4 lore raw

Re: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support

Hi Arnd,

Quoted 16 lines · Arnd Bergmann

drivers/pci/host/pcie-iproc-platform.c line 39

Quoted 12 lines · On Friday 27 November 2015 09:37:45 Ray Jui wrote:
drivers/pci/host/pcie-iproc-platform.c +11 -0
+static const struct of_device_id iproc_pcie_of_match_table[] = {
+       {
+               .compatible = "brcm,iproc-pcie",
+               .data = (int *)IPROC_PCIE_PAXB,
+       }, {
+               .compatible = "brcm,iproc-pcie-paxc",
+               .data = (int *)IPROC_PCIE_PAXC,
+       },
+       { /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, iproc_pcie_of_match_table);

You seem to only need the identifiers in order to set a single pointer, so just point to that array directly.

Not just a single pointer. This sets the iProc PCIe interface type, which is an enum. Based on the interface type, both the iProc PCIe core driver and the iProc PCIe MSI driver can 1) load the correct set of register offsets; 2) skip/invoke some of the link detection, controller reset related behaviors; 3) setting up the correct number of MSI event queue regions and MSI address regions.

Quoted 4 lines

Alternatively,
do the more common thing and point to a structure of function pointers and have different implementations of the low-level access functions there.

I thought about replacing the reset and link check functions in pcie-iproc.c with callbacks but felt it has not reached the point where we need callbacks to completely differentiate the operations. I'm hoping to wait until I see the next revision of PAXC (PAXC v2) implementation in the next-gen iProc SoC to make the call....

Note even with the callback approach, it would not change much of the pcie-iproc-platform.c here, which still needs to pass in just the interface information but no more than that. Main reason being the core driver and MSI drivers are used by both the platform driver and BCMA driver, and I do not want to duplicate the callback implementation at the bus driver level.

Quoted 3 lines

Arnd

Thanks,

Ray

#5 lore raw

Re: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support

Quoted 10 lines · On Friday 27 November 2015 12:56:05 Ray Jui wrote:
Quoted 3 lines

You seem to only need the identifiers in order to set a single pointer, so just point to that array directly.

Not just a single pointer. This sets the iProc PCIe interface type, which is an enum. Based on the interface type, both the iProc PCIe core driver and the iProc PCIe MSI driver can 1) load the correct set of register offsets; 2) skip/invoke some of the link detection, controller reset related behaviors; 3) setting up the correct number of MSI event queue regions and MSI address regions.

Ok, fair enough. I had missed 2) and 3) here. It could still be done as a structure (rather than function pointers) with individual named members for each register offset along with bool flags for 2) and integers to describe 3).

Please that suggestion some thought, and then decide for yourself, I won't complain if you keep the current version.

Arnd

author maintainer
#6 lore raw

Re: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support

Hi Arnd,

Quoted 17 lines · Arnd Bergmann
Quoted 10 lines · On Friday 27 November 2015 12:56:05 Ray Jui wrote:
Quoted 3 lines

You seem to only need the identifiers in order to set a single pointer, so just point to that array directly.

Not just a single pointer. This sets the iProc PCIe interface type, which is an enum. Based on the interface type, both the iProc PCIe core driver and the iProc PCIe MSI driver can 1) load the correct set of register offsets; 2) skip/invoke some of the link detection, controller reset related behaviors; 3) setting up the correct number of MSI event queue regions and MSI address regions.

Ok, fair enough. I had missed 2) and 3) here. It could still be done as a structure (rather than function pointers) with individual named members for each register offset along with bool flags for 2) and integers to describe 3).

Quoted 5 lines

Please that suggestion some thought, and then decide for yourself, I won't complain if you keep the current version.

Arnd

I'll definitely re-visit this when I need to add support to the next revision of PAXC interface. At that time, what you have suggested may become almost mandatory in order to make the core code clean and more maintainable.

For now, I'll keep this as it is.

Thanks!

Ray

author maintainer
#7 lore raw

PATCH v4 3/5 PCI: iproc: Add iProc PCIe MSI device tree binding

In reply to Ray Jui

This patch updates the iProc PCIe device tree bindings with added binding information for MSI

---
 .../devicetree/bindings/pci/brcm,iproc-pcie.txt    | 35 ++++++++++++++++++++++
 1 file changed, 35 insertions(+)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +35 -0
index 06eae0f..acbdbad 100644
@@ -35,6 +35,28 @@ Optional:
 - brcm,pcie-ob-oarr-size: Some iProc SoCs need the OARR size bit to be set to
 increase the outbound window size
 
+MSI support (optional):
+
+For older platforms without MSI integrated in the GIC, iProc PCIe core provides
+an event queue based MSI support. The iProc MSI uses host memories to store
+MSI posted writes in the event queues
+
+- msi-parent: Link to the device node of the MSI controller. On newer iProc
+platforms, the MSI controller may be gicv2m or gicv3-its. On older iProc
+platforms without MSI support in its interrupt controller, one may use the
+event queue based MSI support integrated within the iProc PCIe core
+
+When the iProc event queue based MSI is used, one needs to define the
+following properties in the MSI device node:
+- compatible: Must be "brcm,iproc-msi"
+- msi-controller: claims itself as an MSI controller
+- interrupt-parent: Link to its parent interrupt device
+- interrupts: List of interrupt IDs from its parent interrupt device
+
+Optional properties:
+- brcm,pcie-msi-inten: Needs to be present for some older iProc platforms that
+require the interrupt enable registers to be set explicitly to enable MSI
+
 Example:
 	pcie0: pcie@18012000 {
 		compatible = "brcm,iproc-pcie";
@@ -61,6 +83,19 @@ Example:
 		brcm,pcie-ob-oarr-size;
 		brcm,pcie-ob-axi-offset = <0x00000000>;
 		brcm,pcie-ob-window-size = <256>;
+
+		msi-parent = <&msi0>;
+
+		/* iProc event queue based MSI */
+		msi0: msi@18012000 {
+			compatible = "brcm,iproc-msi";
+			msi-controller;
+			interrupt-parent = <&gic>;
+			interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>,
+				     <GIC_SPI 97 IRQ_TYPE_NONE>,
+				     <GIC_SPI 98 IRQ_TYPE_NONE>,
+				     <GIC_SPI 99 IRQ_TYPE_NONE>,
+		};
 	};
 
 	pcie1: pcie@18013000 {
Signature · 1 line
1.9.1
author maintainer
#8 lore raw

PATCH v4 4/5 PCI: iproc: Add iProc PCIe MSI support

In reply to Ray Jui

This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms

The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing

Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt

MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU

Therefore, the actual number of supported MSI vectors is:

M * 64 / N

where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores

This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)

---
 drivers/pci/host/Kconfig               |   9 +
 drivers/pci/host/Makefile              |   1 +
 drivers/pci/host/pcie-iproc-bcma.c     |   1 +
 drivers/pci/host/pcie-iproc-msi.c      | 675 +++++++++++++++++++++++++++++++++
 drivers/pci/host/pcie-iproc-platform.c |   1 +
 drivers/pci/host/pcie-iproc.c          |  26 ++
 drivers/pci/host/pcie-iproc.h          |  23 +-
 7 files changed, 734 insertions(+), 2 deletions(-)
 create mode 100644 drivers/pci/host/pcie-iproc-msi.c
drivers/pci/host/Kconfig +9 -0
index f131ba9..972e906 100644
@@ -126,6 +126,15 @@ config PCIE_IPROC
 	  iProc family of SoCs. An appropriate bus interface driver also needs
 	  to be enabled
 
+config PCIE_IPROC_MSI
+	bool "Broadcom iProc PCIe MSI support"
+	depends on ARCH_BCM_IPROC && PCI_MSI
+	select PCI_MSI_IRQ_DOMAIN
+	default ARCH_BCM_IPROC
+	help
+	  Say Y here if you want to enable MSI support for Broadcom's iProc
+	  PCIe controller
+
 config PCIE_IPROC_PLATFORM
 	tristate "Broadcom iProc PCIe platform bus driver"
 	depends on ARCH_BCM_IPROC || (ARM && COMPILE_TEST)
drivers/pci/host/Makefile +1 -0
index 9d4d3c6..0e4e95e 100644
@@ -15,6 +15,7 @@ obj-$(CONFIG_PCI_XGENE_MSI) += pci-xgene-msi.o
 obj-$(CONFIG_PCI_LAYERSCAPE) += pci-layerscape.o
 obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o
 obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
+obj-$(CONFIG_PCIE_IPROC_MSI) += pcie-iproc-msi.o
 obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
 obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
 obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
drivers/pci/host/pcie-iproc-bcma.c +1 -0
index 96a7d99..0d7bee4 100644
@@ -55,6 +55,7 @@ static int iproc_pcie_bcma_probe(struct bcma_device *bdev)
 	bcma_set_drvdata(bdev, pcie);
 
 	pcie->base = bdev->io_addr;
+	pcie->base_addr = bdev->addr;
 
 	res_mem.start = bdev->addr_s[0];
 	res_mem.end = bdev->addr_s[0] + SZ_128M - 1;
drivers/pci/host/pcie-iproc-msi.c new file +675 -0

675 lines not on this page · raw message

drivers/pci/host/pcie-iproc-platform.c +1 -0
index e8b32d8..1738c52 100644
@@ -71,6 +71,7 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 		dev_err(pcie->dev, "unable to map controller registers\n");
 		return -ENOMEM;
 	}
+	pcie->base_addr = reg.start;
 
 	if (of_property_read_bool(np, "brcm,pcie-ob")) {
 		u32 val;
drivers/pci/host/pcie-iproc.c +26 -0
index 24d5b62..b7fda356 100644
@@ -440,6 +440,26 @@ static int iproc_pcie_map_ranges(struct iproc_pcie *pcie,
 	return 0;
 }
 
+static int iproc_pcie_msi_enable(struct iproc_pcie *pcie)
+{
+	struct device_node *msi_node;
+
+	msi_node = of_parse_phandle(pcie->dev->of_node, "msi-parent", 0);
+	if (!msi_node)
+		return -ENODEV;
+
+	/*
+	 * If another MSI controller is being used, the call below should fail
+	 * but that is okay
+	 */
+	return iproc_msi_init(pcie, msi_node);
+}
+
+static void iproc_pcie_msi_disable(struct iproc_pcie *pcie)
+{
+	iproc_msi_exit(pcie);
+}
+
 int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res)
 {
 	int ret;
@@ -507,6 +527,10 @@ int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res)
 
 	iproc_pcie_enable(pcie);
 
+	if (IS_ENABLED(CONFIG_PCI_MSI))
+		if (iproc_pcie_msi_enable(pcie))
+			dev_info(pcie->dev, "not using iProc MSI\n");
+
 	pci_scan_child_bus(bus);
 	pci_assign_unassigned_bus_resources(bus);
 	pci_fixup_irqs(pci_common_swizzle, pcie->map_irq);
@@ -531,6 +555,8 @@ int iproc_pcie_remove(struct iproc_pcie *pcie)
 	pci_stop_root_bus(pcie->root_bus);
 	pci_remove_root_bus(pcie->root_bus);
 
+	iproc_pcie_msi_disable(pcie);
+
 	phy_power_off(pcie->phy);
 	phy_exit(pcie->phy);
 
drivers/pci/host/pcie-iproc.h +21 -2
index 051b651..c41db15 100644
@@ -41,6 +41,8 @@ struct iproc_pcie_ob {
 	resource_size_t window_size;
 };
 
+struct iproc_msi;
+
 /**
  * iProc PCIe device
  *
@@ -48,19 +50,21 @@ struct iproc_pcie_ob {
  * @type: iProc PCIe interface type
  * @reg_offsets: register offsets
  * @base: PCIe host controller I/O register base
+ * @base_addr: PCIe host controller register base physical address
  * @sysdata: Per PCI controller data (ARM-specific)
  * @root_bus: pointer to root bus
  * @phy: optional PHY device that controls the Serdes
- * @irqs: interrupt IDs
  * @map_irq: function callback to map interrupts
- * @need_ob_cfg: indidates SW needs to configure the outbound mapping window
+ * @need_ob_cfg: indicates SW needs to configure the outbound mapping window
  * @ob: outbound mapping parameters
+ * @msi: MSI data
  */
 struct iproc_pcie {
 	struct device *dev;
 	enum iproc_pcie_type type;
 	const u16 *reg_offsets;
 	void __iomem *base;
+	phys_addr_t base_addr;
 #ifdef CONFIG_ARM
 	struct pci_sys_data sysdata;
 #endif
@@ -69,9 +73,24 @@ struct iproc_pcie {
 	int (*map_irq)(const struct pci_dev *, u8, u8);
 	bool need_ob_cfg;
 	struct iproc_pcie_ob ob;
+	struct iproc_msi *msi;
 };
 
 int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res);
 int iproc_pcie_remove(struct iproc_pcie *pcie);
 
+#ifdef CONFIG_PCI_MSI
+int iproc_msi_init(struct iproc_pcie *pcie, struct device_node *node);
+void iproc_msi_exit(struct iproc_pcie *pcie);
+#else
+static inline int iproc_msi_init(struct iproc_pcie *pcie,
+				 struct device_node *node)
+{
+	return -ENODEV;
+}
+static void iproc_msi_exit(struct iproc_pcie *pcie)
+{
+}
+#endif
+
 #endif /* _PCIE_IPROC_H */
Signature · 1 line
1.9.1
#9 lore raw

Re: [PATCH v4 4/5] PCI: iproc: Add iProc PCIe MSI support

Quoted 47 lines · Ray Jui

This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms

The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing

Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt

MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU

Therefore, the actual number of supported MSI vectors is:

M * 64 / N

where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores

This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)

---
 drivers/pci/host/Kconfig               |   9 +
 drivers/pci/host/Makefile              |   1 +
 drivers/pci/host/pcie-iproc-bcma.c     |   1 +
 drivers/pci/host/pcie-iproc-msi.c      | 675 +++++++++++++++++++++++++++++++++
 drivers/pci/host/pcie-iproc-platform.c |   1 +
 drivers/pci/host/pcie-iproc.c          |  26 ++
 drivers/pci/host/pcie-iproc.h          |  23 +-
 7 files changed, 734 insertions(+), 2 deletions(-)
 create mode 100644 drivers/pci/host/pcie-iproc-msi.c

.....

Quoted 4 lines

int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res); int iproc_pcie_remove(struct iproc_pcie *pcie);

drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 91

Quoted 10 lines
drivers/pci/host/pcie-iproc.h +10 -0
+#ifdef CONFIG_PCI_MSI
+int iproc_msi_init(struct iproc_pcie *pcie, struct device_node *node);
+void iproc_msi_exit(struct iproc_pcie *pcie);
+#else
+static inline int iproc_msi_init(struct iproc_pcie *pcie,
+				 struct device_node *node)
+{
+	return -ENODEV;
+}
+static void iproc_msi_exit(struct iproc_pcie *pcie)

Please use static inline here.

drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 96

Quoted 6 lines
drivers/pci/host/pcie-iproc.h +4 -0
+{
+}
+#endif
+
 #endif /* _PCIE_IPROC_H */

Hauke

author maintainer
#10 lore raw

Re: [PATCH v4 4/5] PCI: iproc: Add iProc PCIe MSI support

Quoted 54 lines · Hauke Mehrtens
Quoted 47 lines · Ray Jui

This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms

The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing

Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt

MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU

Therefore, the actual number of supported MSI vectors is:

M * 64 / N

where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores

This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)

---
  drivers/pci/host/Kconfig               |   9 +
  drivers/pci/host/Makefile              |   1 +
  drivers/pci/host/pcie-iproc-bcma.c     |   1 +
  drivers/pci/host/pcie-iproc-msi.c      | 675 +++++++++++++++++++++++++++++++++
  drivers/pci/host/pcie-iproc-platform.c |   1 +
  drivers/pci/host/pcie-iproc.c          |  26 ++
  drivers/pci/host/pcie-iproc.h          |  23 +-
  7 files changed, 734 insertions(+), 2 deletions(-)

create mode 100644 drivers/pci/host/pcie-iproc-msi.c

.....

Quoted 4 lines
  int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res);
  int iproc_pcie_remove(struct iproc_pcie *pcie);
Quoted 13 lines

drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 91

Quoted 10 lines
drivers/pci/host/pcie-iproc.h +10 -0
+#ifdef CONFIG_PCI_MSI
+int iproc_msi_init(struct iproc_pcie *pcie, struct device_node *node);
+void iproc_msi_exit(struct iproc_pcie *pcie);
+#else
+static inline int iproc_msi_init(struct iproc_pcie *pcie,
+				 struct device_node *node)
+{
+	return -ENODEV;
+}
+static void iproc_msi_exit(struct iproc_pcie *pcie)

Please use static inline here.

Right. Will fix. Thanks!

Quoted 9 lines

drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 96

Quoted 6 lines
drivers/pci/host/pcie-iproc.h +4 -0
+{
+}
+#endif
+
  #endif /* _PCIE_IPROC_H */

Hauke

Ray

author maintainer
#11 lore raw

PATCH v4 5/5 ARM: dts: Enable MSI support for Broadcom Cygnus

In reply to Ray Jui

Enable MSI support for Broadcom Cygnus platforms

---
 arch/arm/boot/dts/bcm-cygnus.dtsi | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/arm/boot/dts/bcm-cygnus.dtsi +22 -0
index 2778533..789b519 100644
@@ -162,6 +162,17 @@
 				  0x82000000 0 0x20000000 0x20000000 0 0x04000000>;
 
 			status = "disabled";
+
+			msi-parent = <&msi0>;
+			msi0: msi@18012000 {
+				compatible = "brcm,iproc-msi";
+				msi-controller;
+				interrupt-parent = <&gic>;
+				interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>,
+					     <GIC_SPI 97 IRQ_TYPE_NONE>,
+					     <GIC_SPI 98 IRQ_TYPE_NONE>,
+					     <GIC_SPI 99 IRQ_TYPE_NONE>;
+			};
 		};
 
 		pcie1: pcie@18013000 {
@@ -183,6 +194,17 @@
 				  0x82000000 0 0x40000000 0x40000000 0 0x04000000>;
 
 			status = "disabled";
+
+			msi-parent = <&msi1>;
+			msi1: msi@18013000 {
+				compatible = "brcm,iproc-msi";
+				msi-controller;
+				interrupt-parent = <&gic>;
+				interrupts = <GIC_SPI 102 IRQ_TYPE_NONE>,
+					     <GIC_SPI 103 IRQ_TYPE_NONE>,
+					     <GIC_SPI 104 IRQ_TYPE_NONE>,
+					     <GIC_SPI 105 IRQ_TYPE_NONE>;
+			};
 		};
 
 		uart0: serial@18020000 {
Signature · 1 line
1.9.1
author maintainer
#12 lore raw
In reply to Ray Jui

Hi Bjorn,

Is there any more change that you want me to make from the current patch set besides the inline comment from Hauke on the MSI patch?

Thanks,

Ray

Quoted 68 lines · Ray Jui

This patch series adds support for the iProc PAXC interface and support for event queue based MSI, integrated in the iProc PCIe core

This patch series is based on Linux v4.4-rc1 and is avaliable here:
https://github.com/Broadcom/cygnus-linux/tree/iproc-msi-v4

Between v4 and v3, changes are low risk. Sanity tested below:
PAXB:
- Broadcom NS2 SVK board with Intel e1000e network card

Changes from v3:
- Detect the number of possible CPUs instead online CPUs in the driver. Note CPU notifier based implementation still needs to be added in the future, for proper support of MSI IRQ affinity when CPU is brought online/offline at runtime. Support in this dirver will be added when we support CPU hotplug in one of iProc family of SoCs so the changes can be tested
- Use dma_zalloc_coherent for event queue host memory allocation and zeoring

Changes from v2:
- Improved descriptions in the iProc MSI commit message
- Removed redundant host memory used for MSI address. The MSI posted writes never really hit the memory. Use iProc PCIe controller base address instead
- Fixed deadlock when MSI vectors are used up
- Enforced the number of MSI groups to always be multiple of the number of CPUs
- Improved the efficiency of MSI event processing by only updating the head pointer after finishing processing all outstanding events
- Fixed error handling code to make sure all configurations are rolled back
- Added code to zero the host memory used for event queues after allocation
- Removed redundant 'brcm,num-eq-region' and 'brcm,num-msi-msg-region' DT properties. Now determine the number of regions based on interface type
- Other misc. changes

Changes from v1:
- Fixed incorrect 1-to-1 mapping between MSI vector and GIC interrupt. Now the driver supports multiple MSI vectors per GIC interrupt
- Added MSI IRQ affinity support by distributing GIC interrupts across available CPU cores and dynamically steer MSI vectors to the target CPU
- replace readl/writel with readl_relaxed/writel_relaxed since all register accesses within the iProc MSI driver are to/from the same I/O block, i.e., the iProc PCIe core
- Removed all redundant irq_chip callback assignments
- Changed to use uncached host memory for both MSI posted writes and event queues
- Add functions to free resources in error/exit cases
- In pcie-iproc-platform.c, pass in interface type through OF device data
- Moved define for max number of interrupts from pcie-iproc.h to pcie-iproc-msi.c
- Other misc. changes

Ray Jui (5):
PCI: iproc: Update iProc PCIe device tree binding
PCI: iproc: Add PAXC interface support
PCI: iproc: Add iProc PCIe MSI device tree binding
PCI: iproc: Add iProc PCIe MSI support
ARM: dts: Enable MSI support for Broadcom Cygnus

  .../devicetree/bindings/pci/brcm,iproc-pcie.txt    |  40 +-
  arch/arm/boot/dts/bcm-cygnus.dtsi                  |  22 +
  drivers/pci/host/Kconfig                           |   9 +
  drivers/pci/host/Makefile                          |   1 +
  drivers/pci/host/pcie-iproc-bcma.c                 |   1 +
  drivers/pci/host/pcie-iproc-msi.c                  | 675 +++++++++++++++++++++
  drivers/pci/host/pcie-iproc-platform.c             |  25 +-
  drivers/pci/host/pcie-iproc.c                      | 228 +++++--
  drivers/pci/host/pcie-iproc.h                      |  42 +-
  9 files changed, 1000 insertions(+), 43 deletions(-)

create mode 100644 drivers/pci/host/pcie-iproc-msi.c