Skip to contentSkip to conversation

Add iProc PCIe PAXC and MSI support

PATCH Superseded Ray Jui 2 participants 8 messages Nov 26, 2015 – Nov 27, 2015

Subsystems (4)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Ray Jui

author maintainer
#0 lore raw

v3 cover Add iProc PCIe PAXC and MSI support

This patch series adds support for the iProc PAXC interface and support for event queue based MSI, integrated in the iProc PCIe core

This patch series is based on Linux v4.4-rc1 and is avaliable here:
https://github.com/Broadcom/cygnus-linux/tree/iproc-msi-v3

This patch series is tested on the following platforms:

Changes from v2:
- Improved descriptions in the iProc MSI commit message
- Removed redundant host memory used for MSI address. The MSI posted writes never really hit the memory. Use iProc PCIe controller base address instead
- Fixed deadlock when MSI vectors are used up
- Enforced the number of MSI groups to always be multiple of the number of CPUs
- Improved the efficiency of MSI event processing by only updating the head pointer after finishing processing all outstanding events
- Fixed error handling code to make sure all configurations are rolled back
- Added code to zero the host memory used for event queues after allocation
- Removed redundant 'brcm,num-eq-region' and 'brcm,num-msi-msg-region' DT properties. Now determine the number of regions based on interface type
- Other misc. changes

PAXB:
- Broadcom Cygnus wireless audio board with Intel e1000e network card
- Broadcom NS2 SVK board with Intel e1000e network card

PAXC:
- Broadcom NS2 SVK board with integrated dual 10 Gig Ethernet ports

Changes from v1:
- Fixed incorrect 1-to-1 mapping between MSI vector and GIC interrupt. Now the driver supports multiple MSI vectors per GIC interrupt
- Added MSI IRQ affinity support by distributing GIC interrupts across available CPU cores and dynamically steer MSI vectors to the target CPU
- replace readl/writel with readl_relaxed/writel_relaxed since all register accesses within the iProc MSI driver are to/from the same I/O block, i.e., the iProc PCIe core
- Removed all redundant irq_chip callback assignments
- Changed to use uncached host memory for both MSI posted writes and event queues
- Add functions to free resources in error/exit cases
- In pcie-iproc-platform.c, pass in interface type through OF device data
- Moved define for max number of interrupts from pcie-iproc.h to pcie-iproc-msi.c
- Other misc. changes

Ray Jui (5):
PCI: iproc: Update iProc PCIe device tree binding
PCI: iproc: Add PAXC interface support
PCI: iproc: Add iProc PCIe MSI device tree binding
PCI: iproc: Add iProc PCIe MSI support
ARM: dts: Enable MSI support for Broadcom Cygnus

 .../devicetree/bindings/pci/brcm,iproc-pcie.txt    |  40 +-
 arch/arm/boot/dts/bcm-cygnus.dtsi                  |  22 +
 drivers/pci/host/Kconfig                           |   9 +
 drivers/pci/host/Makefile                          |   1 +
 drivers/pci/host/pcie-iproc-bcma.c                 |   1 +
 drivers/pci/host/pcie-iproc-msi.c                  | 678 +++++++++++++++++++++
 drivers/pci/host/pcie-iproc-platform.c             |  25 +-
 drivers/pci/host/pcie-iproc.c                      | 228 +++++--
 drivers/pci/host/pcie-iproc.h                      |  42 +-
 9 files changed, 1003 insertions(+), 43 deletions(-)
 create mode 100644 drivers/pci/host/pcie-iproc-msi.c
Signature · 1 line
1.9.1
author maintainer
#1 lore raw

PATCH v3 1/5 PCI: iproc: Update iProc PCIe device tree binding

Add a new compatible string "brcm,iproc-pcie-paxc", for PAXC based iProc PCIe root complex. A PAXC based PCIe root complex is connected to emulated endpoint devices internal to the ASIC

---
 Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +4 -1
index 45c2a80..06eae0f 100644
@@ -1,7 +1,10 @@
 * Broadcom iProc PCIe controller with the platform bus interface
 
 Required properties:
-- compatible: Must be "brcm,iproc-pcie"
+- compatible: Must be "brcm,iproc-pcie" for PAXB, or "brcm,iproc-pcie-paxc"
+  for PAXC. PAXB based root complex is used for external endpoint devices.
+  PAXC based root complex is connected to emulated endpoint devices
+  internal to the ASIC
 - reg: base address and length of the PCIe controller I/O register space
 - #interrupt-cells: set to <1>
 - interrupt-map-mask and interrupt-map, standard PCI properties to define the
Signature · 1 line
1.9.1
author maintainer
#2 lore raw

PATCH v3 2/5 PCI: iproc: Add PAXC interface support

In reply to Ray Jui

Traditionally, all iProc PCIe root complexes use PAXB based wrapper, with an integrated on-chip Serdes to support external endpoint devices. On newer iProc platforms, a PAXC based wrapper is introduced, for connection with internally emulated PCIe endpoint devices in the ASIC

This patch adds support for PAXC based iProc PCIe root complex in the iProc PCIe core driver. This change fators out common logic between PAXB and PAXC, and use tables to store register offsets that are different between PAXB and PAXC. This allows the driver to be scaled to support subsequent PAXC revisions in the future

---
 drivers/pci/host/pcie-iproc-platform.c |  24 +++-
 drivers/pci/host/pcie-iproc.c          | 202 +++++++++++++++++++++++++++------
 drivers/pci/host/pcie-iproc.h          |  19 ++++
 3 files changed, 205 insertions(+), 40 deletions(-)
drivers/pci/host/pcie-iproc-platform.c +18 -6
index c9550dc..e8b32d8 100644
@@ -26,8 +26,21 @@
 
 #include "pcie-iproc.h"
 
+static const struct of_device_id iproc_pcie_of_match_table[] = {
+	{
+		.compatible = "brcm,iproc-pcie",
+		.data = (int *)IPROC_PCIE_PAXB,
+	}, {
+		.compatible = "brcm,iproc-pcie-paxc",
+		.data = (int *)IPROC_PCIE_PAXC,
+	},
+	{ /* sentinel */ }
+};
+MODULE_DEVICE_TABLE(of, iproc_pcie_of_match_table);
+
 static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 {
+	const struct of_device_id *of_id;
 	struct iproc_pcie *pcie;
 	struct device_node *np = pdev->dev.of_node;
 	struct resource reg;
@@ -35,11 +48,16 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 	LIST_HEAD(res);
 	int ret;
 
+	of_id = of_match_device(iproc_pcie_of_match_table, &pdev->dev);
+	if (!of_id)
+		return -EINVAL;
+
 	pcie = devm_kzalloc(&pdev->dev, sizeof(struct iproc_pcie), GFP_KERNEL);
 	if (!pcie)
 		return -ENOMEM;
 
 	pcie->dev = &pdev->dev;
+	pcie->type = (enum iproc_pcie_type)of_id->data;
 	platform_set_drvdata(pdev, pcie);
 
 	ret = of_address_to_resource(np, 0, &reg);
@@ -114,12 +132,6 @@ static int iproc_pcie_pltfm_remove(struct platform_device *pdev)
 	return iproc_pcie_remove(pcie);
 }
 
-static const struct of_device_id iproc_pcie_of_match_table[] = {
-	{ .compatible = "brcm,iproc-pcie", },
-	{ /* sentinel */ }
-};
-MODULE_DEVICE_TABLE(of, iproc_pcie_of_match_table);
-
 static struct platform_driver iproc_pcie_pltfm_driver = {
 	.driver = {
 		.name = "iproc-pcie",
drivers/pci/host/pcie-iproc.c +168 -34

295 lines not on this page · raw message

drivers/pci/host/pcie-iproc.h +19 -0
index d3dc940..051b651 100644
@@ -15,6 +15,20 @@
 #define _PCIE_IPROC_H
 
 /**
+ * iProc PCIe interface type
+ *
+ * PAXB is the wrapper used in root complex that can be connected to an
+ * external endpoint device
+ *
+ * PAXC is the wrapper used in root complex dedicated for internal emulated
+ * endpoint devices
+ */
+enum iproc_pcie_type {
+	IPROC_PCIE_PAXB = 0,
+	IPROC_PCIE_PAXC,
+};
+
+/**
  * iProc PCIe outbound mapping
  * @set_oarr_size: indicates the OARR size bit needs to be set
  * @axi_offset: offset from the AXI address to the internal address used by
@@ -29,7 +43,10 @@ struct iproc_pcie_ob {
 
 /**
  * iProc PCIe device
+ *
  * @dev: pointer to device data structure
+ * @type: iProc PCIe interface type
+ * @reg_offsets: register offsets
  * @base: PCIe host controller I/O register base
  * @sysdata: Per PCI controller data (ARM-specific)
  * @root_bus: pointer to root bus
@@ -41,6 +58,8 @@ struct iproc_pcie_ob {
  */
 struct iproc_pcie {
 	struct device *dev;
+	enum iproc_pcie_type type;
+	const u16 *reg_offsets;
 	void __iomem *base;
 #ifdef CONFIG_ARM
 	struct pci_sys_data sysdata;
Signature · 1 line
1.9.1
author maintainer
#3 lore raw

PATCH v3 3/5 PCI: iproc: Add iProc PCIe MSI device tree binding

In reply to Ray Jui

This patch updates the iProc PCIe device tree bindings with added binding information for MSI

---
 .../devicetree/bindings/pci/brcm,iproc-pcie.txt    | 35 ++++++++++++++++++++++
 1 file changed, 35 insertions(+)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +35 -0
index 06eae0f..acbdbad 100644
@@ -35,6 +35,28 @@ Optional:
 - brcm,pcie-ob-oarr-size: Some iProc SoCs need the OARR size bit to be set to
 increase the outbound window size
 
+MSI support (optional):
+
+For older platforms without MSI integrated in the GIC, iProc PCIe core provides
+an event queue based MSI support. The iProc MSI uses host memories to store
+MSI posted writes in the event queues
+
+- msi-parent: Link to the device node of the MSI controller. On newer iProc
+platforms, the MSI controller may be gicv2m or gicv3-its. On older iProc
+platforms without MSI support in its interrupt controller, one may use the
+event queue based MSI support integrated within the iProc PCIe core
+
+When the iProc event queue based MSI is used, one needs to define the
+following properties in the MSI device node:
+- compatible: Must be "brcm,iproc-msi"
+- msi-controller: claims itself as an MSI controller
+- interrupt-parent: Link to its parent interrupt device
+- interrupts: List of interrupt IDs from its parent interrupt device
+
+Optional properties:
+- brcm,pcie-msi-inten: Needs to be present for some older iProc platforms that
+require the interrupt enable registers to be set explicitly to enable MSI
+
 Example:
 	pcie0: pcie@18012000 {
 		compatible = "brcm,iproc-pcie";
@@ -61,6 +83,19 @@ Example:
 		brcm,pcie-ob-oarr-size;
 		brcm,pcie-ob-axi-offset = <0x00000000>;
 		brcm,pcie-ob-window-size = <256>;
+
+		msi-parent = <&msi0>;
+
+		/* iProc event queue based MSI */
+		msi0: msi@18012000 {
+			compatible = "brcm,iproc-msi";
+			msi-controller;
+			interrupt-parent = <&gic>;
+			interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>,
+				     <GIC_SPI 97 IRQ_TYPE_NONE>,
+				     <GIC_SPI 98 IRQ_TYPE_NONE>,
+				     <GIC_SPI 99 IRQ_TYPE_NONE>,
+		};
 	};
 
 	pcie1: pcie@18013000 {
Signature · 1 line
1.9.1
author maintainer
#4 lore raw

PATCH v3 4/5 PCI: iproc: Add iProc PCIe MSI support

In reply to Ray Jui

This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms

The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing

Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt

MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU

Therefore, the actual number of supported MSI vectors is:

M * 64 / N

where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores

This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)

---
 drivers/pci/host/Kconfig               |   9 +
 drivers/pci/host/Makefile              |   1 +
 drivers/pci/host/pcie-iproc-bcma.c     |   1 +
 drivers/pci/host/pcie-iproc-msi.c      | 678 +++++++++++++++++++++++++++++++++
 drivers/pci/host/pcie-iproc-platform.c |   1 +
 drivers/pci/host/pcie-iproc.c          |  26 ++
 drivers/pci/host/pcie-iproc.h          |  23 +-
 7 files changed, 737 insertions(+), 2 deletions(-)
 create mode 100644 drivers/pci/host/pcie-iproc-msi.c
drivers/pci/host/Kconfig +9 -0
index f131ba9..972e906 100644
@@ -126,6 +126,15 @@ config PCIE_IPROC
 	  iProc family of SoCs. An appropriate bus interface driver also needs
 	  to be enabled
 
+config PCIE_IPROC_MSI
+	bool "Broadcom iProc PCIe MSI support"
+	depends on ARCH_BCM_IPROC && PCI_MSI
+	select PCI_MSI_IRQ_DOMAIN
+	default ARCH_BCM_IPROC
+	help
+	  Say Y here if you want to enable MSI support for Broadcom's iProc
+	  PCIe controller
+
 config PCIE_IPROC_PLATFORM
 	tristate "Broadcom iProc PCIe platform bus driver"
 	depends on ARCH_BCM_IPROC || (ARM && COMPILE_TEST)
drivers/pci/host/Makefile +1 -0
index 9d4d3c6..0e4e95e 100644
@@ -15,6 +15,7 @@ obj-$(CONFIG_PCI_XGENE_MSI) += pci-xgene-msi.o
 obj-$(CONFIG_PCI_LAYERSCAPE) += pci-layerscape.o
 obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o
 obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o
+obj-$(CONFIG_PCIE_IPROC_MSI) += pcie-iproc-msi.o
 obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o
 obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o
 obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
drivers/pci/host/pcie-iproc-bcma.c +1 -0
index 96a7d99..0d7bee4 100644
@@ -55,6 +55,7 @@ static int iproc_pcie_bcma_probe(struct bcma_device *bdev)
 	bcma_set_drvdata(bdev, pcie);
 
 	pcie->base = bdev->io_addr;
+	pcie->base_addr = bdev->addr;
 
 	res_mem.start = bdev->addr_s[0];
 	res_mem.end = bdev->addr_s[0] + SZ_128M - 1;
drivers/pci/host/pcie-iproc-msi.c new file +678 -0

678 lines not on this page · raw message

drivers/pci/host/pcie-iproc-platform.c +1 -0
index e8b32d8..1738c52 100644
@@ -71,6 +71,7 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)
 		dev_err(pcie->dev, "unable to map controller registers\n");
 		return -ENOMEM;
 	}
+	pcie->base_addr = reg.start;
 
 	if (of_property_read_bool(np, "brcm,pcie-ob")) {
 		u32 val;
drivers/pci/host/pcie-iproc.c +26 -0
index 24d5b62..b7fda356 100644
@@ -440,6 +440,26 @@ static int iproc_pcie_map_ranges(struct iproc_pcie *pcie,
 	return 0;
 }
 
+static int iproc_pcie_msi_enable(struct iproc_pcie *pcie)
+{
+	struct device_node *msi_node;
+
+	msi_node = of_parse_phandle(pcie->dev->of_node, "msi-parent", 0);
+	if (!msi_node)
+		return -ENODEV;
+
+	/*
+	 * If another MSI controller is being used, the call below should fail
+	 * but that is okay
+	 */
+	return iproc_msi_init(pcie, msi_node);
+}
+
+static void iproc_pcie_msi_disable(struct iproc_pcie *pcie)
+{
+	iproc_msi_exit(pcie);
+}
+
 int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res)
 {
 	int ret;
@@ -507,6 +527,10 @@ int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res)
 
 	iproc_pcie_enable(pcie);
 
+	if (IS_ENABLED(CONFIG_PCI_MSI))
+		if (iproc_pcie_msi_enable(pcie))
+			dev_info(pcie->dev, "not using iProc MSI\n");
+
 	pci_scan_child_bus(bus);
 	pci_assign_unassigned_bus_resources(bus);
 	pci_fixup_irqs(pci_common_swizzle, pcie->map_irq);
@@ -531,6 +555,8 @@ int iproc_pcie_remove(struct iproc_pcie *pcie)
 	pci_stop_root_bus(pcie->root_bus);
 	pci_remove_root_bus(pcie->root_bus);
 
+	iproc_pcie_msi_disable(pcie);
+
 	phy_power_off(pcie->phy);
 	phy_exit(pcie->phy);
 
drivers/pci/host/pcie-iproc.h +21 -2
index 051b651..c41db15 100644
@@ -41,6 +41,8 @@ struct iproc_pcie_ob {
 	resource_size_t window_size;
 };
 
+struct iproc_msi;
+
 /**
  * iProc PCIe device
  *
@@ -48,19 +50,21 @@ struct iproc_pcie_ob {
  * @type: iProc PCIe interface type
  * @reg_offsets: register offsets
  * @base: PCIe host controller I/O register base
+ * @base_addr: PCIe host controller register base physical address
  * @sysdata: Per PCI controller data (ARM-specific)
  * @root_bus: pointer to root bus
  * @phy: optional PHY device that controls the Serdes
- * @irqs: interrupt IDs
  * @map_irq: function callback to map interrupts
- * @need_ob_cfg: indidates SW needs to configure the outbound mapping window
+ * @need_ob_cfg: indicates SW needs to configure the outbound mapping window
  * @ob: outbound mapping parameters
+ * @msi: MSI data
  */
 struct iproc_pcie {
 	struct device *dev;
 	enum iproc_pcie_type type;
 	const u16 *reg_offsets;
 	void __iomem *base;
+	phys_addr_t base_addr;
 #ifdef CONFIG_ARM
 	struct pci_sys_data sysdata;
 #endif
@@ -69,9 +73,24 @@ struct iproc_pcie {
 	int (*map_irq)(const struct pci_dev *, u8, u8);
 	bool need_ob_cfg;
 	struct iproc_pcie_ob ob;
+	struct iproc_msi *msi;
 };
 
 int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res);
 int iproc_pcie_remove(struct iproc_pcie *pcie);
 
+#ifdef CONFIG_PCI_MSI
+int iproc_msi_init(struct iproc_pcie *pcie, struct device_node *node);
+void iproc_msi_exit(struct iproc_pcie *pcie);
+#else
+static inline int iproc_msi_init(struct iproc_pcie *pcie,
+				 struct device_node *node)
+{
+	return -ENODEV;
+}
+static void iproc_msi_exit(struct iproc_pcie *pcie)
+{
+}
+#endif
+
 #endif /* _PCIE_IPROC_H */
Signature · 1 line
1.9.1
#5 lore raw

Re: [PATCH v3 4/5] PCI: iproc: Add iProc PCIe MSI support

Quoted 47 lines · Ray Jui

This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms

The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing

Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt

MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU

Therefore, the actual number of supported MSI vectors is:

M * 64 / N

where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores

This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)

---
 drivers/pci/host/Kconfig               |   9 +
 drivers/pci/host/Makefile              |   1 +
 drivers/pci/host/pcie-iproc-bcma.c     |   1 +
 drivers/pci/host/pcie-iproc-msi.c      | 678 +++++++++++++++++++++++++++++++++
 drivers/pci/host/pcie-iproc-platform.c |   1 +
 drivers/pci/host/pcie-iproc.c          |  26 ++
 drivers/pci/host/pcie-iproc.h          |  23 +-
 7 files changed, 737 insertions(+), 2 deletions(-)
 create mode 100644 drivers/pci/host/pcie-iproc-msi.c

[...]

Quoted 5 lines
drivers/pci/host/pcie-iproc-msi.c new file +0 -0
new file mode 100644
index 0000000..f64399a

[...]

Quoted 14 lines
drivers/pci/host/pcie-iproc-msi.c new file +14 -0
+int iproc_msi_init(struct iproc_pcie *pcie, struct device_node *node)
+{
+	struct iproc_msi *msi;
+	int i, ret;
+	unsigned int cpu;
+
+	if (!of_device_is_compatible(node, "brcm,iproc-msi"))
+		return -ENODEV;
+
+	if (!of_find_property(node, "msi-controller", NULL))
+		return -ENODEV;
+
+	if (pcie->msi)
+		return -EBUSY;

drivers/pci/host/pcie-iproc-msi.c line 539

Quoted 10 lines
drivers/pci/host/pcie-iproc-msi.c new file +10 -0
+
+	msi = devm_kzalloc(pcie->dev, sizeof(*msi), GFP_KERNEL);
+	if (!msi)
+		return -ENOMEM;
+
+	msi->pcie = pcie;
+	pcie->msi = msi;
+	msi->msi_addr = pcie->base_addr;
+	mutex_init(&msi->bitmap_lock);
+	msi->nr_cpus = num_online_cpus();

What if some of the CPUs are offline at that time, but come back online later? My guess is that you need to have num_possible_cpus().

Quoted 70 lines
drivers/pci/host/pcie-iproc-msi.c new file +70 -0
+
+	msi->nr_irqs = of_irq_count(node);
+	if (!msi->nr_irqs) {
+		dev_err(pcie->dev, "found no MSI GIC interrupt\n");
+		return -ENODEV;
+	}
+
+	if (msi->nr_irqs > NR_HW_IRQS) {
+		dev_warn(pcie->dev, "too many MSI GIC interrupts defined %d\n",
+			 msi->nr_irqs);
+		msi->nr_irqs = NR_HW_IRQS;
+	}
+
+	if (msi->nr_irqs < msi->nr_cpus) {
+		dev_err(pcie->dev,
+			"not enough GIC interrupts for MSI affinity\n");
+		return -EINVAL;
+	}
+
+	if (msi->nr_irqs % msi->nr_cpus != 0) {
+		msi->nr_irqs -= msi->nr_irqs % msi->nr_cpus;
+		dev_warn(pcie->dev, "Reducing number of interrupts to %d\n",
+			 msi->nr_irqs);
+	}
+
+	switch (pcie->type) {
+	case IPROC_PCIE_PAXB:
+		msi->reg_offsets = iproc_msi_reg_paxb;
+		msi->nr_eq_region = 1;
+		msi->nr_msi_region = 1;
+		break;
+	case IPROC_PCIE_PAXC:
+		msi->reg_offsets = iproc_msi_reg_paxc;
+		msi->nr_eq_region = msi->nr_irqs;
+		msi->nr_msi_region = msi->nr_irqs;
+		break;
+	default:
+		dev_err(pcie->dev, "incompatible iProc PCIe interface\n");
+		return -EINVAL;
+	}
+
+	if (of_find_property(node, "brcm,pcie-msi-inten", NULL))
+		msi->has_inten_reg = true;
+
+	msi->nr_msi_vecs = msi->nr_irqs * EQ_LEN;
+	msi->bitmap = devm_kcalloc(pcie->dev, BITS_TO_LONGS(msi->nr_msi_vecs),
+				   sizeof(*msi->bitmap), GFP_KERNEL);
+	if (!msi->bitmap)
+		return -ENOMEM;
+
+	msi->grps = devm_kcalloc(pcie->dev, msi->nr_irqs, sizeof(*msi->grps),
+				 GFP_KERNEL);
+	if (!msi->grps)
+		return -ENOMEM;
+
+	for (i = 0; i < msi->nr_irqs; i++) {
+		unsigned int irq = irq_of_parse_and_map(node, i);
+
+		if (!irq) {
+			dev_err(pcie->dev, "unable to parse/map interrupt\n");
+			ret = -ENODEV;
+			goto free_irqs;
+		}
+		msi->grps[i].gic_irq = irq;
+		msi->grps[i].msi = msi;
+		msi->grps[i].eq = i;
+	}
+
+	/* reserve memory for MSI event queue */
+	msi->eq_cpu = dma_alloc_coherent(pcie->dev,

drivers/pci/host/pcie-iproc-msi.c line 619

Quoted 10 lines
drivers/pci/host/pcie-iproc-msi.c new file +10 -0
+					 msi->nr_eq_region * EQ_MEM_REGION_SIZE,
+					 &msi->eq_dma, GFP_KERNEL);
+	if (!msi->eq_cpu) {
+		ret = -ENOMEM;
+		goto free_irqs;
+	}
+
+	/* zero out all memory contents of the MSI event queues */
+	memset(msi->eq_cpu, 0, msi->nr_eq_region * EQ_MEM_REGION_SIZE);
+

Please use dma_zalloc_coherent instead of memsetting the memory.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
author maintainer
#6 lore raw

Re: [PATCH v3 4/5] PCI: iproc: Add iProc PCIe MSI support

Hi Marc,

Quoted 78 lines · Marc Zyngier
Quoted 47 lines · Ray Jui

This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms

The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing

Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt

MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU

Therefore, the actual number of supported MSI vectors is:

M * 64 / N

where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores

This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)

---
  drivers/pci/host/Kconfig               |   9 +
  drivers/pci/host/Makefile              |   1 +
  drivers/pci/host/pcie-iproc-bcma.c     |   1 +
  drivers/pci/host/pcie-iproc-msi.c      | 678 +++++++++++++++++++++++++++++++++
  drivers/pci/host/pcie-iproc-platform.c |   1 +
  drivers/pci/host/pcie-iproc.c          |  26 ++
  drivers/pci/host/pcie-iproc.h          |  23 +-
  7 files changed, 737 insertions(+), 2 deletions(-)

create mode 100644 drivers/pci/host/pcie-iproc-msi.c

[...]

Quoted 5 lines
drivers/pci/host/pcie-iproc-msi.c new file +0 -0
new file mode 100644
index 0000000..f64399a

[...]

Quoted 19 lines
drivers/pci/host/pcie-iproc-msi.c new file +19 -0
+int iproc_msi_init(struct iproc_pcie *pcie, struct device_node *node)
+{
+	struct iproc_msi *msi;
+	int i, ret;
+	unsigned int cpu;
+
+	if (!of_device_is_compatible(node, "brcm,iproc-msi"))
+		return -ENODEV;
+
+	if (!of_find_property(node, "msi-controller", NULL))
+		return -ENODEV;
+
+	if (pcie->msi)
+		return -EBUSY;
+
+	msi = devm_kzalloc(pcie->dev, sizeof(*msi), GFP_KERNEL);
+	if (!msi)
+		return -ENOMEM;
+
Quoted 9 lines

drivers/pci/host/pcie-iproc-msi.c line 539

Quoted 5 lines
drivers/pci/host/pcie-iproc-msi.c new file +5 -0
+	msi->pcie = pcie;
+	pcie->msi = msi;
+	msi->msi_addr = pcie->base_addr;
+	mutex_init(&msi->bitmap_lock);
+	msi->nr_cpus = num_online_cpus();

What if some of the CPUs are offline at that time, but come back online later? My guess is that you need to have num_possible_cpus().

Okay let me change this back to num_possible_cpus().

Quoted 74 lines
Quoted 74 lines
drivers/pci/host/pcie-iproc-msi.c new file +74 -0
+
+	msi->nr_irqs = of_irq_count(node);
+	if (!msi->nr_irqs) {
+		dev_err(pcie->dev, "found no MSI GIC interrupt\n");
+		return -ENODEV;
+	}
+
+	if (msi->nr_irqs > NR_HW_IRQS) {
+		dev_warn(pcie->dev, "too many MSI GIC interrupts defined %d\n",
+			 msi->nr_irqs);
+		msi->nr_irqs = NR_HW_IRQS;
+	}
+
+	if (msi->nr_irqs < msi->nr_cpus) {
+		dev_err(pcie->dev,
+			"not enough GIC interrupts for MSI affinity\n");
+		return -EINVAL;
+	}
+
+	if (msi->nr_irqs % msi->nr_cpus != 0) {
+		msi->nr_irqs -= msi->nr_irqs % msi->nr_cpus;
+		dev_warn(pcie->dev, "Reducing number of interrupts to %d\n",
+			 msi->nr_irqs);
+	}
+
+	switch (pcie->type) {
+	case IPROC_PCIE_PAXB:
+		msi->reg_offsets = iproc_msi_reg_paxb;
+		msi->nr_eq_region = 1;
+		msi->nr_msi_region = 1;
+		break;
+	case IPROC_PCIE_PAXC:
+		msi->reg_offsets = iproc_msi_reg_paxc;
+		msi->nr_eq_region = msi->nr_irqs;
+		msi->nr_msi_region = msi->nr_irqs;
+		break;
+	default:
+		dev_err(pcie->dev, "incompatible iProc PCIe interface\n");
+		return -EINVAL;
+	}
+
+	if (of_find_property(node, "brcm,pcie-msi-inten", NULL))
+		msi->has_inten_reg = true;
+
+	msi->nr_msi_vecs = msi->nr_irqs * EQ_LEN;
+	msi->bitmap = devm_kcalloc(pcie->dev, BITS_TO_LONGS(msi->nr_msi_vecs),
+				   sizeof(*msi->bitmap), GFP_KERNEL);
+	if (!msi->bitmap)
+		return -ENOMEM;
+
+	msi->grps = devm_kcalloc(pcie->dev, msi->nr_irqs, sizeof(*msi->grps),
+				 GFP_KERNEL);
+	if (!msi->grps)
+		return -ENOMEM;
+
+	for (i = 0; i < msi->nr_irqs; i++) {
+		unsigned int irq = irq_of_parse_and_map(node, i);
+
+		if (!irq) {
+			dev_err(pcie->dev, "unable to parse/map interrupt\n");
+			ret = -ENODEV;
+			goto free_irqs;
+		}
+		msi->grps[i].gic_irq = irq;
+		msi->grps[i].msi = msi;
+		msi->grps[i].eq = i;
+	}
+
+	/* reserve memory for MSI event queue */
+	msi->eq_cpu = dma_alloc_coherent(pcie->dev,
+					 msi->nr_eq_region * EQ_MEM_REGION_SIZE,
+					 &msi->eq_dma, GFP_KERNEL);
+	if (!msi->eq_cpu) {
+		ret = -ENOMEM;
Quoted 8 lines

drivers/pci/host/pcie-iproc-msi.c line 619

Quoted 6 lines
drivers/pci/host/pcie-iproc-msi.c new file +6 -0
+		goto free_irqs;
+	}
+
+	/* zero out all memory contents of the MSI event queues */
+	memset(msi->eq_cpu, 0, msi->nr_eq_region * EQ_MEM_REGION_SIZE);
+

Please use dma_zalloc_coherent instead of memsetting the memory.

Definitely. Will do.

Quoted 5 lines

Thanks,

	M.

Thanks, Marc!

Ray

author maintainer
#7 lore raw

PATCH v3 5/5 ARM: dts: Enable MSI support for Broadcom Cygnus

In reply to Ray Jui

Enable MSI support for Broadcom Cygnus platforms

---
 arch/arm/boot/dts/bcm-cygnus.dtsi | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/arm/boot/dts/bcm-cygnus.dtsi +22 -0
index 2778533..789b519 100644
@@ -162,6 +162,17 @@
 				  0x82000000 0 0x20000000 0x20000000 0 0x04000000>;
 
 			status = "disabled";
+
+			msi-parent = <&msi0>;
+			msi0: msi@18012000 {
+				compatible = "brcm,iproc-msi";
+				msi-controller;
+				interrupt-parent = <&gic>;
+				interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>,
+					     <GIC_SPI 97 IRQ_TYPE_NONE>,
+					     <GIC_SPI 98 IRQ_TYPE_NONE>,
+					     <GIC_SPI 99 IRQ_TYPE_NONE>;
+			};
 		};
 
 		pcie1: pcie@18013000 {
@@ -183,6 +194,17 @@
 				  0x82000000 0 0x40000000 0x40000000 0 0x04000000>;
 
 			status = "disabled";
+
+			msi-parent = <&msi1>;
+			msi1: msi@18013000 {
+				compatible = "brcm,iproc-msi";
+				msi-controller;
+				interrupt-parent = <&gic>;
+				interrupts = <GIC_SPI 102 IRQ_TYPE_NONE>,
+					     <GIC_SPI 103 IRQ_TYPE_NONE>,
+					     <GIC_SPI 104 IRQ_TYPE_NONE>,
+					     <GIC_SPI 105 IRQ_TYPE_NONE>;
+			};
 		};
 
 		uart0: serial@18020000 {
Signature · 1 line
1.9.1