Add iProc PCIe PAXC and MSI support
Subsystems (4)
Ray Jui
maintainerv3 cover Add iProc PCIe PAXC and MSI support
This patch series adds support for the iProc PAXC interface and support for event queue based MSI, integrated in the iProc PCIe core
This patch series is based on Linux v4.4-rc1 and is avaliable here:
https://github.com/Broadcom/cygnus-linux/tree/iproc-msi-v3
This patch series is tested on the following platforms:
Changes from v2:
- Improved descriptions in the iProc MSI commit message
- Removed redundant host memory used for MSI address. The MSI posted writes never really hit the memory. Use iProc PCIe controller base address instead
- Fixed deadlock when MSI vectors are used up
- Enforced the number of MSI groups to always be multiple of the number of CPUs
- Improved the efficiency of MSI event processing by only updating the head pointer after finishing processing all outstanding events
- Fixed error handling code to make sure all configurations are rolled back
- Added code to zero the host memory used for event queues after allocation
- Removed redundant 'brcm,num-eq-region' and 'brcm,num-msi-msg-region' DT properties. Now determine the number of regions based on interface type
- Other misc. changes
PAXB:
- Broadcom Cygnus wireless audio board with Intel e1000e network card
- Broadcom NS2 SVK board with Intel e1000e network card
PAXC:
- Broadcom NS2 SVK board with integrated dual 10 Gig Ethernet ports
Changes from v1:
- Fixed incorrect 1-to-1 mapping between MSI vector and GIC interrupt. Now the driver supports multiple MSI vectors per GIC interrupt
- Added MSI IRQ affinity support by distributing GIC interrupts across available CPU cores and dynamically steer MSI vectors to the target CPU
- replace readl/writel with readl_relaxed/writel_relaxed since all register accesses within the iProc MSI driver are to/from the same I/O block, i.e., the iProc PCIe core
- Removed all redundant irq_chip callback assignments
- Changed to use uncached host memory for both MSI posted writes and event queues
- Add functions to free resources in error/exit cases
- In pcie-iproc-platform.c, pass in interface type through OF device data
- Moved define for max number of interrupts from pcie-iproc.h to pcie-iproc-msi.c
- Other misc. changes
Ray Jui (5):
PCI: iproc: Update iProc PCIe device tree binding
PCI: iproc: Add PAXC interface support
PCI: iproc: Add iProc PCIe MSI device tree binding
PCI: iproc: Add iProc PCIe MSI support
ARM: dts: Enable MSI support for Broadcom Cygnus
.../devicetree/bindings/pci/brcm,iproc-pcie.txt | 40 +- arch/arm/boot/dts/bcm-cygnus.dtsi | 22 + drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 678 +++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 25 +- drivers/pci/host/pcie-iproc.c | 228 +++++-- drivers/pci/host/pcie-iproc.h | 42 +- 9 files changed, 1003 insertions(+), 43 deletions(-) create mode 100644 drivers/pci/host/pcie-iproc-msi.c
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v3 1/5 PCI: iproc: Update iProc PCIe device tree binding
Add a new compatible string "brcm,iproc-pcie-paxc", for PAXC based iProc PCIe root complex. A PAXC based PCIe root complex is connected to emulated endpoint devices internal to the ASIC
--- Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +4 -1
@@ -1,7 +1,10 @@ * Broadcom iProc PCIe controller with the platform bus interface Required properties: -- compatible: Must be "brcm,iproc-pcie" +- compatible: Must be "brcm,iproc-pcie" for PAXB, or "brcm,iproc-pcie-paxc" + for PAXC. PAXB based root complex is used for external endpoint devices. + PAXC based root complex is connected to emulated endpoint devices + internal to the ASIC - reg: base address and length of the PCIe controller I/O register space - #interrupt-cells: set to <1> - interrupt-map-mask and interrupt-map, standard PCI properties to define the
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v3 2/5 PCI: iproc: Add PAXC interface support
In reply to Ray JuiTraditionally, all iProc PCIe root complexes use PAXB based wrapper, with an integrated on-chip Serdes to support external endpoint devices. On newer iProc platforms, a PAXC based wrapper is introduced, for connection with internally emulated PCIe endpoint devices in the ASIC
This patch adds support for PAXC based iProc PCIe root complex in the iProc PCIe core driver. This change fators out common logic between PAXB and PAXC, and use tables to store register offsets that are different between PAXB and PAXC. This allows the driver to be scaled to support subsequent PAXC revisions in the future
--- drivers/pci/host/pcie-iproc-platform.c | 24 +++- drivers/pci/host/pcie-iproc.c | 202 +++++++++++++++++++++++++++------ drivers/pci/host/pcie-iproc.h | 19 ++++ 3 files changed, 205 insertions(+), 40 deletions(-)
drivers/pci/host/pcie-iproc-platform.c +18 -6
@@ -26,8 +26,21 @@#include "pcie-iproc.h" +static const struct of_device_id iproc_pcie_of_match_table[] = { + { + .compatible ="brcm,iproc-pcie" , + .data = (int *)IPROC_PCIE_PAXB , + }, { + .compatible ="brcm,iproc-pcie-paxc" , + .data = (int *)IPROC_PCIE_PAXC , + }, + {/* sentinel */ } +}; +MODULE_DEVICE_TABLE (of, iproc_pcie_of_match_table); +static int iproc_pcie_pltfm_probe (struct platform_device *pdev) { +const struct of_device_id *of_id;struct iproc_pcie *pcie;struct device_node *np = pdev->dev.of_node;struct resource reg;
@@ -35,11 +48,16 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)LIST_HEAD (res);int ret; + of_id =of_match_device (iproc_pcie_of_match_table, &pdev->dev); +if (!of_id) +return -EINVAL ; + pcie =devm_kzalloc (&pdev->dev,sizeof (struct iproc_pcie ),GFP_KERNEL );if (!pcie)return -ENOMEM ; pcie->dev = &pdev->dev; + pcie->type = (enum iproc_pcie_type )of_id->data;platform_set_drvdata (pdev, pcie); ret =of_address_to_resource (np,0 , ®);
@@ -114,12 +132,6 @@ static int iproc_pcie_pltfm_remove(struct platform_device *pdev)return iproc_pcie_remove (pcie); } -static const struct of_device_id iproc_pcie_of_match_table[] = { - { .compatible ="brcm,iproc-pcie" , }, - {/* sentinel */ } -}; -MODULE_DEVICE_TABLE (of, iproc_pcie_of_match_table); -static struct platform_driver iproc_pcie_pltfm_driver = { .driver = { .name ="iproc-pcie" ,
drivers/pci/host/pcie-iproc.c +168 -34
295 lines not on this page · raw message
drivers/pci/host/pcie-iproc.h +19 -0
@@ -15,6 +15,20 @@#define _PCIE_IPROC_H/** +* iProc PCIe interface type +* +* PAXB is the wrapper used in root complex that can be connected to an +* external endpoint device +* +* PAXC is the wrapper used in root complex dedicated for internal emulated +* endpoint devices +*/ +enum iproc_pcie_type { +IPROC_PCIE_PAXB =0 , +IPROC_PCIE_PAXC , +}; + +/** * iProc PCIe outbound mapping * @set_oarr_size: indicates the OARR size bit needs to be set * @axi_offset: offset from the AXI address to the internal address used by
@@ -29,7 +43,10 @@ struct iproc_pcie_ob {/** * iProc PCIe device +* * @dev: pointer to device data structure +* @type: iProc PCIe interface type +* @reg_offsets: register offsets * @base: PCIe host controller I/O register base * @sysdata: Per PCI controller data (ARM-specific) * @root_bus: pointer to root bus
@@ -41,6 +58,8 @@ struct iproc_pcie_ob {*/ struct iproc_pcie {struct device *dev; +enum iproc_pcie_type type; +const u16 *reg_offsets;void __iomem *base;#ifdef CONFIG_ARM struct pci_sys_data sysdata;
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v3 3/5 PCI: iproc: Add iProc PCIe MSI device tree binding
In reply to Ray JuiThis patch updates the iProc PCIe device tree bindings with added binding information for MSI
--- .../devicetree/bindings/pci/brcm,iproc-pcie.txt | 35 ++++++++++++++++++++++ 1 file changed, 35 insertions(+)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +35 -0
@@ -35,6 +35,28 @@ Optional: - brcm,pcie-ob-oarr-size: Some iProc SoCs need the OARR size bit to be set to increase the outbound window size +MSI support (optional): + +For older platforms without MSI integrated in the GIC, iProc PCIe core provides +an event queue based MSI support. The iProc MSI uses host memories to store +MSI posted writes in the event queues + +- msi-parent: Link to the device node of the MSI controller. On newer iProc +platforms, the MSI controller may be gicv2m or gicv3-its. On older iProc +platforms without MSI support in its interrupt controller, one may use the +event queue based MSI support integrated within the iProc PCIe core + +When the iProc event queue based MSI is used, one needs to define the +following properties in the MSI device node: +- compatible: Must be "brcm,iproc-msi" +- msi-controller: claims itself as an MSI controller +- interrupt-parent: Link to its parent interrupt device +- interrupts: List of interrupt IDs from its parent interrupt device + +Optional properties: +- brcm,pcie-msi-inten: Needs to be present for some older iProc platforms that +require the interrupt enable registers to be set explicitly to enable MSI + Example: pcie0: pcie@18012000 { compatible = "brcm,iproc-pcie";
@@ -61,6 +83,19 @@ Example: brcm,pcie-ob-oarr-size; brcm,pcie-ob-axi-offset = <0x00000000>; brcm,pcie-ob-window-size = <256>; + + msi-parent = <&msi0>; + + /* iProc event queue based MSI */ + msi0: msi@18012000 { + compatible = "brcm,iproc-msi"; + msi-controller; + interrupt-parent = <&gic>; + interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>, + <GIC_SPI 97 IRQ_TYPE_NONE>, + <GIC_SPI 98 IRQ_TYPE_NONE>, + <GIC_SPI 99 IRQ_TYPE_NONE>, + }; }; pcie1: pcie@18013000 {
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v3 4/5 PCI: iproc: Add iProc PCIe MSI support
In reply to Ray JuiThis patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms
The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing
Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt
MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU
Therefore, the actual number of supported MSI vectors is:
M * 64 / N
where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores
This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)
--- drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 678 +++++++++++++++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 1 + drivers/pci/host/pcie-iproc.c | 26 ++ drivers/pci/host/pcie-iproc.h | 23 +- 7 files changed, 737 insertions(+), 2 deletions(-) create mode 100644 drivers/pci/host/pcie-iproc-msi.c
drivers/pci/host/Kconfig +9 -0
@@ -126,6 +126,15 @@ config PCIE_IPROC iProc family of SoCs. An appropriate bus interface driver also needs to be enabled +config PCIE_IPROC_MSI + bool "Broadcom iProc PCIe MSI support" + depends on ARCH_BCM_IPROC && PCI_MSI + select PCI_MSI_IRQ_DOMAIN + default ARCH_BCM_IPROC + help + Say Y here if you want to enable MSI support for Broadcom's iProc + PCIe controller + config PCIE_IPROC_PLATFORM tristate "Broadcom iProc PCIe platform bus driver" depends on ARCH_BCM_IPROC || (ARM && COMPILE_TEST)
drivers/pci/host/Makefile +1 -0
@@ -15,6 +15,7 @@ obj-$(CONFIG_PCI_XGENE_MSI) += pci-xgene-msi.o obj-$(CONFIG_PCI_LAYERSCAPE) += pci-layerscape.o obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o +obj-$(CONFIG_PCIE_IPROC_MSI) += pcie-iproc-msi.o obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
drivers/pci/host/pcie-iproc-bcma.c +1 -0
@@ -55,6 +55,7 @@ static int iproc_pcie_bcma_probe(struct bcma_device *bdev)bcma_set_drvdata (bdev, pcie); pcie->base = bdev->io_addr; + pcie->base_addr = bdev->addr; res_mem.start = bdev->addr_s[0 ]; res_mem.end = bdev->addr_s[0 ] +SZ_128M -1 ;
drivers/pci/host/pcie-iproc-msi.c new file +678 -0
678 lines not on this page · raw message
drivers/pci/host/pcie-iproc-platform.c +1 -0
@@ -71,6 +71,7 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)dev_err (pcie->dev,"unable to map controller registers\n" );return -ENOMEM ; } + pcie->base_addr = reg.start;if (of_property_read_bool (np,"brcm,pcie-ob" )) {u32 val;
drivers/pci/host/pcie-iproc.c +26 -0
@@ -440,6 +440,26 @@ static int iproc_pcie_map_ranges(struct iproc_pcie *pcie,return 0 ; } +static int iproc_pcie_msi_enable (struct iproc_pcie *pcie) +{ +struct device_node *msi_node; + + msi_node =of_parse_phandle (pcie->dev->of_node,"msi-parent" ,0 ); +if (!msi_node) +return -ENODEV ; + +/* +* If another MSI controller is being used, the call below should fail +* but that is okay +*/ +return iproc_msi_init (pcie, msi_node); +} + +static void iproc_pcie_msi_disable (struct iproc_pcie *pcie) +{ +iproc_msi_exit (pcie); +} +int iproc_pcie_setup (struct iproc_pcie *pcie,struct list_head *res) {int ret;
@@ -507,6 +527,10 @@ int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res)iproc_pcie_enable (pcie); +if (IS_ENABLED (CONFIG_PCI_MSI )) +if (iproc_pcie_msi_enable (pcie)) +dev_info (pcie->dev,"not using iProc MSI\n" ); +pci_scan_child_bus (bus);pci_assign_unassigned_bus_resources (bus);pci_fixup_irqs (pci_common_swizzle, pcie->map_irq);
@@ -531,6 +555,8 @@ int iproc_pcie_remove(struct iproc_pcie *pcie)pci_stop_root_bus (pcie->root_bus);pci_remove_root_bus (pcie->root_bus); +iproc_pcie_msi_disable (pcie); +phy_power_off (pcie->phy);phy_exit (pcie->phy);
drivers/pci/host/pcie-iproc.h +21 -2
@@ -41,6 +41,8 @@ struct iproc_pcie_ob {resource_size_t window_size; }; +struct iproc_msi ; +/** * iProc PCIe device *
@@ -48,19 +50,21 @@ struct iproc_pcie_ob {* @type: iProc PCIe interface type * @reg_offsets: register offsets * @base: PCIe host controller I/O register base +* @base_addr: PCIe host controller register base physical address * @sysdata: Per PCI controller data (ARM-specific) * @root_bus: pointer to root bus * @phy: optional PHY device that controls the Serdes -* @irqs: interrupt IDs * @map_irq: function callback to map interrupts -* @need_ob_cfg: indidates SW needs to configure the outbound mapping window +* @need_ob_cfg: indicates SW needs to configure the outbound mapping window * @ob: outbound mapping parameters +* @msi: MSI data */ struct iproc_pcie {struct device *dev;enum iproc_pcie_type type;const u16 *reg_offsets;void __iomem *base; +phys_addr_t base_addr;#ifdef CONFIG_ARM struct pci_sys_data sysdata;#endif
@@ -69,9 +73,24 @@ struct iproc_pcie {int (*map_irq)(const struct pci_dev *,u8 ,u8 );bool need_ob_cfg;struct iproc_pcie_ob ob; + struct iproc_msi *msi; };int iproc_pcie_setup (struct iproc_pcie *pcie,struct list_head *res);int iproc_pcie_remove (struct iproc_pcie *pcie); +#ifdef CONFIG_PCI_MSI +int iproc_msi_init (struct iproc_pcie *pcie,struct device_node *node); +void iproc_msi_exit (struct iproc_pcie *pcie); +#else +static inline int iproc_msi_init (struct iproc_pcie *pcie, +struct device_node *node) +{ +return -ENODEV ; +} +static void iproc_msi_exit (struct iproc_pcie *pcie) +{ +} +#endif +#endif /* _PCIE_IPROC_H */
Signature · 1 line
1.9.1
Re: [PATCH v3 4/5] PCI: iproc: Add iProc PCIe MSI support
Quoted 47 lines · Ray Jui
This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms
The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing
Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt
MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU
Therefore, the actual number of supported MSI vectors is:
M * 64 / N
where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores
This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)
Signed-off-by Ray Jui <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Vikram Prakash <[email protected]>Reviewed-by Scott Branden <[email protected]>--- drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 678 +++++++++++++++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 1 + drivers/pci/host/pcie-iproc.c | 26 ++ drivers/pci/host/pcie-iproc.h | 23 +- 7 files changed, 737 insertions(+), 2 deletions(-) create mode 100644 drivers/pci/host/pcie-iproc-msi.c
[...]
Quoted 5 lines
drivers/pci/host/pcie-iproc-msi.c new file +0
-0
[...]
Quoted 14 lines
drivers/pci/host/pcie-iproc-msi.c new file +14
-0+int iproc_msi_init (struct iproc_pcie *pcie,struct device_node *node) +{ +struct iproc_msi *msi; +int i, ret; +unsigned int cpu; + +if (!of_device_is_compatible (node,"brcm,iproc-msi" )) +return -ENODEV ; + +if (!of_find_property (node,"msi-controller" ,NULL )) +return -ENODEV ; + +if (pcie->msi) +return -EBUSY ;
drivers/pci/host/pcie-iproc-msi.c line 539
Quoted 10 lines
drivers/pci/host/pcie-iproc-msi.c new file +10
-0+ + msi =devm_kzalloc (pcie->dev,sizeof (*msi),GFP_KERNEL ); +if (!msi) +return -ENOMEM ; + + msi->pcie = pcie; + pcie->msi = msi; + msi->msi_addr = pcie->base_addr; +mutex_init (&msi->bitmap_lock); + msi->nr_cpus =num_online_cpus ();
What if some of the CPUs are offline at that time, but come back online later? My guess is that you need to have num_possible_cpus().
Quoted 70 lines
drivers/pci/host/pcie-iproc-msi.c new file +70
-0+ + msi->nr_irqs =of_irq_count (node); +if (!msi->nr_irqs) { +dev_err (pcie->dev,"found no MSI GIC interrupt\n" ); +return -ENODEV ; + } + +if (msi->nr_irqs >NR_HW_IRQS ) { +dev_warn (pcie->dev,"too many MSI GIC interrupts defined %d\n" , + msi->nr_irqs); + msi->nr_irqs =NR_HW_IRQS ; + } + +if (msi->nr_irqs < msi->nr_cpus) { +dev_err (pcie->dev, +"not enough GIC interrupts for MSI affinity\n" ); +return -EINVAL ; + } + +if (msi->nr_irqs % msi->nr_cpus !=0 ) { + msi->nr_irqs -= msi->nr_irqs % msi->nr_cpus; +dev_warn (pcie->dev,"Reducing number of interrupts to %d\n" , + msi->nr_irqs); + } + +switch (pcie->type) { +case IPROC_PCIE_PAXB : + msi->reg_offsets = iproc_msi_reg_paxb; + msi->nr_eq_region =1 ; + msi->nr_msi_region =1 ; +break ; +case IPROC_PCIE_PAXC : + msi->reg_offsets = iproc_msi_reg_paxc; + msi->nr_eq_region = msi->nr_irqs; + msi->nr_msi_region = msi->nr_irqs; +break ; +default : +dev_err (pcie->dev,"incompatible iProc PCIe interface\n" ); +return -EINVAL ; + } + +if (of_find_property (node,"brcm,pcie-msi-inten" ,NULL )) + msi->has_inten_reg = true; + + msi->nr_msi_vecs = msi->nr_irqs *EQ_LEN ; + msi->bitmap =devm_kcalloc (pcie->dev,BITS_TO_LONGS (msi->nr_msi_vecs), +sizeof (*msi->bitmap),GFP_KERNEL ); +if (!msi->bitmap) +return -ENOMEM ; + + msi->grps =devm_kcalloc (pcie->dev, msi->nr_irqs,sizeof (*msi->grps), +GFP_KERNEL ); +if (!msi->grps) +return -ENOMEM ; + +for (i =0 ; i < msi->nr_irqs; i++) { +unsigned int irq =irq_of_parse_and_map (node, i); + +if (!irq) { +dev_err (pcie->dev,"unable to parse/map interrupt\n" ); + ret = -ENODEV ; + goto free_irqs; + } + msi->grps[i].gic_irq = irq; + msi->grps[i].msi = msi; + msi->grps[i].eq = i; + } + +/* reserve memory for MSI event queue */ + msi->eq_cpu = dma_alloc_coherent(pcie->dev,
drivers/pci/host/pcie-iproc-msi.c line 619
Quoted 10 lines
drivers/pci/host/pcie-iproc-msi.c new file +10
-0+ msi->nr_eq_region *EQ_MEM_REGION_SIZE , + &msi->eq_dma, GFP_KERNEL); +if (!msi->eq_cpu) { + ret = -ENOMEM ; + goto free_irqs; + } + +/* zero out all memory contents of the MSI event queues */ +memset (msi->eq_cpu,0 , msi->nr_eq_region *EQ_MEM_REGION_SIZE ); +
Please use dma_zalloc_coherent instead of memsetting the memory.
Thanks,
M.
Signature · 1 line
Jazz is not dead. It just smells funny...
Ray Jui
maintainerRe: [PATCH v3 4/5] PCI: iproc: Add iProc PCIe MSI support
Hi Marc,
Quoted 78 lines · Marc Zyngier
Quoted 47 lines · Ray Jui
This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms
The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing
Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt
MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU
Therefore, the actual number of supported MSI vectors is:
M * 64 / N
where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores
This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)
Signed-off-by Ray Jui <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Vikram Prakash <[email protected]>Reviewed-by Scott Branden <[email protected]>--- drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 678 +++++++++++++++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 1 + drivers/pci/host/pcie-iproc.c | 26 ++ drivers/pci/host/pcie-iproc.h | 23 +- 7 files changed, 737 insertions(+), 2 deletions(-)create mode 100644
drivers/pci/host/pcie-iproc-msi.c[...]
Quoted 5 lines
drivers/pci/host/pcie-iproc-msi.c new file +0
-0[...]
Quoted 19 lines
drivers/pci/host/pcie-iproc-msi.c new file +19
-0+int iproc_msi_init (struct iproc_pcie *pcie,struct device_node *node) +{ +struct iproc_msi *msi; +int i, ret; +unsigned int cpu; + +if (!of_device_is_compatible (node,"brcm,iproc-msi" )) +return -ENODEV ; + +if (!of_find_property (node,"msi-controller" ,NULL )) +return -ENODEV ; + +if (pcie->msi) +return -EBUSY ; + + msi =devm_kzalloc (pcie->dev,sizeof (*msi),GFP_KERNEL ); +if (!msi) +return -ENOMEM ; +
Quoted 9 lines
drivers/pci/host/pcie-iproc-msi.c line 539
Quoted 5 lines
drivers/pci/host/pcie-iproc-msi.c new file +5
-0+ msi->pcie = pcie; + pcie->msi = msi; + msi->msi_addr = pcie->base_addr; +mutex_init (&msi->bitmap_lock); + msi->nr_cpus =num_online_cpus ();What if some of the CPUs are offline at that time, but come back online later? My guess is that you need to have
num_possible_cpus().
Okay let me change this back to num_possible_cpus().
Quoted 74 lines
Quoted 74 lines
drivers/pci/host/pcie-iproc-msi.c new file +74
-0+ + msi->nr_irqs =of_irq_count (node); +if (!msi->nr_irqs) { +dev_err (pcie->dev,"found no MSI GIC interrupt\n" ); +return -ENODEV ; + } + +if (msi->nr_irqs >NR_HW_IRQS ) { +dev_warn (pcie->dev,"too many MSI GIC interrupts defined %d\n" , + msi->nr_irqs); + msi->nr_irqs =NR_HW_IRQS ; + } + +if (msi->nr_irqs < msi->nr_cpus) { +dev_err (pcie->dev, +"not enough GIC interrupts for MSI affinity\n" ); +return -EINVAL ; + } + +if (msi->nr_irqs % msi->nr_cpus !=0 ) { + msi->nr_irqs -= msi->nr_irqs % msi->nr_cpus; +dev_warn (pcie->dev,"Reducing number of interrupts to %d\n" , + msi->nr_irqs); + } + +switch (pcie->type) { +case IPROC_PCIE_PAXB : + msi->reg_offsets = iproc_msi_reg_paxb; + msi->nr_eq_region =1 ; + msi->nr_msi_region =1 ; +break ; +case IPROC_PCIE_PAXC : + msi->reg_offsets = iproc_msi_reg_paxc; + msi->nr_eq_region = msi->nr_irqs; + msi->nr_msi_region = msi->nr_irqs; +break ; +default : +dev_err (pcie->dev,"incompatible iProc PCIe interface\n" ); +return -EINVAL ; + } + +if (of_find_property (node,"brcm,pcie-msi-inten" ,NULL )) + msi->has_inten_reg = true; + + msi->nr_msi_vecs = msi->nr_irqs *EQ_LEN ; + msi->bitmap =devm_kcalloc (pcie->dev,BITS_TO_LONGS (msi->nr_msi_vecs), +sizeof (*msi->bitmap),GFP_KERNEL ); +if (!msi->bitmap) +return -ENOMEM ; + + msi->grps =devm_kcalloc (pcie->dev, msi->nr_irqs,sizeof (*msi->grps), +GFP_KERNEL ); +if (!msi->grps) +return -ENOMEM ; + +for (i =0 ; i < msi->nr_irqs; i++) { +unsigned int irq =irq_of_parse_and_map (node, i); + +if (!irq) { +dev_err (pcie->dev,"unable to parse/map interrupt\n" ); + ret = -ENODEV ; + goto free_irqs; + } + msi->grps[i].gic_irq = irq; + msi->grps[i].msi = msi; + msi->grps[i].eq = i; + } + +/* reserve memory for MSI event queue */ + msi->eq_cpu =dma_alloc_coherent (pcie->dev, + msi->nr_eq_region *EQ_MEM_REGION_SIZE , + &msi->eq_dma,GFP_KERNEL ); +if (!msi->eq_cpu) { + ret = -ENOMEM ;
Quoted 8 lines
drivers/pci/host/pcie-iproc-msi.c line 619
Quoted 6 lines
drivers/pci/host/pcie-iproc-msi.c new file +6
-0+ goto free_irqs; + } + +/* zero out all memory contents of the MSI event queues */ +memset (msi->eq_cpu,0 , msi->nr_eq_region *EQ_MEM_REGION_SIZE ); +Please use
dma_zalloc_coherentinstead of memsetting the memory.
Definitely. Will do.
Quoted 5 lines
Thanks,
M.
Thanks, Marc!
Ray
Ray Jui
maintainerPATCH v3 5/5 ARM: dts: Enable MSI support for Broadcom Cygnus
In reply to Ray JuiEnable MSI support for Broadcom Cygnus platforms
--- arch/arm/boot/dts/bcm-cygnus.dtsi | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/arm/boot/dts/bcm-cygnus.dtsi +22 -0
@@ -162,6 +162,17 @@ 0x82000000 0 0x20000000 0x20000000 0 0x04000000>; status = "disabled"; + + msi-parent = <&msi0>; + msi0: msi@18012000 { + compatible = "brcm,iproc-msi"; + msi-controller; + interrupt-parent = <&gic>; + interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>, + <GIC_SPI 97 IRQ_TYPE_NONE>, + <GIC_SPI 98 IRQ_TYPE_NONE>, + <GIC_SPI 99 IRQ_TYPE_NONE>; + }; }; pcie1: pcie@18013000 {
@@ -183,6 +194,17 @@ 0x82000000 0 0x40000000 0x40000000 0 0x04000000>; status = "disabled"; + + msi-parent = <&msi1>; + msi1: msi@18013000 { + compatible = "brcm,iproc-msi"; + msi-controller; + interrupt-parent = <&gic>; + interrupts = <GIC_SPI 102 IRQ_TYPE_NONE>, + <GIC_SPI 103 IRQ_TYPE_NONE>, + <GIC_SPI 104 IRQ_TYPE_NONE>, + <GIC_SPI 105 IRQ_TYPE_NONE>; + }; }; uart0: serial@18020000 {
Signature · 1 line
1.9.1