Add iProc PCIe PAXC and MSI support
Subsystems (4)
Ray Jui
maintainerv4 cover Add iProc PCIe PAXC and MSI support
This patch series adds support for the iProc PAXC interface and support for event queue based MSI, integrated in the iProc PCIe core
This patch series is based on Linux v4.4-rc1 and is avaliable here:
https://github.com/Broadcom/cygnus-linux/tree/iproc-msi-v4
Between v4 and v3, changes are low risk. Sanity tested below:
PAXB:
- Broadcom NS2 SVK board with Intel e1000e network card
Changes from v3:
- Detect the number of possible CPUs instead online CPUs in the driver. Note CPU notifier based implementation still needs to be added in the future, for proper support of MSI IRQ affinity when CPU is brought online/offline at runtime. Support in this dirver will be added when we support CPU hotplug in one of iProc family of SoCs so the changes can be tested
- Use dma_zalloc_coherent for event queue host memory allocation and zeoring
Changes from v2:
- Improved descriptions in the iProc MSI commit message
- Removed redundant host memory used for MSI address. The MSI posted writes never really hit the memory. Use iProc PCIe controller base address instead
- Fixed deadlock when MSI vectors are used up
- Enforced the number of MSI groups to always be multiple of the number of CPUs
- Improved the efficiency of MSI event processing by only updating the head pointer after finishing processing all outstanding events
- Fixed error handling code to make sure all configurations are rolled back
- Added code to zero the host memory used for event queues after allocation
- Removed redundant 'brcm,num-eq-region' and 'brcm,num-msi-msg-region' DT properties. Now determine the number of regions based on interface type
- Other misc. changes
Changes from v1:
- Fixed incorrect 1-to-1 mapping between MSI vector and GIC interrupt. Now the driver supports multiple MSI vectors per GIC interrupt
- Added MSI IRQ affinity support by distributing GIC interrupts across available CPU cores and dynamically steer MSI vectors to the target CPU
- replace readl/writel with readl_relaxed/writel_relaxed since all register accesses within the iProc MSI driver are to/from the same I/O block, i.e., the iProc PCIe core
- Removed all redundant irq_chip callback assignments
- Changed to use uncached host memory for both MSI posted writes and event queues
- Add functions to free resources in error/exit cases
- In pcie-iproc-platform.c, pass in interface type through OF device data
- Moved define for max number of interrupts from pcie-iproc.h to pcie-iproc-msi.c
- Other misc. changes
Ray Jui (5):
PCI: iproc: Update iProc PCIe device tree binding
PCI: iproc: Add PAXC interface support
PCI: iproc: Add iProc PCIe MSI device tree binding
PCI: iproc: Add iProc PCIe MSI support
ARM: dts: Enable MSI support for Broadcom Cygnus
.../devicetree/bindings/pci/brcm,iproc-pcie.txt | 40 +- arch/arm/boot/dts/bcm-cygnus.dtsi | 22 + drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 675 +++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 25 +- drivers/pci/host/pcie-iproc.c | 228 +++++-- drivers/pci/host/pcie-iproc.h | 42 +- 9 files changed, 1000 insertions(+), 43 deletions(-) create mode 100644 drivers/pci/host/pcie-iproc-msi.c
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v4 1/5 PCI: iproc: Update iProc PCIe device tree binding
Add a new compatible string "brcm,iproc-pcie-paxc", for PAXC based iProc PCIe root complex. A PAXC based PCIe root complex is connected to emulated endpoint devices internal to the ASIC
--- Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +4 -1
@@ -1,7 +1,10 @@ * Broadcom iProc PCIe controller with the platform bus interface Required properties: -- compatible: Must be "brcm,iproc-pcie" +- compatible: Must be "brcm,iproc-pcie" for PAXB, or "brcm,iproc-pcie-paxc" + for PAXC. PAXB based root complex is used for external endpoint devices. + PAXC based root complex is connected to emulated endpoint devices + internal to the ASIC - reg: base address and length of the PCIe controller I/O register space - #interrupt-cells: set to <1> - interrupt-map-mask and interrupt-map, standard PCI properties to define the
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v4 2/5 PCI: iproc: Add PAXC interface support
In reply to Ray JuiTraditionally, all iProc PCIe root complexes use PAXB based wrapper, with an integrated on-chip Serdes to support external endpoint devices. On newer iProc platforms, a PAXC based wrapper is introduced, for connection with internally emulated PCIe endpoint devices in the ASIC
This patch adds support for PAXC based iProc PCIe root complex in the iProc PCIe core driver. This change fators out common logic between PAXB and PAXC, and use tables to store register offsets that are different between PAXB and PAXC. This allows the driver to be scaled to support subsequent PAXC revisions in the future
--- drivers/pci/host/pcie-iproc-platform.c | 24 +++- drivers/pci/host/pcie-iproc.c | 202 +++++++++++++++++++++++++++------ drivers/pci/host/pcie-iproc.h | 19 ++++ 3 files changed, 205 insertions(+), 40 deletions(-)
drivers/pci/host/pcie-iproc-platform.c +18 -6
@@ -26,8 +26,21 @@#include "pcie-iproc.h" +static const struct of_device_id iproc_pcie_of_match_table[] = { + { + .compatible ="brcm,iproc-pcie" , + .data = (int *)IPROC_PCIE_PAXB , + }, { + .compatible ="brcm,iproc-pcie-paxc" , + .data = (int *)IPROC_PCIE_PAXC , + }, + {/* sentinel */ } +}; +MODULE_DEVICE_TABLE (of, iproc_pcie_of_match_table); +static int iproc_pcie_pltfm_probe (struct platform_device *pdev) { +const struct of_device_id *of_id;struct iproc_pcie *pcie;struct device_node *np = pdev->dev.of_node;struct resource reg;
@@ -35,11 +48,16 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)LIST_HEAD (res);int ret; + of_id =of_match_device (iproc_pcie_of_match_table, &pdev->dev); +if (!of_id) +return -EINVAL ; + pcie =devm_kzalloc (&pdev->dev,sizeof (struct iproc_pcie ),GFP_KERNEL );if (!pcie)return -ENOMEM ; pcie->dev = &pdev->dev; + pcie->type = (enum iproc_pcie_type )of_id->data;platform_set_drvdata (pdev, pcie); ret =of_address_to_resource (np,0 , ®);
@@ -114,12 +132,6 @@ static int iproc_pcie_pltfm_remove(struct platform_device *pdev)return iproc_pcie_remove (pcie); } -static const struct of_device_id iproc_pcie_of_match_table[] = { - { .compatible ="brcm,iproc-pcie" , }, - {/* sentinel */ } -}; -MODULE_DEVICE_TABLE (of, iproc_pcie_of_match_table); -static struct platform_driver iproc_pcie_pltfm_driver = { .driver = { .name ="iproc-pcie" ,
drivers/pci/host/pcie-iproc.c +168 -34
295 lines not on this page · raw message
drivers/pci/host/pcie-iproc.h +19 -0
@@ -15,6 +15,20 @@#define _PCIE_IPROC_H/** +* iProc PCIe interface type +* +* PAXB is the wrapper used in root complex that can be connected to an +* external endpoint device +* +* PAXC is the wrapper used in root complex dedicated for internal emulated +* endpoint devices +*/ +enum iproc_pcie_type { +IPROC_PCIE_PAXB =0 , +IPROC_PCIE_PAXC , +}; + +/** * iProc PCIe outbound mapping * @set_oarr_size: indicates the OARR size bit needs to be set * @axi_offset: offset from the AXI address to the internal address used by
@@ -29,7 +43,10 @@ struct iproc_pcie_ob {/** * iProc PCIe device +* * @dev: pointer to device data structure +* @type: iProc PCIe interface type +* @reg_offsets: register offsets * @base: PCIe host controller I/O register base * @sysdata: Per PCI controller data (ARM-specific) * @root_bus: pointer to root bus
@@ -41,6 +58,8 @@ struct iproc_pcie_ob {*/ struct iproc_pcie {struct device *dev; +enum iproc_pcie_type type; +const u16 *reg_offsets;void __iomem *base;#ifdef CONFIG_ARM struct pci_sys_data sysdata;
Signature · 1 line
1.9.1
Re: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support
drivers/pci/host/pcie-iproc-platform.c line 39
Quoted 12 lines · On Friday 27 November 2015 09:37:45 Ray Jui wrote:
drivers/pci/host/pcie-iproc-platform.c +11
-0+static const struct of_device_id iproc_pcie_of_match_table[] = { + { + .compatible ="brcm,iproc-pcie" , + .data = (int *)IPROC_PCIE_PAXB , + }, { + .compatible ="brcm,iproc-pcie-paxc" , + .data = (int *)IPROC_PCIE_PAXC , + }, + {/* sentinel */ } +}; +MODULE_DEVICE_TABLE (of, iproc_pcie_of_match_table);
You seem to only need the identifiers in order to set a single pointer, so just point to that array directly. Alternatively, do the more common thing and point to a structure of function pointers and have different implementations of the low-level access functions there.
Arnd
Ray Jui
maintainerRe: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support
Hi Arnd,
Quoted 16 lines · Arnd Bergmann
drivers/pci/host/pcie-iproc-platform.c line 39
Quoted 12 lines · On Friday 27 November 2015 09:37:45 Ray Jui wrote:
drivers/pci/host/pcie-iproc-platform.c +11
-0+static const struct of_device_id iproc_pcie_of_match_table[] = { + { + .compatible ="brcm,iproc-pcie" , + .data = (int *)IPROC_PCIE_PAXB , + }, { + .compatible ="brcm,iproc-pcie-paxc" , + .data = (int *)IPROC_PCIE_PAXC , + }, + {/* sentinel */ } +}; +MODULE_DEVICE_TABLE (of, iproc_pcie_of_match_table);You seem to only need the identifiers in order to set a single pointer, so just point to that array directly.
Not just a single pointer. This sets the iProc PCIe interface type, which is an enum. Based on the interface type, both the iProc PCIe core driver and the iProc PCIe MSI driver can 1) load the correct set of register offsets; 2) skip/invoke some of the link detection, controller reset related behaviors; 3) setting up the correct number of MSI event queue regions and MSI address regions.
Quoted 4 lines
Alternatively,
do the more common thing and point to a structure of function pointers and have different implementations of the low-level access functions there.
I thought about replacing the reset and link check functions in pcie-iproc.c with callbacks but felt it has not reached the point where we need callbacks to completely differentiate the operations. I'm hoping to wait until I see the next revision of PAXC (PAXC v2) implementation in the next-gen iProc SoC to make the call....
Note even with the callback approach, it would not change much of the pcie-iproc-platform.c here, which still needs to pass in just the interface information but no more than that. Main reason being the core driver and MSI drivers are used by both the platform driver and BCMA driver, and I do not want to duplicate the callback implementation at the bus driver level.
Quoted 3 lines
Arnd
Thanks,
Ray
Re: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support
Quoted 10 lines · On Friday 27 November 2015 12:56:05 Ray Jui wrote:
Quoted 3 lines
You seem to only need the identifiers in order to set a single pointer, so just point to that array directly.
Not just a single pointer. This sets the iProc PCIe interface type, which is an enum. Based on the interface type, both the iProc PCIe core driver and the iProc PCIe MSI driver can 1) load the correct set of register offsets; 2) skip/invoke some of the link detection, controller reset related behaviors; 3) setting up the correct number of MSI event queue regions and MSI address regions.
Ok, fair enough. I had missed 2) and 3) here. It could still be done as a structure (rather than function pointers) with individual named members for each register offset along with bool flags for 2) and integers to describe 3).
Please that suggestion some thought, and then decide for yourself, I won't complain if you keep the current version.
Arnd
Ray Jui
maintainerRe: [PATCH v4 2/5] PCI: iproc: Add PAXC interface support
Hi Arnd,
Quoted 17 lines · Arnd Bergmann
Quoted 10 lines · On Friday 27 November 2015 12:56:05 Ray Jui wrote:
Quoted 3 lines
You seem to only need the identifiers in order to set a single pointer, so just point to that array directly.
Not just a single pointer. This sets the iProc PCIe interface type, which is an enum. Based on the interface type, both the iProc PCIe core driver and the iProc PCIe MSI driver can 1) load the correct set of register offsets; 2) skip/invoke some of the link detection, controller reset related behaviors; 3) setting up the correct number of MSI event queue regions and MSI address regions.
Ok, fair enough. I had missed 2) and 3) here. It could still be done as a structure (rather than function pointers) with individual named members for each register offset along with bool flags for 2) and integers to describe 3).
Quoted 5 lines
Please that suggestion some thought, and then decide for yourself, I won't complain if you keep the current version.
Arnd
I'll definitely re-visit this when I need to add support to the next revision of PAXC interface. At that time, what you have suggested may become almost mandatory in order to make the core code clean and more maintainable.
For now, I'll keep this as it is.
Thanks!
Ray
Ray Jui
maintainerPATCH v4 3/5 PCI: iproc: Add iProc PCIe MSI device tree binding
In reply to Ray JuiThis patch updates the iProc PCIe device tree bindings with added binding information for MSI
--- .../devicetree/bindings/pci/brcm,iproc-pcie.txt | 35 ++++++++++++++++++++++ 1 file changed, 35 insertions(+)
Documentation/devicetree/bindings/pci/brcm,iproc-pcie.txt +35 -0
@@ -35,6 +35,28 @@ Optional: - brcm,pcie-ob-oarr-size: Some iProc SoCs need the OARR size bit to be set to increase the outbound window size +MSI support (optional): + +For older platforms without MSI integrated in the GIC, iProc PCIe core provides +an event queue based MSI support. The iProc MSI uses host memories to store +MSI posted writes in the event queues + +- msi-parent: Link to the device node of the MSI controller. On newer iProc +platforms, the MSI controller may be gicv2m or gicv3-its. On older iProc +platforms without MSI support in its interrupt controller, one may use the +event queue based MSI support integrated within the iProc PCIe core + +When the iProc event queue based MSI is used, one needs to define the +following properties in the MSI device node: +- compatible: Must be "brcm,iproc-msi" +- msi-controller: claims itself as an MSI controller +- interrupt-parent: Link to its parent interrupt device +- interrupts: List of interrupt IDs from its parent interrupt device + +Optional properties: +- brcm,pcie-msi-inten: Needs to be present for some older iProc platforms that +require the interrupt enable registers to be set explicitly to enable MSI + Example: pcie0: pcie@18012000 { compatible = "brcm,iproc-pcie";
@@ -61,6 +83,19 @@ Example: brcm,pcie-ob-oarr-size; brcm,pcie-ob-axi-offset = <0x00000000>; brcm,pcie-ob-window-size = <256>; + + msi-parent = <&msi0>; + + /* iProc event queue based MSI */ + msi0: msi@18012000 { + compatible = "brcm,iproc-msi"; + msi-controller; + interrupt-parent = <&gic>; + interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>, + <GIC_SPI 97 IRQ_TYPE_NONE>, + <GIC_SPI 98 IRQ_TYPE_NONE>, + <GIC_SPI 99 IRQ_TYPE_NONE>, + }; }; pcie1: pcie@18013000 {
Signature · 1 line
1.9.1
Ray Jui
maintainerPATCH v4 4/5 PCI: iproc: Add iProc PCIe MSI support
In reply to Ray JuiThis patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms
The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing
Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt
MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU
Therefore, the actual number of supported MSI vectors is:
M * 64 / N
where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores
This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)
--- drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 675 +++++++++++++++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 1 + drivers/pci/host/pcie-iproc.c | 26 ++ drivers/pci/host/pcie-iproc.h | 23 +- 7 files changed, 734 insertions(+), 2 deletions(-) create mode 100644 drivers/pci/host/pcie-iproc-msi.c
drivers/pci/host/Kconfig +9 -0
@@ -126,6 +126,15 @@ config PCIE_IPROC iProc family of SoCs. An appropriate bus interface driver also needs to be enabled +config PCIE_IPROC_MSI + bool "Broadcom iProc PCIe MSI support" + depends on ARCH_BCM_IPROC && PCI_MSI + select PCI_MSI_IRQ_DOMAIN + default ARCH_BCM_IPROC + help + Say Y here if you want to enable MSI support for Broadcom's iProc + PCIe controller + config PCIE_IPROC_PLATFORM tristate "Broadcom iProc PCIe platform bus driver" depends on ARCH_BCM_IPROC || (ARM && COMPILE_TEST)
drivers/pci/host/Makefile +1 -0
@@ -15,6 +15,7 @@ obj-$(CONFIG_PCI_XGENE_MSI) += pci-xgene-msi.o obj-$(CONFIG_PCI_LAYERSCAPE) += pci-layerscape.o obj-$(CONFIG_PCI_VERSATILE) += pci-versatile.o obj-$(CONFIG_PCIE_IPROC) += pcie-iproc.o +obj-$(CONFIG_PCIE_IPROC_MSI) += pcie-iproc-msi.o obj-$(CONFIG_PCIE_IPROC_PLATFORM) += pcie-iproc-platform.o obj-$(CONFIG_PCIE_IPROC_BCMA) += pcie-iproc-bcma.o obj-$(CONFIG_PCIE_ALTERA) += pcie-altera.o
drivers/pci/host/pcie-iproc-bcma.c +1 -0
@@ -55,6 +55,7 @@ static int iproc_pcie_bcma_probe(struct bcma_device *bdev)bcma_set_drvdata (bdev, pcie); pcie->base = bdev->io_addr; + pcie->base_addr = bdev->addr; res_mem.start = bdev->addr_s[0 ]; res_mem.end = bdev->addr_s[0 ] +SZ_128M -1 ;
drivers/pci/host/pcie-iproc-msi.c new file +675 -0
675 lines not on this page · raw message
drivers/pci/host/pcie-iproc-platform.c +1 -0
@@ -71,6 +71,7 @@ static int iproc_pcie_pltfm_probe(struct platform_device *pdev)dev_err (pcie->dev,"unable to map controller registers\n" );return -ENOMEM ; } + pcie->base_addr = reg.start;if (of_property_read_bool (np,"brcm,pcie-ob" )) {u32 val;
drivers/pci/host/pcie-iproc.c +26 -0
@@ -440,6 +440,26 @@ static int iproc_pcie_map_ranges(struct iproc_pcie *pcie,return 0 ; } +static int iproc_pcie_msi_enable (struct iproc_pcie *pcie) +{ +struct device_node *msi_node; + + msi_node =of_parse_phandle (pcie->dev->of_node,"msi-parent" ,0 ); +if (!msi_node) +return -ENODEV ; + +/* +* If another MSI controller is being used, the call below should fail +* but that is okay +*/ +return iproc_msi_init (pcie, msi_node); +} + +static void iproc_pcie_msi_disable (struct iproc_pcie *pcie) +{ +iproc_msi_exit (pcie); +} +int iproc_pcie_setup (struct iproc_pcie *pcie,struct list_head *res) {int ret;
@@ -507,6 +527,10 @@ int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res)iproc_pcie_enable (pcie); +if (IS_ENABLED (CONFIG_PCI_MSI )) +if (iproc_pcie_msi_enable (pcie)) +dev_info (pcie->dev,"not using iProc MSI\n" ); +pci_scan_child_bus (bus);pci_assign_unassigned_bus_resources (bus);pci_fixup_irqs (pci_common_swizzle, pcie->map_irq);
@@ -531,6 +555,8 @@ int iproc_pcie_remove(struct iproc_pcie *pcie)pci_stop_root_bus (pcie->root_bus);pci_remove_root_bus (pcie->root_bus); +iproc_pcie_msi_disable (pcie); +phy_power_off (pcie->phy);phy_exit (pcie->phy);
drivers/pci/host/pcie-iproc.h +21 -2
@@ -41,6 +41,8 @@ struct iproc_pcie_ob {resource_size_t window_size; }; +struct iproc_msi ; +/** * iProc PCIe device *
@@ -48,19 +50,21 @@ struct iproc_pcie_ob {* @type: iProc PCIe interface type * @reg_offsets: register offsets * @base: PCIe host controller I/O register base +* @base_addr: PCIe host controller register base physical address * @sysdata: Per PCI controller data (ARM-specific) * @root_bus: pointer to root bus * @phy: optional PHY device that controls the Serdes -* @irqs: interrupt IDs * @map_irq: function callback to map interrupts -* @need_ob_cfg: indidates SW needs to configure the outbound mapping window +* @need_ob_cfg: indicates SW needs to configure the outbound mapping window * @ob: outbound mapping parameters +* @msi: MSI data */ struct iproc_pcie {struct device *dev;enum iproc_pcie_type type;const u16 *reg_offsets;void __iomem *base; +phys_addr_t base_addr;#ifdef CONFIG_ARM struct pci_sys_data sysdata;#endif
@@ -69,9 +73,24 @@ struct iproc_pcie {int (*map_irq)(const struct pci_dev *,u8 ,u8 );bool need_ob_cfg;struct iproc_pcie_ob ob; + struct iproc_msi *msi; };int iproc_pcie_setup (struct iproc_pcie *pcie,struct list_head *res);int iproc_pcie_remove (struct iproc_pcie *pcie); +#ifdef CONFIG_PCI_MSI +int iproc_msi_init (struct iproc_pcie *pcie,struct device_node *node); +void iproc_msi_exit (struct iproc_pcie *pcie); +#else +static inline int iproc_msi_init (struct iproc_pcie *pcie, +struct device_node *node) +{ +return -ENODEV ; +} +static void iproc_msi_exit (struct iproc_pcie *pcie) +{ +} +#endif +#endif /* _PCIE_IPROC_H */
Signature · 1 line
1.9.1
Re: [PATCH v4 4/5] PCI: iproc: Add iProc PCIe MSI support
Quoted 47 lines · Ray Jui
This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms
The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing
Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt
MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU
Therefore, the actual number of supported MSI vectors is:
M * 64 / N
where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores
This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)
Signed-off-by Ray Jui <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Vikram Prakash <[email protected]>Reviewed-by Scott Branden <[email protected]>--- drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 675 +++++++++++++++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 1 + drivers/pci/host/pcie-iproc.c | 26 ++ drivers/pci/host/pcie-iproc.h | 23 +- 7 files changed, 734 insertions(+), 2 deletions(-) create mode 100644 drivers/pci/host/pcie-iproc-msi.c
.....
Quoted 4 lines
int
iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res); intiproc_pcie_remove(struct iproc_pcie *pcie);
drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 91
Quoted 10 lines
drivers/pci/host/pcie-iproc.h +10
-0+#ifdef CONFIG_PCI_MSI +int iproc_msi_init (struct iproc_pcie *pcie,struct device_node *node); +void iproc_msi_exit (struct iproc_pcie *pcie); +#else +static inline int iproc_msi_init (struct iproc_pcie *pcie, +struct device_node *node) +{ +return -ENODEV ; +} +static void iproc_msi_exit (struct iproc_pcie *pcie)
Please use static inline here.
drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 96
Quoted 6 lines
drivers/pci/host/pcie-iproc.h +4
-0+{ +} +#endif +#endif /* _PCIE_IPROC_H */
Hauke
Ray Jui
maintainerRe: [PATCH v4 4/5] PCI: iproc: Add iProc PCIe MSI support
Quoted 54 lines · Hauke Mehrtens
Quoted 47 lines · Ray Jui
This patch adds PCIe MSI support for both PAXB and PAXC interfaces on all iProc based platforms
The iProc PCIe MSI support deploys an event queue based implementation. Each event queue is serviced by a GIC interrupt and can support up to 64 MSI vectors. Host memory is allocated for the event queues, and each event queue consists of 64 word-sized entries. MSI data is written to the lower 16-bit of each entry, whereas the upper 16-bit of the entry is reserved for the controller for internal processing
Each event queue is tracked by a head pointer and tail pointer. Head pointer indicates the next entry in the event queue to be processed by the driver and is updated by the driver after processing is done. The controller uses the tail pointer as the next MSI data insertion point. The controller ensures MSI data is flushed to host memory before updating the tail pointer and then triggering the interrupt
MSI IRQ affinity is supported by evenly distributing the interrupts to each CPU core. MSI vector is moved from one GIC interrupt to another in order to steer to the target CPU
Therefore, the actual number of supported MSI vectors is:
M * 64 / N
where M denotes the number of GIC interrupts (event queues), and N denotes the number of CPU cores
This iProc event queue based MSI support should not be used with newer platforms with integrated MSI support in the GIC (e.g., giv2m or gicv3-its)
Signed-off-by Ray Jui <[email protected]>Reviewed-by Anup Patel <[email protected]>Reviewed-by Vikram Prakash <[email protected]>Reviewed-by Scott Branden <[email protected]>--- drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 675 +++++++++++++++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 1 + drivers/pci/host/pcie-iproc.c | 26 ++ drivers/pci/host/pcie-iproc.h | 23 +- 7 files changed, 734 insertions(+), 2 deletions(-)create mode 100644
drivers/pci/host/pcie-iproc-msi.c.....
Quoted 4 lines
int iproc_pcie_setup(struct iproc_pcie *pcie, struct list_head *res); int iproc_pcie_remove(struct iproc_pcie *pcie);
Quoted 13 lines
drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 91
Quoted 10 lines
drivers/pci/host/pcie-iproc.h +10
-0+#ifdef CONFIG_PCI_MSI +int iproc_msi_init (struct iproc_pcie *pcie,struct device_node *node); +void iproc_msi_exit (struct iproc_pcie *pcie); +#else +static inline int iproc_msi_init (struct iproc_pcie *pcie, +struct device_node *node) +{ +return -ENODEV ; +} +static void iproc_msi_exit (struct iproc_pcie *pcie)Please use static inline here.
Right. Will fix. Thanks!
Quoted 9 lines
drivers/pci/host/pcie-iproc.h struct iproc_pcie { · line 96
Quoted 6 lines
drivers/pci/host/pcie-iproc.h +4
-0+{ +} +#endif +#endif /* _PCIE_IPROC_H */ Hauke
Ray
Ray Jui
maintainerPATCH v4 5/5 ARM: dts: Enable MSI support for Broadcom Cygnus
In reply to Ray JuiEnable MSI support for Broadcom Cygnus platforms
--- arch/arm/boot/dts/bcm-cygnus.dtsi | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+)
arch/arm/boot/dts/bcm-cygnus.dtsi +22 -0
@@ -162,6 +162,17 @@ 0x82000000 0 0x20000000 0x20000000 0 0x04000000>; status = "disabled"; + + msi-parent = <&msi0>; + msi0: msi@18012000 { + compatible = "brcm,iproc-msi"; + msi-controller; + interrupt-parent = <&gic>; + interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>, + <GIC_SPI 97 IRQ_TYPE_NONE>, + <GIC_SPI 98 IRQ_TYPE_NONE>, + <GIC_SPI 99 IRQ_TYPE_NONE>; + }; }; pcie1: pcie@18013000 {
@@ -183,6 +194,17 @@ 0x82000000 0 0x40000000 0x40000000 0 0x04000000>; status = "disabled"; + + msi-parent = <&msi1>; + msi1: msi@18013000 { + compatible = "brcm,iproc-msi"; + msi-controller; + interrupt-parent = <&gic>; + interrupts = <GIC_SPI 102 IRQ_TYPE_NONE>, + <GIC_SPI 103 IRQ_TYPE_NONE>, + <GIC_SPI 104 IRQ_TYPE_NONE>, + <GIC_SPI 105 IRQ_TYPE_NONE>; + }; }; uart0: serial@18020000 {
Signature · 1 line
1.9.1
Ray Jui
maintainerHi Bjorn,
Is there any more change that you want me to make from the current patch set besides the inline comment from Hauke on the MSI patch?
Thanks,
Ray
Quoted 68 lines · Ray Jui
This patch series adds support for the iProc PAXC interface and support for event queue based MSI, integrated in the iProc PCIe core
This patch series is based on Linux v4.4-rc1 and is avaliable here:
https://github.com/Broadcom/cygnus-linux/tree/iproc-msi-v4Between v4 and v3, changes are low risk. Sanity tested below:
PAXB:
- Broadcom NS2 SVK board with Intel e1000e network cardChanges from v3:
- Detect the number of possible CPUs instead online CPUs in the driver. Note CPU notifier based implementation still needs to be added in the future, for proper support of MSI IRQ affinity when CPU is brought online/offline at runtime. Support in this dirver will be added when we support CPU hotplug in one of iProc family of SoCs so the changes can be tested
- Usedma_zalloc_coherentfor event queue host memory allocation and zeoringChanges from v2:
- Improved descriptions in the iProc MSI commit message
- Removed redundant host memory used for MSI address. The MSI posted writes never really hit the memory. Use iProc PCIe controller base address instead
- Fixed deadlock when MSI vectors are used up
- Enforced the number of MSI groups to always be multiple of the number of CPUs
- Improved the efficiency of MSI event processing by only updating the head pointer after finishing processing all outstanding events
- Fixed error handling code to make sure all configurations are rolled back
- Added code to zero the host memory used for event queues after allocation
- Removed redundant 'brcm,num-eq-region' and 'brcm,num-msi-msg-region' DT properties. Now determine the number of regions based on interface type
- Other misc. changesChanges from v1:
- Fixed incorrect 1-to-1 mapping between MSI vector and GIC interrupt. Now the driver supports multiple MSI vectors per GIC interrupt
- Added MSI IRQ affinity support by distributing GIC interrupts across available CPU cores and dynamically steer MSI vectors to the target CPU
- replace readl/writel with readl_relaxed/writel_relaxed since all register accesses within the iProc MSI driver are to/from the same I/O block, i.e., the iProc PCIe core
- Removed all redundantirq_chipcallback assignments
- Changed to use uncached host memory for both MSI posted writes and event queues
- Add functions to free resources in error/exit cases
- Inpcie-iproc-platform.c, pass in interface type through OF device data
- Moved define for max number of interrupts frompcie-iproc.htopcie-iproc-msi.c
- Other misc. changesRay Jui (5):
PCI: iproc: Update iProc PCIe device tree binding
PCI: iproc: Add PAXC interface support
PCI: iproc: Add iProc PCIe MSI device tree binding
PCI: iproc: Add iProc PCIe MSI support
ARM: dts: Enable MSI support for Broadcom Cygnus.../devicetree/bindings/pci/brcm,iproc-pcie.txt | 40 +- arch/arm/boot/dts/bcm-cygnus.dtsi | 22 + drivers/pci/host/Kconfig | 9 + drivers/pci/host/Makefile | 1 + drivers/pci/host/pcie-iproc-bcma.c | 1 + drivers/pci/host/pcie-iproc-msi.c | 675 +++++++++++++++++++++ drivers/pci/host/pcie-iproc-platform.c | 25 +- drivers/pci/host/pcie-iproc.c | 228 +++++-- drivers/pci/host/pcie-iproc.h | 42 +- 9 files changed, 1000 insertions(+), 43 deletions(-)create mode 100644
drivers/pci/host/pcie-iproc-msi.c