MSI support for Marvell EBU PCIe driver

Hello,

This set of patches introduces Message Signaled Interrupt support in the Marvell EBU PCIe driver. It has been successfully tested on the Armada XP GP platform with an Intel e1000e PCIe network card that supports MSI.

Compared to v8, this is a minor update with some build fixes and more Acked-by.

This patch set is intended for merging in 3.12, so the respective maintainers of the different areas are invited to review/ack the patches, see below for the details.

The patches do the following:

* Patch 1 reworks how the architecture-specific MSI functions can be overriden by architecture-specific code, by using weak functions. It was suggested by Bjorn Helgaas.

This patch has been acked by Bjorn Helgaas, the PCI maintainter.

* Patch 2 removes the ARCH_SUPPORTS_MSI hidden kconfig boolean which is no longer needed now that we have weak functions for all MSI architecture-specific hooks.

This patch has been acked by Bjorn Helgaas, the PCI maintainer.

* Patch 3 adds a minimal msi_chip infrastructure, that allows a pci_bus to be connected to a msi_chip, and that provides default implementations of the architecture-specific MSI functions to use msi_chip.

This patch has been acked by Bjorn Helgaas, the PCI maintainter.

* Patch 4 adds a simple registry of msi_chip <-> of_node in drivers/of.

This patch has been Acked by Rob Herring.

* Patch 5 makes some not very interesting preparation in the Armada 370/XP IRQ controller driver.

This patch needs the Ack from Thomas Gleixner, as the drivers/irqchip maintainer.

It has been Acked-by Gregory Clement, as a mvebu maintainer.

* Patch 6 implements the MSI support in the Armada 370/XP IRQ controller driver. It registers an msi_chip using the msi_chip_add() function added in PATCH 3.

This patch needs the Ack from Thomas Gleixner, as the drivers/irqchip maintainer.

It has been Acked-by Gregory Clement, as a mvebu maintainer.

* Patch 7 extends the ARM PCI core to expose ->add_bus() and ->remove_bus() hooks to PCI drivers. This was suggested by Bjorn Helgaas to allow the PCI driver to connected the PCI busses with the corresponding MSI chip.

This patch has been Acked by Russell King, the ARM maintainer.

* Patch 8 adjust the Armada 370/XP Device Tree to indicate that the MPIC is not only an interrupt-controller, but also an msi-controller.

It has been Acked-by Gregory Clement, as a mvebu maintainer.

* Patch 9 adds MSI support in the Marvell PCIe host controller driver. The work to do here is minimal: get a reference to the msi-parent controller thanks to msi_chip_find_by_of_node(), and link it to the pci_bus structure before the bus gets enumerated.

This patch has been acked by Bjorn Helgaas, the PCI maintainer.

* Patch 10 adjusts the Armada 370/XP Device Tree to add the msi-parent properties in the PCIe controller nodes.

It has been Acked-by Gregory Clement, as a mvebu maintainer.

This version 9 follows:
* PATCH version 8 sent on August, 9th 2013
* PATCH version 7 sent on August, 7th 2013
* PATCH version 6 sent on August, 1st 2013
* PATCH version 5 sent on July, 15th 2013
* PATCH version 4 sent on July, 1st 2013
* PATCH version 3 sent on June, 19th 2013
* PATCH version 2 sent on June, 6th 2013
* RFC version 1 sent on March, 26th 2013

Changes since v8:

* Fix !CONFIG_PCI_MSI build on x86 (by removing some unneeded defines in arch/x86/include/asm/pci.h, and adding some compile-time conditions in arch/x86/kernel/x86_init.c).

* Fix !CONFIG_PCI_MSI && !CONFIG_OF build by adding a missing ';' in include/linux/of_pci.h.

* Added Acked-by from Benjamin Herrenschmidt on patches:
PCI: use weak functions for MSI arch-specific functions
PCI: remove ARCH_SUPPORTS_MSI kconfig option
of: pci: add registry of MSI chips

* Re-added Acked-by from Rob Herring on patch:
of: pci: add registry of MSI chips
it had been lost in the process, but Rob did give his ack on this patch on July, 9th.

* Added Acked-by from Gregory Clement on patches:
irqchip: armada-370-xp: properly request resources
irqchip: armada-370-xp: implement MSI support
ARM: mvebu: the MPIC now provides MSI controller features
ARM: mvebu: link PCIe controllers to the MSI controller

Changes since v7:

* Instead of using an irq_domain based allocation of MSI interrupts, revert back to a bitmap-based solution directly implemented inside the IRQ controller driver. This is identical to what was done in Version 2 of this patch set.

* Instead of tightly coupling msi_chip and irq_domain, revert back to the simple msi_chip <-> of_node registry in drivers/of, which was part of Version 3, Version 4 and Version 5 of this patch set.

* Remove Tested-by from Daniel Price on the two modified patches.

* Fix the indentation problem noticed by Thierry Reding on the 'ARM:
pci: add ->add_bus() and ->remove_bus() hooks to hw_pci' patch.

Changes since v6:

* Fixed x86 build failure related to default_restore_msi_irqs(), noticed by Stephen Rothwell on linux-next.

* Added Acked-by from Bjorn Helgaas on 'PCI: remove ARCH_SUPPORTS_MSI kconfig option'.

* Fixed a compile-time issue in drivers/irqchip/irq-armada-370-xp.c in the !CONFIG_PCI_MSI case.

* Fixed one of the stylistic issue raised by Thierry Reding on 'ARM:
pci: add ->add_bus() and ->remove_bus() hooks to hw_pci' (i.e do not align '=' when assigning elements of the pci_sys_data structure). I however didn't fix the tab vs. spaces indentation issue noticed by Thierry, since I couldn't see it (to me the indentation was identical in both locations).

* Added Tested-by from Daniel Price on all patches.

Changes since v5:

* Rebased on top of 3.11-rc3.

* Implemented the suggestions of Grant Likely on "irqdomain: add irq_alloc_mapping() function" and added his Acked-by.

* Added Bjorn Helgaas Acked-by on "PCI: use weak functions for MSI arch-specific functions".

* Added Bjorn Helgaas Acked-by on "PCI: Introduce new MSI chip infrastructure".

* Remove the drivers/of global registry of irq_chip, and as suggested by Grant Likely, create an association between an irq_domain and a msi_chip. This required refactoring the __irq_domain_add() function, adding a 'msi_chip' pointer to 'irq_domain', adding an irq_domain_add_msi() function and a irq_find_msi_host() function.

* Added Russell King Acked-by on "ARM: pci: add ->add_bus() and ->remove_bus() hooks to hw_pci"

Changes since v4:

* Rebased on top of 3.11-rc1.

* Slightly rework the implementation of arch_setup_msi_irq(), arch_teardown_msi_irq and arch_msi_check_device() according to the suggestions of Bjorn Helgaas (PATCH 4)

* Make a few improvements to the msi_chip registry, according to the suggestion of Bjorn Helgaas and Rob Herring: coding style fixes, usage of mutex while traversing the list of msi_chip, usage of of_pci_msi_* prefix instead of just of_msi_*.

Changes since v3:

* Keep only a default_teardown_msi_irqs() function needed for the Xen PCI x86 code, and remove all other default_*() function and put the default behavior directly in the weak functions. Suggested by Thierry Redding and Bjorn Helgaas.

* Misc small improvements the MSI chip registry code: compiled only under CONFIG_PCI_MSI, functions exported to modules, addition of an of_msi_chip_remove() function, renaming of of_msi_chip_find_by_node() to of_find_msi_chip_by_node(), move the test of the "msi-controller" property to the of_msi_chip_add() function, renamed the list_head field from link to list, added dummy functions in the header files when !CONFIG_PCI_MSI && !CONFIG_OF. All suggested by Thierry Redding.

* Add a patch that entirely removes the ARCH_SUPPORTS_MSI, as we now have weak functions, so even if an arch doesn't actually use/support MSI, the code will build properly. Suggested by Thierry Redding.

* Added Device Tree binding documentation updates for the IRQ controller and the PCIe controller.

Changes since v2:

* Add an IRQ domain function that allows to allocate dynamically a free hwirq number from an IRQ domain, and create a virq mapping to it. It was suggested by Grant Likely in order to remove the hwirq allocation code from the IRQ driver itself. (PATCH 1)

* Separate the use of weak functions from the introduction of the msi_chip infrastructure, and use weak functions for all architecture-specific MSI hooks. Suggested by Bjorn Helgaas (PATCH 2).

* Move the msi_chip registry to drivers/of. Suggested by Bjorn Helgaas (PATCH 4).

* Use pcibios_add_bus() and pcibios_remove_bus() to connect the PCI busses to their msi_chip, as suggested by Bjorn Helgaas. Requires some ARM PCI core changes (PATCH 7), and then changes to the PCI driver itself (PATCH 9).

This set of patches applies on top of 3.11-rc3, and is available as a Git branch at
https://github.com/MISL-EBU-System-SW/mainline-public/tree/marvell-pcie-msi-v9.

Thanks,

Thomas

Thierry Reding (1):
PCI: Introduce new MSI chip infrastructure

Thomas Petazzoni (9):
PCI: use weak functions for MSI arch-specific functions
PCI: remove ARCH_SUPPORTS_MSI kconfig option
of: pci: add registry of MSI chips
irqchip: armada-370-xp: properly request resources
irqchip: armada-370-xp: implement MSI support
ARM: pci: add ->add_bus() and ->remove_bus() hooks to hw_pci
ARM: mvebu: the MPIC now provides MSI controller features
PCI: mvebu: add support for MSI
ARM: mvebu: link PCIe controllers to the MSI controller

 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   3 +
 .../devicetree/bindings/pci/mvebu-pci.txt          |   3 +
 arch/arm/Kconfig                                   |   1 -
 arch/arm/boot/dts/armada-370-xp.dtsi               |   1 +
 arch/arm/boot/dts/armada-370.dtsi                  |   1 +
 arch/arm/boot/dts/armada-xp-mv78230.dtsi           |   1 +
 arch/arm/boot/dts/armada-xp-mv78260.dtsi           |   1 +
 arch/arm/boot/dts/armada-xp-mv78460.dtsi           |   1 +
 arch/arm/include/asm/mach/pci.h                    |   4 +
 arch/arm/kernel/bios32.c                           |  16 ++
 arch/ia64/Kconfig                                  |   1 -
 arch/mips/Kconfig                                  |   2 -
 arch/mips/include/asm/pci.h                        |   5 -
 arch/powerpc/Kconfig                               |   1 -
 arch/powerpc/include/asm/pci.h                     |   5 -
 arch/s390/Kconfig                                  |   1 -
 arch/s390/include/asm/pci.h                        |   4 -
 arch/sparc/Kconfig                                 |   1 -
 arch/tile/Kconfig                                  |   1 -
 arch/x86/Kconfig                                   |   1 -
 arch/x86/include/asm/pci.h                         |  30 ---
 arch/x86/kernel/x86_init.c                         |  24 +++
 drivers/irqchip/irq-armada-370-xp.c                | 202 ++++++++++++++++++++-
 drivers/of/of_pci.c                                |  45 +++++
 drivers/pci/Kconfig                                |   4 -
 drivers/pci/host/pci-mvebu.c                       |  26 +++
 drivers/pci/msi.c                                  |  69 ++++---
 drivers/pci/probe.c                                |   1 +
 include/linux/msi.h                                |  21 ++-
 include/linux/of_pci.h                             |  12 ++
 include/linux/pci.h                                |   1 +
 31 files changed, 403 insertions(+), 86 deletions(-)

1.8.1.2

Until now, the MSI architecture-specific functions could be overloaded using a fairly complex set of #define and compile-time conditionals. In order to prepare for the introduction of the msi_chip infrastructure, it is desirable to switch all those functions to use the 'weak' mechanism. This commit converts all the architectures that were overidding those MSI functions to use the new strategy.

Note that we keep two separate, non-weak, functions default_teardown_msi_irqs() and default_restore_msi_irqs() for the default behavior of the arch_teardown_msi_irqs() and arch_restore_msi_irqs(), as the default behavior is needed by x86 PCI code.

---
 arch/mips/include/asm/pci.h    |  5 -----
 arch/powerpc/include/asm/pci.h |  5 -----
 arch/s390/include/asm/pci.h    |  4 ----
 arch/x86/include/asm/pci.h     | 30 --------------------------
 arch/x86/kernel/x86_init.c     | 24 +++++++++++++++++++++
 drivers/pci/msi.c              | 48 +++++++++++++++++++++---------------------
 include/linux/msi.h            |  8 ++++++-
 7 files changed, 55 insertions(+), 69 deletions(-)

index fa8e0aa..f194c08 100644
@@ -136,11 +136,6 @@ static inline int pci_get_legacy_ide_irq(struct pci_dev *dev, int channel)
 	return channel ? 15 : 14;
 }
 
-#ifdef CONFIG_CPU_CAVIUM_OCTEON
-/* MSI arch hook for OCTEON */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
-#endif
-
 extern char * (*pcibios_plat_setup)(char *str);
 
 #ifdef CONFIG_OF

index 6653f27..95145a1 100644
@@ -113,11 +113,6 @@ extern int pci_domain_nr(struct pci_bus *bus);
 /* Decide whether to display the domain number in /proc */
 extern int pci_proc_domain(struct pci_bus *bus);
 
-/* MSI arch hooks */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
-#define arch_teardown_msi_irqs arch_teardown_msi_irqs
-#define arch_msi_check_device arch_msi_check_device
-
 struct vm_area_struct;
 /* Map a range of PCI memory or I/O space for a device into user space */
 int pci_mmap_page_range(struct pci_dev *pdev, struct vm_area_struct *vma,

index 6e577ba..262b91b 100644
@@ -21,10 +21,6 @@ void pci_iounmap(struct pci_dev *, void __iomem *);
 int pci_domain_nr(struct pci_bus *);
 int pci_proc_domain(struct pci_bus *);
 
-/* MSI arch hooks */
-#define arch_setup_msi_irqs	arch_setup_msi_irqs
-#define arch_teardown_msi_irqs	arch_teardown_msi_irqs
-
 #define ZPCI_BUS_NR			0	/* default bus number */
 #define ZPCI_DEVFN			0	/* default device number */
 

index d9e9e6c..7d74432 100644
@@ -100,29 +100,6 @@ static inline void early_quirks(void) { }
 extern void pci_iommu_alloc(void);
 
 #ifdef CONFIG_PCI_MSI
-/* MSI arch specific hooks */
-static inline int x86_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
-{
-	return x86_msi.setup_msi_irqs(dev, nvec, type);
-}
-
-static inline void x86_teardown_msi_irqs(struct pci_dev *dev)
-{
-	x86_msi.teardown_msi_irqs(dev);
-}
-
-static inline void x86_teardown_msi_irq(unsigned int irq)
-{
-	x86_msi.teardown_msi_irq(irq);
-}
-static inline void x86_restore_msi_irqs(struct pci_dev *dev, int irq)
-{
-	x86_msi.restore_msi_irqs(dev, irq);
-}
-#define arch_setup_msi_irqs x86_setup_msi_irqs
-#define arch_teardown_msi_irqs x86_teardown_msi_irqs
-#define arch_teardown_msi_irq x86_teardown_msi_irq
-#define arch_restore_msi_irqs x86_restore_msi_irqs
 /* implemented in arch/x86/kernel/apic/io_apic. */
 struct msi_desc;
 int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);

@@ -130,16 +107,9 @@ void native_teardown_msi_irq(unsigned int irq);
 void native_restore_msi_irqs(struct pci_dev *dev, int irq);
 int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc,
 		  unsigned int irq_base, unsigned int irq_offset);
-/* default to the implementation in drivers/lib/msi.c */
-#define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
-#define HAVE_DEFAULT_MSI_RESTORE_IRQS
-void default_teardown_msi_irqs(struct pci_dev *dev);
-void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 #else
 #define native_setup_msi_irqs		NULL
 #define native_teardown_msi_irq		NULL
-#define default_teardown_msi_irqs	NULL
-#define default_restore_msi_irqs	NULL
 #endif
 
 #define PCI_DMA_BUS_IS_PHYS (dma_ops->is_phys)

index 5f24c71..8ce0072 100644
@@ -107,6 +107,8 @@ struct x86_platform_ops x86_platform = {
 };
 
 EXPORT_SYMBOL_GPL(x86_platform);
+
+#if defined(CONFIG_PCI_MSI)
 struct x86_msi_ops x86_msi = {
 	.setup_msi_irqs		= native_setup_msi_irqs,
 	.compose_msi_msg	= native_compose_msi_msg,

@@ -116,6 +118,28 @@ struct x86_msi_ops x86_msi = {
 	.setup_hpet_msi		= default_setup_hpet_msi,
 };
 
+/* MSI arch specific hooks */
+int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	return x86_msi.setup_msi_irqs(dev, nvec, type);
+}
+
+void arch_teardown_msi_irqs(struct pci_dev *dev)
+{
+	x86_msi.teardown_msi_irqs(dev);
+}
+
+void arch_teardown_msi_irq(unsigned int irq)
+{
+	x86_msi.teardown_msi_irq(irq);
+}
+
+void arch_restore_msi_irqs(struct pci_dev *dev, int irq)
+{
+	x86_msi.restore_msi_irqs(dev, irq);
+}
+#endif
+
 struct x86_io_apic_ops x86_io_apic_ops = {
 	.init			= native_io_apic_init_mappings,
 	.read			= native_io_apic_read,

index ee66f3a..271dfd1 100644
@@ -51,12 +51,18 @@ struct msi_desc {
 };
 
 /*
- * The arch hook for setup up msi irqs
+ * The arch hooks to setup up msi irqs. Those functions are
+ * implemented as weak symbols so that they /can/ be overriden by
+ * architecture specific code if needed.
  */
 int arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc);
 void arch_teardown_msi_irq(unsigned int irq);
 int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
 void arch_teardown_msi_irqs(struct pci_dev *dev);
 int arch_msi_check_device(struct pci_dev* dev, int nvec, int type);
+void arch_restore_msi_irqs(struct pci_dev *dev, int irq);
+
+void default_teardown_msi_irqs(struct pci_dev *dev);
+void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 
 #endif /* LINUX_MSI_H */

1.8.1.2

Now that we have weak versions for each of the PCI MSI architecture functions, we can actually build the MSI support for all platforms, regardless of whether they provide or not architecture-specific versions of those functions. For this reason, the ARCH_SUPPORTS_MSI hidden kconfig boolean becomes useless, and this patch gets rid of it.

---
 arch/arm/Kconfig     | 1 -
 arch/ia64/Kconfig    | 1 -
 arch/mips/Kconfig    | 2 --
 arch/powerpc/Kconfig | 1 -
 arch/s390/Kconfig    | 1 -
 arch/sparc/Kconfig   | 1 -
 arch/tile/Kconfig    | 1 -
 arch/x86/Kconfig     | 1 -
 drivers/pci/Kconfig  | 4 ----
 9 files changed, 13 deletions(-)

index 37c0f4e..41b6c96 100644
@@ -441,7 +441,6 @@ config ARCH_NETX
 config ARCH_IOP13XX
 	bool "IOP13xx-based"
 	depends on MMU
-	select ARCH_SUPPORTS_MSI
 	select CPU_XSC3
 	select NEED_MACH_MEMORY_H
 	select NEED_RET_TO_USER

index 5a768ad..098602b 100644
@@ -9,7 +9,6 @@ config IA64
 	select PCI if (!IA64_HP_SIM)
 	select ACPI if (!IA64_HP_SIM)
 	select PM if (!IA64_HP_SIM)
-	select ARCH_SUPPORTS_MSI
 	select HAVE_UNSTABLE_SCHED_CLOCK
 	select HAVE_IDE
 	select HAVE_OPROFILE

index c3abed3..01b5f5a 100644
@@ -726,7 +726,6 @@ config CAVIUM_OCTEON_SOC
 	select SYS_HAS_CPU_CAVIUM_OCTEON
 	select SWAP_IO_SPACE
 	select HW_HAS_PCI
-	select ARCH_SUPPORTS_MSI
 	select ZONE_DMA32
 	select USB_ARCH_HAS_OHCI
 	select USB_ARCH_HAS_EHCI

@@ -762,7 +761,6 @@ config NLM_XLR_BOARD
 	select CEVT_R4K
 	select CSRC_R4K
 	select IRQ_CPU
-	select ARCH_SUPPORTS_MSI
 	select ZONE_DMA32 if 64BIT
 	select SYNC_R4K
 	select SYS_HAS_EARLY_PRINTK

index 3bf72cd..183a165 100644
@@ -727,7 +727,6 @@ config PCI
 	default y if !40x && !CPM2 && !8xx && !PPC_83xx \
 		&& !PPC_85xx && !PPC_86xx && !GAMECUBE_COMMON
 	default PCI_QSPAN if !4xx && !CPM2 && 8xx
-	select ARCH_SUPPORTS_MSI
 	select GENERIC_PCI_IOMAP
 	help
 	  Find out whether your system includes a PCI bus. PCI is the name of

index 22f75b5..e9982a3 100644
@@ -428,7 +428,6 @@ menuconfig PCI
 	bool "PCI support"
 	default n
 	depends on 64BIT
-	select ARCH_SUPPORTS_MSI
 	select PCI_MSI
 	help
 	  Enable PCI support.

index a00cbd3..1570ad2 100644
@@ -52,7 +52,6 @@ config SPARC32
 
 config SPARC64
 	def_bool 64BIT
-	select ARCH_SUPPORTS_MSI
 	select HAVE_FUNCTION_TRACER
 	select HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_GRAPH_FP_TEST

index 24565a7..74dff90 100644
@@ -380,7 +380,6 @@ config PCI
 	select PCI_DOMAINS
 	select GENERIC_PCI_IOMAP
 	select TILE_GXIO_TRIO if TILEGX
-	select ARCH_SUPPORTS_MSI if TILEGX
 	select PCI_MSI if TILEGX
 	---help---
 	  Enable PCI root complex support, so PCIe endpoint devices can

index b32ebf9..5db62ef 100644
@@ -2014,7 +2014,6 @@ menu "Bus options (PCI etc.)"
 config PCI
 	bool "PCI support"
 	default y
-	select ARCH_SUPPORTS_MSI if (X86_LOCAL_APIC && X86_IO_APIC)
 	---help---
 	  Find out whether you have a PCI motherboard. PCI is the name of a
 	  bus system, i.e. the way the CPU talks to the other stuff inside

index 81944fb..b6a99f7 100644
@@ -1,13 +1,9 @@
 #
 # PCI configuration
 #
-config ARCH_SUPPORTS_MSI
-	bool
-
 config PCI_MSI
 	bool "Message Signaled Interrupts (MSI and MSI-X)"
 	depends on PCI
-	depends on ARCH_SUPPORTS_MSI
 	help
 	   This allows device drivers to enable MSI (Message Signaled
 	   Interrupts).  Message Signaled Interrupts enable a device to

1.8.1.2

From: Thierry Reding <[email protected]>

The new struct msi_chip is used to associated an MSI controller with a PCI bus. It is automatically handed down from the root to its children during bus enumeration.

This patch provides default (weak) implementations for the architecture- specific MSI functions (arch_setup_msi_irq(), arch_teardown_msi_irq() and arch_msi_check_device()) which check if a PCI device's bus has an attached MSI chip and forward the call appropriately.

---
 drivers/pci/msi.c   | 27 +++++++++++++++++++++++++--
 drivers/pci/probe.c |  1 +
 include/linux/msi.h | 11 +++++++++++
 include/linux/pci.h |  1 +
 4 files changed, 38 insertions(+), 2 deletions(-)

index 823c386..2837285 100644
@@ -32,16 +32,39 @@ static int pci_msi_enable = 1;
 
 int __weak arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc)
 {
-	return -EINVAL;
+	struct msi_chip *chip = dev->bus->msi;
+	int err;
+
+	if (!chip || !chip->setup_irq)
+		return -EINVAL;
+
+	err = chip->setup_irq(chip, dev, desc);
+	if (err < 0)
+		return err;
+
+	irq_set_chip_data(desc->irq, chip);
+
+	return 0;
 }
 
 void __weak arch_teardown_msi_irq(unsigned int irq)
 {
+	struct msi_chip *chip = irq_get_chip_data(irq);
+
+	if (!chip || !chip->teardown_irq)
+		return;
+
+	chip->teardown_irq(chip, irq);
 }
 
 int __weak arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
 {
-	return 0;
+	struct msi_chip *chip = dev->bus->msi;
+
+	if (!chip || !chip->check_device)
+		return 0;
+
+	return chip->check_device(chip, dev, nvec, type);
 }
 
 int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)

index 46ada5c..b8eaa81 100644
@@ -666,6 +666,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
 	child->parent = parent;
 	child->ops = parent->ops;
+	child->msi = parent->msi;
 	child->sysdata = parent->sysdata;
 	child->bus_flags = parent->bus_flags;
 

index 271dfd1..090ddad 100644
@@ -65,4 +65,15 @@ void arch_restore_msi_irqs(struct pci_dev *dev, int irq);
 void default_teardown_msi_irqs(struct pci_dev *dev);
 void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 
+struct msi_chip {
+	struct module *owner;
+	struct device *dev;
+
+	int (*setup_irq)(struct msi_chip *chip, struct pci_dev *dev,
+			 struct msi_desc *desc);
+	void (*teardown_irq)(struct msi_chip *chip, unsigned int irq);
+	int (*check_device)(struct msi_chip *chip, struct pci_dev *dev,
+			    int nvec, int type);
+};
+
 #endif /* LINUX_MSI_H */

index 0fd1f15..4044e3c 100644
@@ -433,6 +433,7 @@ struct pci_bus {
 	struct resource busn_res;	/* bus numbers routed to this bus */
 
 	struct pci_ops	*ops;		/* configuration access functions */
+	struct msi_chip	*msi;		/* MSI controller */
 	void		*sysdata;	/* hook for sys-specific extension */
 	struct proc_dir_entry *procdir;	/* directory entry in /proc/bus/pci */
 

1.8.1.2

This commit adds a very basic registry of msi_chip structures, so that an IRQ controller driver can register an msi_chip, and a PCIe host controller can find it, based on a 'struct device_node'.

---
 drivers/of/of_pci.c    | 45 +++++++++++++++++++++++++++++++++++++++++++++
 include/linux/msi.h    |  2 ++
 include/linux/of_pci.h | 12 ++++++++++++
 3 files changed, 59 insertions(+)

index 42c687a..e5ca008 100644
@@ -89,3 +89,48 @@ int of_pci_parse_bus_range(struct device_node *node, struct resource *res)
 	return 0;
 }
 EXPORT_SYMBOL_GPL(of_pci_parse_bus_range);
+
+#ifdef CONFIG_PCI_MSI
+
+static LIST_HEAD(of_pci_msi_chip_list);
+static DEFINE_MUTEX(of_pci_msi_chip_mutex);
+
+int of_pci_msi_chip_add(struct msi_chip *chip)
+{
+	if (!of_property_read_bool(chip->of_node, "msi-controller"))
+		return -EINVAL;
+
+	mutex_lock(&of_pci_msi_chip_mutex);
+	list_add(&chip->list, &of_pci_msi_chip_list);
+	mutex_unlock(&of_pci_msi_chip_mutex);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(of_pci_msi_chip_add);
+
+void of_pci_msi_chip_remove(struct msi_chip *chip)
+{
+	mutex_lock(&of_pci_msi_chip_mutex);
+	list_del(&chip->list);
+	mutex_unlock(&of_pci_msi_chip_mutex);
+}
+EXPORT_SYMBOL_GPL(of_pci_msi_chip_remove);
+
+struct msi_chip *of_pci_find_msi_chip_by_node(struct device_node *of_node)
+{
+	struct msi_chip *c;
+
+	mutex_lock(&of_pci_msi_chip_mutex);
+	list_for_each_entry(c, &of_pci_msi_chip_list, list) {
+		if (c->of_node == of_node) {
+			mutex_unlock(&of_pci_msi_chip_mutex);
+			return c;
+		}
+	}
+	mutex_unlock(&of_pci_msi_chip_mutex);
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(of_pci_find_msi_chip_by_node);
+
+#endif /* CONFIG_PCI_MSI */

index 090ddad..b17ead8 100644
@@ -68,6 +68,8 @@ void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 struct msi_chip {
 	struct module *owner;
 	struct device *dev;
+	struct device_node *of_node;
+	struct list_head list;
 
 	int (*setup_irq)(struct msi_chip *chip, struct pci_dev *dev,
 			 struct msi_desc *desc);

index 7a04826..fd9c408 100644
@@ -2,6 +2,7 @@
 #define __OF_PCI_H
 
 #include <linux/pci.h>
+#include <linux/msi.h>
 
 struct pci_dev;
 struct of_irq;

@@ -13,4 +14,15 @@ struct device_node *of_pci_find_child_device(struct device_node *parent,
 int of_pci_get_devfn(struct device_node *np);
 int of_pci_parse_bus_range(struct device_node *node, struct resource *res);
 
+#if defined(CONFIG_OF) && defined(CONFIG_PCI_MSI)
+int of_pci_msi_chip_add(struct msi_chip *chip);
+void of_pci_msi_chip_remove(struct msi_chip *chip);
+struct msi_chip *of_pci_find_msi_chip_by_node(struct device_node *of_node);
+#else
+static inline int of_pci_msi_chip_add(struct msi_chip *chip) { return -EINVAL; }
+static inline void of_pci_msi_chip_remove(struct msi_chip *chip) { }
+static inline struct msi_chip *
+of_pci_find_msi_chip_by_node(struct device_node *of_node) { return NULL; }
+#endif
+
 #endif

1.8.1.2

Instead of using of_iomap(), we now use of_address_to_resource(), request_mem_region() and ioremap(). This allows the corresponding I/O regions to be properly requested and visible in /proc/iomem.

The main motivation for this change is that the introduction of the MSI support requires us to get the physical address of the main interrupt controller registers, so we will need the corresponding 'struct resource' anyway.

We also take this opportunity to change a panic() to BUG_ON(), in order to be consistent with the rest of the driver.

---
 drivers/irqchip/irq-armada-370-xp.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

index bb328a3..26adc74 100644
@@ -248,12 +248,25 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 					     struct device_node *parent)
 {
+	struct resource main_int_res, per_cpu_int_res;
 	u32 control;
 
-	main_int_base = of_iomap(node, 0);
-	per_cpu_int_base = of_iomap(node, 1);
+	BUG_ON(of_address_to_resource(node, 0, &main_int_res));
+	BUG_ON(of_address_to_resource(node, 1, &per_cpu_int_res));
 
+	BUG_ON(!request_mem_region(main_int_res.start,
+				   resource_size(&main_int_res),
+				   node->full_name));
+	BUG_ON(!request_mem_region(per_cpu_int_res.start,
+				   resource_size(&per_cpu_int_res),
+				   node->full_name));
+
+	main_int_base = ioremap(main_int_res.start,
+				resource_size(&main_int_res));
 	BUG_ON(!main_int_base);
+
+	per_cpu_int_base = ioremap(per_cpu_int_res.start,
+				   resource_size(&per_cpu_int_res));
 	BUG_ON(!per_cpu_int_base);
 
 	control = readl(main_int_base + ARMADA_370_XP_INT_CONTROL);

@@ -262,8 +275,7 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 		irq_domain_add_linear(node, (control >> 2) & 0x3ff,
 				&armada_370_xp_mpic_irq_ops, NULL);
 
-	if (!armada_370_xp_mpic_domain)
-		panic("Unable to add Armada_370_Xp MPIC irq domain (DT)\n");
+	BUG_ON(!armada_370_xp_mpic_domain);
 
 	irq_set_default_host(armada_370_xp_mpic_domain);
 

1.8.1.2

This commit introduces the support for the MSI interrupts in the armada-370-xp interrupt controller driver. It registers an MSI chip to the MSI chip registry, which will be used by the Marvell PCIe host controller driver.

The MSI interrupts use the 16 high doorbells, and are therefore notified using IRQ1 of the main interrupt controller.

---
 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   3 +
 drivers/irqchip/irq-armada-370-xp.c                | 182 ++++++++++++++++++++-
 2 files changed, 184 insertions(+), 1 deletion(-)

index 61df564..d74091a 100644
@@ -4,6 +4,8 @@ Marvell Armada 370 and Armada XP Interrupt Controller
 Required properties:
 - compatible: Should be "marvell,mpic"
 - interrupt-controller: Identifies the node as an interrupt controller.
+- msi-controller: Identifies the node as an PCI Message Signaled
+  Interrupt controller.
 - #interrupt-cells: The number of cells to define the interrupts. Should be 1.
   The cell is the IRQ number
 

@@ -24,6 +26,7 @@ Example:
               #address-cells = <1>;
               #size-cells = <1>;
               interrupt-controller;
+              msi-controller;
               reg = <0xd0020a00 0x1d0>,
                     <0xd0021070 0x58>;
         };

1.8.1.2

Some PCI drivers may need to adjust the pci_bus structure after it has been allocated by the Linux PCI core. The PCI core allows architectures to implement the pcibios_add_bus() and pcibios_remove_bus() for this purpose. This commit therefore extends the hw_pci and pci_sys_data structures of the ARM PCI core to allow PCI drivers to register ->add_bus() and ->remove_bus() in hw_pci, which will get called when a bus is added or removed from the system.

This will be used for example by the Marvell PCIe driver to connect a particular PCI bus with its corresponding MSI chip to handle Message Signaled Interrupts.

---
 arch/arm/include/asm/mach/pci.h |  4 ++++
 arch/arm/kernel/bios32.c        | 16 ++++++++++++++++
 2 files changed, 20 insertions(+)

index a1c90d7..454d642 100644
@@ -36,6 +36,8 @@ struct hw_pci {
 					  resource_size_t start,
 					  resource_size_t size,
 					  resource_size_t align);
+	void		(*add_bus)(struct pci_bus *bus);
+	void		(*remove_bus)(struct pci_bus *bus);
 };
 
 /*

@@ -63,6 +65,8 @@ struct pci_sys_data {
 					  resource_size_t start,
 					  resource_size_t size,
 					  resource_size_t align);
+	void		(*add_bus)(struct pci_bus *bus);
+	void		(*remove_bus)(struct pci_bus *bus);
 	void		*private_data;	/* platform controller private data	*/
 };
 

index 261fcc8..1ec9c87 100644
@@ -363,6 +363,20 @@ void pcibios_fixup_bus(struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pcibios_fixup_bus);
 
+void pcibios_add_bus(struct pci_bus *bus)
+{
+	struct pci_sys_data *sys = bus->sysdata;
+	if (sys->add_bus)
+		sys->add_bus(bus);
+}
+
+void pcibios_remove_bus(struct pci_bus *bus)
+{
+	struct pci_sys_data *sys = bus->sysdata;
+	if (sys->remove_bus)
+		sys->remove_bus(bus);
+}
+
 /*
  * Swizzle the device pin each time we cross a bridge.  If a platform does
  * not provide a swizzle function, we perform the standard PCI swizzling.

@@ -464,6 +478,8 @@ static void pcibios_init_hw(struct device *parent, struct hw_pci *hw,
 		sys->swizzle = hw->swizzle;
 		sys->map_irq = hw->map_irq;
 		sys->align_resource = hw->align_resource;
+		sys->add_bus = hw->add_bus;
+		sys->remove_bus = hw->remove_bus;
 		INIT_LIST_HEAD(&sys->resources);
 
 		if (hw->private_data)

1.8.1.2

Adds the 'msi-controller' property to the main interrupt controller Device Tree node, to indicate that it can now behave as a MSI controller.

---
 arch/arm/boot/dts/armada-370-xp.dtsi | 1 +
 1 file changed, 1 insertion(+)

index 90b1176..ac0bbc0 100644
@@ -56,6 +56,7 @@
 				#interrupt-cells = <1>;
 				#size-cells = <1>;
 				interrupt-controller;
+				msi-controller;
 			};
 
 			coherency-fabric at 20200 {

1.8.1.2

This commit adds support for Message Signaled Interrupts in the Marvell PCIe host controller. The work is very simple: it simply gets a reference to the msi_chip associated to the PCIe controller thanks to the msi-parent DT property, and stores this reference in the pci_bus structure. This is enough to let the Linux PCI core use the functions of msi_chip to setup and teardown MSIs.

---
 .../devicetree/bindings/pci/mvebu-pci.txt          |  3 +++
 drivers/pci/host/pci-mvebu.c                       | 26 ++++++++++++++++++++++
 2 files changed, 29 insertions(+)

index f8d4058..77e0ffe 100644
@@ -12,6 +12,8 @@ Mandatory properties:
 - device_type, set to "pci"
 - ranges: ranges for the PCI memory and I/O regions, as well as the
   MMIO registers to control the PCIe interfaces.
+- msi-parent: Link to the hardware entity that serves as the Message
+  Signaled Interrupt controller for this PCI controller.
 
 In addition, the Device Tree node must have sub-nodes describing each
 PCIe interface, having the following mandatory properties:

@@ -46,6 +48,7 @@ pcie-controller {
 	#size-cells = <2>;
 
 	bus-range = <0x00 0xff>;
+	msi-parent = <&mpic>;
 
 	ranges = <0x82000000 0 0xd0040000 0xd0040000 0 0x00002000   /* Port 0.0 registers */
 		  0x82000000 0 0xd0042000 0xd0042000 0 0x00002000   /* Port 2.0 registers */

index 13a633b..5715a97 100644
@@ -11,6 +11,7 @@
 #include <linux/clk.h>
 #include <linux/module.h>
 #include <linux/mbus.h>
+#include <linux/msi.h>
 #include <linux/slab.h>
 #include <linux/platform_device.h>
 #include <linux/of_address.h>

@@ -107,6 +108,7 @@ struct mvebu_pcie_port;
 struct mvebu_pcie {
 	struct platform_device *pdev;
 	struct mvebu_pcie_port *ports;
+	struct msi_chip *msi;
 	struct resource io;
 	struct resource realio;
 	struct resource mem;

@@ -695,6 +697,12 @@ static struct pci_bus *mvebu_pcie_scan_bus(int nr, struct pci_sys_data *sys)
 	return bus;
 }
 
+void mvebu_pcie_add_bus(struct pci_bus *bus)
+{
+	struct mvebu_pcie *pcie = sys_to_pcie(bus->sysdata);
+	bus->msi = pcie->msi;
+}
+
 resource_size_t mvebu_pcie_align_resource(struct pci_dev *dev,
 					  const struct resource *res,
 					  resource_size_t start,

@@ -731,6 +739,7 @@ static void __init mvebu_pcie_enable(struct mvebu_pcie *pcie)
 	hw.map_irq        = mvebu_pcie_map_irq;
 	hw.ops            = &mvebu_pcie_ops;
 	hw.align_resource = mvebu_pcie_align_resource;
+	hw.add_bus        = mvebu_pcie_add_bus;
 
 	pci_common_init(&hw);
 }

@@ -755,6 +764,21 @@ mvebu_pcie_map_registers(struct platform_device *pdev,
 	return devm_request_and_ioremap(&pdev->dev, &regs);
 }
 
+static void __init mvebu_pcie_msi_enable(struct mvebu_pcie *pcie)
+{
+	struct device_node *msi_node;
+
+	msi_node = of_parse_phandle(pcie->pdev->dev.of_node,
+				    "msi-parent", 0);
+	if (!msi_node)
+		return;
+
+	pcie->msi = of_pci_find_msi_chip_by_node(msi_node);
+
+	if (pcie->msi)
+		pcie->msi->dev = &pcie->pdev->dev;
+}
+
 static int __init mvebu_pcie_probe(struct platform_device *pdev)
 {
 	struct mvebu_pcie *pcie;

@@ -879,6 +903,8 @@ static int __init mvebu_pcie_probe(struct platform_device *pdev)
 		i++;
 	}
 
+	mvebu_pcie_msi_enable(pcie);
+
 	mvebu_pcie_enable(pcie);
 
 	return 0;

1.8.1.2

This commit adjusts the Armada 370 and Armada XP PCIe controllers Device Tree informations to reference their MSI controller. In the case of this platform, the MSI controller is implemented by the MPIC.

---
 arch/arm/boot/dts/armada-370.dtsi        | 1 +
 arch/arm/boot/dts/armada-xp-mv78230.dtsi | 1 +
 arch/arm/boot/dts/armada-xp-mv78260.dtsi | 1 +
 arch/arm/boot/dts/armada-xp-mv78460.dtsi | 1 +
 4 files changed, 4 insertions(+)

index fa3dfc6..a60018d 100644
@@ -178,6 +178,7 @@
 				#address-cells = <3>;
 				#size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

index f8eaa38..4a9a305 100644
@@ -91,6 +91,7 @@
 #address-cells = <3>;
 #size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

index 2d9335d..dd23e96 100644
@@ -111,6 +111,7 @@
 				#address-cells = <3>;
 				#size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

index c7b1f4d..77db2bc 100644
@@ -127,6 +127,7 @@
 				#address-cells = <3>;
 				#size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

1.8.1.2

Thomas,

Whole series applied as follows:

+---+ mvebu/msi_pci (1-4,7)
    |
    +---+---+ mvebu/msi_irq (5,6)
            |
            +---+---+ mvebu/msi_mvebu (8-10)

mvebu/msi_pci built successfully with Randy's config for x86_64.

Thierry, I've tagged mvebu/msi_pci. Unless it all falls apart in -next, those commit-id's won't change from what will be in arm-soc.

Unless I hear any objection, I'll be submitting the PRs mid-week or so.

thx,

Jason.

So, the branch still isn't stable, hence I shouldn't use it as a baseline to apply Thierry's PCIe patches?

It's as stable as I can make it. I've tested it for all known build regressions (discovered in -next), and I see no reason to change it. Anything needing fixed after today, I'll append the fix to it's HEAD if appropriate.

thx,

Jason.

OK, if you will only append fixes rather than rebase the branch, then it's fine; what you said above implied that the commit IDs still might change.

So, I'll try pulling this in with Thierry's patches today.

Thanks.

Well, that's being generous. ;-) What I said was muddy as hell. Thanks for making me clear that up.

Great!

thx,

Jason.