MSI support for Marvell EBU PCIe driver

Hello,

This set of patches introduces Message Signaled Interrupt support in the Marvell EBU PCIe driver. It has been successfully tested on the Armada XP GP platform with an Intel e1000e PCIe network card that supports MSI.

This patch set is intended for merging in 3.12, so the respective maintainers of the different areas are invited to review/ack the patches, see below for the details.

There are still missing ACKs from Benjamin Herrenschmidt (5, 6, 7), Thomas Gleixner (8, 9).

The patches do the following:

* Patch 1 reworks how the architecture-specific MSI functions can be overriden by architecture-specific code, by using weak functions. It was suggested by Bjorn Helgaas.

This patch has been acked by Bjorn Helgaas, the PCI maintainter.

* Patch 2 removes the ARCH_SUPPORTS_MSI hidden kconfig boolean which is no longer needed now that we have weak functions for all MSI architecture-specific hooks.

This patch has been acked by Bjorn Helgaas, the PCI maintainer.

* Patch 3 adds a minimal msi_chip infrastructure, that allows a pci_bus to be connected to a msi_chip, and that provides default implementations of the architecture-specific MSI functions to use msi_chip.

This patch has been acked by Bjorn Helgaas, the PCI maintainter.

* Patch 4 adds an IRQ domain function that allows to allocate dynamically a free hwirq number from an IRQ domain, and create a virq mapping to it. It was suggested by Grant Likely in order to remove the hwirq allocation code from the IRQ driver itself.

Patch has been Acked by Grant Likely, former irq_domain maintainer.

* Patch 5 refactors the __irq_domain_add() function in the irq_domain code, in preparation for the introduction of MSI support in irq_domain.

This patch needs the ACK of the irq_domain maintainer Benjamin Herrenschmidt <[email protected]>.

* Patch 6 allows to create an irq_domain associated with a msi_chip, thanks to the new irq_domain_add_msi() function.

This patch needs the ACK of the irq_domain maintainer Benjamin Herrenschmidt <[email protected]>.

* Patch 7 adds a new function irq_find_msi_host() to find the MSI-type interrupt controller associated to a given DT node, and adjusts irq_find_host() to not match MSI-type interrupt controllers.

This patch needs the ACK of the irq_domain maintainer Benjamin Herrenschmidt <[email protected]>.

* Patch 8 makes some not very interesting preparation in the Armada 370/XP IRQ controller driver.

This patch needs the Ack from Thomas Gleixner, as the drivers/irqchip maintainer.

* Patch 9 implements the MSI support in the Armada 370/XP IRQ controller driver. It registers an msi_chip using the msi_chip_add() function added in PATCH 3.

This patch needs the Ack from Thomas Gleixner, as the drivers/irqchip maintainer.

* Patch 10 extends the ARM PCI core to expose ->add_bus() and ->remove_bus() hooks to PCI drivers. This was suggested by Bjorn Helgaas to allow the PCI driver to connected the PCI busses with the corresponding MSI chip.

This patch has been Acked by Russell King, the ARM maintainer.

* Patch 11 adjust the Armada 370/XP Device Tree to indicate that the MPIC is not only an interrupt-controller, but also an msi-controller.

This patch needs the Ack from Jason Cooper, Gregory Clement and/or Andrew Lunn, the Marvell maintainers.

* Patch 12 adds MSI support in the Marvell PCIe host controller driver. The work to do here is minimal: get a reference to the msi-parent controller thanks to msi_chip_find_by_of_node(), and link it to the pci_bus structure before the bus gets enumerated.

This patch has been acked by Bjorn Helgaas, the PCI maintainer.

* Patch 13 adjusts the Armada 370/XP Device Tree to add the msi-parent properties in the PCIe controller nodes.

This patch needs the Ack from Jason Cooper, Gregory Clement and/or Andrew Lunn, the Marvell Maintainers.

This version 7 follows:
* PATCH version 6 sent on August, 1st 2013
* PATCH version 5 sent on July, 15th 2013
* PATCH version 4 sent on July, 1st 2013
* PATCH version 3 sent on June, 19th 2013
* PATCH version 2 sent on June, 6th 2013
* RFC version 1 sent on March, 26th 2013

Changes since v6:

* Fixed x86 build failure related to default_restore_msi_irqs(), noticed by Stephen Rothwell on linux-next.

* Added Acked-by from Bjorn Helgaas on 'PCI: remove ARCH_SUPPORTS_MSI kconfig option'.

* Fixed a compile-time issue in drivers/irqchip/irq-armada-370-xp.c in the !CONFIG_PCI_MSI case.

* Fixed one of the stylistic issue raised by Thierry Reding on 'ARM:
pci: add ->add_bus() and ->remove_bus() hooks to hw_pci' (i.e do not align '=' when assigning elements of the pci_sys_data structure). I however didn't fix the tab vs. spaces indentation issue noticed by Thierry, since I couldn't see it (to me the indentation was identical in both locations).

* Added Tested-by from Daniel Price on all patches.

Changes since v5:

* Rebased on top of 3.11-rc3.

* Implemented the suggestions of Grant Likely on "irqdomain: add irq_alloc_mapping() function" and added his Acked-by.

* Added Bjorn Helgaas Acked-by on "PCI: use weak functions for MSI arch-specific functions".

* Added Bjorn Helgaas Acked-by on "PCI: Introduce new MSI chip infrastructure".

* Remove the drivers/of global registry of irq_chip, and as suggested by Grant Likely, create an association between an irq_domain and a msi_chip. This required refactoring the __irq_domain_add() function, adding a 'msi_chip' pointer to 'irq_domain', adding an irq_domain_add_msi() function and a irq_find_msi_host() function.

* Added Russell King Acked-by on "ARM: pci: add ->add_bus() and ->remove_bus() hooks to hw_pci"

Changes since v4:

* Rebased on top of 3.11-rc1.

* Slightly rework the implementation of arch_setup_msi_irq(), arch_teardown_msi_irq and arch_msi_check_device() according to the suggestions of Bjorn Helgaas (PATCH 4)

* Make a few improvements to the msi_chip registry, according to the suggestion of Bjorn Helgaas and Rob Herring: coding style fixes, usage of mutex while traversing the list of msi_chip, usage of of_pci_msi_* prefix instead of just of_msi_*.

Changes since v3:

* Keep only a default_teardown_msi_irqs() function needed for the Xen PCI x86 code, and remove all other default_*() function and put the default behavior directly in the weak functions. Suggested by Thierry Redding and Bjorn Helgaas.

* Misc small improvements the MSI chip registry code: compiled only under CONFIG_PCI_MSI, functions exported to modules, addition of an of_msi_chip_remove() function, renaming of of_msi_chip_find_by_node() to of_find_msi_chip_by_node(), move the test of the "msi-controller" property to the of_msi_chip_add() function, renamed the list_head field from link to list, added dummy functions in the header files when !CONFIG_PCI_MSI && !CONFIG_OF. All suggested by Thierry Redding.

* Add a patch that entirely removes the ARCH_SUPPORTS_MSI, as we now have weak functions, so even if an arch doesn't actually use/support MSI, the code will build properly. Suggested by Thierry Redding.

* Added Device Tree binding documentation updates for the IRQ controller and the PCIe controller.

Changes since v2:

* Add an IRQ domain function that allows to allocate dynamically a free hwirq number from an IRQ domain, and create a virq mapping to it. It was suggested by Grant Likely in order to remove the hwirq allocation code from the IRQ driver itself. (PATCH 1)

* Separate the use of weak functions from the introduction of the msi_chip infrastructure, and use weak functions for all architecture-specific MSI hooks. Suggested by Bjorn Helgaas (PATCH 2).

* Move the msi_chip registry to drivers/of. Suggested by Bjorn Helgaas (PATCH 4).

* Use pcibios_add_bus() and pcibios_remove_bus() to connect the PCI busses to their msi_chip, as suggested by Bjorn Helgaas. Requires some ARM PCI core changes (PATCH 7), and then changes to the PCI driver itself (PATCH 9).

This set of patches applies on top of 3.11-rc3, and is available as a Git branch at
https://github.com/MISL-EBU-System-SW/mainline-public/tree/marvell-pcie-msi-v7.

Thanks,

Thomas

Thierry Reding (1):
PCI: Introduce new MSI chip infrastructure

Thomas Petazzoni (12):
PCI: use weak functions for MSI arch-specific functions
PCI: remove ARCH_SUPPORTS_MSI kconfig option
irqdomain: add irq_alloc_mapping() function
irqdomain: refactor __irq_domain_add()
irqdomain: add support to associate an irq_domain with a msi_chip
irqdomain: add function to find a MSI irq_domain
irqchip: armada-370-xp: properly request resources
irqchip: armada-370-xp: implement MSI support
ARM: pci: add ->add_bus() and ->remove_bus() hooks to hw_pci
ARM: mvebu: the MPIC now provides MSI controller features
PCI: mvebu: add support for MSI
ARM: mvebu: link PCIe controllers to the MSI controller

 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   3 +
 .../devicetree/bindings/pci/mvebu-pci.txt          |   3 +
 arch/arm/Kconfig                                   |   1 -
 arch/arm/boot/dts/armada-370-xp.dtsi               |   1 +
 arch/arm/boot/dts/armada-370.dtsi                  |   1 +
 arch/arm/boot/dts/armada-xp-mv78230.dtsi           |   1 +
 arch/arm/boot/dts/armada-xp-mv78260.dtsi           |   1 +
 arch/arm/boot/dts/armada-xp-mv78460.dtsi           |   1 +
 arch/arm/include/asm/mach/pci.h                    |   4 +
 arch/arm/kernel/bios32.c                           |  16 +++
 arch/ia64/Kconfig                                  |   1 -
 arch/mips/Kconfig                                  |   2 -
 arch/mips/include/asm/pci.h                        |   5 -
 arch/powerpc/Kconfig                               |   1 -
 arch/powerpc/include/asm/pci.h                     |   5 -
 arch/s390/Kconfig                                  |   1 -
 arch/s390/include/asm/pci.h                        |   4 -
 arch/sparc/Kconfig                                 |   1 -
 arch/tile/Kconfig                                  |   1 -
 arch/x86/Kconfig                                   |   1 -
 arch/x86/include/asm/pci.h                         |  28 ----
 arch/x86/kernel/x86_init.c                         |  21 +++
 drivers/irqchip/irq-armada-370-xp.c                | 158 ++++++++++++++++++++-
 drivers/pci/Kconfig                                |   4 -
 drivers/pci/host/pci-mvebu.c                       |  31 ++++
 drivers/pci/msi.c                                  |  69 ++++++---
 drivers/pci/probe.c                                |   1 +
 include/linux/irqdomain.h                          |  62 +++++++-
 include/linux/msi.h                                |  19 ++-
 include/linux/pci.h                                |   1 +
 kernel/irq/irqdomain.c                             |  89 +++++++++---
 31 files changed, 432 insertions(+), 105 deletions(-)

1.8.1.2

Until now, the MSI architecture-specific functions could be overloaded using a fairly complex set of #define and compile-time conditionals. In order to prepare for the introduction of the msi_chip infrastructure, it is desirable to switch all those functions to use the 'weak' mechanism. This commit converts all the architectures that were overidding those MSI functions to use the new strategy.

Note that we keep two separate, non-weak, functions default_teardown_msi_irqs() and default_restore_msi_irqs() for the default behavior of the arch_teardown_msi_irqs() and arch_restore_msi_irqs(), as the default behavior is needed by x86 PCI code.

---
 arch/mips/include/asm/pci.h    |  5 -----
 arch/powerpc/include/asm/pci.h |  5 -----
 arch/s390/include/asm/pci.h    |  4 ----
 arch/x86/include/asm/pci.h     | 28 ------------------------
 arch/x86/kernel/x86_init.c     | 21 ++++++++++++++++++
 drivers/pci/msi.c              | 48 +++++++++++++++++++++---------------------
 include/linux/msi.h            |  8 ++++++-
 7 files changed, 52 insertions(+), 67 deletions(-)

index fa8e0aa..f194c08 100644
@@ -136,11 +136,6 @@ static inline int pci_get_legacy_ide_irq(struct pci_dev *dev, int channel)
 	return channel ? 15 : 14;
 }
 
-#ifdef CONFIG_CPU_CAVIUM_OCTEON
-/* MSI arch hook for OCTEON */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
-#endif
-
 extern char * (*pcibios_plat_setup)(char *str);
 
 #ifdef CONFIG_OF

index 6653f27..95145a1 100644
@@ -113,11 +113,6 @@ extern int pci_domain_nr(struct pci_bus *bus);
 /* Decide whether to display the domain number in /proc */
 extern int pci_proc_domain(struct pci_bus *bus);
 
-/* MSI arch hooks */
-#define arch_setup_msi_irqs arch_setup_msi_irqs
-#define arch_teardown_msi_irqs arch_teardown_msi_irqs
-#define arch_msi_check_device arch_msi_check_device
-
 struct vm_area_struct;
 /* Map a range of PCI memory or I/O space for a device into user space */
 int pci_mmap_page_range(struct pci_dev *pdev, struct vm_area_struct *vma,

index 6e577ba..262b91b 100644
@@ -21,10 +21,6 @@ void pci_iounmap(struct pci_dev *, void __iomem *);
 int pci_domain_nr(struct pci_bus *);
 int pci_proc_domain(struct pci_bus *);
 
-/* MSI arch hooks */
-#define arch_setup_msi_irqs	arch_setup_msi_irqs
-#define arch_teardown_msi_irqs	arch_teardown_msi_irqs
-
 #define ZPCI_BUS_NR			0	/* default bus number */
 #define ZPCI_DEVFN			0	/* default device number */
 

index d9e9e6c..8c61de0 100644
@@ -100,29 +100,6 @@ static inline void early_quirks(void) { }
 extern void pci_iommu_alloc(void);
 
 #ifdef CONFIG_PCI_MSI
-/* MSI arch specific hooks */
-static inline int x86_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
-{
-	return x86_msi.setup_msi_irqs(dev, nvec, type);
-}
-
-static inline void x86_teardown_msi_irqs(struct pci_dev *dev)
-{
-	x86_msi.teardown_msi_irqs(dev);
-}
-
-static inline void x86_teardown_msi_irq(unsigned int irq)
-{
-	x86_msi.teardown_msi_irq(irq);
-}
-static inline void x86_restore_msi_irqs(struct pci_dev *dev, int irq)
-{
-	x86_msi.restore_msi_irqs(dev, irq);
-}
-#define arch_setup_msi_irqs x86_setup_msi_irqs
-#define arch_teardown_msi_irqs x86_teardown_msi_irqs
-#define arch_teardown_msi_irq x86_teardown_msi_irq
-#define arch_restore_msi_irqs x86_restore_msi_irqs
 /* implemented in arch/x86/kernel/apic/io_apic. */
 struct msi_desc;
 int native_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);

@@ -130,11 +107,6 @@ void native_teardown_msi_irq(unsigned int irq);
 void native_restore_msi_irqs(struct pci_dev *dev, int irq);
 int setup_msi_irq(struct pci_dev *dev, struct msi_desc *msidesc,
 		  unsigned int irq_base, unsigned int irq_offset);
-/* default to the implementation in drivers/lib/msi.c */
-#define HAVE_DEFAULT_MSI_TEARDOWN_IRQS
-#define HAVE_DEFAULT_MSI_RESTORE_IRQS
-void default_teardown_msi_irqs(struct pci_dev *dev);
-void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 #else
 #define native_setup_msi_irqs		NULL
 #define native_teardown_msi_irq		NULL

index 5f24c71..4c374a9 100644
@@ -116,6 +116,27 @@ struct x86_msi_ops x86_msi = {
 	.setup_hpet_msi		= default_setup_hpet_msi,
 };
 
+/* MSI arch specific hooks */
+int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)
+{
+	return x86_msi.setup_msi_irqs(dev, nvec, type);
+}
+
+void arch_teardown_msi_irqs(struct pci_dev *dev)
+{
+	x86_msi.teardown_msi_irqs(dev);
+}
+
+void arch_teardown_msi_irq(unsigned int irq)
+{
+	x86_msi.teardown_msi_irq(irq);
+}
+
+void arch_restore_msi_irqs(struct pci_dev *dev, int irq)
+{
+	x86_msi.restore_msi_irqs(dev, irq);
+}
+
 struct x86_io_apic_ops x86_io_apic_ops = {
 	.init			= native_io_apic_init_mappings,
 	.read			= native_io_apic_read,

index ee66f3a..271dfd1 100644
@@ -51,12 +51,18 @@ struct msi_desc {
 };
 
 /*
- * The arch hook for setup up msi irqs
+ * The arch hooks to setup up msi irqs. Those functions are
+ * implemented as weak symbols so that they /can/ be overriden by
+ * architecture specific code if needed.
  */
 int arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc);
 void arch_teardown_msi_irq(unsigned int irq);
 int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
 void arch_teardown_msi_irqs(struct pci_dev *dev);
 int arch_msi_check_device(struct pci_dev* dev, int nvec, int type);
+void arch_restore_msi_irqs(struct pci_dev *dev, int irq);
+
+void default_teardown_msi_irqs(struct pci_dev *dev);
+void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 
 #endif /* LINUX_MSI_H */

1.8.1.2

Now that we have weak versions for each of the PCI MSI architecture functions, we can actually build the MSI support for all platforms, regardless of whether they provide or not architecture-specific versions of those functions. For this reason, the ARCH_SUPPORTS_MSI hidden kconfig boolean becomes useless, and this patch gets rid of it.

---
 arch/arm/Kconfig     | 1 -
 arch/ia64/Kconfig    | 1 -
 arch/mips/Kconfig    | 2 --
 arch/powerpc/Kconfig | 1 -
 arch/s390/Kconfig    | 1 -
 arch/sparc/Kconfig   | 1 -
 arch/tile/Kconfig    | 1 -
 arch/x86/Kconfig     | 1 -
 drivers/pci/Kconfig  | 4 ----
 9 files changed, 13 deletions(-)

index 37c0f4e..41b6c96 100644
@@ -441,7 +441,6 @@ config ARCH_NETX
 config ARCH_IOP13XX
 	bool "IOP13xx-based"
 	depends on MMU
-	select ARCH_SUPPORTS_MSI
 	select CPU_XSC3
 	select NEED_MACH_MEMORY_H
 	select NEED_RET_TO_USER

index 5a768ad..098602b 100644
@@ -9,7 +9,6 @@ config IA64
 	select PCI if (!IA64_HP_SIM)
 	select ACPI if (!IA64_HP_SIM)
 	select PM if (!IA64_HP_SIM)
-	select ARCH_SUPPORTS_MSI
 	select HAVE_UNSTABLE_SCHED_CLOCK
 	select HAVE_IDE
 	select HAVE_OPROFILE

index c3abed3..01b5f5a 100644
@@ -726,7 +726,6 @@ config CAVIUM_OCTEON_SOC
 	select SYS_HAS_CPU_CAVIUM_OCTEON
 	select SWAP_IO_SPACE
 	select HW_HAS_PCI
-	select ARCH_SUPPORTS_MSI
 	select ZONE_DMA32
 	select USB_ARCH_HAS_OHCI
 	select USB_ARCH_HAS_EHCI

@@ -762,7 +761,6 @@ config NLM_XLR_BOARD
 	select CEVT_R4K
 	select CSRC_R4K
 	select IRQ_CPU
-	select ARCH_SUPPORTS_MSI
 	select ZONE_DMA32 if 64BIT
 	select SYNC_R4K
 	select SYS_HAS_EARLY_PRINTK

index 3bf72cd..183a165 100644
@@ -727,7 +727,6 @@ config PCI
 	default y if !40x && !CPM2 && !8xx && !PPC_83xx \
 		&& !PPC_85xx && !PPC_86xx && !GAMECUBE_COMMON
 	default PCI_QSPAN if !4xx && !CPM2 && 8xx
-	select ARCH_SUPPORTS_MSI
 	select GENERIC_PCI_IOMAP
 	help
 	  Find out whether your system includes a PCI bus. PCI is the name of

index 22f75b5..e9982a3 100644
@@ -428,7 +428,6 @@ menuconfig PCI
 	bool "PCI support"
 	default n
 	depends on 64BIT
-	select ARCH_SUPPORTS_MSI
 	select PCI_MSI
 	help
 	  Enable PCI support.

index a00cbd3..1570ad2 100644
@@ -52,7 +52,6 @@ config SPARC32
 
 config SPARC64
 	def_bool 64BIT
-	select ARCH_SUPPORTS_MSI
 	select HAVE_FUNCTION_TRACER
 	select HAVE_FUNCTION_GRAPH_TRACER
 	select HAVE_FUNCTION_GRAPH_FP_TEST

index 24565a7..74dff90 100644
@@ -380,7 +380,6 @@ config PCI
 	select PCI_DOMAINS
 	select GENERIC_PCI_IOMAP
 	select TILE_GXIO_TRIO if TILEGX
-	select ARCH_SUPPORTS_MSI if TILEGX
 	select PCI_MSI if TILEGX
 	---help---
 	  Enable PCI root complex support, so PCIe endpoint devices can

index b32ebf9..5db62ef 100644
@@ -2014,7 +2014,6 @@ menu "Bus options (PCI etc.)"
 config PCI
 	bool "PCI support"
 	default y
-	select ARCH_SUPPORTS_MSI if (X86_LOCAL_APIC && X86_IO_APIC)
 	---help---
 	  Find out whether you have a PCI motherboard. PCI is the name of a
 	  bus system, i.e. the way the CPU talks to the other stuff inside

index 81944fb..b6a99f7 100644
@@ -1,13 +1,9 @@
 #
 # PCI configuration
 #
-config ARCH_SUPPORTS_MSI
-	bool
-
 config PCI_MSI
 	bool "Message Signaled Interrupts (MSI and MSI-X)"
 	depends on PCI
-	depends on ARCH_SUPPORTS_MSI
 	help
 	   This allows device drivers to enable MSI (Message Signaled
 	   Interrupts).  Message Signaled Interrupts enable a device to

1.8.1.2

From: Thierry Reding <[email protected]>

The new struct msi_chip is used to associated an MSI controller with a PCI bus. It is automatically handed down from the root to its children during bus enumeration.

This patch provides default (weak) implementations for the architecture- specific MSI functions (arch_setup_msi_irq(), arch_teardown_msi_irq() and arch_msi_check_device()) which check if a PCI device's bus has an attached MSI chip and forward the call appropriately.

---
 drivers/pci/msi.c   | 27 +++++++++++++++++++++++++--
 drivers/pci/probe.c |  1 +
 include/linux/msi.h | 11 +++++++++++
 include/linux/pci.h |  1 +
 4 files changed, 38 insertions(+), 2 deletions(-)

index 823c386..2837285 100644
@@ -32,16 +32,39 @@ static int pci_msi_enable = 1;
 
 int __weak arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc)
 {
-	return -EINVAL;
+	struct msi_chip *chip = dev->bus->msi;
+	int err;
+
+	if (!chip || !chip->setup_irq)
+		return -EINVAL;
+
+	err = chip->setup_irq(chip, dev, desc);
+	if (err < 0)
+		return err;
+
+	irq_set_chip_data(desc->irq, chip);
+
+	return 0;
 }
 
 void __weak arch_teardown_msi_irq(unsigned int irq)
 {
+	struct msi_chip *chip = irq_get_chip_data(irq);
+
+	if (!chip || !chip->teardown_irq)
+		return;
+
+	chip->teardown_irq(chip, irq);
 }
 
 int __weak arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
 {
-	return 0;
+	struct msi_chip *chip = dev->bus->msi;
+
+	if (!chip || !chip->check_device)
+		return 0;
+
+	return chip->check_device(chip, dev, nvec, type);
 }
 
 int __weak arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type)

index 46ada5c..b8eaa81 100644
@@ -666,6 +666,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
 	child->parent = parent;
 	child->ops = parent->ops;
+	child->msi = parent->msi;
 	child->sysdata = parent->sysdata;
 	child->bus_flags = parent->bus_flags;
 

index 271dfd1..090ddad 100644
@@ -65,4 +65,15 @@ void arch_restore_msi_irqs(struct pci_dev *dev, int irq);
 void default_teardown_msi_irqs(struct pci_dev *dev);
 void default_restore_msi_irqs(struct pci_dev *dev, int irq);
 
+struct msi_chip {
+	struct module *owner;
+	struct device *dev;
+
+	int (*setup_irq)(struct msi_chip *chip, struct pci_dev *dev,
+			 struct msi_desc *desc);
+	void (*teardown_irq)(struct msi_chip *chip, unsigned int irq);
+	int (*check_device)(struct msi_chip *chip, struct pci_dev *dev,
+			    int nvec, int type);
+};
+
 #endif /* LINUX_MSI_H */

index 0fd1f15..4044e3c 100644
@@ -433,6 +433,7 @@ struct pci_bus {
 	struct resource busn_res;	/* bus numbers routed to this bus */
 
 	struct pci_ops	*ops;		/* configuration access functions */
+	struct msi_chip	*msi;		/* MSI controller */
 	void		*sysdata;	/* hook for sys-specific extension */
 	struct proc_dir_entry *procdir;	/* directory entry in /proc/bus/pci */
 

1.8.1.2

This commit extends the irqdomain subsystem with an irq_alloc_mapping() function which allows to let the irqdomain code find an available hwirq number in the range [ 0 ; domain size ] for the given domain, and create a virq mapping for it.

---
 include/linux/irqdomain.h |  2 ++
 kernel/irq/irqdomain.c    | 36 ++++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

index c983ed1..1ffa336 100644
@@ -195,6 +195,8 @@ static inline unsigned int irq_linear_revmap(struct irq_domain *domain,
 extern unsigned int irq_find_mapping(struct irq_domain *host,
 				     irq_hw_number_t hwirq);
 extern unsigned int irq_create_direct_mapping(struct irq_domain *host);
+extern unsigned int irq_alloc_mapping(struct irq_domain *host,
+				      irq_hw_number_t *hwirq);
 extern int irq_create_strict_mappings(struct irq_domain *domain,
 				      unsigned int irq_base,
 				      irq_hw_number_t hwirq_base, int count);

index 706724e..034bbac 100644
@@ -375,6 +375,42 @@ unsigned int irq_create_direct_mapping(struct irq_domain *domain)
 EXPORT_SYMBOL_GPL(irq_create_direct_mapping);
 
 /**
+ * irq_alloc_mapping() - Allocate an irq for mapping
+ * @domain: domain to allocate the irq for or NULL for default domain
+ * @hwirq:  reference to the returned hwirq
+ *
+ * This routine are used for irq controllers which can choose the
+ * hardware interrupt number from a range [ 0 ; domain size ], such as
+ * is often the case with PCI MSI controllers. The function will
+ * returned the allocated hwirq number in the hwirq pointer, and the
+ * corresponding virq number as the return value.
+ */
+unsigned int irq_alloc_mapping(struct irq_domain *domain,
+			       irq_hw_number_t *out_hwirq)
+{
+	irq_hw_number_t hwirq;
+	int rc;
+
+	pr_debug("irq_alloc_mapping(0x%p)\n", domain);
+
+	for (hwirq = 0; hwirq < domain->revmap_size; hwirq++)
+		if (domain->linear_revmap[hwirq] == 0)
+			break;
+
+	if (hwirq == domain->hwirq_max) {
+		pr_debug("-> no available hwirq found\n");
+		return 0;
+	}
+
+	rc = irq_create_mapping(domain, hwirq);
+	if (rc)
+		*out_hwirq = hwirq;
+
+	return rc;
+}
+EXPORT_SYMBOL_GPL(irq_alloc_mapping);
+
+/**
  * irq_create_mapping() - Map a hardware interrupt into linux irq space
  * @domain: domain owning this hardware interrupt or NULL for default domain
  * @hwirq: hardware irq number in that domain space

1.8.1.2

Currently, the __irq_domain_add() function does both the irq_domain allocation *and* registration. This is a problem for irq_domain_add_*() helpers that need to do something between the allocation and the registration, such as for example setting a d->msi_chip pointer in the irq_domain structure before this structure is made globally visible by being registered.

Moreover, the comment in __irq_domain_add() also suggests that an irq_domain_register() function should exist.

Therefore, this commit:

* Splits __irq_domain_add() into __irq_domain_alloc() (allocation and initialization of the irq_domain) and __irq_domain_register() (registration of the irq_domain to the global list)

* Modifies the existing users of __irq_domain_add() to use those two functions.

A next commit will make another use of it when introducing support for msi_chip in the irqdomain code.

---
 include/linux/irqdomain.h | 21 +++++++++++++++++----
 kernel/irq/irqdomain.c    | 40 +++++++++++++++++++++++++++-------------
 2 files changed, 44 insertions(+), 17 deletions(-)

index 1ffa336..7113941 100644
@@ -111,10 +111,11 @@ struct irq_domain {
 };
 
 #ifdef CONFIG_IRQ_DOMAIN
-struct irq_domain *__irq_domain_add(struct device_node *of_node, int size,
+struct irq_domain *__irq_domain_alloc(struct device_node *of_node, int size,
 				    irq_hw_number_t hwirq_max, int direct_max,
 				    const struct irq_domain_ops *ops,
 				    void *host_data);
+void __irq_domain_register(struct irq_domain *domain);
 struct irq_domain *irq_domain_add_simple(struct device_node *of_node,
 					 unsigned int size,
 					 unsigned int first_irq,

@@ -141,14 +142,22 @@ static inline struct irq_domain *irq_domain_add_linear(struct device_node *of_no
 					 const struct irq_domain_ops *ops,
 					 void *host_data)
 {
-	return __irq_domain_add(of_node, size, size, 0, ops, host_data);
+	struct irq_domain *d;
+	d = __irq_domain_alloc(of_node, size, size, 0, ops, host_data);
+	if (d)
+		__irq_domain_register(d);
+	return d;
 }
 static inline struct irq_domain *irq_domain_add_nomap(struct device_node *of_node,
 					 unsigned int max_irq,
 					 const struct irq_domain_ops *ops,
 					 void *host_data)
 {
-	return __irq_domain_add(of_node, 0, max_irq, max_irq, ops, host_data);
+	struct irq_domain *d;
+	d = __irq_domain_alloc(of_node, 0, max_irq, max_irq, ops, host_data);
+	if (d)
+		__irq_domain_register(d);
+	return d;
 }
 static inline struct irq_domain *irq_domain_add_legacy_isa(
 				struct device_node *of_node,

@@ -162,7 +171,11 @@ static inline struct irq_domain *irq_domain_add_tree(struct device_node *of_node
 					 const struct irq_domain_ops *ops,
 					 void *host_data)
 {
-	return __irq_domain_add(of_node, 0, ~0, 0, ops, host_data);
+	struct irq_domain *d;
+	d = __irq_domain_alloc(of_node, 0, ~0, 0, ops, host_data);
+	if (d)
+		__irq_domain_register(d);
+	return d;
 }
 
 extern void irq_domain_remove(struct irq_domain *host);

index 034bbac..8d02af7 100644
@@ -23,7 +23,7 @@ static DEFINE_MUTEX(revmap_trees_mutex);
 static struct irq_domain *irq_default_domain;
 
 /**
- * __irq_domain_add() - Allocate a new irq_domain data structure
+ * __irq_domain_alloc() - Allocate a new irq_domain data structure
  * @of_node: optional device-tree node of the interrupt controller
  * @size: Size of linear map; 0 for radix mapping only
  * @direct_max: Maximum value of direct maps; Use ~0 for no limit; 0 for no

@@ -31,14 +31,15 @@ static struct irq_domain *irq_default_domain;
  * @ops: map/unmap domain callbacks
  * @host_data: Controller private data pointer
  *
- * Allocates and initialize and irq_domain structure.  Caller is expected to
- * register allocated irq_domain with irq_domain_register().  Returns pointer
- * to IRQ domain, or NULL on failure.
+ * Allocates and initializes an irq_domain structure.  Caller is
+ * expected to register the allocated irq_domain with
+ * __irq_domain_register().  Returns pointer to IRQ domain, or NULL on
+ * failure.
  */
-struct irq_domain *__irq_domain_add(struct device_node *of_node, int size,
-				    irq_hw_number_t hwirq_max, int direct_max,
-				    const struct irq_domain_ops *ops,
-				    void *host_data)
+struct irq_domain *__irq_domain_alloc(struct device_node *of_node, int size,
+				      irq_hw_number_t hwirq_max, int direct_max,
+				      const struct irq_domain_ops *ops,
+				      void *host_data)
 {
 	struct irq_domain *domain;
 

@@ -56,14 +57,24 @@ struct irq_domain *__irq_domain_add(struct device_node *of_node, int size,
 	domain->revmap_size = size;
 	domain->revmap_direct_max_irq = direct_max;
 
+	return domain;
+}
+EXPORT_SYMBOL_GPL(__irq_domain_alloc);
+
+/**
+ * __irq_domain_register() - Register a new irq_domain that has been
+ * previously allocated with __irq_domain_alloc().
+ * @domain: irq_domain to registers
+ */
+void __irq_domain_register(struct irq_domain *domain)
+{
 	mutex_lock(&irq_domain_mutex);
 	list_add(&domain->link, &irq_domain_list);
 	mutex_unlock(&irq_domain_mutex);
 
 	pr_debug("Added domain %s\n", domain->name);
-	return domain;
 }
-EXPORT_SYMBOL_GPL(__irq_domain_add);
+EXPORT_SYMBOL_GPL(__irq_domain_register);
 
 /**
  * irq_domain_remove() - Remove an irq domain.

@@ -127,7 +138,7 @@ struct irq_domain *irq_domain_add_simple(struct device_node *of_node,
 {
 	struct irq_domain *domain;
 
-	domain = __irq_domain_add(of_node, size, size, 0, ops, host_data);
+	domain = __irq_domain_alloc(of_node, size, size, 0, ops, host_data);
 	if (!domain)
 		return NULL;
 

@@ -143,6 +154,8 @@ struct irq_domain *irq_domain_add_simple(struct device_node *of_node,
 		irq_domain_associate_many(domain, first_irq, 0, size);
 	}
 
+	__irq_domain_register(domain);
+
 	return domain;
 }
 EXPORT_SYMBOL_GPL(irq_domain_add_simple);

@@ -171,12 +184,13 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 {
 	struct irq_domain *domain;
 
-	domain = __irq_domain_add(of_node, first_hwirq + size,
-				  first_hwirq + size, 0, ops, host_data);
+	domain = __irq_domain_alloc(of_node, first_hwirq + size,
+				    first_hwirq + size, 0, ops, host_data);
 	if (!domain)
 		return NULL;
 
 	irq_domain_associate_many(domain, first_irq, first_hwirq, size);
+	__irq_domain_register(domain);
 
 	return domain;
 }

1.8.1.2

Message Signaled Interrupts are a PCI-specific mechanism that allows PCI devices to notify interrupts to the CPU using in-band messages. The PCI subsystem represents an MSI-capable interrupt controller as an msi_chip structure, and this patch improves the irqdomain subsystem with a new pointer associating an irq_domain with the corresponding msi_chip.

The irq_domain structure gains a pointer to a msi_chip structure, and a new helper function irq_domain_add_msi() is added to help in allocating, initializing and registering an irq_domain for a MSI-type interrupt controller.

---
 include/linux/irqdomain.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

index 7113941..b0504ff 100644
@@ -38,6 +38,7 @@
 struct device_node;
 struct irq_domain;
 struct of_device_id;
+struct msi_chip;
 
 /* Number of irqs reserved for a legacy isa controller */
 #define NUM_ISA_INTERRUPTS	16

@@ -101,6 +102,7 @@ struct irq_domain {
 	/* Optional data */
 	struct device_node *of_node;
 	struct irq_domain_chip_generic *gc;
+	struct msi_chip *msi_chip;
 
 	/* reverse map data. The linear map gets appended to the irq_domain */
 	irq_hw_number_t hwirq_max;

@@ -177,6 +179,22 @@ static inline struct irq_domain *irq_domain_add_tree(struct device_node *of_node
 		__irq_domain_register(d);
 	return d;
 }
+static inline struct irq_domain *irq_domain_add_msi(struct device_node *of_node,
+						    unsigned int size,
+						    const struct irq_domain_ops *ops,
+						    struct msi_chip *msi_chip,
+						    void *host_data)
+{
+	struct irq_domain *d;
+	d = __irq_domain_alloc(of_node, size, size, 0, ops, host_data);
+	if (d) {
+		d->msi_chip = msi_chip;
+		__irq_domain_register(d);
+	}
+
+	return d;
+}
+
 
 extern void irq_domain_remove(struct irq_domain *host);
 

1.8.1.2

Now that an irq_domain can be associated to a msi_chip structure, a given PCIe driver will want to find this irq_domain, based on the Device Tree node of the interrupt controller, as pointed by the 'msi-controller' DT property.

However, since on those platforms a single piece of hardware, represented by a single DT node can provide both a "normal" IRQ domain and a MSI-type IRQ domain, we need separate lookup functions to distinguish them.

This patch makes irq_find_host() find only non-MSI-type IRQ domains, and introduces irq_find_msi_host() to find only MSI-type IRQ domains. It does so by factorizing the irq_find_host() logic into __irq_find_host().

---
 include/linux/irqdomain.h | 21 ++++++++++++++++++++-
 kernel/irq/irqdomain.c    | 13 ++++++++++---
 2 files changed, 30 insertions(+), 4 deletions(-)

index b0504ff..fc669b4 100644
@@ -129,7 +129,8 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 					 irq_hw_number_t first_hwirq,
 					 const struct irq_domain_ops *ops,
 					 void *host_data);
-extern struct irq_domain *irq_find_host(struct device_node *node);
+struct irq_domain *__irq_find_host(struct device_node *node,
+				   bool findmsi);
 extern void irq_set_default_host(struct irq_domain *host);
 
 /**

@@ -196,6 +197,24 @@ static inline struct irq_domain *irq_domain_add_msi(struct device_node *of_node,
 }
 
 
+/**
+ * irq_find_host() - Locates a domain for a given device node
+ * @node: device-tree node of the interrupt controller
+ */
+static inline struct irq_domain *irq_find_host(struct device_node *node)
+{
+	return __irq_find_host(node, false);
+}
+
+/**
+ * irq_find_msi_host() - Locates a MSI domain for a given device node
+ * @node: device-tree node of the interrupt controller
+ */
+static inline struct irq_domain *irq_find_msi_host(struct device_node *node)
+{
+	return __irq_find_host(node, true);
+}
+
 extern void irq_domain_remove(struct irq_domain *host);
 
 extern int irq_domain_associate(struct irq_domain *domain, unsigned int irq,

index 8d02af7..6d066e2 100644
@@ -197,10 +197,14 @@ struct irq_domain *irq_domain_add_legacy(struct device_node *of_node,
 EXPORT_SYMBOL_GPL(irq_domain_add_legacy);
 
 /**
- * irq_find_host() - Locates a domain for a given device node
+ * __irq_find_host() - Locates a domain for a given device node,
+ * taking into account whether the domain is of MSI-type or not.
  * @node: device-tree node of the interrupt controller
+ * @findmsi: true when the domain being search is of MSI-type, false
+ * otherwise.
  */
-struct irq_domain *irq_find_host(struct device_node *node)
+struct irq_domain *__irq_find_host(struct device_node *node,
+				   bool findmsi)
 {
 	struct irq_domain *h, *found = NULL;
 	int rc;

@@ -212,6 +216,9 @@ struct irq_domain *irq_find_host(struct device_node *node)
 	 */
 	mutex_lock(&irq_domain_mutex);
 	list_for_each_entry(h, &irq_domain_list, link) {
+		if ((findmsi && !h->msi_chip) ||
+		    (!findmsi && h->msi_chip))
+			continue;
 		if (h->ops->match)
 			rc = h->ops->match(h, node);
 		else

@@ -225,7 +232,7 @@ struct irq_domain *irq_find_host(struct device_node *node)
 	mutex_unlock(&irq_domain_mutex);
 	return found;
 }
-EXPORT_SYMBOL_GPL(irq_find_host);
+EXPORT_SYMBOL_GPL(__irq_find_host);
 
 /**
  * irq_set_default_host() - Set a "default" irq domain

1.8.1.2

I still don't quite understand why you have to do all that.

At least on power we have cases where an mpic does both MSIs and LSIs, we have the XICS that happily mixes both in a single large domain, etc... and never needed any of that.

I don't quite understand what problem that stuff is trying to solve really. Are you trying to avoid having an added MSI bitmap allocator for the MSI side of the PIC and use the irq domain stuff both as your virq and your hwirq allocator ?

Dear Benjamin Herrenschmidt,

Yes. Originally, the allocator was in the driver, but Grant Likely suggested that it should be done by the irqdomain code itself. If needed, I can dig the link, but I already gave you the pointers to each iteration of this patch series, and the comment from Grant is definitely part of one of those threads.

Basically, the PCIe driver in drivers/pci/host/pci-mvebu.c needs to give the Linux PCI core a pointer to a msi_chip, which contains operations to setup/teardown an MSI irq.

On Marvell platforms, it's directly the main interrupt controller that provides the MSI mechanism. So the msi_chip is created and handled by the drivers/irqchip/irq-armada-370-xp.c driver. It implements the ->setup_irq() and ->teardown_irq() operations of the msi_chip structure, using irq_domain functions, as Grant suggested that the irq_domain infrastructure should be used instead of having another bitmap allocator in each MSI driver.

Then, we need to "connect" the PCI driver and the IRQ driver. Arnd Bergmann suggested that the DT should look like this:

	mpic {
		interrupt-controller;
		msi-controller;
		...
	};

	pcie-controller {
		msi-parent = <&mpic>;
		...
	};

So, the pcie-controller needs to be able, from a DT node pointer to "mpic" to get the corresponding msi_chip. The way to do this has gone through different steps over the different iteration of the patch series:

(1) A small registry in drivers/pci that associates a device_node* with a msi_chip*. The IRQ driver registers the msi_chip* with the corresponding device_node*, and the PCI driver lookups the right msi_chip* using the device_node* pointer by the 'msi-parent' property.

Bjorn Helgaas disliked that and suggested that the registry should be in drivers/of/

(2) The same registry was moved to drivers/of/, with the same principle, except that the functions were renamed to match the conventions of drivers/of/.

Rob Herring (drivers/of maintainer) ACKed this approach. However, later, Grant Likely came in the discussion and NAKed this approach, and suggested instead that the irq_domain should be associated with the msi_chip directly.

(3) The current approach, where irq_domain contains a pointer to the msi_chip. So the PCI driver can look up an irq_domain using the device_node* pointer to by its 'msi-parent' property. However, there are *two* irq_domain that are related to the same 'mpic'
node: the irq_domain for normal interrupts, and the irq_domain for MSI interrupts. And they should not be confused together. This is what this patch does: it makes sure that irq_find_host() matches only "normal irq_domain", while irq_find_msi_host() matches only "MSI irq_domain".

Again, this has been discussed at lengths in the previous iterations, for which I already gave you all the links, as you requested in a private e-mail. It'd be great if this discussion was read seriously, because I really have the feeling we are restarting from zero on this whole MSI thing...

Thanks,

Thomas Petazzoni

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Well, two things here:

- You don't need my ack since I am not the maintainer of the irqdomain
code anymore, Grant is :-)

- I still don't like it. I find that it's looking more and more like
over engineering. I don't like having any kind of infrastructure relationship between MSI stuff and irqdomain, ie, a PCI/PCIe specific construct and a generic interrupt remapper.

Trying to use irqdomain for HW number allocation seems to be pushing it where it wasn't designed to go. Are those interrupts really different domains ? Do they have separate number spaces, separate DT encodings and overall characteristics ?

What's wrong with the bitmap allocator in the PIC driver ? It's simple, and does the job just fine. If anything, take it from powerpc and sparc and move it to generic. It's already a "generic" (ie shared) infrastructure in powerpc.

Let's ask somebody of well known taste ... Thomas ! :-) (Yes, you tglx, I know you are lurking ...). What do you reckon ?

That series makes me feel nervous, it feels like a hack. I really don't like creating that relationship between msi_chip and irqdomain. In fact, I think it makes it harder to understand what's happening in the code and following things.

It's a LOT clearer to me to have an irq domain for the PIC and an explicit bitmap allocation for MSIs, I see where things come from, I can follow the code path etc... much more easily.

I suspect we have a case of over-abstracting happening here. This is a dangerous illness and can be contagious :-)

Cheers,
Ben.

Hrm, I'm being told Grant isn't anymore... I can step in and take it all back but you might not like the result ....

Ben.

Oh but it looks like MAINTAINERS says I am :-)

Ok so from that perspective, I don't like it at all. You can try to convince me otherwise but I don't think we need to introduce a dependency to something like msi into the core remapper. It's already to complex.

Why don't you move the powerpc bitmap allocator over to a generic place ? I feel like it would be actually simpler but feel free to prove me wrong.

Ben.

Dear Benjamin Herrenschmidt,

I'm sorry, but I'm not buying this. There must be some continuity when the maintenance of one subsystem transitions from one maintainer to another. I'm perfectly ok with accepting some hick-ups, but not radical changes in design decisions.

What you're asking me to do is to go completely backwards compared to the comments and review Grant made. The irqdomain-based allocator was suggested by Grant (see my previous e-mail, or Grant reply at
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-June/175430.html)
and was even Acked-by Grant in
http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/187082.html.

Note that this patch set has been posted at the following dates:

* PATCH version 7 sent on August, 7th 2013
* PATCH version 6 sent on August, 1st 2013
* PATCH version 5 sent on July, 15th 2013
* PATCH version 4 sent on July, 1st 2013
* PATCH version 3 sent on June, 19th 2013
* PATCH version 2 sent on June, 6th 2013
* RFC version 1 sent on March, 26th 2013

So it has been around since 4 months, I've taken into account all the comments from the various maintainers who were involved, and especially the comments from Grant. You cannot ask me now, as we are approaching the next merge window for which this code is intended, to take completely opposite design choices than what the previous irqdomain maintainer was suggesting.

Could you contact Grant and align with him on those design decisions? It would also be good if you could read the past discussions on this patch set, because all what you're pointing at has already been discussed at length, as I pointed out in my previous e-mail.

Thanks,

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Well, I wrote it in the first place :-)

I'm really sorry and I feel your pain. I have not actively been monitoring any of that stuff, and you might have gotten away without CC'ing me or asking for my point of view but you did (and I thank you for that), and sadly this is my opinion.

I can and I do. However, I also leave the opportunity of bringing in a third party into the debate with a well known track record to overrule me if he thinks I'm being unnecessarily obstructive.

I can try... Grant, are you around ? (I've added you to the CC list), we might be able to catch up on IRC and discuss it ...

Cheers,
Ben.

Dear Benjamin Herrenschmidt,

This is *exactly* the opposite of what Grant Likely said in:

http://lists.infradead.org/pipermail/linux-arm-kernel/2013-July/187083.html
[PATCHv5 05/11] of: pci: add registry of MSI chips

Grant said:

"""
Actually, I'm going to disagree on this one and say NAK. I don't think it is a good idea to create a completely separate registry of msi_chips for binding to dt nodes. I think it would be better to include the msi_chip pointer directly into the irq_domain which has to be there anyway. It then becomes another feature for irq controllers if it can support doing MSI.
"""

So Grant is completely in favor of a strong relationship between MSI stuff and irqdomain.

This is also *exactly* the opoosite of what Grant Likely said in:

http://lists.infradead.org/pipermail/linux-arm-kernel/2013-June/175430.html
[PATCH v2 4/8] irqchip: armada-370-xp: implement MSI support

He was replying to the patch that did a bitmap based allocation scheme, within the IRQ controller driver, for MSI interrupts. And Grant said:

"""
This looks like something that the irq_domain should handle for you. It would be good to have a variant of irq_create_mapping() that keeps track of available hwirqs, allocates a free one, and maps it all with one function call. I can see that being useful by other MSI interrupt controllers and would eliminate needing to open-code it above. """

See above. Grant said to *NOT* implement a bitmap allocator. He even said it would be useful for other MSI interrupt controllers, and that ultimately they should be migrated to not use the bitmap allocator that you're talking about, but instead rely on irqdomain based allocations.

Please see above, you're going completely backwards to what Grant Likely was saying.

Best regards,

Thomas

Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

Grant and I tend to disagree from time to time :-)

And I disagree. If you consider msi_chip purely as a mechanism for binding dt-nodes, then he *might* have some kind of argument but I think that's the wrong approach.

msi-chip looks like the right low level abstraction for providing different hooks for different MSI HW implementations that may exist in an architecture, and in fact as such it's welcome. We currently have arch specific function pointers we could then easily replace.

However this has always been my cardinal rule with the DT stuff from the ground up on powerpc: Such a mechanism must be independent from and orthogonal to the device-tree.

For example the generic irq_chip is orthogonal to irq remapping via irq_domain. It's possible to instanciate irq_chips without device nodes, and with a completely different firmware representation (ACPI ?). It should be the same with msi-chip.

Now, having some kind of HW allocator within the irq domain is actually not completely idiotic I suppose since it somewhat keeps track of the HW interrupts, but it's pushing beyond its original design, and even if that was to be something we would allow, I would *still* keep it separate from msi-chip.

A given msi-chip implementation may use that irq-domain provided facility if it exist, indeed, but I don't want a strong tie between the two.

And that ends up into a bloody cathedral instead of nicely separated and individually useful small components. I actually disagree with Grant pretty strongly on that.

Or Grant is going completely backward from what I've always wanted that stuff to be... I've been too busy to monitor closely and frankly, I'm also too busy to actively do so now either.

I suggest we bring in the opinion of Thomas Gleixner. I trust his sense of taste and will bow if he says that I'm full of shit.

Cheers,
Ben.

In fact, to a large extent, the original irq_domain was also orthogonal to the device-tree ...

I did add the ability to match a device-node with an irq domain but that has always been just an optional addition, it was possible (and should still be though I haven't looked in a while) to create irq domains completely independently of the device-tree.

Now there is one thing that might sway me ... if you can show me (sorry don't have the bandwidth to look in details and scrutinize the patch) that overall, having the msi_chip in the domain as an optional facility does indeed overall make the code *much* smaller than keeping them separate, and for more than just your use case.

One reason I don't like the allocator being in irq domain is that it really only is useful for a subset of the different types of domains around.

For example, on power server, I have a unique domain accross the fabric (irqs are special powerbus messages that are encoded in a 24 bit number), but each "source" (a PCI host bridge for example) gets a subset of that domain, typically a fixed range.

So your allocator would only be useful to that case if:

- It can be used to allocate within specific boundaries

- It works with radix based domains

This is just an example... I don't like bolting a facility (allocation) in a lawyer originally designed to do something else (mapping) unless that facility is directly useful to the vast majority of the users of the layer in question.

In fact, there is an argument to be made to provide a generic bitmap allocator specialized for MSIs. MSIs have quirks ... alignment constraints for multiple MSI-non-X for example, which might potentially benefit in having an allocator with some smarts to limit fragmentation. That sort of things....

Cheers,
Ben.

Instead of using of_iomap(), we now use of_address_to_resource(), request_mem_region() and ioremap(). This allows the corresponding I/O regions to be properly requested and visible in /proc/iomem.

The main motivation for this change is that the introduction of the MSI support requires us to get the physical address of the main interrupt controller registers, so we will need the corresponding 'struct resource' anyway.

We also take this opportunity to change a panic() to BUG_ON(), in order to be consistent with the rest of the driver.

---
 drivers/irqchip/irq-armada-370-xp.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

index bb328a3..26adc74 100644
@@ -248,12 +248,25 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 					     struct device_node *parent)
 {
+	struct resource main_int_res, per_cpu_int_res;
 	u32 control;
 
-	main_int_base = of_iomap(node, 0);
-	per_cpu_int_base = of_iomap(node, 1);
+	BUG_ON(of_address_to_resource(node, 0, &main_int_res));
+	BUG_ON(of_address_to_resource(node, 1, &per_cpu_int_res));
 
+	BUG_ON(!request_mem_region(main_int_res.start,
+				   resource_size(&main_int_res),
+				   node->full_name));
+	BUG_ON(!request_mem_region(per_cpu_int_res.start,
+				   resource_size(&per_cpu_int_res),
+				   node->full_name));
+
+	main_int_base = ioremap(main_int_res.start,
+				resource_size(&main_int_res));
 	BUG_ON(!main_int_base);
+
+	per_cpu_int_base = ioremap(per_cpu_int_res.start,
+				   resource_size(&per_cpu_int_res));
 	BUG_ON(!per_cpu_int_base);
 
 	control = readl(main_int_base + ARMADA_370_XP_INT_CONTROL);

@@ -262,8 +275,7 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 		irq_domain_add_linear(node, (control >> 2) & 0x3ff,
 				&armada_370_xp_mpic_irq_ops, NULL);
 
-	if (!armada_370_xp_mpic_domain)
-		panic("Unable to add Armada_370_Xp MPIC irq domain (DT)\n");
+	BUG_ON(!armada_370_xp_mpic_domain);
 
 	irq_set_default_host(armada_370_xp_mpic_domain);
 

1.8.1.2

This commit introduces the support for the MSI interrupts in the armada-370-xp interrupt controller driver. It registers an MSI chip to the MSI chip registry, which will be used by the Marvell PCIe host controller driver.

The MSI interrupts use the 16 high doorbells, and are therefore notified using IRQ1 of the main interrupt controller.

---
 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   3 +
 drivers/irqchip/irq-armada-370-xp.c                | 138 ++++++++++++++++++++-
 2 files changed, 140 insertions(+), 1 deletion(-)

index 61df564..d74091a 100644
@@ -4,6 +4,8 @@ Marvell Armada 370 and Armada XP Interrupt Controller
 Required properties:
 - compatible: Should be "marvell,mpic"
 - interrupt-controller: Identifies the node as an interrupt controller.
+- msi-controller: Identifies the node as an PCI Message Signaled
+  Interrupt controller.
 - #interrupt-cells: The number of cells to define the interrupts. Should be 1.
   The cell is the IRQ number
 

@@ -24,6 +26,7 @@ Example:
               #address-cells = <1>;
               #size-cells = <1>;
               interrupt-controller;
+              msi-controller;
               reg = <0xd0020a00 0x1d0>,
                     <0xd0021070 0x58>;
         };

1.8.1.2

Some PCI drivers may need to adjust the pci_bus structure after it has been allocated by the Linux PCI core. The PCI core allows architectures to implement the pcibios_add_bus() and pcibios_remove_bus() for this purpose. This commit therefore extends the hw_pci and pci_sys_data structures of the ARM PCI core to allow PCI drivers to register ->add_bus() and ->remove_bus() in hw_pci, which will get called when a bus is added or removed from the system.

This will be used for example by the Marvell PCIe driver to connect a particular PCI bus with its corresponding MSI chip to handle Message Signaled Interrupts.

---
 arch/arm/include/asm/mach/pci.h |  4 ++++
 arch/arm/kernel/bios32.c        | 16 ++++++++++++++++
 2 files changed, 20 insertions(+)

index a1c90d7..487155c 100644
@@ -36,6 +36,8 @@ struct hw_pci {
 					  resource_size_t start,
 					  resource_size_t size,
 					  resource_size_t align);
+	void            (*add_bus)(struct pci_bus *bus);
+	void            (*remove_bus)(struct pci_bus *bus);
 };
 
 /*

@@ -63,6 +65,8 @@ struct pci_sys_data {
 					  resource_size_t start,
 					  resource_size_t size,
 					  resource_size_t align);
+	void            (*add_bus)(struct pci_bus *bus);
+	void            (*remove_bus)(struct pci_bus *bus);
 	void		*private_data;	/* platform controller private data	*/
 };
 

index 261fcc8..1ec9c87 100644
@@ -363,6 +363,20 @@ void pcibios_fixup_bus(struct pci_bus *bus)
 }
 EXPORT_SYMBOL(pcibios_fixup_bus);
 
+void pcibios_add_bus(struct pci_bus *bus)
+{
+	struct pci_sys_data *sys = bus->sysdata;
+	if (sys->add_bus)
+		sys->add_bus(bus);
+}
+
+void pcibios_remove_bus(struct pci_bus *bus)
+{
+	struct pci_sys_data *sys = bus->sysdata;
+	if (sys->remove_bus)
+		sys->remove_bus(bus);
+}
+
 /*
  * Swizzle the device pin each time we cross a bridge.  If a platform does
  * not provide a swizzle function, we perform the standard PCI swizzling.

@@ -464,6 +478,8 @@ static void pcibios_init_hw(struct device *parent, struct hw_pci *hw,
 		sys->swizzle = hw->swizzle;
 		sys->map_irq = hw->map_irq;
 		sys->align_resource = hw->align_resource;
+		sys->add_bus = hw->add_bus;
+		sys->remove_bus = hw->remove_bus;
 		INIT_LIST_HEAD(&sys->resources);
 
 		if (hw->private_data)

1.8.1.2

Adds the 'msi-controller' property to the main interrupt controller Device Tree node, to indicate that it can now behave as a MSI controller.

---
 arch/arm/boot/dts/armada-370-xp.dtsi | 1 +
 1 file changed, 1 insertion(+)

index 90b1176..ac0bbc0 100644
@@ -56,6 +56,7 @@
 				#interrupt-cells = <1>;
 				#size-cells = <1>;
 				interrupt-controller;
+				msi-controller;
 			};
 
 			coherency-fabric at 20200 {

1.8.1.2

This commit adds support for Message Signaled Interrupts in the Marvell PCIe host controller. The work is very simple: it simply gets a reference to the msi_chip associated to the PCIe controller thanks to the msi-parent DT property, and stores this reference in the pci_bus structure. This is enough to let the Linux PCI core use the functions of msi_chip to setup and teardown MSIs.

---
 .../devicetree/bindings/pci/mvebu-pci.txt          |  3 +++
 drivers/pci/host/pci-mvebu.c                       | 31 ++++++++++++++++++++++
 2 files changed, 34 insertions(+)

index f8d4058..77e0ffe 100644
@@ -12,6 +12,8 @@ Mandatory properties:
 - device_type, set to "pci"
 - ranges: ranges for the PCI memory and I/O regions, as well as the
   MMIO registers to control the PCIe interfaces.
+- msi-parent: Link to the hardware entity that serves as the Message
+  Signaled Interrupt controller for this PCI controller.
 
 In addition, the Device Tree node must have sub-nodes describing each
 PCIe interface, having the following mandatory properties:

@@ -46,6 +48,7 @@ pcie-controller {
 	#size-cells = <2>;
 
 	bus-range = <0x00 0xff>;
+	msi-parent = <&mpic>;
 
 	ranges = <0x82000000 0 0xd0040000 0xd0040000 0 0x00002000   /* Port 0.0 registers */
 		  0x82000000 0 0xd0042000 0xd0042000 0 0x00002000   /* Port 2.0 registers */

index 13a633b..884cf82 100644
@@ -11,6 +11,7 @@
 #include <linux/clk.h>
 #include <linux/module.h>
 #include <linux/mbus.h>
+#include <linux/msi.h>
 #include <linux/slab.h>
 #include <linux/platform_device.h>
 #include <linux/of_address.h>

@@ -107,6 +108,7 @@ struct mvebu_pcie_port;
 struct mvebu_pcie {
 	struct platform_device *pdev;
 	struct mvebu_pcie_port *ports;
+	struct msi_chip *msi;
 	struct resource io;
 	struct resource realio;
 	struct resource mem;

@@ -695,6 +697,12 @@ static struct pci_bus *mvebu_pcie_scan_bus(int nr, struct pci_sys_data *sys)
 	return bus;
 }
 
+void mvebu_pcie_add_bus(struct pci_bus *bus)
+{
+	struct mvebu_pcie *pcie = sys_to_pcie(bus->sysdata);
+	bus->msi = pcie->msi;
+}
+
 resource_size_t mvebu_pcie_align_resource(struct pci_dev *dev,
 					  const struct resource *res,
 					  resource_size_t start,

@@ -731,6 +739,7 @@ static void __init mvebu_pcie_enable(struct mvebu_pcie *pcie)
 	hw.map_irq        = mvebu_pcie_map_irq;
 	hw.ops            = &mvebu_pcie_ops;
 	hw.align_resource = mvebu_pcie_align_resource;
+	hw.add_bus        = mvebu_pcie_add_bus;
 
 	pci_common_init(&hw);
 }

@@ -755,6 +764,26 @@ mvebu_pcie_map_registers(struct platform_device *pdev,
 	return devm_request_and_ioremap(&pdev->dev, &regs);
 }
 
+static void __init mvebu_pcie_msi_enable(struct mvebu_pcie *pcie)
+{
+	struct device_node *msi_node;
+	struct irq_domain *msi_domain;
+
+	msi_node = of_parse_phandle(pcie->pdev->dev.of_node,
+				    "msi-parent", 0);
+	if (!msi_node)
+		return;
+
+	msi_domain = irq_find_msi_host(msi_node);
+	if (!msi_domain)
+		return;
+
+	pcie->msi = msi_domain->msi_chip;
+
+	if (pcie->msi)
+		pcie->msi->dev = &pcie->pdev->dev;
+}
+
 static int __init mvebu_pcie_probe(struct platform_device *pdev)
 {
 	struct mvebu_pcie *pcie;

@@ -879,6 +908,8 @@ static int __init mvebu_pcie_probe(struct platform_device *pdev)
 		i++;
 	}
 
+	mvebu_pcie_msi_enable(pcie);
+
 	mvebu_pcie_enable(pcie);
 
 	return 0;

1.8.1.2

This commit adjusts the Armada 370 and Armada XP PCIe controllers Device Tree informations to reference their MSI controller. In the case of this platform, the MSI controller is implemented by the MPIC.

---
 arch/arm/boot/dts/armada-370.dtsi        | 1 +
 arch/arm/boot/dts/armada-xp-mv78230.dtsi | 1 +
 arch/arm/boot/dts/armada-xp-mv78260.dtsi | 1 +
 arch/arm/boot/dts/armada-xp-mv78460.dtsi | 1 +
 4 files changed, 4 insertions(+)

index fa3dfc6..a60018d 100644
@@ -178,6 +178,7 @@
 				#address-cells = <3>;
 				#size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

index f8eaa38..4a9a305 100644
@@ -91,6 +91,7 @@
 #address-cells = <3>;
 #size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

index 2d9335d..dd23e96 100644
@@ -111,6 +111,7 @@
 				#address-cells = <3>;
 				#size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

index c7b1f4d..77db2bc 100644
@@ -127,6 +127,7 @@
 				#address-cells = <3>;
 				#size-cells = <2>;
 
+				msi-parent = <&mpic>;
 				bus-range = <0x00 0xff>;
 
 				ranges = <0x82000000 0 0x40000 0x40000 0 0x00002000   /* Port 0.0 registers */

1.8.1.2

Thomas,

Ok, take two. :) I've applied (again, *tentatively*) this series in the following fashion:

+ v3.11-rc1
\

 +---+ mvebu/msi_common (1-3,10)
     |
     +---+---+ mvebu/msi_irq (4-9)
             |
             +---+---+ mvebu/msi_mvebu (11-13)

If it makes it through -next tonight (and it should, I built the bejeezuz out of it), I'll tag mvebu/msi_common and send a PR for that so that Thierry can base off of it.

I believe our only blocker is waiting for an Ack from Benjamin.

thx,

Jason.