Skip to contentSkip to conversation

MSI support for Marvell EBU PCIe driver

RFC New Thomas Petazzoni 10 participants 49 messages Mar 26, 2013 – May 30, 2013 1 message not in the archive

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover MSI support for Marvell EBU PCIe driver

Hello,

This set of patches introduces Message Signaled Interrupt support in the Marvell EBU PCIe driver. It has been successfully tested on the Armada XP GP platform and the Armada 370 DB platform with an Intel e1000e PCIe network card that supports MSI.

This is based on work done by Lior Amsalem <[email protected]>.

The patches do the following:

* Patches 1, 2 and 3 move the IRQ controller driver of Armada 370/XP platforms from arch/arm/mach-mvebu/ into drivers/irqchip/ and use the proper irqchip infrastructure. Those changes are not strictly needed to add MSI interrupts support, but since we will be touching the IRQ controller driver anyway, it sounded like the right time to do this move.

* Patches 4 and 5 do some minor cleanups that will help for the remainer of the changes.

* Patch 6 makes the MPIC node a subnode of the interrupt controller node, since the MSI interrupt controller will be added as a second subnode. The details on why this is needed are given in the commit log. It is important to read those details to understand why such a Device Tree representation was used.

* Patch 7 modifies the IRQ controller driver to support MSI.

* Patch 8 is a patch from Thierry Reding that introduces a small infrastructure based on 'struct msi_chip' which makes the MSI support usable on multiplatform kernel.

* Patch 9 adds the MSI support to the Marvell EBU PCIe driver.

* Patch 10 and 11 respectively update the Device Tree and the defconfig of Armada 370/XP platforms to enable MSI.

It is for now sent as a RFC in order to define whether the Device Tree representation is correct or not. Again, patch 6 gives the rationale for this choice.

This patch set is meant to be applied on top of the Marvell PCIe driver. I keep it distinct from the PCIe driver patch set because the PCIe driver itself is mostly reading for merging (it has gone through multiple iterations over the last 3 months), while this patch set is brand new and will most likely need a bunch of iterations as well.

Thanks,

Thomas

Thierry Reding (1):
PCI: Introduce new MSI chip infrastructure

Thomas Petazzoni (10):
arm: mvebu: move L2 cache initialization in init_early()
irqchip: move IRQ driver for Armada 370/XP
irqchip: armada-370-xp: move IRQ handler to avoid forward declaration
irqchip: armada-370-xp: slightly cleanup irq controller driver
arm: mvebu: do not duplicate the mpic alias
irqchip: armada-370-xp: use a separate mpic node
irqchip: armada-370-xp: add MSI support to interrupt controller
driver
pci: mvebu: add MSI support
arm: mvebu: enable MSI support in DT
arm: mvebu: enable PCI MSI support in defconfig

 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   39 +++-
 .../devicetree/bindings/pci/mvebu-pci.txt          |    5 +
 arch/arm/boot/dts/armada-370-xp.dtsi               |   15 +-
 arch/arm/boot/dts/armada-370.dtsi                  |    3 +-
 arch/arm/boot/dts/armada-xp-mv78230.dtsi           |    1 +
 arch/arm/boot/dts/armada-xp-mv78260.dtsi           |    1 +
 arch/arm/boot/dts/armada-xp-mv78460.dtsi           |    1 +
 arch/arm/boot/dts/armada-xp.dtsi                   |    2 +-
 arch/arm/configs/mvebu_defconfig                   |    1 +
 arch/arm/mach-mvebu/Kconfig                        |    1 +
 arch/arm/mach-mvebu/Makefile                       |    2 +-
 arch/arm/mach-mvebu/armada-370-xp.c                |    9 +-
 drivers/irqchip/Makefile                           |    1 +
 .../irqchip}/irq-armada-370-xp.c                   |  202 ++++++++++++++------
 drivers/pci/host/pci-mvebu.c                       |  128 +++++++++++++
 drivers/pci/msi.c                                  |   35 +++-
 drivers/pci/probe.c                                |    1 +
 include/linux/msi.h                                |   10 +
 include/linux/pci.h                                |    1 +
 19 files changed, 384 insertions(+), 74 deletions(-)
 rename {arch/arm/mach-mvebu => drivers/irqchip}/irq-armada-370-xp.c (68%)
Signature · 1 line
1.7.9.5
#1 lore raw

RFC 1/11 arm: mvebu: move L2 cache initialization in init_early()

In preparation for moving the IRQ controller driver to drivers/irqchip/, we don't want the IRQ controller driver to be responsible for initializing the L2 cache. Instead, let's initialize the L2 cache at the init_early() level, like mach-exynos/common.c is doing.

---
 arch/arm/mach-mvebu/armada-370-xp.c     |    5 +++++
 arch/arm/mach-mvebu/irq-armada-370-xp.c |    4 ----
 2 files changed, 5 insertions(+), 4 deletions(-)
arch/arm/mach-mvebu/armada-370-xp.c +5 -0
index 12d3655..d98a0ca 100644
@@ -20,6 +20,7 @@
 #include <linux/clk/mvebu.h>
 #include <linux/dma-mapping.h>
 #include <linux/mbus.h>
+#include <asm/hardware/cache-l2x0.h>
 #include <asm/mach/arch.h>
 #include <asm/mach/map.h>
 #include <asm/mach/time.h>
@@ -72,6 +73,10 @@ void __init armada_370_xp_init_early(void)
 			ARMADA_370_XP_MBUS_WINS_SIZE,
 			ARMADA_370_XP_SDRAM_WINS_BASE,
 			ARMADA_370_XP_SDRAM_WINS_SIZE);
+
+#ifdef CONFIG_CACHE_L2X0
+	l2x0_of_init(0, ~0UL);
+#endif
 }
 
 static void __init armada_370_xp_dt_init(void)
arch/arm/mach-mvebu/irq-armada-370-xp.c +0 -4
index 6a9195e..f6699f3 100644
@@ -25,7 +25,6 @@
 #include <asm/mach/arch.h>
 #include <asm/exception.h>
 #include <asm/smp_plat.h>
-#include <asm/hardware/cache-l2x0.h>
 
 /* Interrupt Controller Registers Map */
 #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
@@ -292,7 +291,4 @@ static const struct of_device_id mpic_of_match[] __initconst = {
 void __init armada_370_xp_init_irq(void)
 {
 	of_irq_init(mpic_of_match);
-#ifdef CONFIG_CACHE_L2X0
-	l2x0_of_init(0, ~0UL);
-#endif
 }
Signature · 1 line
1.7.9.5
#2 lore raw

RFC 1/11 arm: mvebu: move L2 cache initialization in init_early()

arch/arm/mach-mvebu/armada-370-xp.c void __init armada_370_xp_init_early(void) · line 79

Quoted 8 lines · Thomas Petazzoni
arch/arm/mach-mvebu/armada-370-xp.c +4 -0
@@ -72,6 +73,10 @@ void __init armada_370_xp_init_early(void)
                        ARMADA_370_XP_MBUS_WINS_SIZE,
                        ARMADA_370_XP_SDRAM_WINS_BASE,
                        ARMADA_370_XP_SDRAM_WINS_SIZE);
+
+#ifdef CONFIG_CACHE_L2X0
+       l2x0_of_init(0, ~0UL);
+#endif

I guess you should remove the #ifdef as well here, it's not needed.

Arnd

#3 lore raw

RFC 1/11 arm: mvebu: move L2 cache initialization in init_early()

Dear Arnd Bergmann,

Quoted 11 lines · Arnd Bergmann

arch/arm/mach-mvebu/armada-370-xp.c void __init armada_370_xp_init_early(void) · line 79

Quoted 8 lines · Thomas Petazzoni
arch/arm/mach-mvebu/armada-370-xp.c +4 -0
@@ -72,6 +73,10 @@ void __init armada_370_xp_init_early(void)
                        ARMADA_370_XP_MBUS_WINS_SIZE,
                        ARMADA_370_XP_SDRAM_WINS_BASE,
                        ARMADA_370_XP_SDRAM_WINS_SIZE);
+
+#ifdef CONFIG_CACHE_L2X0
+       l2x0_of_init(0, ~0UL);
+#endif

I guess you should remove the #ifdef as well here, it's not needed.

Right, will fix for the next version.

Thanks,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#4 lore raw

RFC 1/11 arm: mvebu: move L2 cache initialization in init_early()

In reply to Thomas Petazzoni
Quoted 25 lines · Thomas Petazzoni

In preparation for moving the IRQ controller driver to drivers/irqchip/, we don't want the IRQ controller driver to be responsible for initializing the L2 cache. Instead, let's initialize the L2 cache at the init_early() level, like mach-exynos/common.c is doing.

---
 arch/arm/mach-mvebu/armada-370-xp.c     |    5 +++++
 arch/arm/mach-mvebu/irq-armada-370-xp.c |    4 ----
 2 files changed, 5 insertions(+), 4 deletions(-)
arch/arm/mach-mvebu/armada-370-xp.c +1 -0
index 12d3655..d98a0ca 100644
@@ -20,6 +20,7 @@
 #include <linux/clk/mvebu.h>
 #include <linux/dma-mapping.h>
 #include <linux/mbus.h>
+#include <asm/hardware/cache-l2x0.h>
 #include <asm/mach/arch.h>
 #include <asm/mach/map.h>
 #include <asm/mach/time.h>

arch/arm/mach-mvebu/armada-370-xp.c void __init armada_370_xp_init_early(void) · line 79

Quoted 7 lines
arch/arm/mach-mvebu/armada-370-xp.c +4 -0
@@ -72,6 +73,10 @@ void __init armada_370_xp_init_early(void)
 			ARMADA_370_XP_MBUS_WINS_SIZE,
 			ARMADA_370_XP_SDRAM_WINS_BASE,
 			ARMADA_370_XP_SDRAM_WINS_SIZE);
+
+#ifdef CONFIG_CACHE_L2X0
+	l2x0_of_init(0, ~0UL);
+#endif

Have you actually tested this? I don't think the ioremap in here will work during init_early.

Rob

Quoted 24 lines
arch/arm/mach-mvebu/armada-370-xp.c +0 -0
 }
 
 static void __init armada_370_xp_dt_init(void)
arch/arm/mach-mvebu/irq-armada-370-xp.c +0 -4
index 6a9195e..f6699f3 100644
@@ -25,7 +25,6 @@
 #include <asm/mach/arch.h>
 #include <asm/exception.h>
 #include <asm/smp_plat.h>
-#include <asm/hardware/cache-l2x0.h>
 
 /* Interrupt Controller Registers Map */
 #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
@@ -292,7 +291,4 @@ static const struct of_device_id mpic_of_match[] __initconst = {
 void __init armada_370_xp_init_irq(void)
 {
 	of_irq_init(mpic_of_match);
-#ifdef CONFIG_CACHE_L2X0
-	l2x0_of_init(0, ~0UL);
-#endif
 }

A message that is not in the archive (sent off-list or not yet received): 1364316746-8702-1-git-send-email-thomas.petazzoni-wi1+55ScJUtKEb57/[email protected]

#6 lore raw

RFC 2/11 irqchip: move IRQ driver for Armada 370/XP

When the Marvell Armada 370/XP support was included in the kernel, the drivers/irqchip/ directory didn't exist and the minimal infrastructure in it also didn't exist. Now that we have those things in place, we move the Armada 370/XP IRQ controller driver from arch/arm/mach-mvebu/irq-armada-370-xp.c to drivers/irqchip/irq-armada-370-xp.c.

Note in order to reduce code movement and therefore ease the review of this patch, we intentionally introduce a forward declaration of armada_370_xp_handle_irq(). It is in fact not needed because this handler can now simply be implemented before armada_370_xp_mpic_of_init(). That will be done in the next commit.

---
 arch/arm/mach-mvebu/Makefile                       |    2 +-
 arch/arm/mach-mvebu/armada-370-xp.c                |    4 ++--
 drivers/irqchip/Makefile                           |    1 +
 .../irqchip}/irq-armada-370-xp.c                   |   22 ++++++++++----------
 4 files changed, 15 insertions(+), 14 deletions(-)
 rename {arch/arm/mach-mvebu => drivers/irqchip}/irq-armada-370-xp.c (95%)
arch/arm/mach-mvebu/Makefile +1 -1
index ba769e0..2d04f0e 100644
@@ -5,6 +5,6 @@ AFLAGS_coherency_ll.o		:= -Wa,-march=armv7-a
 
 obj-y				 += system-controller.o
 obj-$(CONFIG_MACH_ARMADA_370_XP) += armada-370-xp.o
-obj-$(CONFIG_ARCH_MVEBU)	 += coherency.o coherency_ll.o pmsu.o irq-armada-370-xp.o
+obj-$(CONFIG_ARCH_MVEBU)	 += coherency.o coherency_ll.o pmsu.o
 obj-$(CONFIG_SMP)                += platsmp.o headsmp.o
 obj-$(CONFIG_HOTPLUG_CPU)        += hotplug.o
arch/arm/mach-mvebu/armada-370-xp.c +2 -2
index d98a0ca..42a4cb3 100644
@@ -20,6 +20,7 @@
 #include <linux/clk/mvebu.h>
 #include <linux/dma-mapping.h>
 #include <linux/mbus.h>
+#include <linux/irqchip.h>
 #include <asm/hardware/cache-l2x0.h>
 #include <asm/mach/arch.h>
 #include <asm/mach/map.h>
@@ -95,8 +96,7 @@ DT_MACHINE_START(ARMADA_XP_DT, "Marvell Armada 370/XP (Device Tree)")
 	.init_machine	= armada_370_xp_dt_init,
 	.map_io		= armada_370_xp_map_io,
 	.init_early	= armada_370_xp_init_early,
-	.init_irq	= armada_370_xp_init_irq,
-	.handle_irq     = armada_370_xp_handle_irq,
+	.init_irq	= irqchip_init,
 	.init_time	= armada_370_xp_timer_and_clk_init,
 	.restart	= mvebu_restart,
 	.dt_compat	= armada_370_xp_dt_compat,
drivers/irqchip/Makefile +1 -0
index 98e3b87..dae27a7 100644
@@ -2,6 +2,7 @@ obj-$(CONFIG_IRQCHIP)			+= irqchip.o
 
 obj-$(CONFIG_ARCH_BCM2835)		+= irq-bcm2835.o
 obj-$(CONFIG_ARCH_EXYNOS)		+= exynos-combiner.o
+obj-$(CONFIG_ARCH_MVEBU)		+= irq-armada-370-xp.o
 obj-$(CONFIG_METAG)			+= irq-metag-ext.o
 obj-$(CONFIG_METAG_PERFCOUNTER_IRQS)	+= irq-metag.o
 obj-$(CONFIG_ARCH_SUNXI)		+= irq-sunxi.o
drivers/irqchip/irq-armada-370-xp.c renamed from arch/arm/mach-mvebu/irq-armada-370-xp.c +11 -11
similarity index 95%
rename from arch/arm/mach-mvebu/irq-armada-370-xp.c
rename to drivers/irqchip/irq-armada-370-xp.c
index f6699f3..d20a832 100644
@@ -25,6 +25,9 @@
 #include <asm/mach/arch.h>
 #include <asm/exception.h>
 #include <asm/smp_plat.h>
+#include <asm/mach/irq.h>
+
+#include "irqchip.h"
 
 /* Interrupt Controller Registers Map */
 #define ARMADA_370_XP_INT_SET_MASK_OFFS		(0x48)
@@ -202,6 +205,9 @@ static struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
 	.xlate = irq_domain_xlate_onecell,
 };
 
+static asmlinkage void __exception_irq_entry
+armada_370_xp_handle_irq(struct pt_regs *regs);
+
 static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 					     struct device_node *parent)
 {
@@ -237,11 +243,13 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 
 #endif
 
+	set_handle_irq(armada_370_xp_handle_irq);
+
 	return 0;
 }
 
-asmlinkage void __exception_irq_entry armada_370_xp_handle_irq(struct pt_regs
-							       *regs)
+static asmlinkage void __exception_irq_entry
+armada_370_xp_handle_irq(struct pt_regs *regs)
 {
 	u32 irqstat, irqnr;
 
@@ -283,12 +291,4 @@ asmlinkage void __exception_irq_entry armada_370_xp_handle_irq(struct pt_regs
 	} while (1);
 }
 
-static const struct of_device_id mpic_of_match[] __initconst = {
-	{.compatible = "marvell,mpic", .data = armada_370_xp_mpic_of_init},
-	{},
-};
-
-void __init armada_370_xp_init_irq(void)
-{
-	of_irq_init(mpic_of_match);
-}
+IRQCHIP_DECLARE(armada_370_xp_mpic, "marvell,mpic", armada_370_xp_mpic_of_init);
Signature · 1 line
1.7.9.5
#7 lore raw

RFC 2/11 irqchip: move IRQ driver for Armada 370/XP

Quoted 7 lines · Thomas Petazzoni

When the Marvell Armada 370/XP support was included in the kernel, the drivers/irqchip/ directory didn't exist and the minimal infrastructure in it also didn't exist. Now that we have those things in place, we move the Armada 370/XP IRQ controller driver from arch/arm/mach-mvebu/irq-armada-370-xp.c to drivers/irqchip/irq-armada-370-xp.c.

Quoted 7 lines

Note in order to reduce code movement and therefore ease the review of this patch, we intentionally introduce a forward declaration of armada_370_xp_handle_irq(). It is in fact not needed because this handler can now simply be implemented before armada_370_xp_mpic_of_init(). That will be done in the next commit.

#8 lore raw

RFC 3/11 irqchip: armada-370-xp: move IRQ handler to avoid forward declaration

In reply to a message not in the archive

If we move the IRQ handler function above the initialization function, we avoid a forward declaration. This wasn't done as part of the previous commit, in order to increase the readibility of the previous commit, who was also moving the IRQ controller driver from arch/arm to drivers/irqchip.

---
 drivers/irqchip/irq-armada-370-xp.c |   83 +++++++++++++++++------------------
 1 file changed, 40 insertions(+), 43 deletions(-)
drivers/irqchip/irq-armada-370-xp.c +40 -43
index d20a832..1115bf8 100644
@@ -206,49 +206,6 @@ static struct irq_domain_ops armada_370_xp_mpic_irq_ops = {
 };
 
 static asmlinkage void __exception_irq_entry
-armada_370_xp_handle_irq(struct pt_regs *regs);
-
-static int __init armada_370_xp_mpic_of_init(struct device_node *node,
-					     struct device_node *parent)
-{
-	u32 control;
-
-	main_int_base = of_iomap(node, 0);
-	per_cpu_int_base = of_iomap(node, 1);
-
-	BUG_ON(!main_int_base);
-	BUG_ON(!per_cpu_int_base);
-
-	control = readl(main_int_base + ARMADA_370_XP_INT_CONTROL);
-
-	armada_370_xp_mpic_domain =
-		irq_domain_add_linear(node, (control >> 2) & 0x3ff,
-				&armada_370_xp_mpic_irq_ops, NULL);
-
-	if (!armada_370_xp_mpic_domain)
-		panic("Unable to add Armada_370_Xp MPIC irq domain (DT)\n");
-
-	irq_set_default_host(armada_370_xp_mpic_domain);
-
-#ifdef CONFIG_SMP
-	armada_xp_mpic_smp_cpu_init();
-
-	/*
-	 * Set the default affinity from all CPUs to the boot cpu.
-	 * This is required since the MPIC doesn't limit several CPUs
-	 * from acknowledging the same interrupt.
-	 */
-	cpumask_clear(irq_default_affinity);
-	cpumask_set_cpu(smp_processor_id(), irq_default_affinity);
-
-#endif
-
-	set_handle_irq(armada_370_xp_handle_irq);
-
-	return 0;
-}
-
-static asmlinkage void __exception_irq_entry
 armada_370_xp_handle_irq(struct pt_regs *regs)
 {
 	u32 irqstat, irqnr;
@@ -291,4 +248,44 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 	} while (1);
 }
 
+static int __init armada_370_xp_mpic_of_init(struct device_node *node,
+					     struct device_node *parent)
+{
+	u32 control;
+
+	main_int_base = of_iomap(node, 0);
+	per_cpu_int_base = of_iomap(node, 1);
+
+	BUG_ON(!main_int_base);
+	BUG_ON(!per_cpu_int_base);
+
+	control = readl(main_int_base + ARMADA_370_XP_INT_CONTROL);
+
+	armada_370_xp_mpic_domain =
+		irq_domain_add_linear(node, (control >> 2) & 0x3ff,
+				&armada_370_xp_mpic_irq_ops, NULL);
+
+	if (!armada_370_xp_mpic_domain)
+		panic("Unable to add Armada_370_Xp MPIC irq domain (DT)\n");
+
+	irq_set_default_host(armada_370_xp_mpic_domain);
+
+#ifdef CONFIG_SMP
+	armada_xp_mpic_smp_cpu_init();
+
+	/*
+	 * Set the default affinity from all CPUs to the boot cpu.
+	 * This is required since the MPIC doesn't limit several CPUs
+	 * from acknowledging the same interrupt.
+	 */
+	cpumask_clear(irq_default_affinity);
+	cpumask_set_cpu(smp_processor_id(), irq_default_affinity);
+
+#endif
+
+	set_handle_irq(armada_370_xp_handle_irq);
+
+	return 0;
+}
+
 IRQCHIP_DECLARE(armada_370_xp_mpic, "marvell,mpic", armada_370_xp_mpic_of_init);
Signature · 1 line
1.7.9.5
#9 lore raw

RFC 4/11 irqchip: armada-370-xp: slightly cleanup irq controller driver

In reply to a message not in the archive

In preparation for the introduction of MSI support in the IRQ controller driver, we clarify the implementation of IPI using additional defines for the manipulation of doorbells. Just like IPIs are implemented using doorbells, MSIs will also use doorbells, so it makes sense to do this preparatory cleanup first.

---
 drivers/irqchip/irq-armada-370-xp.c |   13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
drivers/irqchip/irq-armada-370-xp.c +8 -5
index 1115bf8..ad1e642 100644
@@ -48,7 +48,9 @@
 
 #define ARMADA_370_XP_TIMER0_PER_CPU_IRQ	(5)
 
-#define ACTIVE_DOORBELLS			(8)
+#define IPI_DOORBELL_START                      (0)
+#define IPI_DOORBELL_END                        (8)
+#define IPI_DOORBELL_MASK                       0xFF
 
 static DEFINE_RAW_SPINLOCK(irq_controller_lock);
 
@@ -192,7 +194,7 @@ void armada_xp_mpic_smp_cpu_init(void)
 	writel(0, per_cpu_int_base + ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS);
 
 	/* Enable first 8 IPIs */
-	writel((1 << ACTIVE_DOORBELLS) - 1, per_cpu_int_base +
+	writel(IPI_DOORBELL_MASK, per_cpu_int_base +
 		ARMADA_370_XP_IN_DRBEL_MSK_OFFS);
 
 	/* Unmask IPI interrupt */
@@ -231,13 +233,14 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 
 			ipimask = readl_relaxed(per_cpu_int_base +
 						ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS)
-				& 0xFF;
+				& IPI_DOORBELL_MASK;
 
-			writel(0x0, per_cpu_int_base +
+			writel(~IPI_DOORBELL_MASK, per_cpu_int_base +
 				ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS);
 
 			/* Handle all pending doorbells */
-			for (ipinr = 0; ipinr < ACTIVE_DOORBELLS; ipinr++) {
+			for (ipinr = IPI_DOORBELL_START;
+			     ipinr < IPI_DOORBELL_END; ipinr++) {
 				if (ipimask & (0x1 << ipinr))
 					handle_IPI(ipinr, regs);
 			}
Signature · 1 line
1.7.9.5
#10 lore raw

RFC 5/11 arm: mvebu: do not duplicate the mpic alias

In reply to Thomas Petazzoni

The mpic alias is already defined in the common armada-370-xp.dtsi, so there's no need to repeat it at the armada-xp.dtsi and armada-370.dtsi level. Moreover, we're going to slightly change how the interrupt controller is declared in the common armada-370-xp.dtsi file.

---
 arch/arm/boot/dts/armada-370.dtsi |    2 +-
 arch/arm/boot/dts/armada-xp.dtsi  |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
arch/arm/boot/dts/armada-370.dtsi +1 -1
index b0468fd..a23a74f 100644
@@ -33,7 +33,7 @@
 		gpio2 = &gpio2;
 	};
 
-	mpic: interrupt-controller@d0020000 {
+	interrupt-controller@d0020000 {
 	      reg = <0xd0020a00 0x1d0>,
 		    <0xd0021870 0x58>;
 	};
arch/arm/boot/dts/armada-xp.dtsi +1 -1
index ca00d83..2fed73b 100644
@@ -29,7 +29,7 @@
 		wt-override;
 	};
 
-	mpic: interrupt-controller@d0020000 {
+	interrupt-controller@d0020000 {
 	      reg = <0xd0020a00 0x2d0>,
 		    <0xd0021070 0x58>;
 	};
Signature · 1 line
1.7.9.5
#11 lore raw

RFC 6/11 irqchip: armada-370-xp: use a separate mpic node

In reply to Thomas Petazzoni

In the Marvell hardware, MSI interrupts are supported using doorbells, and those doorbells are handled through registers that are part of the registers managed by the IRQ controller driver: they are the same registers used for handling IPI interrupts. Therefore, it is clearly the responsability of the Armada 370/XP IRQ controller driver to expose the functionality of handling MSI interrupts.

However, we need the driver to expose two different IRQ domains: one for the main interrupt controller itself, and one for the MSI interrupt controller. In order to achieve this, we will create two subnodes in the interrupt-controller@d0020000 node: one subnode for the main interrupt controller, and one subnode for the MSI interrupt controller. The two irq domains can't be registered on the same DT node, otherwise when irq_find_host() gets used by of_irq_map_one() to resolve IRQs of devices, they may find the MSI interrupt controller instead of the main interrupt controller.

Note that both the parent and the child node need to have the 'interrupt-controller' empty property:

* The interrupt-controller property is needed in the main interrupt controller node (interrupt-controller@d0020000) because the of_irq_init() function skips nodes that are matching the given compatible string, but that don't have the interrupt-controller property.

* The interrupt-controller property is needed in the child interrupt controller node (main-intc@d0020000) otherwise the resolution done by of_irq_map_one() doesn't work.

This commit is a preparation to this, by adding the subnode for the main interrupt controller, and by ensuring that the IRQ domain that corresponds to this IRQ controller is associated to this IRQ node.

Note that the Device Tree binding documentation is updated in the following patch, when the MSI interrupt support is added to the armada-370-xp irqchip driver.

---
 arch/arm/boot/dts/armada-370-xp.dtsi |    9 ++++++---
 drivers/irqchip/irq-armada-370-xp.c  |    9 +++++++--
 2 files changed, 13 insertions(+), 5 deletions(-)
arch/arm/boot/dts/armada-370-xp.dtsi +6 -3
index 7704829..de054ed 100644
@@ -28,11 +28,14 @@
 		};
 	};
 
-	mpic: interrupt-controller@d0020000 {
+	interrupt-controller@d0020000 {
 	      compatible = "marvell,mpic";
-	      #interrupt-cells = <1>;
-	      #size-cells = <1>;
 	      interrupt-controller;
+
+	      mpic: main-intc@d0020000 {
+		    #interrupt-cells = <1>;
+		    interrupt-controller;
+	      };
 	};
 
 	coherency-fabric@d0020200 {
drivers/irqchip/irq-armada-370-xp.c +7 -2
index ad1e642..4b2a6d7 100644
@@ -254,6 +254,7 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 					     struct device_node *parent)
 {
+	struct device_node *mpic_node;
 	u32 control;
 
 	main_int_base = of_iomap(node, 0);
@@ -264,9 +265,13 @@ static int __init armada_370_xp_mpic_of_init(struct device_node *node,
 
 	control = readl(main_int_base + ARMADA_370_XP_INT_CONTROL);
 
+	mpic_node = of_get_child_by_name(node, "main-intc");
 	armada_370_xp_mpic_domain =
-		irq_domain_add_linear(node, (control >> 2) & 0x3ff,
-				&armada_370_xp_mpic_irq_ops, NULL);
+		irq_domain_add_linear(mpic_node, (control >> 2) & 0x3ff,
+				      &armada_370_xp_mpic_irq_ops, NULL);
+
+	if (!armada_370_xp_mpic_domain)
+		panic("Unable to add Armada_370_Xp MPIC irq domain (DT)\n");
 
 	if (!armada_370_xp_mpic_domain)
 		panic("Unable to add Armada_370_Xp MPIC irq domain (DT)\n");
Signature · 1 line
1.7.9.5
#12 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

In reply to Thomas Petazzoni

This commit introduces the support for the MSI interrupts in the armada-370-xp interrupt controller driver. It registers an IRQ domain associated with the 'msi' node in the Device Tree, which the PCI controller will refer to in a followup commit.

The MSI interrupts use the 16 high doorbells, and are therefore notified using IRQ1 of the main interrupt controller.

The Device Tree binding documentation for the armada-370-xp driver is also updated in the process.

---
 .../devicetree/bindings/arm/armada-370-xp-mpic.txt |   39 +++++++--
 arch/arm/boot/dts/armada-370-xp.dtsi               |    6 ++
 arch/arm/mach-mvebu/Kconfig                        |    1 +
 drivers/irqchip/irq-armada-370-xp.c                |   91 +++++++++++++++++++-
 4 files changed, 130 insertions(+), 7 deletions(-)
Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt +33 -6
index 61df564..2233d20 100644
@@ -4,8 +4,6 @@ Marvell Armada 370 and Armada XP Interrupt Controller
 Required properties:
 - compatible: Should be "marvell,mpic"
 - interrupt-controller: Identifies the node as an interrupt controller.
-- #interrupt-cells: The number of cells to define the interrupts. Should be 1.
-  The cell is the IRQ number
 
 - reg: Should contain PMIC registers location and length. First pair
   for the main interrupt registers, second pair for the per-CPU
@@ -14,16 +12,45 @@ Required properties:
   automatically map to the interrupt controller registers of the
   current CPU)
 
+The node should have two sub-nodes, each describing one aspect of the
+interrupt controller:
 
+ * A first node named main-intc that describes the main interrupt
+   controller itself, receiving interrupts from all devices.
+
+   This sub-node should have the following properties:
+   - interrupt-controller: identifies the code as an interrupt
+     controller.
+   - #interrupt-cells: The number of cells to define the
+     interrupts. Should be 1.  The cell is the IRQ number
+
+ * A second node named msi-intc that is used for Message Signaled
+   Interrupts of the PCIe bus.
+
+   This sub-node should have the following properties:
+   - interrupt-controller: identifies the code as an interrupt
+     controller.
+   - #interrupt-cells: The number of cells to define the
+     interrupts. Should be 1.  The cell is the IRQ number
+   - marvell,doorbell: Gives the physical address at which PCIe
+     devices should write to signal an MSI interrupt.
 
 Example:
 
-        mpic: interrupt-controller@d0020000 {
+        interrupt-controller@d0020000 {
               compatible = "marvell,mpic";
-              #interrupt-cells = <1>;
-              #address-cells = <1>;
-              #size-cells = <1>;
               interrupt-controller;
               reg = <0xd0020a00 0x1d0>,
                     <0xd0021070 0x58>;
+
+              mpic: main-intc@d0020000 {
+                    #interrupt-cells = <1>;
+                    interrupt-controller;
+              };
+
+              msi: msi-intc@d0020000 {
+                    #interrupt-cells = <1>;
+                    interrupt-controller;
+                    marvell,doorbell = <0xd0020a04>;
+              };
         };
arch/arm/boot/dts/armada-370-xp.dtsi +6 -0
index de054ed..de6569cd 100644
@@ -36,6 +36,12 @@
 		    #interrupt-cells = <1>;
 		    interrupt-controller;
 	      };
+
+	      msi: msi-intc@d0020000 {
+		    #interrupt-cells = <1>;
+		    interrupt-controller;
+		    marvell,doorbell = <0xd0020a04>;
+	      };
 	};
 
 	coherency-fabric@d0020200 {
arch/arm/mach-mvebu/Kconfig +1 -0
index d353249..3254b77 100644
@@ -16,6 +16,7 @@ config ARCH_MVEBU
 	select MVEBU_MBUS
 	select MIGHT_HAVE_PCI
 	select PCI_QUIRKS if PCI
+	select ARCH_SUPPORTS_MSI if PCI
 
 if ARCH_MVEBU
 
drivers/irqchip/irq-armada-370-xp.c +90 -1

132 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#13 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt Required properties: · line 55

Quoted 10 lines · Thomas Petazzoni
Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt +10 -0
+              mpic: main-intc at d0020000 {
+                    #interrupt-cells = <1>;
+                    interrupt-controller;
+              };
+
+              msi: msi-intc at d0020000 {
+                    #interrupt-cells = <1>;
+                    interrupt-controller;
+                    marvell,doorbell = <0xd0020a04>;
+              };

I think the @d0020000 part needs to be removed for the nodes that have no reg property.

I think I did not follow the entire discussion. What has led to having two subnodes in the end, rather than a single node like this?

       interrupt-controller at d0020000 {
              compatible = "marvell,mpic";
              #interrupt-cells = <1>;
              #msi-cells = <1>;
              #address-cells = <1>;
              #size-cells = <1>;
              interrupt-controller;
              msi-controller;
              reg = <0xd0020a00 0x1d0>,
                     <0xd0021070 0x58>;
              marvell,doorbell = <0xd0020a04>;
       };


	Arnd
#14 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Arnd Bergmann,

Quoted 2 lines · Arnd Bergmann

I think the @d0020000 part needs to be removed for the nodes that have no reg property.

Sure, will fix.

Quoted 4 lines

I think I did not follow the entire discussion. What has led to having two subnodes in the end, rather than a single node like this?

Quoted 12 lines
       interrupt-controller at d0020000 {
              compatible = "marvell,mpic";
              #interrupt-cells = <1>;
              #msi-cells = <1>;
              #address-cells = <1>;
              #size-cells = <1>;
              interrupt-controller;
              msi-controller;
              reg = <0xd0020a00 0x1d0>,
                     <0xd0021070 0x58>;
              marvell,doorbell = <0xd0020a04>;
       };

I've tried to explain that in the commit log of PATCH 6, which says:

However, we need the driver to expose two different IRQ domains: one for the main interrupt controller itself, and one for the MSI interrupt controller. In order to achieve this, we will create two subnodes in the interrupt-controller at d0020000 node: one subnode for the main interrupt controller, and one subnode for the MSI interrupt controller. The two irq domains can't be registered on the same DT node, otherwise when irq_find_host() gets used by of_irq_map_one() to resolve IRQs of devices, they may find the MSI interrupt controller instead of the main interrupt controller.

Note that both the parent and the child node need to have the 'interrupt-controller' empty property:

* The interrupt-controller property is needed in the main
interrupt controller node (interrupt-controller at d0020000) because the of_irq_init() function skips nodes that are matching the given
compatible string, but that don't have the interrupt-controller property.

* The interrupt-controller property is needed in the child
interrupt controller node (main-intc at d0020000) otherwise the resolution done by of_irq_map_one() doesn't work.

So really, the thing is that irq_domain_add_linear() registers an IRQ domain on a specific DT node, and then irq_find_host() finds back an IRQ domain from a given DT node. So if you have two IRQ domains registered on the same DT node, then you don't know which one will be used.

So if I do the two irq_domain_add_linear() (one for MPIC, one for MSI) on one single DT node, when the timer driver will request its interrupt, it turns out that the MSI IRQ domain is used and not the MPIC IRQ domain, even though the timer has <&mpic> as its interrupt parent.

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#15 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Quoted 12 lines · Thomas Petazzoni

I've tried to explain that in the commit log of PATCH 6, which says:

However, we need the driver to expose two different IRQ domains: one for the main interrupt controller itself, and one for the MSI interrupt controller. In order to achieve this, we will create two subnodes in the interrupt-controller at d0020000 node: one subnode for the main interrupt controller, and one subnode for the MSI interrupt controller. The two irq domains can't be registered on the same DT node, otherwise when irq_find_host() gets used by of_irq_map_one() to resolve IRQs of devices, they may find the MSI interrupt controller instead of the main interrupt controller.

Right, I should have read the commit log better ...

Quoted 9 lines

Note that both the parent and the child node need to have the 'interrupt-controller' empty property:

* The interrupt-controller property is needed in the main
interrupt controller node (interrupt-controller at d0020000) because the of_irq_init() function skips nodes that are matching the given
compatible string, but that don't have the interrupt-controller property.

Quoted 3 lines

* The interrupt-controller property is needed in the child
interrupt controller node (main-intc at d0020000) otherwise the resolution done by of_irq_map_one() doesn't work.

If you add compatible properties to the children, that would work I suppose.

Quoted 11 lines

So really, the thing is that irq_domain_add_linear() registers an IRQ domain on a specific DT node, and then irq_find_host() finds back an IRQ domain from a given DT node. So if you have two IRQ domains registered on the same DT node, then you don't know which one will be used.

So if I do the two irq_domain_add_linear() (one for MPIC, one for MSI) on one single DT node, when the timer driver will request its interrupt, it turns out that the MSI IRQ domain is used and not the MPIC IRQ domain, even though the timer has <&mpic> as its interrupt parent.

I still wonder if the real solution shouldn't instead be to make the irq domain code MSI aware. For instance, you don't really need a cell to describe an interrupt because the interrupt number is not a hardware property. So an MSI using device doesn't really needs an "msis" or "interrupts" property, just an "msi-parent", and we can add code to handle as a separate domain even if you have a single device node that can do both level and message interrupts.

Arnd

#16 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Arnd Bergmann,

Quoted 9 lines · Arnd Bergmann
Quoted 9 lines

Note that both the parent and the child node need to have the 'interrupt-controller' empty property:

* The interrupt-controller property is needed in the main
interrupt controller node (interrupt-controller at d0020000)
because the of_irq_init() function skips nodes that are matching the given compatible string, but that don't have the interrupt-controller property.

Quoted 6 lines
Quoted 3 lines

* The interrupt-controller property is needed in the child
interrupt controller node (main-intc at d0020000) otherwise the resolution done by of_irq_map_one() doesn't work.

If you add compatible properties to the children, that would work I suppose.

To which children? Only to the main-intc children? If so, armada_370_xp_mpic_of_init() would be called with a 'device_node *' that points to the main-intc, correct? Then it would have to go back up in the Device Tree to find the msi node? It's doable of course, but sounds strange, no?

To me, the topology of the hardware is really that a single device provides two features: the main interrupt controller and the MSI interrupt controller. But I will adapt to whatever DT binding you propose.

Quoted 12 lines
Quoted 11 lines

So really, the thing is that irq_domain_add_linear() registers an IRQ domain on a specific DT node, and then irq_find_host() finds back an IRQ domain from a given DT node. So if you have two IRQ domains registered on the same DT node, then you don't know which one will be used.

So if I do the two irq_domain_add_linear() (one for MPIC, one for MSI) on one single DT node, when the timer driver will request its interrupt, it turns out that the MSI IRQ domain is used and not the MPIC IRQ domain, even though the timer has <&mpic> as its interrupt parent.

Quoted 8 lines

I still wonder if the real solution shouldn't instead be to make the irq domain code MSI aware. For instance, you don't really need a cell to describe an interrupt because the interrupt number is not a hardware property. So an MSI using device doesn't really needs an "msis" or "interrupts" property, just an "msi-parent", and we can add code to handle as a separate domain even if you have a single device node that can do both level and message interrupts.

So the msi-parent property should point to the single mpic node? But then the IRQ domain for MSI cannot be registered on this single MPIC node. Then, what would be the first argument of:

        armada_370_xp_msi_domain =
                irq_domain_add_linear(msi_node, PCI_MSI_DOORBELL_NR,
                                      &armada_370_xp_msi_irq_ops, NULL);

and how would the PCIe driver get a pointer to this IRQ domain? (In the currently proposed code, it just does a irq_find_host(), which sounded very simple and straightforward).

To clarify: I really don't mind reworking the code, but unfortunately "make the IRQ domain code MSI aware" is too vague for me to understand the direction you're thinking of.

Thanks for your review and comments!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#17 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Quoted 17 lines · Thomas Petazzoni

Dear Arnd Bergmann,

Quoted 13 lines · Arnd Bergmann
Quoted 12 lines

Note that both the parent and the child node need to have the 'interrupt-controller' empty property:

* The interrupt-controller property is needed in the main
interrupt controller node (interrupt-controller at d0020000)
because the of_irq_init() function skips nodes that are matching the given compatible string, but that don't have the interrupt-controller property.

* The interrupt-controller property is needed in the child
interrupt controller node (main-intc at d0020000) otherwise the resolution done by of_irq_map_one() doesn't work.

Quoted 8 lines
Quoted 2 lines

If you add compatible properties to the children, that would work I suppose.

To which children? Only to the main-intc children? If so, armada_370_xp_mpic_of_init() would be called with a 'device_node *' that points to the main-intc, correct? Then it would have to go back up in the Device Tree to find the msi node? It's doable of course, but sounds strange, no?

I was thinking of registering two init functions as well.

Quoted 4 lines

To me, the topology of the hardware is really that a single device provides two features: the main interrupt controller and the MSI interrupt controller. But I will adapt to whatever DT binding you propose.

My preference would be to have no sub-nodes but rather make the code deal with an interrupt controller that has an interrupt domain for regular IRQs but can also handle MSI.

Quoted 13 lines
Quoted 8 lines

I still wonder if the real solution shouldn't instead be to make the irq domain code MSI aware. For instance, you don't really need a cell to describe an interrupt because the interrupt number is not a hardware property. So an MSI using device doesn't really needs an "msis" or "interrupts" property, just an "msi-parent", and we can add code to handle as a separate domain even if you have a single device node that can do both level and message interrupts.

So the msi-parent property should point to the single mpic node? But then the IRQ domain for MSI cannot be registered on this single MPIC node. Then, what would be the first argument of:

Quoted 7 lines
        armada_370_xp_msi_domain =
                irq_domain_add_linear(msi_node, PCI_MSI_DOORBELL_NR,
                                      &armada_370_xp_msi_irq_ops, NULL);

and how would the PCIe driver get a pointer to this IRQ domain? (In the currently proposed code, it just does a irq_find_host(), which sounded very simple and straightforward).

I guess one way would be to have a single domain that is responsible for the controller and handles both MSI and legacy interrupts. That could probably be done without changes to the interrupt domain code.

Another option would be to add an irq_domain_add_msi() function that creates a domain which is also tied to the same device node but does not interact with it when going through the legacy interrupts. You could then add a matching msi_find_host() or irq_find_msi_host() function to return the domain.

Quoted 3 lines

To clarify: I really don't mind reworking the code, but unfortunately "make the IRQ domain code MSI aware" is too vague for me to understand the direction you're thinking of.

I don't have specific code in mind yet, mainly playing around with the possibilities for now.

Arnd

#18 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Arnd Bergmann,

Quoted 7 lines · Arnd Bergmann
Quoted 5 lines

To which children? Only to the main-intc children? If so, armada_370_xp_mpic_of_init() would be called with a 'device_node *' that points to the main-intc, correct? Then it would have to go back up in the Device Tree to find the msi node? It's doable of course, but sounds strange, no?

I was thinking of registering two init functions as well.

But then which of the two init functions would do the of_iomap() (or whatever ioremap()ing function you use) ?

Remember, the very reason what we have *one* driver for both interrupt controllers is because the registers are completely mixed. So to me it's really the case of *one* device providing *two* features, like a device that would be both an Ethernet interface and a SPI controller. You have a single ->probe() function that gets called when the device is detected, and this ->probe() function registers both an Ethernet interface and a SPI controller.

To me, the case we have here is really identical: we have one single set of registers that provide multiple features.

Quoted 8 lines
Quoted 4 lines

To me, the topology of the hardware is really that a single device provides two features: the main interrupt controller and the MSI interrupt controller. But I will adapt to whatever DT binding you propose.

My preference would be to have no sub-nodes but rather make the code deal with an interrupt controller that has an interrupt domain for regular IRQs but can also handle MSI.

Hum, ok.

Quoted 25 lines
Quoted 13 lines
Quoted 8 lines

I still wonder if the real solution shouldn't instead be to make the irq domain code MSI aware. For instance, you don't really need a cell to describe an interrupt because the interrupt number is not a hardware property. So an MSI using device doesn't really needs an "msis" or "interrupts" property, just an "msi-parent", and we can add code to handle as a separate domain even if you have a single device node that can do both level and message interrupts.

So the msi-parent property should point to the single mpic node? But then the IRQ domain for MSI cannot be registered on this single MPIC node. Then, what would be the first argument of:

Quoted 7 lines
        armada_370_xp_msi_domain =
                irq_domain_add_linear(msi_node, PCI_MSI_DOORBELL_NR,
                                      &armada_370_xp_msi_irq_ops, NULL);

and how would the PCIe driver get a pointer to this IRQ domain? (In the currently proposed code, it just does a irq_find_host(), which sounded very simple and straightforward).

I guess one way would be to have a single domain that is responsible for the controller and handles both MSI and legacy interrupts. That could probably be done without changes to the interrupt domain code.

Quoted 5 lines

Another option would be to add an irq_domain_add_msi() function that creates a domain which is also tied to the same device node but does not interact with it when going through the legacy interrupts. You could then add a matching msi_find_host() or irq_find_msi_host() function to return the domain.

This option seems doable. Not sure yet how to do it exactly, but at least I understand the idea.

Quoted 6 lines
Quoted 3 lines

To clarify: I really don't mind reworking the code, but unfortunately "make the IRQ domain code MSI aware" is too vague for me to understand the direction you're thinking of.

I don't have specific code in mind yet, mainly playing around with the possibilities for now.

Sure, I understand. But I also don't have specific ideas: the current code works fine for me, and I don't find it really ugly. So if you don't point me in the direction that you think would make the code less ugly, then I'm lost.

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#19 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Quoted 8 lines · Thomas Petazzoni
Quoted 6 lines · Arnd Bergmann
Quoted 5 lines

To which children? Only to the main-intc children? If so, armada_370_xp_mpic_of_init() would be called with a 'device_node *' that points to the main-intc, correct? Then it would have to go back up in the Device Tree to find the msi node? It's doable of course, but sounds strange, no?

Quoted 4 lines
Quoted 1 line

I was thinking of registering two init functions as well.

But then which of the two init functions would do the of_iomap() (or whatever ioremap()ing function you use) ?

I agree this is where it get a little fishy, hence my preference to use a single node.

Quoted 10 lines

Remember, the very reason what we have one driver for both interrupt controllers is because the registers are completely mixed. So to me it's really the case of one device providing two features, like a device that would be both an Ethernet interface and a SPI controller. You have a single ->probe() function that gets called when the device is detected, and this ->probe() function registers both an Ethernet interface and a SPI controller.b

To me, the case we have here is really identical: we have one single set of registers that provide multiple features.

The problem here is that the irq subsystem makes certain assumptions about one device node being the controller and mapping to a single irq domain, so you'd have to cheat one way or another. Using the sub-nodes the way you do in your patch is bending the rules for the device tree binding, which I think is more problematic than bending the rules for the code. Regarding the call to of_iomap(), you could work around it by having a static variable in the driver that remembers the mmio address and gets set by whichever device node init function gets called first.

Arnd

#20 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

In reply to Thomas Petazzoni
Quoted 4 lines · Thomas Petazzoni

To me, the topology of the hardware is really that a single device provides two features: the main interrupt controller and the MSI interrupt controller. But I will adapt to whatever DT binding you propose.

No.. the HW is a single device that provides an interrupt on register write capability, so it ideally should be a single DT node..

The need to distinguish MSI vs IPI vs other usage is completely a side effect of how Linux's IRQ and PCI layers are hooked together today.

Quoted 3 lines
Quoted 3 lines

I still wonder if the real solution shouldn't instead be to make the irq domain code MSI aware. For instance, you don't really need a cell to describe an interrupt because the interrupt number is

Some kind of generic way for an irq chip driver to say 'here, I can provide some MSI interrupts' and then for the PCI layer to say 'irq layer, find me a driver that can provision a MSI with XXX properties' ?

This need to stack an irq chip under a MSI is not something I think the kernel has had to support before, so new common code is probably needed...

The interrupt chip should not need to know what the ultimate consumer of the interrupt capability will be, it just needs to tell the consumer 'write D to physical address A and irq I will trigger'.

Jason

#21 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Jason Gunthorpe,

Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines · Thomas Petazzoni

To me, the topology of the hardware is really that a single device provides two features: the main interrupt controller and the MSI interrupt controller. But I will adapt to whatever DT binding you propose.

No.. the HW is a single device that provides an interrupt on register write capability, so it ideally should be a single DT node..

No, here you're only talking about the interrupt on register write capability (used for doorbells), but the interrupt controller is also used for all other devices. Not only IPIs and MSIs are handled by this piece of code.

Quoted 2 lines

The need to distinguish MSI vs IPI vs other usage is completely a side effect of how Linux's IRQ and PCI layers are hooked together today.

Yes. And since I live in today's world, I try to adapt somewhat to it :-)

Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

I still wonder if the real solution shouldn't instead be to make the irq domain code MSI aware. For instance, you don't really need a cell to describe an interrupt because the interrupt number is

Some kind of generic way for an irq chip driver to say 'here, I can provide some MSI interrupts' and then for the PCI layer to say 'irq layer, find me a driver that can provision a MSI with XXX properties' ?

Quoted 7 lines

This need to stack an irq chip under a MSI is not something I think the kernel has had to support before, so new common code is probably needed...

The interrupt chip should not need to know what the ultimate consumer of the interrupt capability will be, it just needs to tell the consumer 'write D to physical address A and irq I will trigger'.

I honestly don't see much difference between what you're saying here and what the proposed code is doing. The IRQ driver says "hay, I provide two IRQ domains, one is named mpic, one is named msi". And then all the regular devices have interrupt-parent = <&mpic> to say "I use legacy interrupts from that controller", and the PCIe controller has msi-parent = <&msi> to say "I use MSI interrupt from that controller".

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#22 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

In reply to Thomas Petazzoni
Quoted 4 lines · Thomas Petazzoni

This commit introduces the support for the MSI interrupts in the armada-370-xp interrupt controller driver. It registers an IRQ domain associated with the 'msi' node in the Device Tree, which the PCI controller will refer to in a followup commit.

I've got some MSI stuff working on Kirkwood using the doorbells, so I can confirm this general approach works in HW.

What do you think it would take to extend this to other Marvell SOCs?

BTW, in my testing on Kirkwood I found that register ..40010 in the PEX had to be set to the internal register base to make this work. Did you find something similar?

Quoted 2 lines

The MSI interrupts use the 16 high doorbells, and are therefore notified using IRQ1 of the main interrupt controller.

I was initially a bit surprised this was the high doorbell bits, but I checked and it is right, the 16 bit MSI maps to the high two bytes in the 32 bit MemWr TLP.

FWIW, MSI-X is not restricted to 16 bits, so if you can detect from the PCI layer if it is setting up MSI or MSI-X you could allocate low bits first to MSI-X and high bits first to MSI, increasing the number of available MSI/MSI-X vectors.

Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt Required properties: · line 36

Quoted 2 lines
Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt +2 -0
+   - marvell,doorbell: Gives the physical address at which PCIe
+     devices should write to signal an MSI interrupt.

Why is this necessary? Can't the doorbell register physical address be computed by the driver? AFAIK there is no possibility for address translation on SOC inbound TLPs.

Thinking about it a bit, maybe less magic code is needed here, be explicit about the available interrupts in the DT:

pcie-controller {
msi-interrupts = <0xd0020a04 (1<<16) &msi 16

                     0xd0020a04 (1<<17) &msi 17
                     [..]
   msi-x-interrupts = <0xd0020a04 (1<<1) &msi 1
                       0xd0020a04 (1<<2) &msi 2
                       [..]

There is a better chance of that supporting other Marvell SOCs.. Not sure, just throwing it out there.

Regarding the sub node, I'm not sure it should be necessary. You don't need to differentiate the MSI vs non-MSI case in the doorbell register@the interrupt controller level. Eg this ..

drivers/irqchip/irq-armada-370-xp.c static int armada_370_xp_mpic_irq_map(struct irq_domain *h, · line 186

Quoted 8 lines
drivers/irqchip/irq-armada-370-xp.c +8 -0
+#ifdef CONFIG_PCI_MSI
+static struct irq_chip armada_370_xp_msi_irq_chip = {
+	.name = "armada_370_xp_msi_irq",
+	.irq_enable = unmask_msi_irq,
+	.irq_disable = mask_msi_irq,
+	.irq_mask = mask_msi_irq,
+	.irq_unmask = unmask_msi_irq,
+};

.. isn't necessary for doorbell. With MSI cases on other archs there is no local MSI interrupt controller, the MSI MemWr TLP directly creates an interrupt at the CPU. This requires using the *_msi_irq functions to control the interrupt at the source, since there is no control at the destination.

However, Marvell's doorbell can be controlled at the destination, so it is better to handle it that way, especially since it creates symmetry with the IPI usage.

I used:

       priv->gc = irq_alloc_generic_chip("kirkwood_doorbell_irq", 1,
                                   irq_base, priv->base,
                                   handle_fasteoi_irq);
       [..]				  
       ct->regs.mask = 4;
       ct->regs.eoi = 0;
       ct->chip.irq_eoi = irq_gc_eoi_inv;
       ct->chip.irq_mask = irq_gc_mask_clr_bit;
       ct->chip.irq_unmask = irq_gc_mask_set_bit;

Which should work correctly for the IPI and MSI cases. The handle_fasteoi_irq is important.

Quoted 16 lines

#ifdef CONFIG_SMP
void armada_mpic_send_doorbell(const struct cpumask *mask, unsigned int irq) {

drivers/irqchip/irq-armada-370-xp.c +3 -1
@@ -220,12 +280,39 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 		if (irqnr > 1022)
 			break;
 
-		if (irqnr > 0) {
+		if (irqnr > 1) {
 			irqnr =	irq_find_mapping(armada_370_xp_mpic_domain,
 					irqnr);
 			handle_IRQ(irqnr, regs);
 			continue;
 		}
+
+#ifdef CONFIG_PCI_MSI

drivers/irqchip/irq-armada-370-xp.c armada_370_xp_handle_irq(struct pt_regs *regs) · line 300

Quoted 10 lines
drivers/irqchip/irq-armada-370-xp.c +10 -0
+		/* MSI handling */
+		if (irqnr == 1) {
+			u32 msimask, msinr;
+
+			msimask = readl_relaxed(per_cpu_int_base +
+					ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS)
+				& PCI_MSI_DOORBELL_MASK;
+
+			writel(~PCI_MSI_DOORBELL_MASK, per_cpu_int_base +
+			       ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS);

Be careful here, the ordering of the EOI in relation to the handler is important. If you use my stuff from above then the MSI and IPI cases are identical and the core code handles ack'ing the doorbell via irq_eoi at the right moment, you can just uniformly iterate over all 32 bits and there is no need to care if it is MSI or IPI.

Also, I'm not super familiar with the irq stuff, but is irq_find_mapping the best way? Most of the drivers I looked at used irq_alloc_descs to get a contiguous range of irq numbers and then just used a simple offset in the handle_irq...

Jason

#23 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Jason Gunthorpe,

Quoted 9 lines · Jason Gunthorpe
Quoted 4 lines

This commit introduces the support for the MSI interrupts in the armada-370-xp interrupt controller driver. It registers an IRQ domain associated with the 'msi' node in the Device Tree, which the PCI controller will refer to in a followup commit.

I've got some MSI stuff working on Kirkwood using the doorbells, so I can confirm this general approach works in HW.

What do you think it would take to extend this to other Marvell SOCs?

A bit of time. It is on my list to use the PCIe driver on Kirkwood and other Marvell SoCs as well.

Quoted 3 lines

BTW, in my testing on Kirkwood I found that register ..40010 in the PEX had to be set to the internal register base to make this work. Did you find something similar?

The original code I used was prepared by Lior Amsalem, and it seems like it was not needed, maybe because the bootloader had already configured the PCIe interfaces. I'll have a look to see if those registers already contain the right values, but certainly it would be safer if the PCIe driver was setting their value. Not a big deal to fix, I believe.

Quoted 6 lines
Quoted 2 lines

The MSI interrupts use the 16 high doorbells, and are therefore notified using IRQ1 of the main interrupt controller.

I was initially a bit surprised this was the high doorbell bits, but I checked and it is right, the 16 bit MSI maps to the high two bytes in the 32 bit MemWr TLP.

Right. The low 8 bits are used for the IPIs, and the high 16 bits are used for MSIs. I think it has been chosen to be like this because then we have two different interrupt numbers for the IPIs and the MSIs, which make the thing easy to handle in the IRQ controller driver.

Quoted 4 lines

FWIW, MSI-X is not restricted to 16 bits, so if you can detect from the PCI layer if it is setting up MSI or MSI-X you could allocate low bits first to MSI-X and high bits first to MSI, increasing the number of available MSI/MSI-X vectors.

This could be an improvement. There are also other, non-per-CPU, doorbell interrupts that could potentially be used. Can we consider this a possible improvement, and not something that is fundamentally necessary? For now, I'm trying to get the current feature set merged, and not necessarily to extend it to cover all possible features of the hardware.

Quoted 6 lines

Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt Required properties: · line 36

Quoted 2 lines
Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt +2 -0
+   - marvell,doorbell: Gives the physical address at which PCIe
+     devices should write to signal an MSI interrupt.

Why is this necessary? Can't the doorbell register physical address be computed by the driver? AFAIK there is no possibility for address translation on SOC inbound TLPs.

It is the responsibility of the PCIe driver to prepare the 'struct msi_msg', which contains the physical address at which the PCIe device should write to trigger an MSI. But this physical address is part of the interrupt controller registers, so there is no way for the PCIe driver to magically know about it.

Please see the code where we're using this:
https://github.com/MISL-EBU-System-SW/mainline-public/blob/marvell-pcie-msi-v1/drivers/pci/host/pci-mvebu.c#L844.

If you think this physical address can be "computed" by the driver, I'm all open to suggestions.

Quoted 11 lines

Thinking about it a bit, maybe less magic code is needed here, be explicit about the available interrupts in the DT:

pcie-controller {
   msi-interrupts = <0xd0020a04 (1<<16) &msi 16
                     0xd0020a04 (1<<17) &msi 17
                     [..]
   msi-x-interrupts = <0xd0020a04 (1<<1) &msi 1
                       0xd0020a04 (1<<2) &msi 2
                       [..]
Quoted 2 lines

There is a better chance of that supporting other Marvell SOCs.. Not sure, just throwing it out there.

Isn't that very verbose, to list each and every MSI interrupt, bit per bit? I'm fine with doing that (except maybe implement both MSI and MSI-X support, I'd like to stick with the current feature set for now), but it sounds like a lot more code in the DT and a lot more code in the driver to parse this... just to get the exact same feature.

Arnd, what is your feeling about this suggestion?

Quoted 19 lines

Regarding the sub node, I'm not sure it should be necessary. You don't need to differentiate the MSI vs non-MSI case in the doorbell register at the interrupt controller level. Eg this ..

drivers/irqchip/irq-armada-370-xp.c static int armada_370_xp_mpic_irq_map(struct irq_domain *h, · line 186

Quoted 8 lines
drivers/irqchip/irq-armada-370-xp.c +8 -0
+#ifdef CONFIG_PCI_MSI
+static struct irq_chip armada_370_xp_msi_irq_chip = {
+	.name = "armada_370_xp_msi_irq",
+	.irq_enable = unmask_msi_irq,
+	.irq_disable = mask_msi_irq,
+	.irq_mask = mask_msi_irq,
+	.irq_unmask = unmask_msi_irq,
+};

.. isn't necessary for doorbell. With MSI cases on other archs there is no local MSI interrupt controller, the MSI MemWr TLP directly creates an interrupt at the CPU. This requires using the *_msi_irq functions to control the interrupt at the source, since there is no control at the destination.

Quoted 3 lines

However, Marvell's doorbell can be controlled at the destination, so it is better to handle it that way, especially since it creates symmetry with the IPI usage.

Hum, ok. But the MSI and IPI are handled quite differently: MSIs have to call handle_IRQ(), while IPIs have to call handle_IPI(), so you'd still have to distinguish between IPIs and MSIs. In the current driver, IPIs don't have an associated irq_chip structure.

Quoted 11 lines

I used:

       priv->gc = irq_alloc_generic_chip("kirkwood_doorbell_irq", 1,
                                   irq_base, priv->base,
                                   handle_fasteoi_irq);
       [..]				  
       ct->regs.mask = 4;
       ct->regs.eoi = 0;
       ct->chip.irq_eoi = irq_gc_eoi_inv;
       ct->chip.irq_mask = irq_gc_mask_clr_bit;
       ct->chip.irq_unmask = irq_gc_mask_set_bit;
Quoted 2 lines

Which should work correctly for the IPI and MSI cases. The handle_fasteoi_irq is important.

I don't understand how this can end up calling handle_IPI() when IPIs are raised. There are no IPIs on Kirkwood, so are you sure the Kirkwood logic can apply to the SMP case of Armada XP, which requires IPIs?

Quoted 26 lines
Quoted 26 lines

#ifdef CONFIG_SMP
void armada_mpic_send_doorbell(const struct cpumask *mask,
unsigned int irq) {

drivers/irqchip/irq-armada-370-xp.c +12 -1
@@ -220,12 +280,39 @@ armada_370_xp_handle_irq(struct pt_regs *regs)
 		if (irqnr > 1022)
 			break;
 
-		if (irqnr > 0) {
+		if (irqnr > 1) {
 			irqnr =
 irq_find_mapping(armada_370_xp_mpic_domain, irqnr);
 			handle_IRQ(irqnr, regs);
 			continue;
 		}
+
+#ifdef CONFIG_PCI_MSI
+		/* MSI handling */
+		if (irqnr == 1) {
+			u32 msimask, msinr;
+
+			msimask = readl_relaxed(per_cpu_int_base +
+
 ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS)
+				& PCI_MSI_DOORBELL_MASK;
+
+			writel(~PCI_MSI_DOORBELL_MASK,
Quoted 8 lines

drivers/irqchip/irq-armada-370-xp.c armada_370_xp_handle_irq(struct pt_regs *regs) · line 300

Quoted 2 lines

per_cpu_int_base +

drivers/irqchip/irq-armada-370-xp.c +1 -0
+			       ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS);

Be careful here, the ordering of the EOI in relation to the handler is important. If you use my stuff from above then the MSI and IPI cases are identical and the core code handles ack'ing the doorbell via irq_eoi at the right moment, you can just uniformly iterate over all 32 bits and there is no need to care if it is MSI or IPI.

Again, I don't see how it's possible to not care whether it's MSI or IPI. IPIs have to call handle_IPI() which is a ARM-specific call, and I don't understand how handle_fasteoi_irq() would end up calling handle_IPI().

Quoted 4 lines

Also, I'm not super familiar with the irq stuff, but is irq_find_mapping the best way? Most of the drivers I looked at used irq_alloc_descs to get a contiguous range of irq numbers and then just used a simple offset in the handle_irq...

I'll let Arnd answer this one, but I'm pretty sure that using IRQ domains is the way to go. The fact that a number of drivers don't yet use IRQ domains is maybe just because they haven't been converted yet.

Thanks,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#24 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Quoted 11 lines · Thomas Petazzoni
Quoted 4 lines

FWIW, MSI-X is not restricted to 16 bits, so if you can detect from the PCI layer if it is setting up MSI or MSI-X you could allocate low bits first to MSI-X and high bits first to MSI, increasing the number of available MSI/MSI-X vectors.

This could be an improvement. There are also other, non-per-CPU, doorbell interrupts that could potentially be used. Can we consider this a possible improvement, and not something that is fundamentally necessary? For now, I'm trying to get the current feature set merged, and not necessarily to extend it to cover all possible features of the hardware.

If we are extending the DT binding for the current feature, we should at least think about how it would look like for future extensions, to make sure it won't be fundamentally incompatible.

Quoted 7 lines
Quoted 6 lines

Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt Required properties: · line 36

Quoted 2 lines
Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt +2 -0
+   - marvell,doorbell: Gives the physical address at which PCIe
+     devices should write to signal an MSI interrupt.

Why is this necessary? Can't the doorbell register physical address be computed by the driver? AFAIK there is no possibility for address translation on SOC inbound TLPs.

Quoted 5 lines

It is the responsibility of the PCIe driver to prepare the 'struct msi_msg', which contains the physical address at which the PCIe device should write to trigger an MSI. But this physical address is part of the interrupt controller registers, so there is no way for the PCIe driver to magically know about it.

If we introduce an irq_find_msi_host() interface, we can also introduce an interface to return the doorbell register, or more. I suppose we could actually have a generic version of your mvebu_pcie_setup_msi_irq() function that looks up the domain from the device node and calls a new irq_domain_ops function, which allocates a free MSI hwirq number, creates a mapping for it, and fills out a struct msi_msg with the doorbell register and data.

Quoted 14 lines
Quoted 11 lines

Thinking about it a bit, maybe less magic code is needed here, be explicit about the available interrupts in the DT:

pcie-controller {
   msi-interrupts = <0xd0020a04 (1<<16) &msi 16
                     0xd0020a04 (1<<17) &msi 17
                     [..]
   msi-x-interrupts = <0xd0020a04 (1<<1) &msi 1
                       0xd0020a04 (1<<2) &msi 2
                       [..]
Quoted 2 lines

There is a better chance of that supporting other Marvell SOCs.. Not sure, just throwing it out there.

Quoted 7 lines

Isn't that very verbose, to list each and every MSI interrupt, bit per bit? I'm fine with doing that (except maybe implement both MSI and MSI-X support, I'd like to stick with the current feature set for now), but it sounds like a lot more code in the DT and a lot more code in the driver to parse this... just to get the exact same feature.

Arnd, what is your feeling about this suggestion?

I think we should only need an msi-parent property and let the details be handled by the irq driver.

Quoted 8 lines
Quoted 4 lines

Also, I'm not super familiar with the irq stuff, but is irq_find_mapping the best way? Most of the drivers I looked at used irq_alloc_descs to get a contiguous range of irq numbers and then just used a simple offset in the handle_irq...

I'll let Arnd answer this one, but I'm pretty sure that using IRQ domains is the way to go. The fact that a number of drivers don't yet use IRQ domains is maybe just because they haven't been converted yet.

Yes, irq_find_mapping is what we should be using here.

Arnd

#25 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Arnd Bergmann,

Quoted 13 lines · Arnd Bergmann
Quoted 11 lines · Thomas Petazzoni
Quoted 4 lines

FWIW, MSI-X is not restricted to 16 bits, so if you can detect from the PCI layer if it is setting up MSI or MSI-X you could allocate low bits first to MSI-X and high bits first to MSI, increasing the number of available MSI/MSI-X vectors.

This could be an improvement. There are also other, non-per-CPU, doorbell interrupts that could potentially be used. Can we consider this a possible improvement, and not something that is fundamentally necessary? For now, I'm trying to get the current feature set merged, and not necessarily to extend it to cover all possible features of the hardware.

Quoted 3 lines

If we are extending the DT binding for the current feature, we should at least think about how it would look like for future extensions, to make sure it won't be fundamentally incompatible.

Sure.

Quoted 13 lines
Quoted 7 lines
Quoted 6 lines

Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt Required properties: · line 36

Quoted 2 lines
Documentation/devicetree/bindings/arm/armada-370-xp-mpic.txt +2 -0
+   - marvell,doorbell: Gives the physical address at which PCIe
+     devices should write to signal an MSI interrupt.

Why is this necessary? Can't the doorbell register physical address be computed by the driver? AFAIK there is no possibility for address translation on SOC inbound TLPs.

Quoted 5 lines

It is the responsibility of the PCIe driver to prepare the 'struct msi_msg', which contains the physical address at which the PCIe device should write to trigger an MSI. But this physical address is part of the interrupt controller registers, so there is no way for the PCIe driver to magically know about it.

Quoted 7 lines

If we introduce an irq_find_msi_host() interface, we can also introduce an interface to return the doorbell register, or more. I suppose we could actually have a generic version of your mvebu_pcie_setup_msi_irq() function that looks up the domain from the device node and calls a new irq_domain_ops function, which allocates a free MSI hwirq number, creates a mapping for it, and fills out a struct msi_msg with the doorbell register and data.

Ok, sounds like a plan. I must admit I'm not very familiar with the IRQ domain code, but I guess I should take this as an opportunity to become a little bit more familiar :)

Quoted 20 lines
Quoted 20 lines
Quoted 11 lines

Thinking about it a bit, maybe less magic code is needed here, be explicit about the available interrupts in the DT:

pcie-controller {
   msi-interrupts = <0xd0020a04 (1<<16) &msi 16
                     0xd0020a04 (1<<17) &msi 17
                     [..]
   msi-x-interrupts = <0xd0020a04 (1<<1) &msi 1
                       0xd0020a04 (1<<2) &msi 2
                       [..]
Quoted 2 lines

There is a better chance of that supporting other Marvell SOCs.. Not sure, just throwing it out there.

Isn't that very verbose, to list each and every MSI interrupt, bit per bit? I'm fine with doing that (except maybe implement both MSI and MSI-X support, I'd like to stick with the current feature set for now), but it sounds like a lot more code in the DT and a lot more code in the driver to parse this... just to get the exact same feature.

Quoted 4 lines
Quoted 1 line

Arnd, what is your feeling about this suggestion?

I think we should only need an msi-parent property and let the details be handled by the irq driver.

Ok. Having the irq driver allocate the MSI interrupt number as you suggested above seems a good idea.

Quoted 10 lines
Quoted 8 lines
Quoted 4 lines

Also, I'm not super familiar with the irq stuff, but is irq_find_mapping the best way? Most of the drivers I looked at used irq_alloc_descs to get a contiguous range of irq numbers and then just used a simple offset in the handle_irq...

I'll let Arnd answer this one, but I'm pretty sure that using IRQ domains is the way to go. The fact that a number of drivers don't yet use IRQ domains is maybe just because they haven't been converted yet.

Yes, irq_find_mapping is what we should be using here.

Ok, thanks.

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#26 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

In reply to Thomas Petazzoni
Quoted 11 lines · Thomas Petazzoni
Quoted 4 lines

FWIW, MSI-X is not restricted to 16 bits, so if you can detect from the PCI layer if it is setting up MSI or MSI-X you could allocate low bits first to MSI-X and high bits first to MSI, increasing the number of available MSI/MSI-X vectors.

This could be an improvement. There are also other, non-per-CPU, doorbell interrupts that could potentially be used. Can we consider this a possible improvement, and not something that is fundamentally necessary? For now, I'm trying to get the current feature set merged, and not necessarily to extend it to cover all possible features of the hardware.

A major point of MSI is to be able to direct interrupts on a CPU by CPU basis. Looks like XP has per-cpu and all-cpu doorbell bits?

Combined with this:

Quoted 3 lines

It is the responsibility of the PCIe driver to prepare the 'struct msi_msg', which contains the physical address at which the PCIe device should write to trigger an MSI. But this physical address is part of

Makes me think the split of responsibility created by moving the MSI ops into the PCI host structure is not correct.

The PCIe host driver just seems to get in the way, it has no knowledge it is adding to the process.

irqchip knows:
- what the physical address of the doorbell is
- how to construct an address that is per-cpu or all-cpu
- which bits in the doorbell registers are allocated and which are free

pci has none of that info.

Looking at this some more, there is tonnes of stuff in linux that when a PCI MSI is allocated a special IRQ number is created for it that has special properties - eg set_affinity on that number actually goes into the MSI table and changes it.

The cleanest would be to keep the doorbell driver purely related to the doorbell and when a request for a PCI MSI comes in allocate a new irq_chip (like arch/x86/kernel/apic/io_apic.c does) that has all the special PCI stuff and chain it to the proper bit in the doorbell.

Optimizing to remove function calls from the interrupt stack could happen later.

Quoted 8 lines
Quoted 3 lines

However, Marvell's doorbell can be controlled at the destination, so it is better to handle it that way, especially since it creates symmetry with the IPI usage.

Hum, ok. But the MSI and IPI are handled quite differently: MSIs have to call handle_IRQ(), while IPIs have to call handle_IPI(), so you'd still have to distinguish between IPIs and MSIs. In the current driver, IPIs don't have an associated irq_chip structure.

Once you have a proper generic interrupt driver you can go ahead and use request_irq to grab N bits of the doorbell register and assign them to a handler that only calls handle_IPI(ipinr,get_irq_regs()).

It is not necessary to keep IPI and the irqchip driver convoluted together.

Quoted 4 lines

Again, I don't see how it's possible to not care whether it's MSI or IPI. IPIs have to call handle_IPI() which is a ARM-specific call, and I don't understand how handle_fasteoi_irq() would end up calling handle_IPI().

- The main IRQ vector is entered, it decodes the main cause register and calls the handler for the doorbell
- The doorbell handler is setup as a chained handler and it uses handle_fasteoi_irq to enter into armada_370_xp_handle_irq
- armada_370_xp_handle_irq then runs through all bits and calls their handlers
- the handler for IPI bits is associated with the IPI handler that simply calls handle_IPI(...)

handle_fasteoi_irq acks's and clears the handled bits in the doorbell register at the proper time.

Quoted 3 lines

I'll let Arnd answer this one, but I'm pretty sure that using IRQ domains is the way to go. The fact that a number of drivers don't yet use IRQ domains is maybe just because they haven't been converted yet.

Maybe, but they have irq domain code as well.. I'm curious about the answer too :)

Jason

#27 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Quoted 6 lines · Jason Gunthorpe
Quoted 3 lines

I'll let Arnd answer this one, but I'm pretty sure that using IRQ domains is the way to go. The fact that a number of drivers don't yet use IRQ domains is maybe just because they haven't been converted yet.

Maybe, but they have irq domain code as well.. I'm curious about the answer too :)

To expand on my previous answer: irq_alloc_descs can require a lot of memory if you have a lot of MSIs. If the driver only has 16 MSIs, there is probably no reason to not assign all of them at once, but if you have 2048 MSIs, creating the mapping only when you need it is much more space efficient.

Arnd

#28 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

In reply to Jason Gunthorpe

Dear Jason Gunthorpe,

Quoted 12 lines · Jason Gunthorpe
Quoted 11 lines
Quoted 4 lines

FWIW, MSI-X is not restricted to 16 bits, so if you can detect from the PCI layer if it is setting up MSI or MSI-X you could allocate low bits first to MSI-X and high bits first to MSI, increasing the number of available MSI/MSI-X vectors.

This could be an improvement. There are also other, non-per-CPU, doorbell interrupts that could potentially be used. Can we consider this a possible improvement, and not something that is fundamentally necessary? For now, I'm trying to get the current feature set merged, and not necessarily to extend it to cover all possible features of the hardware.

Quoted 2 lines

A major point of MSI is to be able to direct interrupts on a CPU by CPU basis. Looks like XP has per-cpu and all-cpu doorbell bits?

Yes. Two per-CPU interrupts, IRQ 0 for the low 16 bits of the first doorbell register, IRQ 1 for the high 16 bits of the first doorbell register. And then, three global interrupts, each with a 32 bits register associated to trigger the interrupt. At least, that's my understanding of the datasheet.

Quoted 12 lines

Combined with this:

Quoted 3 lines

It is the responsibility of the PCIe driver to prepare the 'struct msi_msg', which contains the physical address at which the PCIe device should write to trigger an MSI. But this physical address is part of

Makes me think the split of responsibility created by moving the MSI ops into the PCI host structure is not correct.

The PCIe host driver just seems to get in the way, it has no knowledge it is adding to the process.

Quoted 7 lines

irqchip knows:
- what the physical address of the doorbell is
- how to construct an address that is per-cpu or all-cpu
- which bits in the doorbell registers are allocated and which are free

pci has none of that info.

Wait, wait. Did you look at the Tegra PCIe driver? In the Tegra case, the MSI stuff is handled completely by the PCIe unit, and does not need the special interaction with the IRQ controller driver that I need.

The fact that PCIe and MSI handling are completely separate matters on Marvell may just be a specific situation. On other platforms, the physical register that gets written to by PCIe devices to trigger a MSI may well be located within the PCIe interface registers, and therefore the MSI thing should be handled by the PCIe driver.

Quoted 43 lines

Looking at this some more, there is tonnes of stuff in linux that when a PCI MSI is allocated a special IRQ number is created for it that has special properties - eg set_affinity on that number actually goes into the MSI table and changes it.

The cleanest would be to keep the doorbell driver purely related to the doorbell and when a request for a PCI MSI comes in allocate a new irq_chip (like arch/x86/kernel/apic/io_apic.c does) that has all the special PCI stuff and chain it to the proper bit in the doorbell.

Optimizing to remove function calls from the interrupt stack could happen later.

Quoted 8 lines
Quoted 3 lines

However, Marvell's doorbell can be controlled at the destination, so it is better to handle it that way, especially since it creates symmetry with the IPI usage.

Hum, ok. But the MSI and IPI are handled quite differently: MSIs have to call handle_IRQ(), while IPIs have to call handle_IPI(), so you'd still have to distinguish between IPIs and MSIs. In the current driver, IPIs don't have an associated irq_chip structure.

Once you have a proper generic interrupt driver you can go ahead and use request_irq to grab N bits of the doorbell register and assign them to a handler that only calls handle_IPI(ipinr,get_irq_regs()).

It is not necessary to keep IPI and the irqchip driver convoluted together.

Quoted 4 lines

Again, I don't see how it's possible to not care whether it's MSI or IPI. IPIs have to call handle_IPI() which is a ARM-specific call, and I don't understand how handle_fasteoi_irq() would end up calling handle_IPI().

- The main IRQ vector is entered, it decodes the main cause register and calls the handler for the doorbell
- The doorbell handler is setup as a chained handler and it uses handle_fasteoi_irq to enter into armada_370_xp_handle_irq
- armada_370_xp_handle_irq then runs through all bits and calls their handlers
- the handler for IPI bits is associated with the IPI handler that simply calls handle_IPI(...)

Quoted 2 lines

handle_fasteoi_irq acks's and clears the handled bits in the doorbell register at the proper time.

Could you propose some code that implements this? The existing code works fine, and is modeled after what the GIC IRQ driver is doing. I don't say what you're proposing is not interesting, but just that it looks like every time I come with some code, you're suggesting me to 'reinvent' the whole world, and you've never come up with some code to help in a direction or another.

Thanks,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#29 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Quoted 9 lines · Thomas Petazzoni
Quoted 9 lines

The PCIe host driver just seems to get in the way, it has no knowledge it is adding to the process.

irqchip knows:
- what the physical address of the doorbell is
- how to construct an address that is per-cpu or all-cpu
- which bits in the doorbell registers are allocated and which are free

Quoted 5 lines
Quoted 1 line

pci has none of that info.

Wait, wait. Did you look at the Tegra PCIe driver? In the Tegra case, the MSI stuff is handled completely by the PCIe unit, and does not need the special interaction with the IRQ controller driver that I need.

Yes, but only briefly. It doesn't matter if the mechanisms are in the PCI-E register block or someplace else. If irqchip is providing the interface then the PCI-E driver would have to also instantiate an irqchip block.

This is no different from what would be required to handle INTx on Marvell, the PCI-E driver would have to create a chained irqchip from the PEX interrupt and decode the cause register into INTA/B/C/D.

Quoted 17 lines
Quoted 14 lines
Quoted 4 lines

Again, I don't see how it's possible to not care whether it's MSI or IPI. IPIs have to call handle_IPI() which is a ARM-specific call, and I don't understand how handle_fasteoi_irq() would end up calling handle_IPI().

- The main IRQ vector is entered, it decodes the main cause register and calls the handler for the doorbell
- The doorbell handler is setup as a chained handler and it uses handle_fasteoi_irq to enter into armada_370_xp_handle_irq
- armada_370_xp_handle_irq then runs through all bits and calls their handlers
- the handler for IPI bits is associated with the IPI handler that simply calls handle_IPI(...)

Quoted 2 lines

handle_fasteoi_irq acks's and clears the handled bits in the doorbell register at the proper time.

Quoted 6 lines

Could you propose some code that implements this? The existing code works fine, and is modeled after what the GIC IRQ driver is doing. I don't say what you're proposing is not interesting, but just that it looks like every time I come with some code, you're suggesting me to 'reinvent' the whole world, and you've never come up with some code to help in a direction or another.

I'll send you my doorbell driver for Kirkwood - it doesn't attempt to generically solve the MSI problem, but the basic template shows how to wire up the doorbell concept to irqchip's generic code, and it works in the MSI role with lots of rather ugly hacking..

The XP driver is taking some small shortcuts because it only uses the doorbell for IPI..

Jason

#30 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

In reply to Thomas Petazzoni

drivers/irqchip/irq-armada-370-xp.c armada_370_xp_handle_irq(struct pt_regs *regs) · line 301

Quoted 7 lines · Thomas Petazzoni
drivers/irqchip/irq-armada-370-xp.c +7 -0
+                       msimask = readl_relaxed(per_cpu_int_base +
+                                       ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS)
+                               & PCI_MSI_DOORBELL_MASK;
+
+                       writel(~PCI_MSI_DOORBELL_MASK, per_cpu_int_base +
+                              ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS);
+

Upon reading this code again, I stumbled over the barriers. You use a readl_relaxed() without barrier but a writel() with barrier. Is that intentional? Are you sure that you don't need a full readl() to guarantee that all inbound DMA that was sent by the device before the MSI message has arrived by the time the interrupt handler function is called? It depends on the implementation of the MSI controller whether that guarantee is already made by the fact that you are handling the interrupt.

Arnd

#31 lore raw

RFC 7/11 irqchip: armada-370-xp: add MSI support to interrupt controller driver

Dear Arnd Bergmann,

Quoted 17 lines · Arnd Bergmann

drivers/irqchip/irq-armada-370-xp.c armada_370_xp_handle_irq(struct pt_regs *regs) · line 301

Quoted 7 lines · Thomas Petazzoni
drivers/irqchip/irq-armada-370-xp.c +7 -0
+                       msimask = readl_relaxed(per_cpu_int_base +
+                                       ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS)
+                               & PCI_MSI_DOORBELL_MASK;
+
+                       writel(~PCI_MSI_DOORBELL_MASK, per_cpu_int_base +
+                              ARMADA_370_XP_IN_DRBEL_CAUSE_OFFS);
+

Upon reading this code again, I stumbled over the barriers. You use a readl_relaxed() without barrier but a writel() with barrier. Is that intentional? Are you sure that you don't need a full readl() to guarantee that all inbound DMA that was sent by the device before the MSI message has arrived by the time the interrupt handler function is called? It depends on the implementation of the MSI controller whether that guarantee is already made by the fact that you are handling the interrupt.

This question I will have to raise to the Marvell HW engineers, the datasheet does not go into these considerations. For now, I'm rather trying to validate the 'architecture' of the code: DT binding, interaction between the IRQ controller driver and the PCIe driver, etc.

Thanks,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#32 lore raw

RFC 8/11 PCI: Introduce new MSI chip infrastructure

In reply to Thomas Petazzoni

From: Thierry Reding <[email protected]>

The new struct msi_chip is used to associated an MSI controller with a PCI bus. It is automatically handed down from the root to its children during bus enumeration.

This patch provides default (weak) implementations for the architecture- specific MSI functions (arch_setup_msi_irq(), arch_teardown_msi_irq() and arch_msi_check_device()) which check if a PCI device's bus has an attached MSI chip and forward the call appropriately.

---
 drivers/pci/msi.c   |   35 +++++++++++++++++++++++++++++++----
 drivers/pci/probe.c |    1 +
 include/linux/msi.h |   10 ++++++++++
 include/linux/pci.h |    1 +
 4 files changed, 43 insertions(+), 4 deletions(-)
drivers/pci/msi.c +31 -4
index 00cc78c..fce3549 100644
@@ -26,14 +26,41 @@
 
 static int pci_msi_enable = 1;
 
-/* Arch hooks */
+int __weak arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc)
+{
+	struct msi_chip *chip = dev->bus->msi;
+
+	if (chip && chip->setup_irq) {
+		int err;
+
+		err = chip->setup_irq(chip, dev, desc);
+		if (err < 0)
+			return err;
+
+		irq_set_chip_data(desc->irq, chip);
+		return err;
+	}
+
+	return -EINVAL;
+}
 
-#ifndef arch_msi_check_device
-int arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
+void __weak arch_teardown_msi_irq(unsigned int irq)
 {
+	struct msi_chip *chip = irq_get_chip_data(irq);
+
+	if (chip && chip->teardown_irq)
+		chip->teardown_irq(chip, irq);
+}
+
+int __weak arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
+{
+	struct msi_chip *chip = dev->bus->msi;
+
+	if (chip && chip->check_device)
+		return chip->check_device(chip, dev, nvec, type);
+
 	return 0;
 }
-#endif
 
 #ifndef arch_setup_msi_irqs
 # define arch_setup_msi_irqs default_setup_msi_irqs
drivers/pci/probe.c +1 -0
index b494066..9307550 100644
@@ -634,6 +634,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
 	child->parent = parent;
 	child->ops = parent->ops;
+	child->msi = parent->msi;
 	child->sysdata = parent->sysdata;
 	child->bus_flags = parent->bus_flags;
 
include/linux/msi.h +10 -0
index ce93a34..ea4a5be 100644
@@ -58,5 +58,15 @@ extern int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
 extern void arch_teardown_msi_irqs(struct pci_dev *dev);
 extern int arch_msi_check_device(struct pci_dev* dev, int nvec, int type);
 
+struct msi_chip {
+	struct module *owner;
+	struct device *dev;
+
+	int (*setup_irq)(struct msi_chip *chip, struct pci_dev *dev,
+			 struct msi_desc *desc);
+	void (*teardown_irq)(struct msi_chip *chip, unsigned int irq);
+	int (*check_device)(struct msi_chip *chip, struct pci_dev *dev,
+			    int nvec, int type);
+};
 
 #endif /* LINUX_MSI_H */
include/linux/pci.h +1 -0
index 2461033a..6aca43ea 100644
@@ -416,6 +416,7 @@ struct pci_bus {
 	struct resource busn_res;	/* bus numbers routed to this bus */
 
 	struct pci_ops	*ops;		/* configuration access functions */
+	struct msi_chip	*msi;		/* MSI controller */
 	void		*sysdata;	/* hook for sys-specific extension */
 	struct proc_dir_entry *procdir;	/* directory entry in /proc/bus/pci */
 
Signature · 1 line
1.7.9.5
maintainer
#33 lore raw

RFC 8/11 PCI: Introduce new MSI chip infrastructure

Quoted 48 lines · Thomas Petazzoni

From: Thierry Reding <[email protected]>

The new struct msi_chip is used to associated an MSI controller with a PCI bus. It is automatically handed down from the root to its children during bus enumeration.

This patch provides default (weak) implementations for the architecture- specific MSI functions (arch_setup_msi_irq(), arch_teardown_msi_irq() and arch_msi_check_device()) which check if a PCI device's bus has an attached MSI chip and forward the call appropriately.

---
 drivers/pci/msi.c   |   35 +++++++++++++++++++++++++++++++----
 drivers/pci/probe.c |    1 +
 include/linux/msi.h |   10 ++++++++++
 include/linux/pci.h |    1 +
 4 files changed, 43 insertions(+), 4 deletions(-)
drivers/pci/msi.c +17 -2
index 00cc78c..fce3549 100644
@@ -26,14 +26,41 @@
 
 static int pci_msi_enable = 1;
 
-/* Arch hooks */
+int __weak arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc)
+{
+       struct msi_chip *chip = dev->bus->msi;
+
+       if (chip && chip->setup_irq) {
+               int err;
+
+               err = chip->setup_irq(chip, dev, desc);
+               if (err < 0)
+                       return err;
+
+               irq_set_chip_data(desc->irq, chip);
+               return err;
+       }
+
+       return -EINVAL;
+}
 
-#ifndef arch_msi_check_device

drivers/pci/msi.c line 55

Quoted 10 lines
drivers/pci/msi.c +8 -1
-int arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
+void __weak arch_teardown_msi_irq(unsigned int irq)
 {
+       struct msi_chip *chip = irq_get_chip_data(irq);
+
+       if (chip && chip->teardown_irq)
+               chip->teardown_irq(chip, irq);
+}
+
+int __weak arch_msi_check_device(struct pci_dev *dev, int nvec, int type)

I like the replacement of "#ifndef arch_msi_check_device()" with a weak implementation here, but this only does half the job -- shouldn't we remove the powerpc "#define arch_msi_check_device arch_msi_check_device" at the same time?

And since we're touching all the check_device() implementations, maybe we could come up with a better name. "check_device()" conveys absolutely no information about what we're checking or what the sense of the result is. arch_msi_supported()? pcibios_msi_supported()? I guess it should be consistent with the other arch interfaces, so arch_*() is probably better.

Maybe the ugly #ifdef-ery around arch_setup_msi_irqs, arch_teardown_msi_irqs, and arch_restore_msi_irqs could be cleaned up similarly? Somebody worked pretty hard to obfuscate all that, probably before weak functions were available.

Quoted 31 lines
drivers/pci/msi.c +6 -1
+{
+       struct msi_chip *chip = dev->bus->msi;
+
+       if (chip && chip->check_device)
+               return chip->check_device(chip, dev, nvec, type);
+
        return 0;
 }
-#endif
 
 #ifndef arch_setup_msi_irqs
 # define arch_setup_msi_irqs default_setup_msi_irqs
drivers/pci/probe.c +1 -0
index b494066..9307550 100644
@@ -634,6 +634,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
        child->parent = parent;
        child->ops = parent->ops;
+       child->msi = parent->msi;
        child->sysdata = parent->sysdata;
        child->bus_flags = parent->bus_flags;
 
include/linux/msi.h +0 -0
index ce93a34..ea4a5be 100644
@@ -58,5 +58,15 @@ extern int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
 extern void arch_teardown_msi_irqs(struct pci_dev *dev);
 extern int arch_msi_check_device(struct pci_dev* dev, int nvec, int type);

include/linux/msi.h extern int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type); · line 69

Quoted 10 lines
include/linux/msi.h +9 -0
 
+struct msi_chip {
+       struct module *owner;
+       struct device *dev;
+
+       int (*setup_irq)(struct msi_chip *chip, struct pci_dev *dev,
+                        struct msi_desc *desc);
+       void (*teardown_irq)(struct msi_chip *chip, unsigned int irq);
+       int (*check_device)(struct msi_chip *chip, struct pci_dev *dev,
+                           int nvec, int type);

If we do end up adding interfaces like this (I'm not sure it will work -- see below), I think it would be better to pass just the pci_dev, not the "msi_chip, pci_dev" pair. Passing both pointers avoids a cheap lookup in the caller, but it adds a way that two inseparable things can get out of sync.

include/linux/pci.h struct pci_bus { · line 419

Quoted 12 lines
include/linux/msi.h +1 -0
+};
 
 #endif /* LINUX_MSI_H */
include/linux/pci.h +1 -0
index 2461033a..6aca43ea 100644
@@ -416,6 +416,7 @@ struct pci_bus {
        struct resource busn_res;       /* bus numbers routed to this bus */
 
        struct pci_ops  *ops;           /* configuration access functions */
+       struct msi_chip *msi;           /* MSI controller */

"msi" seems like a too-generic name here; it suggests an interrupt or IRQ, not a controller.

I'm not sure this is the correct place for it. Having it in the struct pci_bus means you need arch code to fill it in, e.g., you added it in mvebu_pcie_scan_bus() in patch 09/11. There's no good way to do that for arches that use pci_scan_root_bus(), which is the direction I'd like to go.

I think it probably should go in sysdata instead. That would mean you can't really do generic weak setup/tear-down functions, because they wouldn't know how to pull the MSI controller info out of the arch-specific sysdata. But there are so many levels of weak-ness going on there, maybe it would be a good thing to get rid of one :)

Bjorn

Quoted 6 lines
include/linux/pci.h +0 -1
        void            *sysdata;       /* hook for sys-specific extension */
        struct proc_dir_entry *procdir; /* directory entry in /proc/bus/pci */
 
--

1.7.9.5

#34 lore raw

RFC 8/11 PCI: Introduce new MSI chip infrastructure

Quoted 24 lines · Bjorn Helgaas

include/linux/pci.h struct pci_bus { · line 419

Quoted 12 lines · Thomas Petazzoni

From: Thierry Reding <[email protected]>

#endif /* LINUX_MSI_H */

include/linux/pci.h +1 -0
index 2461033a..6aca43ea 100644
@@ -416,6 +416,7 @@ struct pci_bus {
        struct resource busn_res;       /* bus numbers routed to this bus */
 
        struct pci_ops  *ops;           /* configuration access functions */
+       struct msi_chip *msi;           /* MSI controller */

"msi" seems like a too-generic name here; it suggests an interrupt or IRQ, not a controller.

I'm not sure this is the correct place for it. Having it in the struct pci_bus means you need arch code to fill it in, e.g., you added it in mvebu_pcie_scan_bus() in patch 09/11. There's no good way to do that for arches that use pci_scan_root_bus(), which is the direction I'd like to go.

Quoted 5 lines

I think it probably should go in sysdata instead. That would mean you can't really do generic weak setup/tear-down functions, because they wouldn't know how to pull the MSI controller info out of the arch-specific sysdata. But there are so many levels of weak-ness going on there, maybe it would be a good thing to get rid of one :)

But generic setup/tear-down functions would allow for architecture independent MSI controllers. This would be useful for MSI controllers that are unrelated to PCI host controllers or a specific architecture. It would make drivers sit more comfortably in drivers/irqchip or drivers/pci/host. Also having the msi_chip in struct pci_bus could allow there to exist multiple MSI controllers on a PCI fabric and is consistent with pci_ops.

Assuming the MSI controller is represented in the device tree and there is a relationship between the controller and the host bridge (phandle/interrupt-parent) then as Thierry suggested[1] previously you could call something like of_find_msi_chip_by_node(node) to locate an msi_chip from a device node. Could this look up exist in pci_scan_root_bus via its struct device.of_node? Perhaps pci_create_root_bus can be changed to take a parameter for msi_chip alongside the ops parameter? The of_find_msi_chip_by_node could walk up the device tree until it finds an MSI controller.

In the case where device tree isn't used - then I guess the weakly defined arch_ callbacks would be replaced with the architectures existing implementation. Or perhaps if MSI controllers are registered (msi_chip_add) then pci_scan_root_bus could use the first controller it finds.

Also I believe pci_alloc_child_bus function would need to be changed to add "b->msi = msi" to inherit msi_chip for child buses in the above patch?

Andrew Murray

[1] http://lkml.org/lkml/2013/3/25/67

#35 lore raw

RFC 8/11 PCI: Introduce new MSI chip infrastructure

On Tue, Apr 09, 2013 at 09:11:19AM +0100, Andrew Murray wrote:
[...]

Quoted 2 lines

Also I believe pci_alloc_child_bus function would need to be changed to add "b->msi = msi" to inherit msi_chip for child buses in the above patch?

The patch already does:

	child->msi = parent->msi;

in pci_alloc_child_bus(), the same way that ops is inherited. I have admittedly only tested this code on Tegra, but I don't see why that wouldn't be enough. Or maybe I haven't understood your question?

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130409/99b8cee1/attachment.sig>

#36 lore raw

RFC 8/11 PCI: Introduce new MSI chip infrastructure

Quoted 5 lines · Thierry Reding

On Tue, Apr 09, 2013 at 09:11:19AM +0100, Andrew Murray wrote:
[...]

Quoted 2 lines

Also I believe pci_alloc_child_bus function would need to be changed to add "b->msi = msi" to inherit msi_chip for child buses in the above patch?

Quoted 7 lines

The patch already does:

	child->msi = parent->msi;

in pci_alloc_child_bus(), the same way that ops is inherited. I have admittedly only tested this code on Tegra, but I don't see why that wouldn't be enough. Or maybe I haven't understood your question?

Sorry I missed that part of your patch - that was exacatly what I was hoping to see.

Andrew Murray

#37 lore raw

RFC 8/11 PCI: Introduce new MSI chip infrastructure

In reply to Bjorn Helgaas
Quoted 49 lines · Bjorn Helgaas
Quoted 47 lines · Thomas Petazzoni

From: Thierry Reding <[email protected]>

The new struct msi_chip is used to associated an MSI controller with a PCI bus. It is automatically handed down from the root to its children during bus enumeration.

This patch provides default (weak) implementations for the architecture- specific MSI functions (arch_setup_msi_irq(), arch_teardown_msi_irq() and arch_msi_check_device()) which check if a PCI device's bus has an attached MSI chip and forward the call appropriately.

---
 drivers/pci/msi.c   |   35 +++++++++++++++++++++++++++++++----
 drivers/pci/probe.c |    1 +
 include/linux/msi.h |   10 ++++++++++
 include/linux/pci.h |    1 +
 4 files changed, 43 insertions(+), 4 deletions(-)
drivers/pci/msi.c +17 -1
index 00cc78c..fce3549 100644
@@ -26,14 +26,41 @@
 
 static int pci_msi_enable = 1;
 
-/* Arch hooks */
+int __weak arch_setup_msi_irq(struct pci_dev *dev, struct msi_desc *desc)
+{
+       struct msi_chip *chip = dev->bus->msi;
+
+       if (chip && chip->setup_irq) {
+               int err;
+
+               err = chip->setup_irq(chip, dev, desc);
+               if (err < 0)
+                       return err;
+
+               irq_set_chip_data(desc->irq, chip);
+               return err;
+       }
+
+       return -EINVAL;
+}
 
Quoted 16 lines

drivers/pci/msi.c line 55

Quoted 11 lines
drivers/pci/msi.c +8 -2
-#ifndef arch_msi_check_device
-int arch_msi_check_device(struct pci_dev *dev, int nvec, int type)
+void __weak arch_teardown_msi_irq(unsigned int irq)
 {
+       struct msi_chip *chip = irq_get_chip_data(irq);
+
+       if (chip && chip->teardown_irq)
+               chip->teardown_irq(chip, irq);
+}
+
+int __weak arch_msi_check_device(struct pci_dev *dev, int nvec, int type)

I like the replacement of "#ifndef arch_msi_check_device()" with a weak implementation here, but this only does half the job -- shouldn't we remove the powerpc "#define arch_msi_check_device arch_msi_check_device" at the same time?

Yes, there are a few things that can be done to cleanup the "mess" around this subsequently. For instance I have a local patch which completely removes the ARCH_SUPPORTS_MSI symbol because it isn't useful after making the symbols weak. One other obvious item is to convert more architectures to implement an MSI chip.

Quoted 6 lines

And since we're touching all the check_device() implementations, maybe we could come up with a better name. "check_device()" conveys absolutely no information about what we're checking or what the sense of the result is. arch_msi_supported()? pcibios_msi_supported()? I guess it should be consistent with the other arch interfaces, so arch_*() is probably better.

At least on PowerPC the arch_msi_check_device() hook also refuses to setup multiple MSI per device because it isn't supported. I can't think of a better alternative than arch_msi_supported(), though so that's fine with me.

Perhaps pcibios_msi_supported() wouldn't be so bad either. Other architecture-specific functions already use that prefix (see the OF support functions) and it might be good to settle on one convention for consistency.

That said, the goal is to eventually get rid of all the arch_msi_*() functions. The only reason they are still there is because I didn't want to convert everything in one go. So I wanted to put the infrastructure in place that we need to support multi-platform on ARM (which is given by this generic MSI chip infrastructure). Later the remaining PCI architectures can be converted to provide an msi_chip as well, at which point the now weak implementations can become non-weak and be renamed to something like pci_{setup,teardown,check}_msi() to make it more obvious that they are generic.

Quoted 4 lines

Maybe the ugly #ifdef-ery around arch_setup_msi_irqs, arch_teardown_msi_irqs, and arch_restore_msi_irqs could be cleaned up similarly? Somebody worked pretty hard to obfuscate all that, probably before weak functions were available.

I think Andrew or Jason at one point commented whether they should be allowed to be implemented by an MSI chip. If so we could use the same approach as for the other functions.

Quoted 20 lines
Quoted 20 lines
drivers/pci/probe.c +1 -0
index b494066..9307550 100644
@@ -634,6 +634,7 @@ static struct pci_bus *pci_alloc_child_bus(struct pci_bus *parent,
 
        child->parent = parent;
        child->ops = parent->ops;
+       child->msi = parent->msi;
        child->sysdata = parent->sysdata;
        child->bus_flags = parent->bus_flags;
 
include/linux/msi.h +0 -0
index ce93a34..ea4a5be 100644
@@ -58,5 +58,15 @@ extern int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type);
 extern void arch_teardown_msi_irqs(struct pci_dev *dev);
 extern int arch_msi_check_device(struct pci_dev* dev, int nvec, int type);
 
Quoted 15 lines

include/linux/msi.h extern int arch_setup_msi_irqs(struct pci_dev *dev, int nvec, int type); · line 69

Quoted 9 lines
include/linux/msi.h +9 -0
+struct msi_chip {
+       struct module *owner;
+       struct device *dev;
+
+       int (*setup_irq)(struct msi_chip *chip, struct pci_dev *dev,
+                        struct msi_desc *desc);
+       void (*teardown_irq)(struct msi_chip *chip, unsigned int irq);
+       int (*check_device)(struct msi_chip *chip, struct pci_dev *dev,
+                           int nvec, int type);

If we do end up adding interfaces like this (I'm not sure it will work -- see below), I think it would be better to pass just the pci_dev, not the "msi_chip, pci_dev" pair. Passing both pointers avoids a cheap lookup in the caller, but it adds a way that two inseparable things can get out of sync.

Yes, that's a good idea and we can easily do that.

Quoted 10 lines
Quoted 10 lines
include/linux/msi.h +1 -0
+};
 
 #endif /* LINUX_MSI_H */
include/linux/pci.h +0 -0
index 2461033a..6aca43ea 100644
@@ -416,6 +416,7 @@ struct pci_bus {
        struct resource busn_res;       /* bus numbers routed to this bus */
 
Quoted 5 lines

include/linux/pci.h struct pci_bus { · line 419

Quoted 2 lines
include/linux/pci.h +1 -0
        struct pci_ops  *ops;           /* configuration access functions */
+       struct msi_chip *msi;           /* MSI controller */

"msi" seems like a too-generic name here; it suggests an interrupt or IRQ, not a controller.

It's short and to the point. The bus abstraction doesn't really have any interrupt functionality, has it? Even for PCI devices the MSI interrupt number is actually stored in the .irq field, so I don't think it's too generic. But if you insist I'll think of another name.

Quoted 5 lines

I'm not sure this is the correct place for it. Having it in the struct pci_bus means you need arch code to fill it in, e.g., you added it in mvebu_pcie_scan_bus() in patch 09/11. There's no good way to do that for arches that use pci_scan_root_bus(), which is the direction I'd like to go.

We could add an argument to pci_scan_root_bus() to pass this information in. It's a bit ugly but see below...

Quoted 5 lines

I think it probably should go in sysdata instead. That would mean you can't really do generic weak setup/tear-down functions, because they wouldn't know how to pull the MSI controller info out of the arch-specific sysdata. But there are so many levels of weak-ness going on there, maybe it would be a good thing to get rid of one :)

Isn't putting more data into sysdata a step in the opposite direction of where we want to go? I seem to remember some discussion about wanting to consolidate the various architecture-specific implementations and move more common code into the PCI core. If we handle the whole MSI business in architecture-specific code we gain little to nothing compared to the current situation.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130409/e25780af/attachment-0001.sig>

#38 lore raw

RFC 9/11 pci: mvebu: add MSI support

In reply to Thomas Petazzoni

This commit adds the MSI support for the Marvell EBU PCIe driver. The driver now looks at the 'msi-parent' property of the PCIe controller DT node, and if it exists, it gets the associated IRQ domain, which should be the MSI interrupt controller registered by the IRQ controller driver.

Using this, the PCIe driver registers the ->setup_irq() and ->teardown_irq() callbacks using the newly introduced msi_chip infrastructure, which allows the kernel PCI core to use the MSI functionality.

---
 .../devicetree/bindings/pci/mvebu-pci.txt          |    5 +
 drivers/pci/host/pci-mvebu.c                       |  128 ++++++++++++++++++++
 2 files changed, 133 insertions(+)
Documentation/devicetree/bindings/pci/mvebu-pci.txt +5 -0
index 192bdfb..53cc437 100644
@@ -14,6 +14,10 @@ Mandatory properties:
   corresponding registers
 - ranges: ranges for the PCI memory and I/O regions
 
+Optional properties:
+- msi-parent: a phandle pointing to the interrupt controller that
+  handles the MSI interrupts.
+
 In addition, the Device Tree node must have sub-nodes describing each
 PCIe interface, having the following mandatory properties:
 - reg: used only for interrupt mapping, so only the first four bytes
@@ -43,6 +47,7 @@ pcie-controller {
 	#address-cells = <3>;
 	#size-cells = <2>;
 
+	msi-parent = <&msi>;
 	bus-range = <0x00 0xff>;
 
 	reg = <0xd0040000 0x2000>, <0xd0042000 0x2000>,
drivers/pci/host/pci-mvebu.c +128 -0

179 lines not on this page · raw message

Signature · 1 line
1.7.9.5
#39 lore raw

RFC 9/11 pci: mvebu: add MSI support

Quoted 69 lines · Thomas Petazzoni

This commit adds the MSI support for the Marvell EBU PCIe driver. The driver now looks at the 'msi-parent' property of the PCIe controller DT node, and if it exists, it gets the associated IRQ domain, which should be the MSI interrupt controller registered by the IRQ controller driver.

Using this, the PCIe driver registers the ->setup_irq() and ->teardown_irq() callbacks using the newly introduced msi_chip infrastructure, which allows the kernel PCI core to use the MSI functionality.

---
 .../devicetree/bindings/pci/mvebu-pci.txt          |    5 +
 drivers/pci/host/pci-mvebu.c                       |  128 ++++++++++++++++++++
 2 files changed, 133 insertions(+)
Documentation/devicetree/bindings/pci/mvebu-pci.txt +5 -0
index 192bdfb..53cc437 100644
@@ -14,6 +14,10 @@ Mandatory properties:
   corresponding registers
 - ranges: ranges for the PCI memory and I/O regions
 
+Optional properties:
+- msi-parent: a phandle pointing to the interrupt controller that
+  handles the MSI interrupts.
+
 In addition, the Device Tree node must have sub-nodes describing each
 PCIe interface, having the following mandatory properties:
 - reg: used only for interrupt mapping, so only the first four bytes
@@ -43,6 +47,7 @@ pcie-controller {
 	#address-cells = <3>;
 	#size-cells = <2>;
 
+	msi-parent = <&msi>;
 	bus-range = <0x00 0xff>;
 
 	reg = <0xd0040000 0x2000>, <0xd0042000 0x2000>,
drivers/pci/host/pci-mvebu.c +6 -0
index 9e6b137..b46fab8 100644
@@ -7,17 +7,23 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
 #include <linux/pci.h>
+#include <linux/msi.h>
 #include <linux/clk.h>
 #include <linux/module.h>
 #include <linux/mbus.h>
 #include <linux/slab.h>
 #include <linux/platform_device.h>
+#include <linux/interrupt.h>
 #include <linux/of_address.h>
 #include <linux/of_pci.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 
+#define INT_PCI_MSI_NR (16)
+
 /*
  * PCIe unit register offsets.
  */

drivers/pci/host/pci-mvebu.c struct mvebu_sw_pci_bridge { · line 109

Quoted 3 lines
drivers/pci/host/pci-mvebu.c +0 -0
@@ -101,10 +107,19 @@ struct mvebu_sw_pci_bridge {
 
 struct mvebu_pcie_port;
 

This structure...

drivers/pci/host/pci-mvebu.c struct mvebu_sw_pci_bridge { · line 116

Quoted 7 lines
drivers/pci/host/pci-mvebu.c +7 -0
+struct mvebu_pcie_msi {
+	DECLARE_BITMAP(used, INT_PCI_MSI_NR);
+	struct irq_domain *domain;
+	struct mutex lock;
+	struct msi_chip chip;
+	phys_addr_t doorbell;
+};

And these functions...

Quoted 59 lines
drivers/pci/host/pci-mvebu.c +59 -0
+#if defined(CONFIG_PCI_MSI)
+static int mvebu_pcie_msi_alloc(struct mvebu_pcie_msi *chip)
+{
+	int msi;
+
+	mutex_lock(&chip->lock);
+
+	msi = find_first_zero_bit(chip->used, INT_PCI_MSI_NR);
+
+	if (msi < INT_PCI_MSI_NR)
+		set_bit(msi, chip->used);
+	else
+		msi = -ENOSPC;
+
+	mutex_unlock(&chip->lock);
+
+	return msi;
+}
+
+static void mvebu_pcie_msi_free(struct mvebu_pcie_msi *chip, unsigned long irq)
+{
+	struct device *dev = chip->chip.dev;
+
+	mutex_lock(&chip->lock);
+
+	if (!test_bit(irq, chip->used))
+		dev_err(dev, "trying to free unused MSI#%lu\n", irq);
+	else
+		clear_bit(irq, chip->used);
+
+	mutex_unlock(&chip->lock);
+}
+
+static int mvebu_pcie_setup_msi_irq(struct msi_chip *chip,
+				    struct pci_dev *pdev,
+				    struct msi_desc *desc)
+{
+	struct mvebu_pcie_msi *msi = to_mvebu_msi(chip);
+	struct msi_msg msg;
+	unsigned int irq;
+	int hwirq;
+
+	hwirq = mvebu_pcie_msi_alloc(msi);
+	if (hwirq < 0)
+		return hwirq;
+
+	irq = irq_create_mapping(msi->domain, hwirq);
+	if (!irq)
+		return -EINVAL;
+
+	irq_set_msi_desc(irq, desc);
+
+	msg.address_lo = msi->doorbell;
+	msg.address_hi = 0;
+	msg.data = 0xf00 | (hwirq + 16);
+
+	write_msi_msg(irq, &msg);
+
+	return 0;

drivers/pci/host/pci-mvebu.c static void __iomem *mvebu_pcie_map_registers(struct platform_device *pdev, · line 879

Quoted 10 lines
drivers/pci/host/pci-mvebu.c +10 -0
+}
+
+static void mvebu_pcie_teardown_msi_irq(struct msi_chip *chip,
+					unsigned int irq)
+{
+	struct mvebu_pcie_msi *msi = to_mvebu_msi(chip);
+	struct irq_data *d = irq_get_irq_data(irq);
+
+	mvebu_pcie_msi_free(msi, d->hwirq);
+}

...are not related to your PCIe driver or PCI. I remember you said that the MSI registers are all inter-mixed with the interrupt controller. Therefore as the above don't interact with PCIe controller registers can you move this out to an irqchip driver or include it inside your interrupt controller?

Quoted 18 lines
drivers/pci/host/pci-mvebu.c +18 -0
+
+static int mvebu_pcie_enable_msi(struct mvebu_pcie *pcie)
+{
+	struct device_node *msi_node;
+	struct mvebu_pcie_msi *msi;
+
+	msi = &pcie->msi;
+
+	mutex_init(&msi->lock);
+
+	msi_node = of_parse_phandle(pcie->pdev->dev.of_node,
+				    "msi-parent", 0);
+	if (!msi_node)
+		return -EINVAL;
+
+	msi->domain = irq_find_host(msi_node);
+	if (!msi->domain)
+		return -EINVAL;

drivers/pci/host/pci-mvebu.c static void __iomem *mvebu_pcie_map_registers(struct platform_device *pdev, · line 907

Quoted 10 lines
drivers/pci/host/pci-mvebu.c +10 -0
+
+	if (of_property_read_u32(msi_node, "marvell,doorbell", &msi->doorbell))
+		return -EINVAL;
+
+	msi->chip.dev = &pcie->pdev->dev;
+	msi->chip.setup_irq = mvebu_pcie_setup_msi_irq;
+	msi->chip.teardown_irq = mvebu_pcie_teardown_msi_irq;
+
+	return 0;
+}

This can change too, as per Thierry's suggestion you can call something like:

msi_chip_add(&msi)

from your interrupt controller. And then in this file you can call something like:

of_find_msi_chip_by_node(msi_node)

and use the returned chip to assign to pcie->msi (and register with the pci_bus). Perhaps then you may not need to expose the doorbell register as that would be knowledge that the interrupt controller may already have (i.e. offset from base address already provided in DT).

Andrew Murray

Quoted 23 lines
drivers/pci/host/pci-mvebu.c +9 -0
+#endif /* CONFIG_PCI_MSI */
+
 static int __init mvebu_pcie_probe(struct platform_device *pdev)
 {
 	struct mvebu_pcie *pcie;
@@ -903,6 +1024,13 @@ static int __init mvebu_pcie_probe(struct platform_device *pdev)
 
 	mvebu_pcie_enable(pcie);
 
+#ifdef CONFIG_PCI_MSI
+	ret = mvebu_pcie_enable_msi(pcie);
+	if (ret)
+		dev_warn(&pdev->dev, "could not enable MSI support: %d\n",
+			 ret);
+#endif
+
 	return 0;
 }
 
Signature · 1 line
1.7.9.5
maintainer
#40 lore raw

RFC 9/11 pci: mvebu: add MSI support

In reply to Thomas Petazzoni
Quoted 101 lines · Thomas Petazzoni

This commit adds the MSI support for the Marvell EBU PCIe driver. The driver now looks at the 'msi-parent' property of the PCIe controller DT node, and if it exists, it gets the associated IRQ domain, which should be the MSI interrupt controller registered by the IRQ controller driver.

Using this, the PCIe driver registers the ->setup_irq() and ->teardown_irq() callbacks using the newly introduced msi_chip infrastructure, which allows the kernel PCI core to use the MSI functionality.

---
 .../devicetree/bindings/pci/mvebu-pci.txt          |    5 +
 drivers/pci/host/pci-mvebu.c                       |  128 ++++++++++++++++++++
 2 files changed, 133 insertions(+)
Documentation/devicetree/bindings/pci/mvebu-pci.txt +5 -0
index 192bdfb..53cc437 100644
@@ -14,6 +14,10 @@ Mandatory properties:
   corresponding registers
 - ranges: ranges for the PCI memory and I/O regions
 
+Optional properties:
+- msi-parent: a phandle pointing to the interrupt controller that
+  handles the MSI interrupts.
+
 In addition, the Device Tree node must have sub-nodes describing each
 PCIe interface, having the following mandatory properties:
 - reg: used only for interrupt mapping, so only the first four bytes
@@ -43,6 +47,7 @@ pcie-controller {
        #address-cells = <3>;
        #size-cells = <2>;
 
+       msi-parent = <&msi>;
        bus-range = <0x00 0xff>;
 
        reg = <0xd0040000 0x2000>, <0xd0042000 0x2000>,
drivers/pci/host/pci-mvebu.c +20 -0
index 9e6b137..b46fab8 100644
@@ -7,17 +7,23 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/irq.h>
+#include <linux/irqdomain.h>
 #include <linux/pci.h>
+#include <linux/msi.h>
 #include <linux/clk.h>
 #include <linux/module.h>
 #include <linux/mbus.h>
 #include <linux/slab.h>
 #include <linux/platform_device.h>
+#include <linux/interrupt.h>
 #include <linux/of_address.h>
 #include <linux/of_pci.h>
 #include <linux/of_irq.h>
 #include <linux/of_platform.h>
 
+#define INT_PCI_MSI_NR (16)
+
 /*
  * PCIe unit register offsets.
  */
@@ -101,10 +107,19 @@ struct mvebu_sw_pci_bridge {
 
 struct mvebu_pcie_port;
 
+struct mvebu_pcie_msi {
+       DECLARE_BITMAP(used, INT_PCI_MSI_NR);
+       struct irq_domain *domain;
+       struct mutex lock;
+       struct msi_chip chip;
+       phys_addr_t doorbell;
+};
+
 /* Structure representing all PCIe interfaces */
 struct mvebu_pcie {
        struct platform_device *pdev;
        struct mvebu_pcie_port *ports;
+       struct mvebu_pcie_msi msi;
        struct resource io;
        struct resource realio;
        struct resource mem;
@@ -546,6 +561,11 @@ static inline struct mvebu_pcie *sys_to_pcie(struct pci_sys_data *sys)
        return sys->private_data;
 }
 
+static inline struct mvebu_pcie_msi *to_mvebu_msi(struct msi_chip *chip)
+{
+       return container_of(chip, struct mvebu_pcie_msi, chip);
+}
+
 /* Find the PCIe interface that corresponds to the given bus */
 static struct mvebu_pcie_port *mvebu_find_port_from_bus(struct mvebu_pcie *pcie,
                                                        int bus)

drivers/pci/host/pci-mvebu.c static struct pci_bus *mvebu_pcie_scan_bus(int nr, struct pci_sys_data *sys) · line 733

Quoted 4 lines
drivers/pci/host/pci-mvebu.c +1 -0
@@ -710,6 +730,8 @@ static struct pci_bus *mvebu_pcie_scan_bus(int nr, struct pci_sys_data *sys)
        if (!bus)
                return NULL;
 
+       bus->msi = &pcie->msi.chip;

As I mentioned in the 08/11 patch, I'd like to keep arch-specific stuff out of the PCI scanning path to preserve the ability to move toward pci_scan_root_bus() eventually.

Quoted 35 lines
drivers/pci/host/pci-mvebu.c +28 -0
        pci_scan_child_bus(bus);
 
        return bus;
@@ -786,6 +808,105 @@ static void __iomem *mvebu_pcie_map_registers(struct platform_device *pdev,
        return devm_request_and_ioremap(&pdev->dev, &regs);
 }
 
+#if defined(CONFIG_PCI_MSI)
+static int mvebu_pcie_msi_alloc(struct mvebu_pcie_msi *chip)
+{
+       int msi;
+
+       mutex_lock(&chip->lock);
+
+       msi = find_first_zero_bit(chip->used, INT_PCI_MSI_NR);
+
+       if (msi < INT_PCI_MSI_NR)
+               set_bit(msi, chip->used);
+       else
+               msi = -ENOSPC;
+
+       mutex_unlock(&chip->lock);
+
+       return msi;
+}
+
+static void mvebu_pcie_msi_free(struct mvebu_pcie_msi *chip, unsigned long irq)
+{
+       struct device *dev = chip->chip.dev;
+
+       mutex_lock(&chip->lock);
+
+       if (!test_bit(irq, chip->used))
+               dev_err(dev, "trying to free unused MSI#%lu\n", irq);
+       else

drivers/pci/host/pci-mvebu.c static void __iomem *mvebu_pcie_map_registers(struct platform_device *pdev, · line 848

Quoted 10 lines
drivers/pci/host/pci-mvebu.c +10 -0
+               clear_bit(irq, chip->used);
+
+       mutex_unlock(&chip->lock);
+}
+
+static int mvebu_pcie_setup_msi_irq(struct msi_chip *chip,
+                                   struct pci_dev *pdev,
+                                   struct msi_desc *desc)
+{
+       struct mvebu_pcie_msi *msi = to_mvebu_msi(chip);

If this took only the pci_dev (not the msi_chip), I think you could do this:

  struct mvebu_pcie_msi *msi = &pdev->bus->sysdata->msi;
Quoted 81 lines
drivers/pci/host/pci-mvebu.c +68 -1
+       struct msi_msg msg;
+       unsigned int irq;
+       int hwirq;
+
+       hwirq = mvebu_pcie_msi_alloc(msi);
+       if (hwirq < 0)
+               return hwirq;
+
+       irq = irq_create_mapping(msi->domain, hwirq);
+       if (!irq)
+               return -EINVAL;
+
+       irq_set_msi_desc(irq, desc);
+
+       msg.address_lo = msi->doorbell;
+       msg.address_hi = 0;
+       msg.data = 0xf00 | (hwirq + 16);
+
+       write_msi_msg(irq, &msg);
+
+       return 0;
+}
+
+static void mvebu_pcie_teardown_msi_irq(struct msi_chip *chip,
+                                       unsigned int irq)
+{
+       struct mvebu_pcie_msi *msi = to_mvebu_msi(chip);
+       struct irq_data *d = irq_get_irq_data(irq);
+
+       mvebu_pcie_msi_free(msi, d->hwirq);
+}
+
+static int mvebu_pcie_enable_msi(struct mvebu_pcie *pcie)
+{
+       struct device_node *msi_node;
+       struct mvebu_pcie_msi *msi;
+
+       msi = &pcie->msi;
+
+       mutex_init(&msi->lock);
+
+       msi_node = of_parse_phandle(pcie->pdev->dev.of_node,
+                                   "msi-parent", 0);
+       if (!msi_node)
+               return -EINVAL;
+
+       msi->domain = irq_find_host(msi_node);
+       if (!msi->domain)
+               return -EINVAL;
+
+       if (of_property_read_u32(msi_node, "marvell,doorbell", &msi->doorbell))
+               return -EINVAL;
+
+       msi->chip.dev = &pcie->pdev->dev;
+       msi->chip.setup_irq = mvebu_pcie_setup_msi_irq;
+       msi->chip.teardown_irq = mvebu_pcie_teardown_msi_irq;
+
+       return 0;
+}
+#endif /* CONFIG_PCI_MSI */
+
 static int __init mvebu_pcie_probe(struct platform_device *pdev)
 {
        struct mvebu_pcie *pcie;
@@ -903,6 +1024,13 @@ static int __init mvebu_pcie_probe(struct platform_device *pdev)
 
        mvebu_pcie_enable(pcie);
 
+#ifdef CONFIG_PCI_MSI
+       ret = mvebu_pcie_enable_msi(pcie);
+       if (ret)
+               dev_warn(&pdev->dev, "could not enable MSI support: %d\n",
+                        ret);
+#endif
+
        return 0;
 }
 
--

1.7.9.5

#41 lore raw

RFC 9/11 pci: mvebu: add MSI support

Quoted 1 line · Bjorn Helgaas

On Tue, Mar 26, 2013 at 10:52 AM, Thomas Petazzoni <[email protected]> wrote:

[...]

Quoted 9 lines

drivers/pci/host/pci-mvebu.c static void __iomem *mvebu_pcie_map_registers(struct platform_device *pdev, · line 848

Quoted 5 lines
drivers/pci/host/pci-mvebu.c +5 -0
+static int mvebu_pcie_setup_msi_irq(struct msi_chip *chip,
+                                   struct pci_dev *pdev,
+                                   struct msi_desc *desc)
+{
+       struct mvebu_pcie_msi *msi = to_mvebu_msi(chip);

If this took only the pci_dev (not the msi_chip), I think you could do this:

  struct mvebu_pcie_msi *msi = &pdev->bus->sysdata->msi;

That would mean that the arch_setup_msi_irq() and friends could still be architecture-agnostic because they only pass around pci_dev, and the driver specific implementations would know how to lookup sysdata and from there the MSI chip.

So I was almost convinced that putting the struct msi_chip pointer into sysdata is a good idea. However that also means that each PCI host bridge driver becomes architecture-specific. If we ever get a driver that can be used on multiple architectures (however unlikely), the only way to make it work would be to #ifdef those parts. We could make that easier to deal with by providing an accessor (pci_sysdata_set_msi_chip() or similar), though.

But maybe it's something we don't need to be concerned about because no PCI host bridge driver will ever support two different architecture?

One related point is compile coverage. If the drivers are completely architecture-agnostic it makes it a lot easier to compile-test all drivers, which might come in useful when doing core changes and such.

Thierry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/linux-arm-kernel/attachments/20130530/062557b1/attachment.sig>

maintainer
#42 lore raw

RFC 9/11 pci: mvebu: add MSI support

Quoted 26 lines · Thierry Reding
Quoted 1 line · Bjorn Helgaas

On Tue, Mar 26, 2013 at 10:52 AM, Thomas Petazzoni <[email protected]> wrote:

[...]

Quoted 9 lines

drivers/pci/host/pci-mvebu.c static void __iomem *mvebu_pcie_map_registers(struct platform_device *pdev, · line 848

Quoted 5 lines
drivers/pci/host/pci-mvebu.c +5 -0
+static int mvebu_pcie_setup_msi_irq(struct msi_chip *chip,
+                                   struct pci_dev *pdev,
+                                   struct msi_desc *desc)
+{
+       struct mvebu_pcie_msi *msi = to_mvebu_msi(chip);

If this took only the pci_dev (not the msi_chip), I think you could do this:

  struct mvebu_pcie_msi *msi = &pdev->bus->sysdata->msi;

That would mean that the arch_setup_msi_irq() and friends could still be architecture-agnostic because they only pass around pci_dev, and the driver specific implementations would know how to lookup sysdata and from there the MSI chip.

So I was almost convinced that putting the struct msi_chip pointer into sysdata is a good idea. However that also means that each PCI host bridge driver becomes architecture-specific. If we ever get a driver that can be used on multiple architectures (however unlikely), the only way to make it work would be to #ifdef those parts. We could make that easier to deal with by providing an accessor (pci_sysdata_set_msi_chip() or similar), though.

Quoted 6 lines

But maybe it's something we don't need to be concerned about because no PCI host bridge driver will ever support two different architecture?

One related point is compile coverage. If the drivers are completely architecture-agnostic it makes it a lot easier to compile-test all drivers, which might come in useful when doing core changes and such.

Hmm, I've forgotten what I was thinking here. (And I'm the *worst* about reviving ancient threads myself). So I drop whatever objection I had :)

Bjorn

#43 lore raw

RFC 10/11 arm: mvebu: enable MSI support in DT

In reply to Thomas Petazzoni

Now that the Marvell EBU PCIe driver supports MSI, we can adjust the Device Tree for the Armada 370 and Armada XP SoCs so that the PCIe controller nodes point to the MSI interrupt controller using the 'msi-parent' property.

---
 arch/arm/boot/dts/armada-370.dtsi        |    1 +
 arch/arm/boot/dts/armada-xp-mv78230.dtsi |    1 +
 arch/arm/boot/dts/armada-xp-mv78260.dtsi |    1 +
 arch/arm/boot/dts/armada-xp-mv78460.dtsi |    1 +
 4 files changed, 4 insertions(+)
arch/arm/boot/dts/armada-370.dtsi +1 -0
index a23a74f..e9cda70 100644
@@ -161,6 +161,7 @@
 			#address-cells = <3>;
 			#size-cells = <2>;
 
+			msi-parent = <&msi>;
 			bus-range = <0x00 0xff>;
 
 			reg = <0xd0040000 0x2000>, <0xd0080000 0x2000>;
arch/arm/boot/dts/armada-xp-mv78230.dtsi +1 -0
index 4b05815..0e0c891 100644
@@ -90,6 +90,7 @@
 			#address-cells = <3>;
 			#size-cells = <2>;
 
+			msi-parent = <&msi>;
 			bus-range = <0x00 0xff>;
 
 			reg = <0xd0040000 0x2000>, <0xd0042000 0x2000>,
arch/arm/boot/dts/armada-xp-mv78260.dtsi +1 -0
index ee6c269..1db9da0 100644
@@ -110,6 +110,7 @@
 			#address-cells = <3>;
 			#size-cells = <2>;
 
+			msi-parent = <&msi>;
 			bus-range = <0x00 0xff>;
 
 			reg = <0xd0040000 0x2000>, <0xd0042000 0x2000>,
arch/arm/boot/dts/armada-xp-mv78460.dtsi +1 -0
index ba54990..1263269 100644
@@ -125,6 +125,7 @@
 			#address-cells = <3>;
 			#size-cells = <2>;
 
+			msi-parent = <&msi>;
 			bus-range = <0x00 0xff>;
 
 			reg = <0xd0040000 0x2000>, <0xd0042000 0x2000>,
Signature · 1 line
1.7.9.5
#44 lore raw

RFC 11/11 arm: mvebu: enable PCI MSI support in defconfig

In reply to Thomas Petazzoni

Now that MSI support is available, both in the IRQ controller driver and in the PCIe driver, let's enable it in the mvebu_defconfig used for Armada 370/XP platforms.

---
 arch/arm/configs/mvebu_defconfig |    1 +
 1 file changed, 1 insertion(+)
arch/arm/configs/mvebu_defconfig +1 -0
index 071a5b1..a9d0fc6 100644
@@ -14,6 +14,7 @@ CONFIG_MACH_ARMADA_XP=y
 # CONFIG_CACHE_L2X0 is not set
 # CONFIG_SWP_EMULATE is not set
 CONFIG_PCI=y
+CONFIG_PCI_MSI=y
 CONFIG_PCI_MVEBU=y
 CONFIG_SMP=y
 CONFIG_AEABI=y
Signature · 1 line
1.7.9.5
#45 lore raw

cover MSI support for Marvell EBU PCIe driver

In reply to Thomas Petazzoni

Hello,

Quoted 4 lines · Thomas Petazzoni

This set of patches introduces Message Signaled Interrupt support in the Marvell EBU PCIe driver. It has been successfully tested on the Armada XP GP platform and the Armada 370 DB platform with an Intel e1000e PCIe network card that supports MSI.

To the readers of LAKML: the mailing list software has, for some reason, decided that all the e-mails in this series had a "Suspicious header". They have all been generated by git format-patch and sent with git send-email, just like the previous set of 17 patches for the PCIe driver, so I have no idea why the mailing list software changed his mind.

Hopefully, the next version I'll send will make it to the list.

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#46 lore raw

cover MSI support for Marvell EBU PCIe driver

Quoted 6 lines · Thomas Petazzoni

To the readers of LAKML: the mailing list software has, for some reason, decided that all the e-mails in this series had a "Suspicious header". They have all been generated by git format-patch and sent with git send-email, just like the previous set of 17 patches for the PCIe driver, so I have no idea why the mailing list software changed his mind.

The mailing list rejects patches that have an in-reply-to header with pointing to a different subject as the new email.
It also has an exception for emails that have the work PATCH in brackets, but not [RFC] or [GIT PULL]. You can work around by using [RFC PATCH 1/11] or [RFC][PATCH 1/11].

Arnd

#47 lore raw

cover MSI support for Marvell EBU PCIe driver

Dear Arnd Bergmann,

Quoted 5 lines · Arnd Bergmann

The mailing list rejects patches that have an in-reply-to header with pointing to a different subject as the new email.
It also has an exception for emails that have the work PATCH in brackets, but not [RFC] or [GIT PULL]. You can work around by using [RFC PATCH 1/11] or [RFC][PATCH 1/11].

Aah, thanks for the tip. Do you think there's a place where we could write this down for future reference? Maybe Documentation/arm/mailing-list or whatever?

Best regards,

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#48 lore raw

cover MSI support for Marvell EBU PCIe driver

In reply to Thomas Petazzoni

Thomas,

Quoted 8 lines · Thomas Petazzoni

This set of patches introduces Message Signaled Interrupt support in the Marvell EBU PCIe driver. It has been successfully tested on the Armada XP GP platform and the Armada 370 DB platform with an Intel e1000e PCIe network card that supports MSI.

This is based on work done by Lior Amsalem <[email protected]>.

Quoted 9 lines

The patches do the following:

* Patches 1, 2 and 3 move the IRQ controller driver of Armada 370/XP platforms from arch/arm/mach-mvebu/ into drivers/irqchip/ and use the proper irqchip infrastructure. Those changes are not strictly needed to add MSI interrupts support, but since we will be touching the IRQ controller driver anyway, it sounded like the right time to do this move.

Given the IRQ controller move to drivers/irqchip is independent of MSI work, and that we've already agreed this move is fine (Arnd has acked patch 2), may I suggest that you resend these three first patches (and perhaps the fourth?) as a separate patchset to be included in v3.10.

This has the advantage that further development on IRQ controller can be done directly on its proper place, and also the MSI patchset can be simplified.

What do you think?

Signature · 3 lines
Ezequiel Garc?a, Free Electrons
Embedded Linux, Kernel and Android Engineering
http://free-electrons.com
#49 lore raw

cover MSI support for Marvell EBU PCIe driver

Dear Ezequiel Garcia,

Quoted 10 lines · Ezequiel Garcia

Given the IRQ controller move to drivers/irqchip is independent of MSI work, and that we've already agreed this move is fine (Arnd has acked patch 2), may I suggest that you resend these three first patches (and perhaps the fourth?) as a separate patchset to be included in v3.10.

This has the advantage that further development on IRQ controller can be done directly on its proper place, and also the MSI patchset can be simplified.

What do you think?

Sounds like a good idea. I'll cook a patch set later today and I'll send it.

Thanks for the suggestion!

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com