Skip to contentSkip to conversation

Level-triggered MSI support

PATCH Superseded Marc Zyngier 5 participants 17 messages Apr 23, 2018 – May 1, 2018

Subsystems (8)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Level-triggered MSI support

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Marc Zyngier (7):
genirq/msi: Allow level-triggered MSIs to be exposed by MSI providers genirq/msi: Limit level-triggered MSI to platform devices irqchip/mvebu-gicp: Use level-triggered MSIs between ICU and GICP
dma-iommu: Fix compilation when !CONFIG_IOMMU_DMA irqchip/gic-v3: Add support for Message Based Interrupts as an MSI
controller
irqchip/gic-v3: Add PCI/MSI support to the GICv3 MBI sub-driver dt-bindings/gic-v3: Add documentation for MBI support

 .../bindings/interrupt-controller/arm,gic-v3.txt   |  17 +
 drivers/base/platform-msi.c                        |   3 +
 drivers/irqchip/Makefile                           |   2 +-
 drivers/irqchip/irq-gic-v3-mbi.c                   | 343 +++++++++++++++++++++
 drivers/irqchip/irq-gic-v3.c                       |   6 +
 drivers/irqchip/irq-mvebu-gicp.c                   |  37 +--
 drivers/irqchip/irq-mvebu-gicp.h                   |  12 -
 drivers/irqchip/irq-mvebu-icu.c                    |  33 +-
 drivers/pci/msi.c                                  |   3 +
 include/linux/dma-iommu.h                          |   1 +
 include/linux/irqchip/arm-gic-v3.h                 |   1 +
 include/linux/msi.h                                |   2 +
 kernel/irq/msi.c                                   |  32 +-
 13 files changed, 427 insertions(+), 65 deletions(-)
 create mode 100644 drivers/irqchip/irq-gic-v3-mbi.c
 delete mode 100644 drivers/irqchip/irq-mvebu-gicp.h
Signature · 1 line
2.14.2
#1 lore raw

PATCH 1/7 genirq/msi: Allow level-triggered MSIs to be exposed by MSI providers

So far, MSIs have been used to signal edge-triggered interrupts, as a write is a good model for an edge (you can't "unwrite" something). On the other hand, routing zillions of wires in an SoC because you need level interrupts is a bit extreme.

People have come up with a variety of schemes to support this, which involves sending two messages: one to signal the interrupt, and one to clear it. Since the kernel cannot represent this, we've ended up with side-band mechanisms that are pretty awful.

Instead, let's acknoledge the requirement, and ensure that, under the right circumstances, the irq_compose_msg and irq_write_msg can take as a parameter an array of two messages instead of a pointer to a single one. We also add some checking that the compose method only clobbers the second message if the MSI domain has been created with the MSI_FLAG_LEVEL_CAPABLE flags.

---
 include/linux/msi.h |  2 ++
 kernel/irq/msi.c    | 32 +++++++++++++++++++++++---------
 2 files changed, 25 insertions(+), 9 deletions(-)
include/linux/msi.h +2 -0
index 1f1bbb5b4679..5839d8062dfc 100644
@@ -289,6 +289,8 @@ enum {
 	 * MSI_FLAG_ACTIVATE_EARLY has been set.
 	 */
 	MSI_FLAG_MUST_REACTIVATE	= (1 << 5),
+	/* Is level-triggered capable, using two messages */
+	MSI_FLAG_LEVEL_CAPABLE		= (1 << 6),
 };
 
 int msi_domain_set_affinity(struct irq_data *data, const struct cpumask *mask,
kernel/irq/msi.c +23 -9
index 2a8571f72b17..a6fbf6008dcb 100644
@@ -76,6 +76,18 @@ static inline void irq_chip_write_msi_msg(struct irq_data *data,
 	data->chip->irq_write_msi_msg(data, msg);
 }
 
+static void msi_check_level(struct irq_domain *domain, struct msi_msg *msg)
+{
+	struct msi_domain_info *info = domain->host_data;
+
+	/*
+	 * If the MSI provider has messed with the second message and
+	 * not advertized that it is level-capable, signal the breakage.
+	 */
+	WARN_ON(!(info->flags & MSI_FLAG_LEVEL_CAPABLE) &&
+		(msg[1].address_lo || msg[1].address_hi || msg[1].data));
+}
+
 /**
  * msi_domain_set_affinity - Generic affinity setter function for MSI domains
  * @irq_data:	The irq data associated to the interrupt
@@ -89,13 +101,14 @@ int msi_domain_set_affinity(struct irq_data *irq_data,
 			    const struct cpumask *mask, bool force)
 {
 	struct irq_data *parent = irq_data->parent_data;
-	struct msi_msg msg;
+	struct msi_msg msg[2] = { [1] = { }, };
 	int ret;
 
 	ret = parent->chip->irq_set_affinity(parent, mask, force);
 	if (ret >= 0 && ret != IRQ_SET_MASK_OK_DONE) {
-		BUG_ON(irq_chip_compose_msi_msg(irq_data, &msg));
-		irq_chip_write_msi_msg(irq_data, &msg);
+		BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
+		msi_check_level(irq_data->domain, msg);
+		irq_chip_write_msi_msg(irq_data, msg);
 	}
 
 	return ret;
@@ -104,20 +117,21 @@ int msi_domain_set_affinity(struct irq_data *irq_data,
 static int msi_domain_activate(struct irq_domain *domain,
 			       struct irq_data *irq_data, bool early)
 {
-	struct msi_msg msg;
+	struct msi_msg msg[2] = { [1] = { }, };
 
-	BUG_ON(irq_chip_compose_msi_msg(irq_data, &msg));
-	irq_chip_write_msi_msg(irq_data, &msg);
+	BUG_ON(irq_chip_compose_msi_msg(irq_data, msg));
+	msi_check_level(irq_data->domain, msg);
+	irq_chip_write_msi_msg(irq_data, msg);
 	return 0;
 }
 
 static void msi_domain_deactivate(struct irq_domain *domain,
 				  struct irq_data *irq_data)
 {
-	struct msi_msg msg;
+	struct msi_msg msg[2];
 
-	memset(&msg, 0, sizeof(msg));
-	irq_chip_write_msi_msg(irq_data, &msg);
+	memset(msg, 0, sizeof(msg));
+	irq_chip_write_msi_msg(irq_data, msg);
 }
 
 static int msi_domain_alloc(struct irq_domain *domain, unsigned int virq,
Signature · 1 line
2.14.2
#2 lore raw

PATCH 2/7 genirq/msi: Limit level-triggered MSI to platform devices

In reply to Marc Zyngier

Nobody would be insane enough to try and use level triggered MSIs on PCI, but let's make sure it doesn't happen. Also, let's mandate that the irqchip backing the platform MSI domain is providing an irq_set_type method (or the whole thing would be a bit useless).

---
 drivers/base/platform-msi.c | 3 +++
 drivers/pci/msi.c           | 3 +++
 2 files changed, 6 insertions(+)
drivers/base/platform-msi.c +3 -0
index 8e22073aeeed..3b9694a6beaa 100644
@@ -101,6 +101,9 @@ static void platform_msi_update_chip_ops(struct msi_domain_info *info)
 		chip->irq_set_affinity = msi_domain_set_affinity;
 	if (!chip->irq_write_msi_msg)
 		chip->irq_write_msi_msg = platform_msi_write_msg;
+	if (WARN_ON((info->flags & MSI_FLAG_LEVEL_CAPABLE) &&
+		    !chip->irq_set_type))
+		info->flags &= ~MSI_FLAG_LEVEL_CAPABLE;
 }
 
 static void platform_msi_free_descs(struct device *dev, int base, int nvec)
drivers/pci/msi.c +3 -0
index 30250631efe7..f45b74fcc059 100644
@@ -1434,6 +1434,9 @@ struct irq_domain *pci_msi_create_irq_domain(struct fwnode_handle *fwnode,
 {
 	struct irq_domain *domain;
 
+	if (WARN_ON(info->flags & MSI_FLAG_LEVEL_CAPABLE))
+		info->flags &= ~MSI_FLAG_LEVEL_CAPABLE;
+
 	if (info->flags & MSI_FLAG_USE_DEF_DOM_OPS)
 		pci_msi_domain_update_dom_ops(info);
 	if (info->flags & MSI_FLAG_USE_DEF_CHIP_OPS)
Signature · 1 line
2.14.2
#3 lore raw

Re: [PATCH 2/7] genirq/msi: Limit level-triggered MSI to platform devices

Quoted 17 lines · Marc Zyngier

Nobody would be insane enough to try and use level triggered MSIs on PCI, but let's make sure it doesn't happen. Also, let's mandate that the irqchip backing the platform MSI domain is providing an irq_set_type method (or the whole thing would be a bit useless).

---
 drivers/base/platform-msi.c | 3 +++
 drivers/pci/msi.c           | 3 +++
 2 files changed, 6 insertions(+)
drivers/base/platform-msi.c +0 -0
index 8e22073aeeed..3b9694a6beaa 100644

drivers/base/platform-msi.c static void platform_msi_update_chip_ops(struct msi_domain_info *info) · line 105

Quoted 5 lines
drivers/base/platform-msi.c +2 -0
@@ -101,6 +101,9 @@ static void platform_msi_update_chip_ops(struct msi_domain_info *info)
 		chip->irq_set_affinity = msi_domain_set_affinity;
 	if (!chip->irq_write_msi_msg)
 		chip->irq_write_msi_msg = platform_msi_write_msg;
+	if (WARN_ON((info->flags & MSI_FLAG_LEVEL_CAPABLE) &&
+		    !chip->irq_set_type))

Shouldn't we just require something like IRQCHIP_SUPPORTS_MSI_LEVEL or such in chip->flags being set instead?

Thanks,

tglx

#4 lore raw

Re: [PATCH 2/7] genirq/msi: Limit level-triggered MSI to platform devices

Quoted 14 lines · Thomas Gleixner
Quoted 12 lines · Marc Zyngier

Nobody would be insane enough to try and use level triggered MSIs on PCI, but let's make sure it doesn't happen. Also, let's mandate that the irqchip backing the platform MSI domain is providing an irq_set_type method (or the whole thing would be a bit useless).

---
 drivers/base/platform-msi.c | 3 +++
 drivers/pci/msi.c           | 3 +++
 2 files changed, 6 insertions(+)
Quoted 13 lines

drivers/base/platform-msi.c static void platform_msi_update_chip_ops(struct msi_domain_info *info) · line 105

Quoted 10 lines
drivers/base/platform-msi.c +2 -0
index 8e22073aeeed..3b9694a6beaa 100644
@@ -101,6 +101,9 @@ static void platform_msi_update_chip_ops(struct msi_domain_info *info)
 		chip->irq_set_affinity = msi_domain_set_affinity;
 	if (!chip->irq_write_msi_msg)
 		chip->irq_write_msi_msg = platform_msi_write_msg;
+	if (WARN_ON((info->flags & MSI_FLAG_LEVEL_CAPABLE) &&
+		    !chip->irq_set_type))

Shouldn't we just require something like IRQCHIP_SUPPORTS_MSI_LEVEL or such in chip->flags being set instead?

That definitely makes sense. Thanks for the suggestion.

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#5 lore raw

PATCH 3/7 irqchip/mvebu-gicp: Use level-triggered MSIs between ICU and GICP

In reply to Marc Zyngier

The ICU and GICP drivers are using an ugly side-band mechanism to find out about the "clear" doorbell when using level interrupts.

Let's convert it to level-triggered MSIs, which result in a nice cleanup.

---
 drivers/irqchip/irq-mvebu-gicp.c | 37 ++++++++++---------------------------
 drivers/irqchip/irq-mvebu-gicp.h | 12 ------------
 drivers/irqchip/irq-mvebu-icu.c  | 33 +++++++++++++++++----------------
 3 files changed, 27 insertions(+), 55 deletions(-)
 delete mode 100644 drivers/irqchip/irq-mvebu-gicp.h
drivers/irqchip/irq-mvebu-gicp.c +10 -27
index 17a4a7b6cdbb..b5a3cc8bf370 100644
@@ -19,8 +19,6 @@
 
 #include <dt-bindings/interrupt-controller/arm-gic.h>
 
-#include "irq-mvebu-gicp.h"
-
 #define GICP_SETSPI_NSR_OFFSET	0x0
 #define GICP_CLRSPI_NSR_OFFSET	0x8
 
@@ -55,34 +53,18 @@ static int gicp_idx_to_spi(struct mvebu_gicp *gicp, int idx)
 	return -EINVAL;
 }
 
-int mvebu_gicp_get_doorbells(struct device_node *dn, phys_addr_t *setspi,
-			     phys_addr_t *clrspi)
-{
-	struct platform_device *pdev;
-	struct mvebu_gicp *gicp;
-
-	pdev = of_find_device_by_node(dn);
-	if (!pdev)
-		return -ENODEV;
-
-	gicp = platform_get_drvdata(pdev);
-	if (!gicp)
-		return -ENODEV;
-
-	*setspi = gicp->res->start + GICP_SETSPI_NSR_OFFSET;
-	*clrspi = gicp->res->start + GICP_CLRSPI_NSR_OFFSET;
-
-	return 0;
-}
-
 static void gicp_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 {
 	struct mvebu_gicp *gicp = data->chip_data;
 	phys_addr_t setspi = gicp->res->start + GICP_SETSPI_NSR_OFFSET;
-
-	msg->data = data->hwirq;
-	msg->address_lo = lower_32_bits(setspi);
-	msg->address_hi = upper_32_bits(setspi);
+	phys_addr_t clrspi = gicp->res->start + GICP_CLRSPI_NSR_OFFSET;
+
+	msg[0].data = data->hwirq;
+	msg[0].address_lo = lower_32_bits(setspi);
+	msg[0].address_hi = upper_32_bits(setspi);
+	msg[1].data = data->hwirq;
+	msg[1].address_lo = lower_32_bits(clrspi);
+	msg[1].address_hi = upper_32_bits(clrspi);
 }
 
 static struct irq_chip gicp_irq_chip = {
@@ -176,7 +158,8 @@ static struct msi_domain_ops gicp_msi_ops = {
 };
 
 static struct msi_domain_info gicp_msi_domain_info = {
-	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS),
+	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
+		   MSI_FLAG_LEVEL_CAPABLE),
 	.ops	= &gicp_msi_ops,
 	.chip	= &gicp_msi_irq_chip,
 };
drivers/irqchip/irq-mvebu-gicp.h deleted +0 -12
deleted file mode 100644
index eaa12fb72102..000000000000
@@ -1,12 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __MVEBU_GICP_H__
-#define __MVEBU_GICP_H__
-
-#include <linux/types.h>
-
-struct device_node;
-
-int mvebu_gicp_get_doorbells(struct device_node *dn, phys_addr_t *setspi,
-			     phys_addr_t *clrspi);
-
-#endif /* __MVEBU_GICP_H__ */
drivers/irqchip/irq-mvebu-icu.c +17 -16
index e18c48d3a92e..13063339b416 100644
@@ -21,8 +21,6 @@
 
 #include <dt-bindings/interrupt-controller/mvebu-icu.h>
 
-#include "irq-mvebu-gicp.h"
-
 /* ICU registers */
 #define ICU_SETSPI_NSR_AL	0x10
 #define ICU_SETSPI_NSR_AH	0x14
@@ -43,6 +41,7 @@ struct mvebu_icu {
 	void __iomem *base;
 	struct irq_domain *domain;
 	struct device *dev;
+	atomic_t initialized;
 };
 
 struct mvebu_icu_irq_data {
@@ -51,6 +50,18 @@ struct mvebu_icu_irq_data {
 	unsigned int type;
 };
 
+static void mvebu_icu_init(struct mvebu_icu *icu, struct msi_msg *msg)
+{
+	if (atomic_cmpxchg(&icu->initialized, false, true))
+		return;
+
+	/* Set Clear/Set ICU SPI message address in AP */
+	writel_relaxed(msg[0].address_hi, icu->base + ICU_SETSPI_NSR_AH);
+	writel_relaxed(msg[0].address_lo, icu->base + ICU_SETSPI_NSR_AL);
+	writel_relaxed(msg[1].address_hi, icu->base + ICU_CLRSPI_NSR_AH);
+	writel_relaxed(msg[1].address_lo, icu->base + ICU_CLRSPI_NSR_AL);
+}
+
 static void mvebu_icu_write_msg(struct msi_desc *desc, struct msi_msg *msg)
 {
 	struct irq_data *d = irq_get_irq_data(desc->irq);
@@ -59,6 +70,8 @@ static void mvebu_icu_write_msg(struct msi_desc *desc, struct msi_msg *msg)
 	unsigned int icu_int;
 
 	if (msg->address_lo || msg->address_hi) {
+		/* One off initialization */
+		mvebu_icu_init(icu, msg);
 		/* Configure the ICU with irq number & type */
 		icu_int = msg->data | ICU_INT_ENABLE;
 		if (icu_irqd->type & IRQ_TYPE_EDGE_RISING)
@@ -197,9 +210,7 @@ static int mvebu_icu_probe(struct platform_device *pdev)
 	struct device_node *node = pdev->dev.of_node;
 	struct device_node *gicp_dn;
 	struct resource *res;
-	phys_addr_t setspi, clrspi;
-	u32 i, icu_int;
-	int ret;
+	int i;
 
 	icu = devm_kzalloc(&pdev->dev, sizeof(struct mvebu_icu),
 			   GFP_KERNEL);
@@ -242,22 +253,12 @@ static int mvebu_icu_probe(struct platform_device *pdev)
 	if (!gicp_dn)
 		return -ENODEV;
 
-	ret = mvebu_gicp_get_doorbells(gicp_dn, &setspi, &clrspi);
-	if (ret)
-		return ret;
-
-	/* Set Clear/Set ICU SPI message address in AP */
-	writel_relaxed(upper_32_bits(setspi), icu->base + ICU_SETSPI_NSR_AH);
-	writel_relaxed(lower_32_bits(setspi), icu->base + ICU_SETSPI_NSR_AL);
-	writel_relaxed(upper_32_bits(clrspi), icu->base + ICU_CLRSPI_NSR_AH);
-	writel_relaxed(lower_32_bits(clrspi), icu->base + ICU_CLRSPI_NSR_AL);
-
 	/*
 	 * Clean all ICU interrupts with type SPI_NSR, required to
 	 * avoid unpredictable SPI assignments done by firmware.
 	 */
 	for (i = 0 ; i < ICU_MAX_IRQS ; i++) {
-		icu_int = readl(icu->base + ICU_INT_CFG(i));
+		u32 icu_int = readl_relaxed(icu->base + ICU_INT_CFG(i));
 		if ((icu_int >> ICU_GROUP_SHIFT) == ICU_GRP_NSR)
 			writel_relaxed(0x0, icu->base + ICU_INT_CFG(i));
 	}
Signature · 1 line
2.14.2
#6 lore raw

PATCH 4/7 dma-iommu: Fix compilation when !CONFIG_IOMMU_DMA

In reply to Marc Zyngier

Inclusion of include/dma-iommu.h when CONFIG_IOMMU_DMA is not selected results in the following splat:

In file included from drivers/irqchip/irq-gic-v3-mbi.c:20:0:
./include/linux/dma-iommu.h:95:69: error: unknown type name ‘dma_addr_t’
static inline int iommu_get_msi_cookie(struct iommu_domain *domain, dma_addr_t base)

                                                                     ^~~~~~~~~~
./include/linux/dma-iommu.h:108:74: warning: ‘struct list_head’ declared inside parameter list will not be visible outside of this definition or declaration
 static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_head *list)
                                                                          ^~~~~~~~~

scripts/Makefile.build:312: recipe for target 'drivers/irqchip/irq-gic-v3-mbi.o' failed

Fix it by including linux/types.h.

---
 include/linux/dma-iommu.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/dma-iommu.h +1 -0
index 92f20832fd28..e8ca5e654277 100644
@@ -17,6 +17,7 @@
 #define __DMA_IOMMU_H
 
 #ifdef __KERNEL__
+#include <linux/types.h>
 #include <asm/errno.h>
 
 #ifdef CONFIG_IOMMU_DMA
Signature · 1 line
2.14.2
#7 lore raw

PATCH 5/7 irqchip/gic-v3: Add support for Message Based Interrupts as an MSI controller

In reply to Marc Zyngier

GICv3 offers the possibility to signal SPIs using a pair of doorbells (SETPI, CLRSPI) under the name of Message Based Interrupts (MBI). They can be used as either traditional (edge) MSIs, or the more exotic level-triggered flavour.

Let's implement support for platform MSI, which is the original intent for this feature.

---
 drivers/irqchip/Makefile           |   2 +-
 drivers/irqchip/irq-gic-v3-mbi.c   | 287 +++++++++++++++++++++++++++++++++++++
 drivers/irqchip/irq-gic-v3.c       |   6 +
 include/linux/irqchip/arm-gic-v3.h |   1 +
 4 files changed, 295 insertions(+), 1 deletion(-)
 create mode 100644 drivers/irqchip/irq-gic-v3-mbi.c
drivers/irqchip/Makefile +1 -1
index 5ed465ab1c76..15f268f646bf 100644
@@ -27,7 +27,7 @@ obj-$(CONFIG_ARM_GIC)			+= irq-gic.o irq-gic-common.o
 obj-$(CONFIG_ARM_GIC_PM)		+= irq-gic-pm.o
 obj-$(CONFIG_ARCH_REALVIEW)		+= irq-gic-realview.o
 obj-$(CONFIG_ARM_GIC_V2M)		+= irq-gic-v2m.o
-obj-$(CONFIG_ARM_GIC_V3)		+= irq-gic-v3.o irq-gic-common.o
+obj-$(CONFIG_ARM_GIC_V3)		+= irq-gic-v3.o irq-gic-v3-mbi.o irq-gic-common.o
 obj-$(CONFIG_ARM_GIC_V3_ITS)		+= irq-gic-v3-its.o irq-gic-v3-its-platform-msi.o irq-gic-v4.o
 obj-$(CONFIG_ARM_GIC_V3_ITS_PCI)	+= irq-gic-v3-its-pci-msi.o
 obj-$(CONFIG_ARM_GIC_V3_ITS_FSL_MC)	+= irq-gic-v3-its-fsl-mc-msi.o
drivers/irqchip/irq-gic-v3-mbi.c new file +287 -0

287 lines not on this page · raw message

drivers/irqchip/irq-gic-v3.c +6 -0
index e5d101418390..1c6d039f22a3 100644
@@ -1112,6 +1112,12 @@ static int __init gic_init_bases(void __iomem *dist_base,
 	pr_info("Distributor has %sRange Selector support\n",
 		gic_data.has_rss ? "" : "no ");
 
+	if (typer & GICD_TYPER_MBIS) {
+		err = mbi_init(handle, gic_data.domain);
+		if (err)
+			pr_err("Failed to initialize MBIs\n");
+	}
+
 	set_handle_irq(gic_handle_irq);
 
 	gic_update_vlpi_properties();
include/linux/irqchip/arm-gic-v3.h +1 -0
index f5af3b594e6e..cbb872c1b607 100644
@@ -587,6 +587,7 @@ struct fwnode_handle;
 int its_cpu_init(void);
 int its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	     struct irq_domain *domain);
+int mbi_init(struct fwnode_handle *fwnode, struct irq_domain *parent);
 
 static inline bool gic_enable_sre(void)
 {
Signature · 1 line
2.14.2
#8 lore raw

Re: [PATCH 5/7] irqchip/gic-v3: Add support for Message Based Interrupts as an MSI controller

drivers/irqchip/irq-gic-v3-mbi.c line 3

Quoted 6 lines · Marc Zyngier
drivers/irqchip/irq-gic-v3-mbi.c new file +3 -0
@@ -0,0 +1,287 @@
+/*
+ * Copyright (C) 2018 ARM Limited, All Rights Reserved.
+ * Author: Marc Zyngier <[email protected]>

Can you please:

1) Add the proper SPDX-License-Identifier

// SPDX-License-Identifier: GPL-2.0

at the first line of the file and

2) Remove the boiler plate? Please talk to Jilayne.

Quoted 11 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +11 -0
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License

drivers/irqchip/irq-gic-v3-mbi.c line 37

Quoted 10 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +9 -0
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
 
+struct mbi_range {
+	u32			spi_start;
+	u32			nr_spis;
+	unsigned long		*bm;
+};
+
+static spinlock_t		mbi_lock;

DEFINE_SPINLOCK() please

Quoted 35 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +35 -0
+static phys_addr_t		mbi_phys_base;
+static struct mbi_range		*mbi_ranges;
+static unsigned int		mbi_range_nr;
+
+static struct irq_chip mbi_irq_chip = {
+	.name			= "MBI",
+	.irq_mask		= irq_chip_mask_parent,
+	.irq_unmask		= irq_chip_unmask_parent,
+	.irq_eoi		= irq_chip_eoi_parent,
+	.irq_set_type		= irq_chip_set_type_parent,
+	.irq_set_affinity	= irq_chip_set_affinity_parent,
+};
+
+static int mbi_irq_gic_domain_alloc(struct irq_domain *domain,
+				       unsigned int virq,
+				       irq_hw_number_t hwirq)
+{
+	struct irq_fwspec fwspec;
+	struct irq_data *d;
+	int err;
+
+	/*
+	 * Using ACPI? There is no MBI support in the spec, you
+	 * shouldn't even be here.
+	 */
+	if (!is_of_node(domain->parent->fwnode))
+		return -EINVAL;
+
+	/*
+	 * Let's default to edge. This is consistent with traditional
+	 * MSIs, and systems requiring level signaling will just
+	 * enforce the trigger on their own.
+	 */
+	fwspec.fwnode = domain->parent->fwnode;
+	fwspec.param_count = 3;

drivers/irqchip/irq-gic-v3-mbi.c line 82

Quoted 10 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +10 -0
+	fwspec.param[0] = 0;
+	fwspec.param[1] = hwirq - 32;
+	fwspec.param[2] = IRQ_TYPE_EDGE_RISING;
+
+	err = irq_domain_alloc_irqs_parent(domain, virq, 1, &fwspec);
+	if (err)
+		return err;
+
+	d = irq_domain_get_irq_data(domain->parent, virq);
+	d->chip->irq_set_type(d, IRQ_TYPE_EDGE_RISING);

irq_chip_set_type_parent(d, ....) ?

Quoted 10 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +10 -0
+
+	return 0;
+}
+
+static void mbi_free_msi(struct mbi_range *mbi, unsigned int hwirq,
+			 int nr_irqs)
+{
+	spin_lock(&mbi_lock);
+	bitmap_release_region(mbi->bm, hwirq - mbi->spi_start,
+			      get_count_order(nr_irqs));

drivers/irqchip/irq-gic-v3-mbi.c line 102

Quoted 10 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +10 -0
+	spin_unlock(&mbi_lock);
+}
+
+static int mbi_irq_domain_alloc(struct irq_domain *domain, unsigned int virq,
+				   unsigned int nr_irqs, void *args)
+{
+	struct mbi_range *mbi = NULL;
+	int hwirq, offset, i, err = 0;
+
+	spin_lock(&mbi_lock);

There is no real reason that this must be a spinlock, right? Make it a mutex please.

Quoted 6 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +6 -0
+	for (i = 0; i < mbi_range_nr; i++) {
+		offset = bitmap_find_free_region(mbi_ranges[i].bm,
+						 mbi_ranges[i].nr_spis,
+						 get_count_order(nr_irqs));
+		if (offset >= 0) {
+			mbi = &mbi_ranges[i];

drivers/irqchip/irq-gic-v3-mbi.c line 175

Quoted 10 lines
drivers/irqchip/irq-gic-v3-mbi.c new file +9 -0
+			break;
+		}
+	}
+	spin_unlock(&mbi_lock);
 
+/* Platform-MSI specific irqchip */
+static struct irq_chip mbi_pmsi_irq_chip = {
+	.name			= "pMSI",
+	.irq_set_type		= irq_chip_set_type_parent,
+	.irq_compose_msi_msg	= mbi_compose_mbi_msg,

Fun. I did not expect this to work w/o any of the other callbacks. Magic!

Other than the above nits, this looks pretty good.

Thanks,

tglx

#9 lore raw

PATCH 6/7 irqchip/gic-v3: Add PCI/MSI support to the GICv3 MBI sub-driver

In reply to Marc Zyngier

You would hope that if you have a GICv3 in your system, you'd use the ITS, as it provides a large interrupt ID space and device isolation. Sadly, some SoC integrations are less than perfect, and the ITS is not usesable on those.

The only solution for these systems is to use the MBI interface, and rely on a very small number of possible vectors.

This patch thus adds minimal support for PCI/MSI on top of the GICv3 MBI driver. Please don't use it if you can avoid it.

---
 drivers/irqchip/irq-gic-v3-mbi.c | 62 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 59 insertions(+), 3 deletions(-)
drivers/irqchip/irq-gic-v3-mbi.c +59 -3
index 20303f54e8c0..acbeb236f157 100644
@@ -23,6 +23,7 @@
 #include <linux/kernel.h>
 #include <linux/msi.h>
 #include <linux/of_address.h>
+#include <linux/of_pci.h>
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 
@@ -157,6 +158,55 @@ static void mbi_compose_msi_msg(struct irq_data *data, struct msi_msg *msg)
 	iommu_dma_map_msi_msg(data->irq, msg);
 }
 
+#ifdef CONFIG_PCI_MSI
+/* PCI-specific irqchip */
+static void mbi_mask_msi_irq(struct irq_data *d)
+{
+	pci_msi_mask_irq(d);
+	irq_chip_mask_parent(d);
+}
+
+static void mbi_unmask_msi_irq(struct irq_data *d)
+{
+	pci_msi_unmask_irq(d);
+	irq_chip_unmask_parent(d);
+}
+
+static struct irq_chip mbi_msi_irq_chip = {
+	.name			= "MSI",
+	.irq_mask		= mbi_mask_msi_irq,
+	.irq_unmask		= mbi_unmask_msi_irq,
+	.irq_eoi		= irq_chip_eoi_parent,
+	.irq_compose_msi_msg	= mbi_compose_msi_msg,
+	.irq_write_msi_msg	= pci_msi_domain_write_msg,
+};
+
+static struct msi_domain_info mbi_msi_domain_info = {
+	.flags	= (MSI_FLAG_USE_DEF_DOM_OPS | MSI_FLAG_USE_DEF_CHIP_OPS |
+		   MSI_FLAG_PCI_MSIX | MSI_FLAG_MULTI_PCI_MSI),
+	.chip	= &mbi_msi_irq_chip,
+};
+
+static int mbi_allocate_pci_domain(struct irq_domain *nexus_domain,
+				   struct irq_domain **pci_domain)
+{
+	*pci_domain = pci_msi_create_irq_domain(nexus_domain->parent->fwnode,
+						&mbi_msi_domain_info,
+						nexus_domain);
+	if (!*pci_domain)
+		return -ENOMEM;
+
+	return 0;
+}
+#else
+static int mbi_allocate_pci_domain(struct irq_domain *nexus_domain,
+				   struct irq_domain **pci_domain)
+{
+	*pci_domain = NULL;
+	return 0;
+}
+#endif
+
 static void mbi_compose_mbi_msg(struct irq_data *data, struct msi_msg *msg)
 {
 	mbi_compose_msi_msg(data, msg);
@@ -187,7 +237,8 @@ static struct msi_domain_info mbi_pmsi_domain_info = {
 
 static int mbi_allocate_domains(struct irq_domain *parent)
 {
-	struct irq_domain *nexus_domain, *plat_domain;
+	struct irq_domain *nexus_domain, *pci_domain, *plat_domain;
+	int err;
 
 	nexus_domain = irq_domain_create_tree(parent->fwnode,
 					      &mbi_domain_ops, NULL);
@@ -197,12 +248,17 @@ static int mbi_allocate_domains(struct irq_domain *parent)
 	irq_domain_update_bus_token(nexus_domain, DOMAIN_BUS_NEXUS);
 	nexus_domain->parent = parent;
 
+	err = mbi_allocate_pci_domain(nexus_domain, &pci_domain);
+
 	plat_domain = platform_msi_create_irq_domain(parent->fwnode,
 						     &mbi_pmsi_domain_info,
 						     nexus_domain);
 
-	if (!plat_domain) {
-		irq_domain_remove(plat_domain);
+	if (err || !plat_domain) {
+		if (plat_domain)
+			irq_domain_remove(plat_domain);
+		if (pci_domain)
+			irq_domain_remove(pci_domain);
 		irq_domain_remove(nexus_domain);
 		return -ENOMEM;
 	}
Signature · 1 line
2.14.2
#10 lore raw

PATCH 7/7 dt-bindings/gic-v3: Add documentation for MBI support

In reply to Marc Zyngier

Add the required properties to support the MBI feature on GICv3.

---
 .../bindings/interrupt-controller/arm,gic-v3.txt        | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
Documentation/devicetree/bindings/interrupt-controller/arm,gic-v3.txt +17 -0
index 0a57f2f4167d..3ea78c4ef887 100644
@@ -57,6 +57,20 @@ Optional
   occupied by the redistributors. Required if more than one such
   region is present.
 
+- msi-controller: Boolean property. Identifies the node as an MSI
+  controller. Only present if the Message Based Interrupt
+  functionnality is being exposed by the HW, and the mbi-ranges
+  property present.
+
+- mbi-ranges: A list of pairs <intid span>, where "intid" is the first
+  SPI of a range that can be used an MBI, and "span" the size of that
+  range. Multiple ranges can be provided. Requires "msi-controller" to
+  be set.
+
+- mbi-alias: Address property. Base address of an alias of the GICD
+  region containing only the {SET,CLR}SPI registers to be used if
+  isolation is required, and if supported by the HW.
+
 Sub-nodes:
 
 PPI affinity can be expressed as a single "ppi-partitions" node,
@@ -99,6 +113,9 @@ Examples:
 		      <0x0 0x2c020000 0 0x2000>;	// GICV
 		interrupts = <1 9 4>;
 
+		msi-controller;
+		mbi-ranges = <256 128>;
+
 		gic-its@2c200000 {
 			compatible = "arm,gic-v3-its";
 			msi-controller;
Signature · 1 line
2.14.2
#11 lore raw
In reply to Marc Zyngier
Quoted 17 lines · Marc Zyngier

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

Quoted 7 lines

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Hi Marc,

I am hitting the splat below when trying this series on SynQuacer, with mbi range <64 32> (which is reserved in the h/w manual but note that I haven't confirmed with Socionext whether these are expected to work or not. However, I don't think that makes any difference regarding the issue below.)

 Unable to handle kernel read from unreadable memory at virtual address 00000018
 Mem abort info:
   ESR = 0x96000004
   Exception class = DABT (current EL), IL = 32 bits
   SET = 0, FnV = 0
   EA = 0, S1PTW = 0
 Data abort info:
   ISV = 0, ISS = 0x00000004
   CM = 0, WnR = 0
 user pgtable: 4k pages, 48-bit VAs, pgdp =         (ptrval)
 [0000000000000018] pgd=0000000000000000
 Internal error: Oops: 96000004 [#1] PREEMPT SMP
 Modules linked in: gpio_keys(+) efivarfs ip_tables x_tables autofs4
ext4 crc16 mbcache jbd2 fscrypto sr_mod cdrom sd_mod ahci xhci_pci
libahci xhci_hcd libata usbcore scsi_mod realtek netsec of_mdio
fixed_phy libphy i2c_synquacer gpio_mb86s7x
 CPU: 19 PID: 398 Comm: systemd-udevd Tainted: G        W
4.17.0-rc2+ #54
 Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build
#101 Apr  2 2018
 pstate: a0400085 (NzCv daIf +PAN -UAO)
 pc : iommu_dma_map_msi_msg+0x40/0x1e8
 lr : iommu_dma_map_msi_msg+0x34/0x1e8
 sp : ffff00000b8db690
 x29: ffff00000b8db690 x28: ffffeca6f07442a0
 x27: 0000000000000000 x26: ffffeca6f07442d4
 x25: ffffeca6f0744398 x24: 0000000000000000
 x23: 0000000000000016 x22: 0000000000000000
 x21: ffffeca6f755ed00 x20: ffff00000b8db770
 x19: 0000000000000016 x18: ffffffffffffffff
 x17: ffff446c203fd000 x16: ffff446c1f3b5108
 x15: ffffeca6f0a095b0 x14: ffffeca6f0a3a587
 x13: ffffeca6f0a3a586 x12: 0000000000000040
 x11: 0000000000000004 x10: 0000000000000016
 x9 : ffffeca6f70009d8 x8 : 0000000000000000
 x7 : ffffeca6f0744200 x6 : ffffeca6f0744200
 x5 : ffffeca6f7000900 x4 : ffffeca6f0744200
 x3 : 0000000000000000 x2 : 0000000000000000
 x1 : ffffeca6f0744258 x0 : 0000000000000000
 Process systemd-udevd (pid: 398, stack limit = 0x        (ptrval))
 Call trace:
  iommu_dma_map_msi_msg+0x40/0x1e8
  mbi_compose_msi_msg+0x54/0x60
  mbi_compose_mbi_msg+0x28/0x68
  irq_chip_compose_msi_msg+0x5c/0x78
  msi_domain_activate+0x40/0x90
  __irq_domain_activate_irq+0x74/0xb8
  __irq_domain_activate_irq+0x3c/0xb8
  irq_domain_activate_irq+0x4c/0x60
  irq_activate+0x40/0x50
  __setup_irq+0x4bc/0x7e0
  request_threaded_irq+0xf0/0x198
  request_any_context_irq+0x6c/0xc0
  devm_request_any_context_irq+0x78/0xf0
  gpio_keys_probe+0x324/0x9a0 [gpio_keys]
  platform_drv_probe+0x60/0xc8
  driver_probe_device+0x2c4/0x490
  __driver_attach+0x10c/0x128
  bus_for_each_dev+0x78/0xe0
  driver_attach+0x30/0x40
  bus_add_driver+0x1d0/0x298
  driver_register+0x68/0x100
  __platform_driver_register+0x54/0x60
  gpio_keys_init+0x24/0x1000 [gpio_keys]
  do_one_initcall+0x68/0x258
  do_init_module+0x64/0x1e0
  load_module+0x1e20/0x21a8
  sys_finit_module+0x108/0x140
 EFI Variables Facility v0.08 2004-May-17
  __sys_trace_return+0x0/0x4
 Code: 97ec1444 b4000ca0 f9400800 f9400800 (f9400c18)
 ---[ end trace f7956dc89d9f3be7 ]---

00000000000014a0 <iommu_dma_map_msi_msg>:

    14a0:       a9ba7bfd        stp     x29, x30, [sp, #-96]!
    14a4:       910003fd        mov     x29, sp
    14a8:       a90153f3        stp     x19, x20, [sp, #16]
    14ac:       a9025bf5        stp     x21, x22, [sp, #32]
    14b0:       a90363f7        stp     x23, x24, [sp, #48]
    14b4:       a9046bf9        stp     x25, x26, [sp, #64]
    14b8:       a90573fb        stp     x27, x28, [sp, #80]
    14bc:       2a0003f3        mov     w19, w0
    14c0:       aa1e03e0        mov     x0, x30
    14c4:       aa0103f4        mov     x20, x1
    14c8:       94000000        bl      0 <_mcount>
    14cc:       2a1303e0        mov     w0, w19
    14d0:       94000000        bl      0 <irq_get_irq_data>
    14d4:       b4000ca0        cbz     x0, 1668 <iommu_dma_map_msi_msg+0x1c8>
    14d8:       f9400800        ldr     x0, [x0, #16]
    14dc:       f9400800        ldr     x0, [x0, #16]
    14e0:       f9400c18        ldr     x24, [x0, #24]
    14e4:       aa1803e0        mov     x0, x24
    14e8:       94000000        bl      0 <iommu_get_domain_for_dev>
    14ec:       aa0003f3        mov     x19, x0
    14f0:       b40008a0        cbz     x0, 1604 <iommu_dma_map_msi_msg+0x164>
    14f4:       f9402015        ldr     x21, [x0, #64]
    14f8:       b4000875        cbz     x21, 1604 <iommu_dma_map_msi_msg+0x164>
    14fc:       911d82b6        add     x22, x21, #0x760
#12 lore raw

HI Ard,

Quoted 26 lines · Ard Biesheuvel
Quoted 17 lines · Marc Zyngier

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

Quoted 7 lines

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Quoted 7 lines

Hi Marc,

I am hitting the splat below when trying this series on SynQuacer, with mbi range <64 32> (which is reserved in the h/w manual but note that I haven't confirmed with Socionext whether these are expected to work or not. However, I don't think that makes any difference regarding the issue below.)

Thanks for giving it a go. Something looks odd below:

Quoted 35 lines
 Unable to handle kernel read from unreadable memory at virtual address 00000018
 Mem abort info:
   ESR = 0x96000004
   Exception class = DABT (current EL), IL = 32 bits
   SET = 0, FnV = 0
   EA = 0, S1PTW = 0
 Data abort info:
   ISV = 0, ISS = 0x00000004
   CM = 0, WnR = 0
 user pgtable: 4k pages, 48-bit VAs, pgdp =         (ptrval)
 [0000000000000018] pgd=0000000000000000
 Internal error: Oops: 96000004 [#1] PREEMPT SMP
 Modules linked in: gpio_keys(+) efivarfs ip_tables x_tables autofs4
ext4 crc16 mbcache jbd2 fscrypto sr_mod cdrom sd_mod ahci xhci_pci
libahci xhci_hcd libata usbcore scsi_mod realtek netsec of_mdio
fixed_phy libphy i2c_synquacer gpio_mb86s7x
 CPU: 19 PID: 398 Comm: systemd-udevd Tainted: G        W
4.17.0-rc2+ #54
 Hardware name: Socionext SynQuacer E-series DeveloperBox, BIOS build
#101 Apr  2 2018
 pstate: a0400085 (NzCv daIf +PAN -UAO)
 pc : iommu_dma_map_msi_msg+0x40/0x1e8
 lr : iommu_dma_map_msi_msg+0x34/0x1e8
 sp : ffff00000b8db690
 x29: ffff00000b8db690 x28: ffffeca6f07442a0
 x27: 0000000000000000 x26: ffffeca6f07442d4
 x25: ffffeca6f0744398 x24: 0000000000000000
 x23: 0000000000000016 x22: 0000000000000000
 x21: ffffeca6f755ed00 x20: ffff00000b8db770
 x19: 0000000000000016 x18: ffffffffffffffff
 x17: ffff446c203fd000 x16: ffff446c1f3b5108
 x15: ffffeca6f0a095b0 x14: ffffeca6f0a3a587
 x13: ffffeca6f0a3a586 x12: 0000000000000040
 x11: 0000000000000004 x10: 0000000000000016
Quoted 8 lines
 x9 : ffffeca6f70009d8 x8 : 0000000000000000
 x7 : ffffeca6f0744200 x6 : ffffeca6f0744200
 x5 : ffffeca6f7000900 x4 : ffffeca6f0744200
 x3 : 0000000000000000 x2 : 0000000000000000
 x1 : ffffeca6f0744258 x0 : 0000000000000000
 Process systemd-udevd (pid: 398, stack limit = 0x        (ptrval))
 Call trace:
  iommu_dma_map_msi_msg+0x40/0x1e8

We die here because irq_get_msi_desc() has returned a NULL pointer. That's not really expected.

Quoted 2 lines
  mbi_compose_msi_msg+0x54/0x60
  mbi_compose_mbi_msg+0x28/0x68

We're requesting a platform MSI...

Quoted 11 lines
  irq_chip_compose_msi_msg+0x5c/0x78
  msi_domain_activate+0x40/0x90
  __irq_domain_activate_irq+0x74/0xb8
  __irq_domain_activate_irq+0x3c/0xb8
  irq_domain_activate_irq+0x4c/0x60
  irq_activate+0x40/0x50
  __setup_irq+0x4bc/0x7e0
  request_threaded_irq+0xf0/0x198
  request_any_context_irq+0x6c/0xc0
  devm_request_any_context_irq+0x78/0xf0
  gpio_keys_probe+0x324/0x9a0 [gpio_keys]

from the gpio_keys driver, which shouldn't be doing platform MSI at all. It looks like we've looked-up the wrong irq domain, and really bad things happen after that.

Could you point me to the device-tree of this machine? I need to understand how we can end-up in such a situation.

Thanks,

	M.
Signature · 1 line
Jazz is not dead. It just smells funny...
#13 lore raw
In reply to Ard Biesheuvel
Quoted 26 lines · Ard Biesheuvel
Quoted 17 lines · Marc Zyngier

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

Quoted 7 lines

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Quoted 7 lines

Hi Marc,

I am hitting the splat below when trying this series on SynQuacer, with mbi range <64 32> (which is reserved in the h/w manual but note that I haven't confirmed with Socionext whether these are expected to work or not. However, I don't think that makes any difference regarding the issue below.)

[...]

For the record: After some IRC debugging with Ard, this turns out to be due to two issues:
- GICv3 is now advertising several domains, all using the same device nodes
- irq_find_host() is picking the first one, which is the wrong one.

A good way to avoid this mess is to:
- Let GICv3 advertise its core domain with DOMAIN_BUS_WIRED (consistent with what the armada-370-xp driver is already doing)
- Let irq_find_host return DOMAIN_BUS_WIRED before trying DOMAIN_BUS_ANY (consistent with the way irq_create_fwspec_mapping works).

I've stashed the whole series at [1] with these two fixes.

	M.

[1] git://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms.git
irq/level-msi

Signature · 1 line
Jazz is not dead. It just smells funny...
Tested-by
#14 lore raw
Quoted 42 lines · Marc Zyngier
Quoted 26 lines · Ard Biesheuvel
Quoted 17 lines · Marc Zyngier

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

Quoted 7 lines

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Quoted 7 lines

Hi Marc,

I am hitting the splat below when trying this series on SynQuacer, with mbi range <64 32> (which is reserved in the h/w manual but note that I haven't confirmed with Socionext whether these are expected to work or not. However, I don't think that makes any difference regarding the issue below.)

[...]

For the record: After some IRC debugging with Ard, this turns out to be due to two issues:
- GICv3 is now advertising several domains, all using the same device nodes
- irq_find_host() is picking the first one, which is the wrong one.

Quoted 8 lines

A good way to avoid this mess is to:
- Let GICv3 advertise its core domain with DOMAIN_BUS_WIRED (consistent with what the armada-370-xp driver is already doing)
- Let irq_find_host return DOMAIN_BUS_WIRED before trying DOMAIN_BUS_ANY (consistent with the way irq_create_fwspec_mapping works).

I've stashed the whole series at [1] with these two fixes.

For the fixed version (1c1eff8a4d6b14aae9a709570e692f5fcd837670)

Tested-by on the v1 series
#15 lore raw
In reply to Marc Zyngier
Quoted 24 lines · Marc Zyngier

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Quoted 10 lines

Marc Zyngier (7):
genirq/msi: Allow level-triggered MSIs to be exposed by MSI providers genirq/msi: Limit level-triggered MSI to platform devices irqchip/mvebu-gicp: Use level-triggered MSIs between ICU and GICP
dma-iommu: Fix compilation when !CONFIG_IOMMU_DMA irqchip/gic-v3: Add support for Message Based Interrupts as an MSI
controller
irqchip/gic-v3: Add PCI/MSI support to the GICv3 MBI sub-driver dt-bindings/gic-v3: Add documentation for MBI support

Thanks for the patches,

Am able to get PCIe MSI interrupts on Qualcomm DragonBoard DB820c with this patchset.

root@linaro-alip:~# cat /proc/interrupts  | grep -i MSI
 44:          0          0          0          0     GICv3 437 Edge 
 qcom-pcie-msi
 45:          0          0          0          0     GICv3 445 Edge 
 qcom-pcie-msi
 46:          0          0          0          0     GICv3 453 Edge 
 qcom-pcie-msi
 80:          0          0          0          0       MSI 134217728 
Edge      PCIe PME, aerdrv
114:          0          0          0          0       MSI 268435456 Edge      PCIe PME, aerdrv
115:          0          0          0          0       MSI 134742016 Edge      ahci[0001:01:00.0]
197:          0          0          0          0       MSI   0 Edge PCIe PME, aerdrv
199:       2108          0          0          0       MSI 524288 Edge 
   ath10k_pci
202:        295          0          0          0       MSI 268959744 Edge      eth0

--srini

Tested-by on the v1 series
#16 lore raw
In reply to Marc Zyngier

Hi Marc,

Quoted 34 lines · Marc Zyngier

This series is a first shot at teaching the kernel about the oxymoron expressed in $SUBJECT. Over the past couple of years, we've seen some SoCs coming up with ways of signalling level interrupts using a new flavor of MSIs, where the MSI controller uses two distinct messages:
one that raises a virtual line, and one that lowers it. The target MSI controller is in charge of maintaining the state of the line.

This allows for a much simplified HW signal routing (no need to have hundreds of discrete lines to signal level interrupts if you already have a memory bus), but results in a departure from the current idea the kernel has of MSIs.

This series takes a minimal approach to the problem, which is to allow MSI controllers to use not only one, but up to two messages at a time. This is controlled by a flag exposed at MSI irq domain creation, and is only supported with platform MSI.

The rest of the series repaints the Marvell ICU/GICP drivers which already make use of this feature with a side-channel, and adds support for the same feature in GICv3. A side effect of the last GICv3 patch is that you can also use SPIs to signal PCI MSIs. This is a last resort measure for SoCs where the ITS is unusable for unspeakable reasons.

Marc Zyngier (7):
genirq/msi: Allow level-triggered MSIs to be exposed by MSI providers genirq/msi: Limit level-triggered MSI to platform devices irqchip/mvebu-gicp: Use level-triggered MSIs between ICU and GICP
dma-iommu: Fix compilation when !CONFIG_IOMMU_DMA irqchip/gic-v3: Add support for Message Based Interrupts as an MSI
controller
irqchip/gic-v3: Add PCI/MSI support to the GICv3 MBI sub-driver dt-bindings/gic-v3: Add documentation for MBI support

Quoted 17 lines
 .../bindings/interrupt-controller/arm,gic-v3.txt   |  17 +
 drivers/base/platform-msi.c                        |   3 +
 drivers/irqchip/Makefile                           |   2 +-
 drivers/irqchip/irq-gic-v3-mbi.c                   | 343 +++++++++++++++++++++
 drivers/irqchip/irq-gic-v3.c                       |   6 +
 drivers/irqchip/irq-mvebu-gicp.c                   |  37 +--
 drivers/irqchip/irq-mvebu-gicp.h                   |  12 -
 drivers/irqchip/irq-mvebu-icu.c                    |  33 +-
 drivers/pci/msi.c                                  |   3 +
 include/linux/dma-iommu.h                          |   1 +
 include/linux/irqchip/arm-gic-v3.h                 |   1 +
 include/linux/msi.h                                |   2 +
 kernel/irq/msi.c                                   |  32 +-
 13 files changed, 427 insertions(+), 65 deletions(-)
 create mode 100644 drivers/irqchip/irq-gic-v3-mbi.c
 delete mode 100644 drivers/irqchip/irq-mvebu-gicp.h

Thanks for the series, I gave it a try on Armada-8040-DB:

# cat /proc/interrupts | grep ICU
 28:          0          0          0          0  ICU.f21e0000 107 Level     ahci[f2540000.sata]
 29:          0          0          0          0  ICU.f41e0000 107 Level     ahci[f4540000.sata]
 35:          2          0          0          0  ICU.f21e0000 129 Level   
 36:          3          0          0          0  ICU.f21e0000  41 Level     eth1
 37:          0          0          0          0  ICU.f21e0000  45 Level     eth1
 38:          0          0          4          0  ICU.f21e0000  49 Level     eth1
 39:          0          0          0          2  ICU.f21e0000  53 Level     eth1
 40:        128          0          0          0  ICU.f21e0000  57 Level     eth1
 47:          2          0          0          0  ICU.f41e0000 129 Level   
 54:         82          0          0          0  ICU.f21e0000 106 Level     xhci-hcd:usb3
 55:         82          0          0          0  ICU.f21e0000 105 Level     xhci-hcd:usb5
 56:          4          0          0          0  ICU.f41e0000 106 Level     xhci-hcd:usb7
 57:          0          0          0          0  ICU.f41e0000 105 Level     xhci-hcd:usb1
 58:          0          0          0          0  ICU.f41e0000  77 Level     f4284000.rtc
 59:        224          0          0          0  ICU.f21e0000 120 Level     mv64xxx_i2c
 60:          0          0          0          0  ICU.f41e0000 120 Level     mv64xxx_i2c
 61:         52          0          0          0  ICU.f21e0000  27 Level     mmc1
 63:          0          0          0          0  ICU.f21e0000  22 Level     armada8k-pcie, PCIe PME, aerdrv
 64:          0          0          0          0  ICU.f21e0000  23 Level     armada8k-pcie, PCIe PME, aerdrv
 65:          0          0          0          0  ICU.f41e0000  22 Level     armada8k-pcie, PCIe PME, aerdrv
 66:          0          0          0          0  ICU.f41e0000  24 Level     armada8k-pcie, PCIe PME, aerdrv
 67:          0          0          0          0  ICU.f41e0000  23 Level     armada8k-pcie, PCIe PME, aerdrv

All these interrupts are NSRs and go through the GICP. I was able to use the network (CP0), an I2C bus (CP0) and the USB host ports (on each CP).

Thanks,
Miquèl

Signature · 3 lines
Miquel Raynal, Bootlin (formerly Free Electrons)
Embedded Linux and Kernel engineering
https://bootlin.com