Skip to contentSkip to conversation

MSI: Track device proxying when allocating MSIs

PATCH Applied Marc Zyngier 3 participants 8 messages Nov 29, 2020 – Dec 11, 2020

Subsystems (4)
lore
Share & tools
lore

Marc Zyngier

author maintainer
#0 lore raw

cover MSI: Track device proxying when allocating MSIs

This series aims to plug a gap found with John's "managed interrupts for platform devices" series.

We have a couple of cases where we allocate MSIs for a device, but the endpoint generating the actual interrupts (let's call it the "logical device") isn't the one being seen generating the MSIs (the "effective device"):

- PCI devices behind a non transparent bridge
- Devices attached to a wire-to-MSI bridge

For interrupt infrastructures such as the ITS (where the effective device ID is part of the translation), it is important decouple the lifetime of the effective device mapping from the interrupt allocation on behalf of the logical device. Yes, I have a headache too.

To achieve this split, let's tag the such allocations with a new flag that convey the fact that the effective device isn't the logical one. This mostly affects the platform-MSI subsystem.

Subsequent patches add handling of this new flag in the GICv3 ITS (though there may be scope for something more generic in the case of the last patch).

Marc Zyngier (3):
platform-msi: Track shared domain allocation irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy
device
irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI
bridge

 drivers/base/platform-msi.c              |  7 +++++++
 drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
 drivers/irqchip/irq-gic-v3-its.c         |  3 +++
 include/asm-generic/msi.h                |  4 ++++
 4 files changed, 22 insertions(+), 3 deletions(-)
Signature · 1 line
2.28.0
author maintainer
#1 lore raw

PATCH 1/3 platform-msi: Track shared domain allocation

We have two flavours of platform-MSI:

- MSIs generated by devices for themselves (the usual case)

- MSIs generated on behalf of other devices, as the generating device is some form of bridge (either a wire-to-MSI bridge, or even a non-transparent PCI bridge that repaints the PCI requester ID).

In the latter case, the underlying interrupt architecture may need to track this in order to keep the mapping alive even when no MSI are currently being generated.

Add a set of flags to the generic msi_alloc_info_t structure, as well as the MSI_ALLOC_FLAGS_PROXY_DEVICE flag that will get advertized by the platform-MSI code when allocating an irqdomain for a device.

---
 drivers/base/platform-msi.c | 7 +++++++
 include/asm-generic/msi.h   | 4 ++++
 2 files changed, 11 insertions(+)
drivers/base/platform-msi.c +7 -0
index c4a17e5edf8b..2c1e2e0c1a59 100644
@@ -59,9 +59,15 @@ static int platform_msi_init(struct irq_domain *domain,
 	return irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
 					     info->chip, info->chip_data);
 }
+
+static void platform_msi_set_proxy_dev(msi_alloc_info_t *arg)
+{
+	arg->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+}
 #else
 #define platform_msi_set_desc		NULL
 #define platform_msi_init		NULL
+#define platform_msi_set_proxy_dev(x)	do {} while(0)
 #endif
 
 static void platform_msi_update_dom_ops(struct msi_domain_info *info)
@@ -343,6 +349,7 @@ __platform_msi_create_device_domain(struct device *dev,
 	if (!domain)
 		goto free_priv;
 
+	platform_msi_set_proxy_dev(&data->arg);
 	err = msi_domain_prepare_irqs(domain->parent, dev, nvec, &data->arg);
 	if (err)
 		goto free_domain;
include/asm-generic/msi.h +4 -0
index e6795f088bdd..1010e74cb8e0 100644
@@ -22,12 +22,16 @@ struct msi_desc;
 typedef struct msi_alloc_info {
 	struct msi_desc			*desc;
 	irq_hw_number_t			hwirq;
+	unsigned long			flags;
 	union {
 		unsigned long		ul;
 		void			*ptr;
 	} scratchpad[NUM_MSI_ALLOC_SCRATCHPAD_REGS];
 } msi_alloc_info_t;
 
+/* Device generating MSIs is proxying for another device */
+#define MSI_ALLOC_FLAGS_PROXY_DEVICE	(1UL << 0)
+
 #define GENERIC_MSI_DOMAIN_OPS		1
 
 #endif
Signature · 1 line
2.28.0
author maintainer
#3 lore raw

PATCH 2/3 irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy device

In reply to Marc Zyngier

The ITS already has some notion of "shared" devices. Let's map the MSI_ALLOC_FLAGS_PROXY_DEVICE flag onto this internal property.

---
 drivers/irqchip/irq-gic-v3-its.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/irqchip/irq-gic-v3-its.c +3 -0
index 0fec31931e11..b99a8ec6585e 100644
@@ -3488,6 +3488,9 @@ static int its_msi_prepare(struct irq_domain *domain, struct device *dev,
 		goto out;
 	}
 
+	if (info->flags & MSI_ALLOC_FLAGS_PROXY_DEVICE)
+		its_dev->shared = true;
+
 	pr_debug("ITT %d entries, %d bits\n", nvec, ilog2(nvec));
 out:
 	mutex_unlock(&its->dev_alloc_lock);
Signature · 1 line
2.28.0
author maintainer
#5 lore raw

PATCH 3/3 irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI bridge

In reply to Marc Zyngier

An aliasing PCI bridge is another case where we should flag the corresponding allocation as "proxied", as MSIs are coming with the bridge's RID, and not the originating device's.

---
 drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/irqchip/irq-gic-v3-its-pci-msi.c +8 -3
index 87711e0f8014..ad2810c017ed 100644
@@ -67,11 +67,16 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
 	/*
 	 * If pdev is downstream of any aliasing bridges, take an upper
 	 * bound of how many other vectors could map to the same DevID.
+	 * Also tell the ITS that the signalling will come from a proxy
+	 * device, and that special allocation rules apply.
 	 */
 	pci_for_each_dma_alias(pdev, its_get_pci_alias, &alias_dev);
-	if (alias_dev != pdev && alias_dev->subordinate)
-		pci_walk_bus(alias_dev->subordinate, its_pci_msi_vec_count,
-			     &alias_count);
+	if (alias_dev != pdev) {
+		if (alias_dev->subordinate)
+			pci_walk_bus(alias_dev->subordinate,
+				     its_pci_msi_vec_count, &alias_count);
+		info->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+	}
 
 	/* ITS specific DeviceID, as the core ITS ignores dev. */
 	info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);
Signature · 1 line
2.28.0
Tested-by on the v1 series
#7 lore raw
In reply to Marc Zyngier
Quoted 16 lines · Marc Zyngier

This series aims to plug a gap found with John's "managed interrupts for platform devices" series.

We have a couple of cases where we allocate MSIs for a device, but the endpoint generating the actual interrupts (let's call it the "logical device") isn't the one being seen generating the MSIs (the "effective device"):

- PCI devices behind a non transparent bridge
- Devices attached to a wire-to-MSI bridge

For interrupt infrastructures such as the ITS (where the effective device ID is part of the translation), it is important decouple the lifetime of the effective device mapping from the interrupt allocation on behalf of the logical device. Yes, I have a headache too.

Quoted 7 lines

To achieve this split, let's tag the such allocations with a new flag that convey the fact that the effective device isn't the logical one. This mostly affects the platform-MSI subsystem.

Subsequent patches add handling of this new flag in the GICv3 ITS (though there may be scope for something more generic in the case of the last patch).

Hi Marc,

Thanks!

Quoted 14 lines

Marc Zyngier (3):
platform-msi: Track shared domain allocation irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy
device
irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI
bridge

  drivers/base/platform-msi.c              |  7 +++++++
  drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
  drivers/irqchip/irq-gic-v3-its.c         |  3 +++
  include/asm-generic/msi.h                |  4 ++++
  4 files changed, 22 insertions(+), 3 deletions(-)
#4 lore raw

[irqchip: irq/irqchip-next] irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy device

In reply to Marc Zyngier
The following commit has been merged into the irq/irqchip-next branch of irqchip:

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     5fe71d271df8c05e1060c0184764eba18b17a96f
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/5fe71d271df8c05e1060c0184764eba18b17a96f
Author:        Marc Zyngier <[email protected]>
AuthorDate:    Sun, 29 Nov 2020 13:52:07 
Committer:     Marc Zyngier <[email protected]>
CommitterDate: Fri, 11 Dec 2020 14:47:50 

irqchip/gic-v3-its: Tag ITS device as shared if allocating for a proxy device

The ITS already has some notion of "shared" devices. Let's map the MSI_ALLOC_FLAGS_PROXY_DEVICE flag onto this internal property.

---
 drivers/irqchip/irq-gic-v3-its.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/irqchip/irq-gic-v3-its.c +3 -0
index d74ef41..c951ad2 100644
@@ -3487,6 +3487,9 @@ static int its_msi_prepare(struct irq_domain *domain, struct device *dev,
 		goto out;
 	}
 
+	if (info->flags & MSI_ALLOC_FLAGS_PROXY_DEVICE)
+		its_dev->shared = true;
+
 	pr_debug("ITT %d entries, %d bits\n", nvec, ilog2(nvec));
 out:
 	mutex_unlock(&its->dev_alloc_lock);
#6 lore raw

[irqchip: irq/irqchip-next] irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI bridge

In reply to Marc Zyngier
The following commit has been merged into the irq/irqchip-next branch of irqchip:

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     34dd263fce3114147f21698f8e55e05b9e8185bd
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/34dd263fce3114147f21698f8e55e05b9e8185bd
Author:        Marc Zyngier <[email protected]>
AuthorDate:    Sun, 29 Nov 2020 13:52:08 
Committer:     Marc Zyngier <[email protected]>
CommitterDate: Fri, 11 Dec 2020 14:47:50 

irqchip/gic-v3-its: Flag device allocation as proxied if behind a PCI bridge

An aliasing PCI bridge is another case where we should flag the corresponding allocation as "proxied", as MSIs are coming with the bridge's RID, and not the originating device's.

---
 drivers/irqchip/irq-gic-v3-its-pci-msi.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)
drivers/irqchip/irq-gic-v3-its-pci-msi.c +8 -3
index 87711e0..ad2810c 100644
@@ -67,11 +67,16 @@ static int its_pci_msi_prepare(struct irq_domain *domain, struct device *dev,
 	/*
 	 * If pdev is downstream of any aliasing bridges, take an upper
 	 * bound of how many other vectors could map to the same DevID.
+	 * Also tell the ITS that the signalling will come from a proxy
+	 * device, and that special allocation rules apply.
 	 */
 	pci_for_each_dma_alias(pdev, its_get_pci_alias, &alias_dev);
-	if (alias_dev != pdev && alias_dev->subordinate)
-		pci_walk_bus(alias_dev->subordinate, its_pci_msi_vec_count,
-			     &alias_count);
+	if (alias_dev != pdev) {
+		if (alias_dev->subordinate)
+			pci_walk_bus(alias_dev->subordinate,
+				     its_pci_msi_vec_count, &alias_count);
+		info->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+	}
 
 	/* ITS specific DeviceID, as the core ITS ignores dev. */
 	info->scratchpad[0].ul = pci_msi_domain_get_msi_rid(domain, pdev);
#2 lore raw

[irqchip: irq/irqchip-next] platform-msi: Track shared domain allocation

In reply to Marc Zyngier
The following commit has been merged into the irq/irqchip-next branch of irqchip:

The following commit has been merged into the irq/irqchip-next branch of irqchip:

Commit-ID:     91f90daa4fb2b77db7aa25ef2e0206f2e3962665
Gitweb:        https://git.kernel.org/pub/scm/linux/kernel/git/maz/arm-platforms/91f90daa4fb2b77db7aa25ef2e0206f2e3962665
Author:        Marc Zyngier <[email protected]>
AuthorDate:    Sun, 29 Nov 2020 13:52:06 
Committer:     Marc Zyngier <[email protected]>
CommitterDate: Fri, 11 Dec 2020 14:47:50 

platform-msi: Track shared domain allocation

We have two flavours of platform-MSI:

- MSIs generated by devices for themselves (the usual case)

- MSIs generated on behalf of other devices, as the generating device is some form of bridge (either a wire-to-MSI bridge, or even a non-transparent PCI bridge that repaints the PCI requester ID).

In the latter case, the underlying interrupt architecture may need to track this in order to keep the mapping alive even when no MSI are currently being generated.

Add a set of flags to the generic msi_alloc_info_t structure, as well as the MSI_ALLOC_FLAGS_PROXY_DEVICE flag that will get advertized by the platform-MSI code when allocating an irqdomain for a device.

---
 drivers/base/platform-msi.c | 7 +++++++
 include/asm-generic/msi.h   | 4 ++++
 2 files changed, 11 insertions(+)
drivers/base/platform-msi.c +7 -0
index c4a17e5..2c1e2e0 100644
@@ -59,9 +59,15 @@ static int platform_msi_init(struct irq_domain *domain,
 	return irq_domain_set_hwirq_and_chip(domain, virq, hwirq,
 					     info->chip, info->chip_data);
 }
+
+static void platform_msi_set_proxy_dev(msi_alloc_info_t *arg)
+{
+	arg->flags |= MSI_ALLOC_FLAGS_PROXY_DEVICE;
+}
 #else
 #define platform_msi_set_desc		NULL
 #define platform_msi_init		NULL
+#define platform_msi_set_proxy_dev(x)	do {} while(0)
 #endif
 
 static void platform_msi_update_dom_ops(struct msi_domain_info *info)
@@ -343,6 +349,7 @@ __platform_msi_create_device_domain(struct device *dev,
 	if (!domain)
 		goto free_priv;
 
+	platform_msi_set_proxy_dev(&data->arg);
 	err = msi_domain_prepare_irqs(domain->parent, dev, nvec, &data->arg);
 	if (err)
 		goto free_domain;
include/asm-generic/msi.h +4 -0
index e6795f0..1010e74 100644
@@ -22,12 +22,16 @@ struct msi_desc;
 typedef struct msi_alloc_info {
 	struct msi_desc			*desc;
 	irq_hw_number_t			hwirq;
+	unsigned long			flags;
 	union {
 		unsigned long		ul;
 		void			*ptr;
 	} scratchpad[NUM_MSI_ALLOC_SCRATCHPAD_REGS];
 } msi_alloc_info_t;
 
+/* Device generating MSIs is proxying for another device */
+#define MSI_ALLOC_FLAGS_PROXY_DEVICE	(1UL << 0)
+
 #define GENERIC_MSI_DOMAIN_OPS		1
 
 #endif