Skip to contentSkip to conversation

powerpc/pci: unmap interrupts when a PHB is removed

PATCH Superseded Cédric Le Goater 2 participants 6 messages Jun 17, 2020 – Aug 7, 2020 PowerPC

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover powerpc/pci: unmap interrupts when a PHB is removed

Hello,

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor expects the guest OS to clear all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails. This is because when the PHBs are scanned, Linux maps automatically some interrupts in the Linux interrupt number space but these are never removed.

To solve this problem, we introduce a PPC platform specific pcibios_remove_bus() routine which clears all interrupt mappings when the bus is removed. This also clears the associated page table entries of the ESB pages when using XIVE.

For this purpose, we record the logical interrupt numbers of the mapped interrupt under the PHB structure and let pcibios_remove_bus() do the clean up.

Tested on :

- PowerNV with PCI, OpenCAPI, CAPI and GPU adapters. I don't know how to inject a failure on a PHB but that would be a good test.
- KVM P8+P9 guests with passthrough PCI adapters, but PHBs can not be removed under QEMU/KVM.
- PowerVM with passthrough PCI adapters (main target)

Thanks,

C.

Changes since v1:

- extended the removal to interrupts other than the legacy INTx.

Cédric Le Goater (2):
powerpc/pci: unmap legacy INTx interrupts when a PHB is removed powerpc/pci: unmap all interrupts when a PHB is removed

 arch/powerpc/include/asm/pci-bridge.h |   6 ++
 arch/powerpc/kernel/pci-common.c      | 114 ++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)
Signature · 1 line
2.25.4
#1 lore raw

PATCH v2 1/2 powerpc/pci: unmap legacy INTx interrupts when a PHB is removed

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor expects the guest OS to clear all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails. This is because when the PHBs are scanned, Linux maps automatically the INTx interrupts in the Linux interrupt number space but these are never removed.

To solve this problem, we introduce a PPC platform specific pcibios_remove_bus() routine which clears all interrupt mappings when the bus is removed. This also clears the associated page table entries of the ESB pages when using XIVE.

For this purpose, we record the logical interrupt numbers of the mapped interrupt under the PHB structure and let pcibios_remove_bus() do the clean up.

---
 arch/powerpc/include/asm/pci-bridge.h |  6 +++
 arch/powerpc/kernel/pci-common.c      | 67 +++++++++++++++++++++++++++
 2 files changed, 73 insertions(+)
arch/powerpc/include/asm/pci-bridge.h +6 -0
index b92e81b256e5..ca75cf264ddf 100644
@@ -48,6 +48,9 @@ struct pci_controller_ops {
 
 /*
  * Structure of a PCI controller (host bridge)
+ *
+ * @irq_count: number of interrupt mappings
+ * @irq_map: interrupt mappings
  */
 struct pci_controller {
 	struct pci_bus *bus;
@@ -127,6 +130,9 @@ struct pci_controller {
 
 	void *private_data;
 	struct npu *npu;
+
+	unsigned int irq_count;
+	unsigned int *irq_map;
 };
 
 /* These are used for config access before all the PCI probing
arch/powerpc/kernel/pci-common.c +67 -0

85 lines not on this page · raw message

Signature · 1 line
2.25.4
#2 lore raw

PATCH v2 2/2 powerpc/pci: unmap all interrupts when a PHB is removed

In reply to Cédric Le Goater

Some PCI adapters, like GPUs, use the "interrupt-map" property to describe interrupt mappings other than the legacy INTx interrupts. There can be more than 4 mappings.

To clear all interrupts when a PHB is removed, we need to increase the 'irq_map' array in which mappings are recorded. Compute the number of interrupt mappings from the "interrupt-map" property and allocate a bigger 'irq_map' array.

---
 arch/powerpc/kernel/pci-common.c | 49 +++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)
arch/powerpc/kernel/pci-common.c +48 -1
index 515480a4bac6..deb831f0ae13 100644
@@ -353,9 +353,56 @@ struct pci_controller *pci_find_controller_for_domain(int domain_nr)
 	return NULL;
 }
 
+/*
+ * Assumption is made on the interrupt parent. All interrupt-map
+ * entries are considered to have the same parent.
+ */
+static int pcibios_irq_map_count(struct pci_controller *phb)
+{
+	const __be32 *imap;
+	int imaplen;
+	struct device_node *parent;
+	u32 intsize, addrsize, parintsize, paraddrsize;
+
+	if (of_property_read_u32(phb->dn, "#interrupt-cells", &intsize))
+		return 0;
+	if (of_property_read_u32(phb->dn, "#address-cells", &addrsize))
+		return 0;
+
+	imap = of_get_property(phb->dn, "interrupt-map", &imaplen);
+	if (!imap) {
+		pr_debug("%pOF : no interrupt-map\n", phb->dn);
+		return 0;
+	}
+	imaplen /= sizeof(u32);
+	pr_debug("%pOF : imaplen=%d\n", phb->dn, imaplen);
+
+	if (imaplen < (addrsize + intsize + 1))
+		return 0;
+
+	imap += intsize + addrsize;
+	parent = of_find_node_by_phandle(be32_to_cpup(imap));
+	if (!parent) {
+		pr_debug("%pOF : no imap parent found !\n", phb->dn);
+		return 0;
+	}
+
+	if (of_property_read_u32(parent, "#interrupt-cells", &parintsize)) {
+		pr_debug("%pOF : parent lacks #interrupt-cells!\n", phb->dn);
+		return 0;
+	}
+
+	if (of_property_read_u32(parent, "#address-cells", &paraddrsize))
+		paraddrsize = 0;
+
+	return imaplen / (addrsize + intsize + 1 + paraddrsize + parintsize);
+}
+
 static void pcibios_irq_map_init(struct pci_controller *phb)
 {
-	phb->irq_count = PCI_NUM_INTX;
+	phb->irq_count = pcibios_irq_map_count(phb);
+	if (phb->irq_count < PCI_NUM_INTX)
+		phb->irq_count = PCI_NUM_INTX;
 
 	pr_debug("%pOF : interrupt map #%d\n", phb->dn, phb->irq_count);
 
Signature · 1 line
2.25.4
#3 lore raw

Re: [PATCH v2 2/2] powerpc/pci: unmap all interrupts when a PHB is removed

Quoted 19 lines · Cédric Le Goater

Some PCI adapters, like GPUs, use the "interrupt-map" property to describe interrupt mappings other than the legacy INTx interrupts. There can be more than 4 mappings.

To clear all interrupts when a PHB is removed, we need to increase the 'irq_map' array in which mappings are recorded. Compute the number of interrupt mappings from the "interrupt-map" property and allocate a bigger 'irq_map' array.

---
 arch/powerpc/kernel/pci-common.c | 49 +++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)
arch/powerpc/kernel/pci-common.c +0 -0
index 515480a4bac6..deb831f0ae13 100644

arch/powerpc/kernel/pci-common.c struct pci_controller *pci_find_controller_for_domain(int domain_nr) · line 360

Quoted 8 lines
arch/powerpc/kernel/pci-common.c +5 -0
@@ -353,9 +353,56 @@ struct pci_controller *pci_find_controller_for_domain(int domain_nr)
 	return NULL;
 }
 
+/*
+ * Assumption is made on the interrupt parent. All interrupt-map
+ * entries are considered to have the same parent.
+ */
+static int pcibios_irq_map_count(struct pci_controller *phb)

I wonder if
int of_irq_count(struct device_node *dev) could work here too. If it does not, then never mind.

Other than that, the only other comment is - merge this one into 1/2 as 1/2 alone won't properly fix the problem but it may look like that it does:

for phyp, the test machine just happens to have 4 entries in the map but this is the phyp implementation detail;

for qemu, there are more but we only unregister 4 but kvm does not care in general so it is ok which is also implementation detail;

and 2/2 just makes these details not matter. Thanks,

Quoted 50 lines
arch/powerpc/kernel/pci-common.c +43 -1
+{
+	const __be32 *imap;
+	int imaplen;
+	struct device_node *parent;
+	u32 intsize, addrsize, parintsize, paraddrsize;
+
+	if (of_property_read_u32(phb->dn, "#interrupt-cells", &intsize))
+		return 0;
+	if (of_property_read_u32(phb->dn, "#address-cells", &addrsize))
+		return 0;
+
+	imap = of_get_property(phb->dn, "interrupt-map", &imaplen);
+	if (!imap) {
+		pr_debug("%pOF : no interrupt-map\n", phb->dn);
+		return 0;
+	}
+	imaplen /= sizeof(u32);
+	pr_debug("%pOF : imaplen=%d\n", phb->dn, imaplen);
+
+	if (imaplen < (addrsize + intsize + 1))
+		return 0;
+
+	imap += intsize + addrsize;
+	parent = of_find_node_by_phandle(be32_to_cpup(imap));
+	if (!parent) {
+		pr_debug("%pOF : no imap parent found !\n", phb->dn);
+		return 0;
+	}
+
+	if (of_property_read_u32(parent, "#interrupt-cells", &parintsize)) {
+		pr_debug("%pOF : parent lacks #interrupt-cells!\n", phb->dn);
+		return 0;
+	}
+
+	if (of_property_read_u32(parent, "#address-cells", &paraddrsize))
+		paraddrsize = 0;
+
+	return imaplen / (addrsize + intsize + 1 + paraddrsize + parintsize);
+}
+
 static void pcibios_irq_map_init(struct pci_controller *phb)
 {
-	phb->irq_count = PCI_NUM_INTX;
+	phb->irq_count = pcibios_irq_map_count(phb);
+	if (phb->irq_count < PCI_NUM_INTX)
+		phb->irq_count = PCI_NUM_INTX;
 
 	pr_debug("%pOF : interrupt map #%d\n", phb->dn, phb->irq_count);
 
Signature · 1 line
Alexey
#4 lore raw

Re: [PATCH v2 2/2] powerpc/pci: unmap all interrupts when a PHB is removed

Quoted 25 lines · Alexey Kardashevskiy
Quoted 22 lines · Cédric Le Goater

Some PCI adapters, like GPUs, use the "interrupt-map" property to describe interrupt mappings other than the legacy INTx interrupts. There can be more than 4 mappings.

To clear all interrupts when a PHB is removed, we need to increase the 'irq_map' array in which mappings are recorded. Compute the number of interrupt mappings from the "interrupt-map" property and allocate a bigger 'irq_map' array.

---
 arch/powerpc/kernel/pci-common.c | 49 +++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)
arch/powerpc/kernel/pci-common.c +0 -0
index 515480a4bac6..deb831f0ae13 100644
@@ -353,9 +353,56 @@ struct pci_controller *pci_find_controller_for_domain(int domain_nr)
 	return NULL;
 }
 
Quoted 9 lines

arch/powerpc/kernel/pci-common.c struct pci_controller *pci_find_controller_for_domain(int domain_nr) · line 360

Quoted 5 lines
arch/powerpc/kernel/pci-common.c +5 -0
+/*
+ * Assumption is made on the interrupt parent. All interrupt-map
+ * entries are considered to have the same parent.
+ */
+static int pcibios_irq_map_count(struct pci_controller *phb)

I wonder if
int of_irq_count(struct device_node *dev) could work here too. If it does not, then never mind.

I wished it would, but no.

Quoted 5 lines

Other than that, the only other comment is - merge this one into 1/2 as 1/2 alone won't properly fix the problem but it may look like that it does:

for phyp, the test machine just happens to have 4 entries in the map but this is the phyp implementation detail;

yes

Quoted 4 lines

for qemu, there are more but we only unregister 4 but kvm does not care in general so it is ok which is also implementation detail;

and 2/2 just makes these details not matter. Thanks,

OK. It will ease backport. Sending a v2.

Thanks for the review Alexey !

C.

Quoted 53 lines
Quoted 52 lines
arch/powerpc/kernel/pci-common.c +43 -1
+{
+	const __be32 *imap;
+	int imaplen;
+	struct device_node *parent;
+	u32 intsize, addrsize, parintsize, paraddrsize;
+
+	if (of_property_read_u32(phb->dn, "#interrupt-cells", &intsize))
+		return 0;
+	if (of_property_read_u32(phb->dn, "#address-cells", &addrsize))
+		return 0;
+
+	imap = of_get_property(phb->dn, "interrupt-map", &imaplen);
+	if (!imap) {
+		pr_debug("%pOF : no interrupt-map\n", phb->dn);
+		return 0;
+	}
+	imaplen /= sizeof(u32);
+	pr_debug("%pOF : imaplen=%d\n", phb->dn, imaplen);
+
+	if (imaplen < (addrsize + intsize + 1))
+		return 0;
+
+	imap += intsize + addrsize;
+	parent = of_find_node_by_phandle(be32_to_cpup(imap));
+	if (!parent) {
+		pr_debug("%pOF : no imap parent found !\n", phb->dn);
+		return 0;
+	}
+
+	if (of_property_read_u32(parent, "#interrupt-cells", &parintsize)) {
+		pr_debug("%pOF : parent lacks #interrupt-cells!\n", phb->dn);
+		return 0;
+	}
+
+	if (of_property_read_u32(parent, "#address-cells", &paraddrsize))
+		paraddrsize = 0;
+
+	return imaplen / (addrsize + intsize + 1 + paraddrsize + parintsize);
+}
+
 static void pcibios_irq_map_init(struct pci_controller *phb)
 {
-	phb->irq_count = PCI_NUM_INTX;
+	phb->irq_count = pcibios_irq_map_count(phb);
+	if (phb->irq_count < PCI_NUM_INTX)
+		phb->irq_count = PCI_NUM_INTX;
 
 	pr_debug("%pOF : interrupt map #%d\n", phb->dn, phb->irq_count);
 
#5 lore raw
In reply to Cédric Le Goater
Quoted 16 lines · Cédric Le Goater

Hello,

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor expects the guest OS to clear all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails. This is because when the PHBs are scanned, Linux maps automatically some interrupts in the Linux interrupt number space but these are never removed.

To solve this problem, we introduce a PPC platform specific pcibios_remove_bus() routine which clears all interrupt mappings when the bus is removed. This also clears the associated page table entries of the ESB pages when using XIVE.

Quoted 8 lines

For this purpose, we record the logical interrupt numbers of the mapped interrupt under the PHB structure and let pcibios_remove_bus() do the clean up.

Tested on :

- PowerNV with PCI, OpenCAPI, CAPI and GPU adapters. I don't know how to inject a failure on a PHB but that would be a good test.

I found out that powering down the slot is enough :

echo 0 > /sys/bus/pci/slots/<slot name>/power

The IRQ cleanup is done as expected on baremetal also.

Cheers,

C.

Quoted 20 lines

- KVM P8+P9 guests with passthrough PCI adapters, but PHBs can not be removed under QEMU/KVM.
- PowerVM with passthrough PCI adapters (main target)

Thanks,

C.

Changes since v1:

- extended the removal to interrupts other than the legacy INTx.

Cédric Le Goater (2):
powerpc/pci: unmap legacy INTx interrupts when a PHB is removed powerpc/pci: unmap all interrupts when a PHB is removed

 arch/powerpc/include/asm/pci-bridge.h |   6 ++
 arch/powerpc/kernel/pci-common.c      | 114 ++++++++++++++++++++++++++
 2 files changed, 120 insertions(+)