Skip to contentSkip to conversation

powerpc/xive: PCI hotplug fixes under PowerVM

PATCH In review Cédric Le Goater 4 participants 11 messages Apr 29, 2020 – Jun 10, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover powerpc/xive: PCI hotplug fixes under PowerVM

Hello,

Here are a couple of fixes for PCI hotplug issues for machines running under the POWER hypervisor using hash MMU and the XIVE interrupt mode.

Commit 1ca3dec2b2df ("powerpc/xive: Prevent page fault issues in the machine crash handler") forced the mapping of the XIVE ESB page and this is now blocking the removal of a passthrough IO adapter because the PCI isolation fails with "valid outstanding translations". Under KVM, the ESB pages for the adapter interrupts are un-mapped from the guest by the hypervisor in the KVM XIVE native device. This is is now redundant but it's harmless.

Last is a fix to disable the XIVE debugfs file when XIVE is disabled.

Thanks,

C.

Cédric Le Goater (3):
powerpc/xive: Clear the page tables for the ESB IO mapping powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters powerpc/xive: Do not expose a debugfs file when XIVE is disabled

 arch/powerpc/kernel/pci-hotplug.c |  2 ++
 arch/powerpc/sysdev/xive/common.c | 10 ++++++++--
 2 files changed, 10 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.4
#1 lore raw

PATCH 1/3 powerpc/xive: Clear the page tables for the ESB IO mapping

Commit 1ca3dec2b2df ("powerpc/xive: Prevent page fault issues in the machine crash handler") fixed an issue in the FW assisted dump of machines using hash MMU and the XIVE interrupt mode under the POWER hypervisor. It forced the mapping of the ESB page of interrupts being mapped in the Linux IRQ number space to make sure the 'crash kexec' sequence worked during such an event. But it didn't handle the un-mapping.

This mapping is now blocking the removal of a passthrough IO adapter under the POWER hypervisor because it expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations".

Remove these mapping in the IRQ data cleanup routine.

Under KVM, this cleanup is not required because the ESB pages for the adapter interrupts are un-mapped from the guest by the hypervisor in the KVM XIVE native device. This is now redundant but it's harmless.

Fixes 1ca3dec2b2df ("powerpc/xive: Prevent page fault issues in the machine crash handler")
---
 arch/powerpc/sysdev/xive/common.c | 5 +++++
 1 file changed, 5 insertions(+)
arch/powerpc/sysdev/xive/common.c +5 -0
index 9603b2830d03..3dbc94cb4380 100644
@@ -19,6 +19,7 @@
 #include <linux/slab.h>
 #include <linux/spinlock.h>
 #include <linux/msi.h>
+#include <linux/vmalloc.h>
 
 #include <asm/debugfs.h>
 #include <asm/prom.h>
@@ -1020,12 +1021,16 @@ EXPORT_SYMBOL_GPL(is_xive_irq);
 void xive_cleanup_irq_data(struct xive_irq_data *xd)
 {
 	if (xd->eoi_mmio) {
+		unmap_kernel_range((unsigned long)xd->eoi_mmio,
+				   1u << xd->esb_shift);
 		iounmap(xd->eoi_mmio);
 		if (xd->eoi_mmio == xd->trig_mmio)
 			xd->trig_mmio = NULL;
 		xd->eoi_mmio = NULL;
 	}
 	if (xd->trig_mmio) {
+		unmap_kernel_range((unsigned long)xd->trig_mmio,
+				   1u << xd->esb_shift);
 		iounmap(xd->trig_mmio);
 		xd->trig_mmio = NULL;
 	}
Signature · 1 line
2.25.4
#2 lore raw

PATCH 2/3 powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

In reply to Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

Cc "Oliver O'Halloran" <[email protected]>
---
 arch/powerpc/kernel/pci-hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kernel/pci-hotplug.c +2 -0
index bf83f76563a3..9e9c6befd7ea 100644
@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)
 	struct pci_controller *phb = pci_bus_to_host(dev->bus);
 	struct pci_dn *pdn = pci_get_pdn(dev);
 
+	irq_dispose_mapping(dev->irq);
+
 	eeh_remove_device(dev);
 
 	if (phb->controller_ops.release_device)
Signature · 1 line
2.25.4
#3 lore raw

Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

Quoted 15 lines · Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

Quoted 2 lines
Cc "Oliver O'Halloran" <[email protected]>

We did some tests with differnt passthrough adapters under LPAPRs (PowerVM) and KVM guests P8 (HPT) and P9 (HPT+Radix). Baremetal behaves correctly. But I would feel more comfortable if someone could give a try (PATCH1-2) on a different system.

Thanks,

C.

Quoted 18 lines
---
 arch/powerpc/kernel/pci-hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kernel/pci-hotplug.c +2 -0
index bf83f76563a3..9e9c6befd7ea 100644
@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)
 	struct pci_controller *phb = pci_bus_to_host(dev->bus);
 	struct pci_dn *pdn = pci_get_pdn(dev);
 
+	irq_dispose_mapping(dev->irq);
+
 	eeh_remove_device(dev);
 
 	if (phb->controller_ops.release_device)
#4 lore raw

Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

In reply to Cédric Le Goater
Quoted 27 lines · Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

Cc "Oliver O'Halloran" <[email protected]>
---
 arch/powerpc/kernel/pci-hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kernel/pci-hotplug.c +0 -0
index bf83f76563a3..9e9c6befd7ea 100644

arch/powerpc/kernel/pci-hotplug.c void pcibios_release_device(struct pci_dev *dev) · line 60

Quoted 4 lines
arch/powerpc/kernel/pci-hotplug.c +1 -0
@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)
        struct pci_controller *phb = pci_bus_to_host(dev->bus);
        struct pci_dn *pdn = pci_get_pdn(dev);
 
+       irq_dispose_mapping(dev->irq);

What does the original mapping? Powerpc arch code or the PCI core? Tearing down the mapping in pcibios_release_device() seems a bit fishy to me since the PCI core has already torn down the device state at that point. If the release is delayed it's possible that another pci_dev has mapped the IRQ before we get here, but maybe that's ok.

Quoted 7 lines
arch/powerpc/kernel/pci-hotplug.c +1 -1
+
        eeh_remove_device(dev);
 
        if (phb->controller_ops.release_device)
--

2.25.4

#5 lore raw

Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

Quoted 31 lines · Oliver O'Halloran
Quoted 30 lines · Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

Cc "Oliver O'Halloran" <[email protected]>
---
 arch/powerpc/kernel/pci-hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kernel/pci-hotplug.c +0 -0
index bf83f76563a3..9e9c6befd7ea 100644
@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)
        struct pci_controller *phb = pci_bus_to_host(dev->bus);
        struct pci_dn *pdn = pci_get_pdn(dev);
 
Quoted 3 lines

arch/powerpc/kernel/pci-hotplug.c void pcibios_release_device(struct pci_dev *dev) · line 60

Quoted 1 line
arch/powerpc/kernel/pci-hotplug.c +1 -0
+       irq_dispose_mapping(dev->irq);

What does the original mapping? Powerpc arch code or the PCI core?

Powerpc. In pci_read_irq_line() when a device is added.

Quoted 4 lines

Tearing down the mapping in pcibios_release_device() seems a bit fishy to me since the PCI core has already torn down the device state at that point. If the release is delayed it's possible that another pci_dev has mapped the IRQ before we get here, but maybe that's ok.

Which scenario would that be ? multiple devices mapping the same INTx interrupt because all are used already ?

Where should we drop the mapping ?

Thanks,

C.

Quoted 7 lines
Quoted 7 lines
arch/powerpc/kernel/pci-hotplug.c +1 -1
+
        eeh_remove_device(dev);
 
        if (phb->controller_ops.release_device)
--

2.25.4

#6 lore raw

Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

In reply to Oliver O'Halloran
Quoted 31 lines · Oliver O'Halloran
Quoted 30 lines · Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

Cc "Oliver O'Halloran" <[email protected]>
---
 arch/powerpc/kernel/pci-hotplug.c | 2 ++
 1 file changed, 2 insertions(+)
arch/powerpc/kernel/pci-hotplug.c +0 -0
index bf83f76563a3..9e9c6befd7ea 100644
@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)
        struct pci_controller *phb = pci_bus_to_host(dev->bus);
        struct pci_dn *pdn = pci_get_pdn(dev);
 
Quoted 7 lines

arch/powerpc/kernel/pci-hotplug.c void pcibios_release_device(struct pci_dev *dev) · line 60

Quoted 1 line
arch/powerpc/kernel/pci-hotplug.c +1 -0
+       irq_dispose_mapping(dev->irq);

What does the original mapping? Powerpc arch code or the PCI core? Tearing down the mapping in pcibios_release_device() seems a bit fishy to me since the PCI core has already torn down the device state at that point. If the release is delayed it's possible that another pci_dev has mapped the IRQ before we get here, but maybe that's ok.

How's that below ? INTx mappings are cleared only when the PHB is removed. It applies to all platforms but we could limit the removal to PHB hotplug on pseries.

C.

From 10794159567552355f87e86e24002641c54e7ab5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= <[email protected]>
Date: Wed, 10 Jun 2020 19:55:24 +0200
Subject: [PATCH] powerpc/pci: unmap legacy INTx interrupts of passthrough IO
adapters
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the ESB pages are cleared.

---
 arch/powerpc/include/asm/pci-bridge.h |  4 +++
 arch/powerpc/kernel/pci-common.c      | 47 +++++++++++++++++++++++++++
 2 files changed, 51 insertions(+)
arch/powerpc/include/asm/pci-bridge.h +4 -0
index b92e81b256e5..9960dd249079 100644
@@ -48,6 +48,8 @@ struct pci_controller_ops {
 
 /*
  * Structure of a PCI controller (host bridge)
+ *
+ * @intx: legacy INTx mappings
  */
 struct pci_controller {
 	struct pci_bus *bus;
@@ -127,6 +129,8 @@ struct pci_controller {
 
 	void *private_data;
 	struct npu *npu;
+
+	unsigned int intx[PCI_NUM_INTX];
 };
 
 /* These are used for config access before all the PCI probing
arch/powerpc/kernel/pci-common.c +47 -0

59 lines not on this page · raw message

Signature · 1 line
2.25.4
#7 lore raw

Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

In reply to Cédric Le Goater

Hello Michael,

Quoted 7 lines · Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

Quoted 7 lines

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

It seems this patch needs more digging to make sure we are handling the IRQ unmapping in the correct PCI handler. Could you please keep it back for the moment ?

Thanks,

C.

#8 lore raw

Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters

Quoted 16 lines · Cédric Le Goater
Quoted 7 lines · Cédric Le Goater

When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.

Quoted 7 lines

INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.

Quoted 3 lines

It seems this patch needs more digging to make sure we are handling the IRQ unmapping in the correct PCI handler. Could you please keep it back for the moment ?

Yep no worries.

cheers

#9 lore raw

PATCH 3/3 powerpc/xive: Do not expose a debugfs file when XIVE is disabled

In reply to Cédric Le Goater

The XIVE interrupt mode can be disabled with the "xive=off" kernel parameter, in which case there is nothing to present to the user in the associated /sys/kernel/debug/powerpc/xive file.

Fixes 930914b7d528 ("powerpc/xive: Add a debugfs file to dump internal XIVE state")
---
 arch/powerpc/sysdev/xive/common.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
arch/powerpc/sysdev/xive/common.c +3 -2
index 3dbc94cb4380..f591be9f01f4 100644
@@ -1664,7 +1664,8 @@ DEFINE_SHOW_ATTRIBUTE(xive_core_debug);
 
 int xive_core_debug_init(void)
 {
-	debugfs_create_file("xive", 0400, powerpc_debugfs_root,
-			    NULL, &xive_core_debug_fops);
+	if (xive_enabled())
+		debugfs_create_file("xive", 0400, powerpc_debugfs_root,
+				    NULL, &xive_core_debug_fops);
 	return 0;
 }
Signature · 1 line
2.25.4
#10 lore raw
In reply to Cédric Le Goater
Quoted 11 lines · Cédric Le Goater

Here are a couple of fixes for PCI hotplug issues for machines running under the POWER hypervisor using hash MMU and the XIVE interrupt mode.

Commit 1ca3dec2b2df ("powerpc/xive: Prevent page fault issues in the machine crash handler") forced the mapping of the XIVE ESB page and this is now blocking the removal of a passthrough IO adapter because the PCI isolation fails with "valid outstanding translations". Under KVM, the ESB pages for the adapter interrupts are un-mapped from the guest by the hypervisor in the KVM XIVE native device. This is is now redundant but it's harmless.

Quoted 1 line

[...]

Patches 1 & 3 pplied to powerpc/next.

[1/3] powerpc/xive: Clear the page tables for the ESB IO mapping

      https://git.kernel.org/powerpc/c/a101950fcb78b0ba20cd487be6627dea58d55c2b
[3/3] powerpc/xive: Do not expose a debugfs file when XIVE is disabled
      https://git.kernel.org/powerpc/c/0755e85570a4615ca674ad6489d44d63916f1f3e

cheers