powerpc/xive: PCI hotplug fixes under PowerVM
Subsystems (2)
cover powerpc/xive: PCI hotplug fixes under PowerVM
Hello,
Here are a couple of fixes for PCI hotplug issues for machines running under the POWER hypervisor using hash MMU and the XIVE interrupt mode.
Commit 1ca3dec2b2df ("powerpc/xive: Prevent page fault issues in the machine crash handler") forced the mapping of the XIVE ESB page and this is now blocking the removal of a passthrough IO adapter because the PCI isolation fails with "valid outstanding translations". Under KVM, the ESB pages for the adapter interrupts are un-mapped from the guest by the hypervisor in the KVM XIVE native device. This is is now redundant but it's harmless.
Last is a fix to disable the XIVE debugfs file when XIVE is disabled.
Thanks,
C.
Cédric Le Goater (3):
powerpc/xive: Clear the page tables for the ESB IO mapping powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters powerpc/xive: Do not expose a debugfs file when XIVE is disabled
arch/powerpc/kernel/pci-hotplug.c | 2 ++ arch/powerpc/sysdev/xive/common.c | 10 ++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
Signature · 1 line
2.25.4
PATCH 1/3 powerpc/xive: Clear the page tables for the ESB IO mapping
Commit 1ca3dec2b2df ("powerpc/xive: Prevent page fault issues in the machine crash handler") fixed an issue in the FW assisted dump of machines using hash MMU and the XIVE interrupt mode under the POWER hypervisor. It forced the mapping of the ESB page of interrupts being mapped in the Linux IRQ number space to make sure the 'crash kexec' sequence worked during such an event. But it didn't handle the un-mapping.
This mapping is now blocking the removal of a passthrough IO adapter under the POWER hypervisor because it expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations".
Remove these mapping in the IRQ data cleanup routine.
Under KVM, this cleanup is not required because the ESB pages for the adapter interrupts are un-mapped from the guest by the hypervisor in the KVM XIVE native device. This is now redundant but it's harmless.
--- arch/powerpc/sysdev/xive/common.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/powerpc/sysdev/xive/common.c +5 -0
@@ -19,6 +19,7 @@#include <linux/slab.h> #include <linux/spinlock.h> #include <linux/msi.h> +#include <linux/vmalloc.h> #include <asm/debugfs.h> #include <asm/prom.h>
@@ -1020,12 +1021,16 @@ EXPORT_SYMBOL_GPL(is_xive_irq);void xive_cleanup_irq_data (struct xive_irq_data *xd) {if (xd->eoi_mmio) { +unmap_kernel_range ((unsigned long )xd->eoi_mmio, +1u << xd->esb_shift);iounmap (xd->eoi_mmio);if (xd->eoi_mmio == xd->trig_mmio) xd->trig_mmio =NULL ; xd->eoi_mmio =NULL ; }if (xd->trig_mmio) { +unmap_kernel_range ((unsigned long )xd->trig_mmio, +1u << xd->esb_shift);iounmap (xd->trig_mmio); xd->trig_mmio =NULL ; }
Signature · 1 line
2.25.4
PATCH 2/3 powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
In reply to Cédric Le GoaterWhen a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
--- arch/powerpc/kernel/pci-hotplug.c | 2 ++ 1 file changed, 2 insertions(+)
arch/powerpc/kernel/pci-hotplug.c +2 -0
@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)struct pci_controller *phb =pci_bus_to_host (dev->bus);struct pci_dn *pdn =pci_get_pdn (dev); +irq_dispose_mapping (dev->irq); +eeh_remove_device (dev);if (phb->controller_ops.release_device)
Signature · 1 line
2.25.4
Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
Quoted 15 lines · Cédric Le Goater
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
Quoted 2 lines
Cc "Oliver O'Halloran" <[email protected]>Signed-off-by Cédric Le Goater <[email protected]>
We did some tests with differnt passthrough adapters under LPAPRs (PowerVM) and KVM guests P8 (HPT) and P9 (HPT+Radix). Baremetal behaves correctly. But I would feel more comfortable if someone could give a try (PATCH1-2) on a different system.
Thanks,
C.
Quoted 18 lines
--- arch/powerpc/kernel/pci-hotplug.c | 2 ++ 1 file changed, 2 insertions(+)arch/powerpc/kernel/pci-hotplug.c +2
-0@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)struct pci_controller *phb =pci_bus_to_host (dev->bus);struct pci_dn *pdn =pci_get_pdn (dev); +irq_dispose_mapping (dev->irq); +eeh_remove_device (dev);if (phb->controller_ops.release_device)
Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
In reply to Cédric Le GoaterQuoted 27 lines · Cédric Le Goater
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
Cc "Oliver O'Halloran" <[email protected]>Signed-off-by Cédric Le Goater <[email protected]>--- arch/powerpc/kernel/pci-hotplug.c | 2 ++ 1 file changed, 2 insertions(+)arch/powerpc/kernel/pci-hotplug.c +0
-0
arch/powerpc/kernel/pci-hotplug.c void pcibios_release_device(struct pci_dev *dev) · line 60
Quoted 4 lines
arch/powerpc/kernel/pci-hotplug.c +1
-0@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)struct pci_controller *phb =pci_bus_to_host (dev->bus);struct pci_dn *pdn =pci_get_pdn (dev); +irq_dispose_mapping (dev->irq);
What does the original mapping? Powerpc arch code or the PCI core? Tearing down the mapping in pcibios_release_device() seems a bit fishy to me since the PCI core has already torn down the device state at that point. If the release is delayed it's possible that another pci_dev has mapped the IRQ before we get here, but maybe that's ok.
Quoted 7 lines
arch/powerpc/kernel/pci-hotplug.c +1
-1+eeh_remove_device (dev);if (phb->controller_ops.release_device) --2.25.4
Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
Quoted 31 lines · Oliver O'Halloran
Quoted 30 lines · Cédric Le Goater
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
Cc "Oliver O'Halloran" <[email protected]>Signed-off-by Cédric Le Goater <[email protected]>--- arch/powerpc/kernel/pci-hotplug.c | 2 ++ 1 file changed, 2 insertions(+)arch/powerpc/kernel/pci-hotplug.c +0
-0@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)struct pci_controller *phb =pci_bus_to_host (dev->bus);struct pci_dn *pdn =pci_get_pdn (dev);
Quoted 3 lines
arch/powerpc/kernel/pci-hotplug.c void pcibios_release_device(struct pci_dev *dev) · line 60
Quoted 1 line
arch/powerpc/kernel/pci-hotplug.c +1
-0+irq_dispose_mapping (dev->irq);What does the original mapping? Powerpc arch code or the PCI core?
Powerpc. In pci_read_irq_line() when a device is added.
Quoted 4 lines
Tearing down the mapping in
pcibios_release_device()seems a bit fishy to me since the PCI core has already torn down the device state at that point. If the release is delayed it's possible that anotherpci_devhas mapped the IRQ before we get here, but maybe that's ok.
Which scenario would that be ? multiple devices mapping the same INTx interrupt because all are used already ?
Where should we drop the mapping ?
Thanks,
C.
Quoted 7 lines
Quoted 7 lines
arch/powerpc/kernel/pci-hotplug.c +1
-1+eeh_remove_device (dev);if (phb->controller_ops.release_device) --2.25.4
Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
In reply to Oliver O'HalloranQuoted 31 lines · Oliver O'Halloran
Quoted 30 lines · Cédric Le Goater
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
Cc "Oliver O'Halloran" <[email protected]>Signed-off-by Cédric Le Goater <[email protected]>--- arch/powerpc/kernel/pci-hotplug.c | 2 ++ 1 file changed, 2 insertions(+)arch/powerpc/kernel/pci-hotplug.c +0
-0@@ -57,6 +57,8 @@ void pcibios_release_device(struct pci_dev *dev)struct pci_controller *phb =pci_bus_to_host (dev->bus);struct pci_dn *pdn =pci_get_pdn (dev);
Quoted 7 lines
arch/powerpc/kernel/pci-hotplug.c void pcibios_release_device(struct pci_dev *dev) · line 60
Quoted 1 line
arch/powerpc/kernel/pci-hotplug.c +1
-0+irq_dispose_mapping (dev->irq);What does the original mapping? Powerpc arch code or the PCI core? Tearing down the mapping in
pcibios_release_device()seems a bit fishy to me since the PCI core has already torn down the device state at that point. If the release is delayed it's possible that anotherpci_devhas mapped the IRQ before we get here, but maybe that's ok.
How's that below ? INTx mappings are cleared only when the PHB is removed. It applies to all platforms but we could limit the removal to PHB hotplug on pseries.
C.
From 10794159567552355f87e86e24002641c54e7ab5 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= <[email protected]>
Date: Wed, 10 Jun 2020 19:55:24 +0200
Subject: [PATCH] powerpc/pci: unmap legacy INTx interrupts of passthrough IO
adapters
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the ESB pages are cleared.
--- arch/powerpc/include/asm/pci-bridge.h | 4 +++ arch/powerpc/kernel/pci-common.c | 47 +++++++++++++++++++++++++++ 2 files changed, 51 insertions(+)
arch/powerpc/include/asm/pci-bridge.h +4 -0
@@ -48,6 +48,8 @@ struct pci_controller_ops {/* * Structure of a PCI controller (host bridge) +* +* @intx: legacy INTx mappings */ struct pci_controller {struct pci_bus *bus;
@@ -127,6 +129,8 @@ struct pci_controller {void *private_data;struct npu *npu; + +unsigned int intx[PCI_NUM_INTX ]; };/* These are used for config access before all the PCI probing
arch/powerpc/kernel/pci-common.c +47 -0
59 lines not on this page · raw message
Signature · 1 line
2.25.4
Re: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
In reply to Cédric Le GoaterHello Michael,
Quoted 7 lines · Cédric Le Goater
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
Quoted 7 lines
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
It seems this patch needs more digging to make sure we are handling the IRQ unmapping in the correct PCI handler. Could you please keep it back for the moment ?
Thanks,
C.
Michael Ellerman
reviewerRe: [PATCH 2/3] powerpc/pci: unmap legacy INTx interrupts of passthrough IO adapters
Quoted 16 lines · Cédric Le Goater
Quoted 7 lines · Cédric Le Goater
When a passthrough IO adapter is removed from a pseries machine using hash MMU and the XIVE interrupt mode, the POWER hypervisor, pHyp, expects the guest OS to have cleared all page table entries related to the adapter. If some are still present, the RTAS call which isolates the PCI slot returns error 9001 "valid outstanding translations" and the removal of the IO adapter fails.
Quoted 7 lines
INTx interrupt numbers need special care because Linux maps the interrupts automatically in the Linux interrupt number space if they are presented in the device tree node describing the IO adapter. These interrupts are not un-mapped automatically and in case of an hot-plug adapter, the PCI hot-plug layer needs to handle the cleanup to make sure that all the page table entries of the XIVE ESB pages are cleared.
Quoted 3 lines
It seems this patch needs more digging to make sure we are handling the IRQ unmapping in the correct PCI handler. Could you please keep it back for the moment ?
Yep no worries.
cheers
PATCH 3/3 powerpc/xive: Do not expose a debugfs file when XIVE is disabled
In reply to Cédric Le GoaterThe XIVE interrupt mode can be disabled with the "xive=off" kernel parameter, in which case there is nothing to present to the user in the associated /sys/kernel/debug/powerpc/xive file.
--- arch/powerpc/sysdev/xive/common.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
arch/powerpc/sysdev/xive/common.c +3 -2
@@ -1664,7 +1664,8 @@ DEFINE_SHOW_ATTRIBUTE(xive_core_debug);int xive_core_debug_init (void ) { -debugfs_create_file ("xive" ,0400 , powerpc_debugfs_root, -NULL , &xive_core_debug_fops); +if (xive_enabled ()) +debugfs_create_file ("xive" ,0400 , powerpc_debugfs_root, +NULL , &xive_core_debug_fops);return 0 ; }
Signature · 1 line
2.25.4
Quoted 11 lines · Cédric Le Goater
Here are a couple of fixes for PCI hotplug issues for machines running under the POWER hypervisor using hash MMU and the XIVE interrupt mode.
Commit
1ca3dec2b2df("powerpc/xive: Prevent page fault issues in the machine crash handler") forced the mapping of the XIVE ESB page and this is now blocking the removal of a passthrough IO adapter because the PCI isolation fails with "valid outstanding translations". Under KVM, the ESB pages for the adapter interrupts are un-mapped from the guest by the hypervisor in the KVM XIVE native device. This is is now redundant but it's harmless.
Quoted 1 line
[...]
Patches 1 & 3 pplied to powerpc/next.
[1/3] powerpc/xive: Clear the page tables for the ESB IO mapping
https://git.kernel.org/powerpc/c/a101950fcb78b0ba20cd487be6627dea58d55c2b [3/3] powerpc/xive: Do not expose a debugfs file when XIVE is disabled https://git.kernel.org/powerpc/c/0755e85570a4615ca674ad6489d44d63916f1f3e
cheers