powerpc/pci: Fix PHB removal/rescan on PowerNV
PATCH 1/1 powerpc/pci: Fix PHB removal/rescan on PowerNV
To fix an issue with PHB hotplug on pSeries machine (HPT/XIVE), commit 3a3181e16fbd introduced a PPC specific pcibios_remove_bus() routine to clear all interrupt mappings when the bus is removed. This routine frees an array allocated in pcibios_scan_phb().
This broke PHB hotplug on PowerNV because, when a PHB is removed and re-scanned through sysfs, the PCI layer un-assigns and re-assigns resources to the PHB but does not destroy and recreate the PCI controller structure. Since pcibios_remove_bus() does not clear the 'irq_map' array pointer, a second removal of the PHB will try to free the array a second time and corrupt memory.
Free the 'irq_map' array in pcibios_free_controller() to fix corruption and clear interrupt mapping after it has been disposed. This to avoid filling up the array with successive remove/rescan of a bus.
---
Michael, I am not sure the Fixes tag is required. Feel free to drop it.
--- arch/powerpc/kernel/pci-common.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
arch/powerpc/kernel/pci-common.c +5 -3
@@ -143,6 +143,8 @@ void pcibios_free_controller(struct pci_controller *phb)list_del (&phb->list_node);spin_unlock (&hose_spinlock); +kfree (phb->irq_map); +if (phb->is_dynamic)kfree (phb); }
@@ -450,10 +452,10 @@ static void pci_irq_map_dispose(struct pci_bus *bus)pr_debug ("PCI: Clearing interrupt mappings for PHB %04x:%02x...\n" ,pci_domain_nr (bus), bus->number); -for (i =0 ; i < phb->irq_count; i++) +for (i =0 ; i < phb->irq_count; i++) {irq_dispose_mapping (phb->irq_map[i]); - -kfree (phb->irq_map); + phb->irq_map[i] =0 ; + } }void pcibios_remove_bus (struct pci_bus *bus)
Signature · 1 line
2.25.4
Quoted 6 lines · Cédric Le Goater
To fix an issue with PHB hotplug on pSeries machine (HPT/XIVE), commit
3a3181e16fbdintroduced a PPC specificpcibios_remove_bus()routine to clear all interrupt mappings when the bus is removed. This routine frees an array allocated inpcibios_scan_phb().
Quoted 6 lines
This broke PHB hotplug on PowerNV because, when a PHB is removed and re-scanned through sysfs, the PCI layer un-assigns and re-assigns resources to the PHB but does not destroy and recreate the PCI controller structure. Since
pcibios_remove_bus()does not clear the 'irq_map' array pointer, a second removal of the PHB will try to free the array a second time and corrupt memory.
"PHB hotplug" and "hot-plugging devices under a PHB" are different things. What you're saying here doesn't make a whole lot of sense to me unless you're conflating the two. The distinction is important since on pseries we can use DLPAR to add and remove actual PHBs (i.e. the pci_controller) at runtime, but there's no corresponding mechanism on PowerNV.
Quoted 4 lines
Free the '
irq_map' array inpcibios_free_controller()to fix corruption and clear interrupt mapping after it has been disposed. This to avoid filling up the array with successive remove/rescan of a bus.
Even with this patch I think we're still broken. With this patch applied the init path is something like:
per-phb init:
allocate phb->irq_map array
per-bus init:
nothing
per-device init:
pcibios_bus_add_device()
pci_read_irq_line()
pci_irq_map_register(pci_dev, virq)
*record the device's interrupt in phb->irq_map*And the teardown path:
per-device teardown:
nothing
per-bus teardown:
pcibios_remove_bus()
pci_irq_map_dispose()
*walk phb->irq_map and dispose of each mapped interrupt*
per-phb teardown:
free(phb->irq_map)There's a lot of asymmetry here, which is a problem in itself, but the real issue is that when removing *any* pci_bus under a PHB we dispose of the LSI\ for *every* device under that PHB. Not good.
Ideally we should be fixing this by having the per-device teardown handle disposing the mapping. Unfortunately, there's no pcibios hook that's called when removing a pci_dev. However, we can register a bus notifier which will be called when the pci_dev is removed from its bus and we already do that for the per-device EEH teardown and to handle IOMMU TCE invalidation when the device is removed.
Quoted 14 lines · Oliver O'Halloran
Quoted 6 lines · Cédric Le Goater
To fix an issue with PHB hotplug on pSeries machine (HPT/XIVE), commit
3a3181e16fbdintroduced a PPC specificpcibios_remove_bus()routine to clear all interrupt mappings when the bus is removed. This routine frees an array allocated inpcibios_scan_phb().Quoted 6 lines
This broke PHB hotplug on PowerNV because, when a PHB is removed and re-scanned through sysfs, the PCI layer un-assigns and re-assigns resources to the PHB but does not destroy and recreate the PCI controller structure. Since
pcibios_remove_bus()does not clear the 'irq_map' array pointer, a second removal of the PHB will try to free the array a second time and corrupt memory.
Quoted 6 lines
"PHB hotplug" and "hot-plugging devices under a PHB" are different things. What you're saying here doesn't make a whole lot of sense to me unless you're conflating the two. The distinction is important since on pseries we can use DLPAR to add and remove actual PHBs (i.e. the
pci_controller) at runtime, but there's no corresponding mechanism on PowerNV.
And it's even different on QEMU.
Quoted 33 lines
Quoted 4 lines
Free the '
irq_map' array inpcibios_free_controller()to fix corruption and clear interrupt mapping after it has been disposed. This to avoid filling up the array with successive remove/rescan of a bus.Even with this patch I think we're still broken. With this patch applied the init path is something like:
per-phb init:
allocate phb->irq_map array per-bus init: nothing per-device init: pcibios_bus_add_device() pci_read_irq_line() pci_irq_map_register(pci_dev, virq) *record the device's interrupt in phb->irq_map*And the teardown path:
per-device teardown:
nothing
per-bus teardown:pcibios_remove_bus() pci_irq_map_dispose() *walk phb->irq_map and dispose of each mapped interrupt* per-phb teardown: free(phb->irq_map)There's a lot of asymmetry here, which is a problem in itself, but the real issue is that when removing *any*
pci_busunder a PHB we dispose of the LSI\ for *every* device under that PHB. Not good.
Quoted 6 lines
Ideally we should be fixing this by having the per-device teardown handle disposing the mapping. Unfortunately, there's no pcibios hook that's called when removing a
pci_dev. However, we can register a bus notifier which will be called when thepci_devis removed from its bus and we already do that for the per-device EEH teardown and to handle IOMMU TCE invalidation when the device is removed.
I lack the knowledge here and I think some else should take over, as I am not doing a good job.
Michael, can you drop the initial patch again :/ It is better not to merge anything.
Thanks,
C.
Quoted 15 lines · On Thu, 8 Oct 2020 06:37:02 +0200 Cédric Le Goater <[email protected]> wrote:
Quoted 14 lines · Oliver O'Halloran
Quoted 12 lines · Cédric Le Goater
To fix an issue with PHB hotplug on pSeries machine (HPT/XIVE), commit
3a3181e16fbdintroduced a PPC specificpcibios_remove_bus()routine to clear all interrupt mappings when the bus is removed. This routine frees an array allocated inpcibios_scan_phb().This broke PHB hotplug on PowerNV because, when a PHB is removed and re-scanned through sysfs, the PCI layer un-assigns and re-assigns resources to the PHB but does not destroy and recreate the PCI controller structure. Since
pcibios_remove_bus()does not clear the 'irq_map' array pointer, a second removal of the PHB will try to free the array a second time and corrupt memory.
Quoted 9 lines
Quoted 6 lines
"PHB hotplug" and "hot-plugging devices under a PHB" are different things. What you're saying here doesn't make a whole lot of sense to me unless you're conflating the two. The distinction is important since on pseries we can use DLPAR to add and remove actual PHBs (i.e. the
pci_controller) at runtime, but there's no corresponding mechanism on PowerNV.And it's even different on QEMU.
If the real HW doesn't have the notion of adding/removing a PHB at runtime, then QEMU should stick to that, ie. setting dc->hotpluggable to false for PNV PHB device types.
Quoted 51 lines
Quoted 33 lines
Quoted 4 lines
Free the '
irq_map' array inpcibios_free_controller()to fix corruption and clear interrupt mapping after it has been disposed. This to avoid filling up the array with successive remove/rescan of a bus.Even with this patch I think we're still broken. With this patch applied the init path is something like:
per-phb init:
allocate phb->irq_map array per-bus init: nothing per-device init: pcibios_bus_add_device() pci_read_irq_line() pci_irq_map_register(pci_dev, virq) *record the device's interrupt in phb->irq_map*And the teardown path:
per-device teardown:
nothing
per-bus teardown:pcibios_remove_bus() pci_irq_map_dispose() *walk phb->irq_map and dispose of each mapped interrupt* per-phb teardown: free(phb->irq_map)There's a lot of asymmetry here, which is a problem in itself, but the real issue is that when removing *any*
pci_busunder a PHB we dispose of the LSI\ for *every* device under that PHB. Not good.Quoted 6 lines
Ideally we should be fixing this by having the per-device teardown handle disposing the mapping. Unfortunately, there's no pcibios hook that's called when removing a
pci_dev. However, we can register a bus notifier which will be called when thepci_devis removed from its bus and we already do that for the per-device EEH teardown and to handle IOMMU TCE invalidation when the device is removed.I lack the knowledge here and I think some else should take over, as I am not doing a good job.
Michael, can you drop the initial patch again :/ It is better not to merge anything.
Thanks,
C.