opencapi: enable card reset and link retraining
Subsystems (4)
v2 cover opencapi: enable card reset and link retraining
This is the linux part of the work to use the PCI hotplug framework to control an opencapi card so that it can be reset and re-read after flashing a new FPGA image. The changes required in skiboot are now upstream. On an old skiboot, this series will do nothing.
A virtual PCI slot is created for the opencapi adapter and its state can be controlled through the pnv-php hotplug driver:
echo 0|1 > /sys/bus/pci/slots/OPENCAPI-<...>/power
Note that the power to the card is not really turned off, as the card needs to stay on to be flashed with a new image. Instead the card is in reset.
The first part of the series mostly deals with the pci/ioda state, as the opencapi devices can now go away and the state needs to be cleaned up.
The second part is modifications to the PCI hotplug driver on powernv, so that a virtual slot is created for the opencapi adapters found in the device tree.
Changelog:
v2:
- rebase on latest kernel
- clarify the ref counting done for NPU devices when the PE is setup
- address comments from Andrew and Alastair
Frederic Barrat (11):
powerpc/powernv/ioda: Fix ref count for devices with their own PE powerpc/powernv/ioda: Protect PE list powerpc/powernv/ioda: set up PE on opencapi device when enabling powerpc/powernv/ioda: Release opencapi device powerpc/powernv/ioda: Find opencapi slot for a device node pci/hotplug/pnv-php: Remove erroneous warning pci/hotplug/pnv-php: Improve error msg on power state change failure pci/hotplug/pnv-php: Register opencapi slots pci/hotplug/pnv-php: Relax check when disabling slot pci/hotplug/pnv-php: Wrap warnings in macro
ocxl: Add PCI hotplug dependency to Kconfig
arch/powerpc/include/asm/pnv-pci.h | 1 + arch/powerpc/platforms/powernv/pci-ioda.c | 115 +++++++++++++++------- arch/powerpc/platforms/powernv/pci.c | 24 +++-- drivers/misc/ocxl/Kconfig | 1 + drivers/pci/hotplug/pnv_php.c | 82 ++++++++------- 5 files changed, 145 insertions(+), 78 deletions(-)
Signature · 1 line
2.21.0
PATCH v2 1/11 powerpc/powernv/ioda: Fix ref count for devices with their own PE
The pci_dn structure used to store a pointer to the struct pci_dev, so taking a reference on the device was required. However, the pci_dev pointer was later removed from the pci_dn structure, but the reference was kept for the npu device.
See commit 902bdc57451c ("powerpc/powernv/idoa: Remove unnecessary pcidev from pci_dn").
We don't need to take a reference on the device when assigning the PE as the struct pnv_ioda_pe is cleaned up at the same time as the (physical) device is released. Doing so prevents the device from being released, which is a problem for opencapi devices, since we want to be able to remove them through PCI hotplug.
Now the ugly part: nvlink npu devices are not meant to be released. Because of the above, we've always leaked a reference and simply removing it now is dangerous and would likely require more work. There's currently no release device callback for nvlink devices for example. So to be safe, this patch leaks a reference on the npu device, but only for nvlink and not opencapi.
---
Changelog:
v2:
- clarify the ref counting (and leak) done on npu devices when setting up the PE
- rework commit message
arch/powerpc/platforms/powernv/pci-ioda.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
arch/powerpc/platforms/powernv/pci-ioda.c +12 -7
@@ -1062,14 +1062,13 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)return NULL ; } -/* NOTE: We get only one ref to the pci_dev for the pdn, not for the -* pointer in the PE data structure, both should be destroyed at the -* same time. However, this needs to be looked at more closely again -* once we actually start removing things (Hotplug, SR-IOV, ...) +/* NOTE: We don't get a reference for the pointer in the PE +* data structure, both the device and PE structures should be +* destroyed at the same time. However, removing nvlink +* devices will need some work. * * At some point we want to remove the PDN completely anyways */ -pci_dev_get (dev); pdn->pe_number = pe->pe_number; pe->flags =PNV_IODA_PE_DEV ; pe->pdev = dev;
@@ -1084,7 +1083,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev)pnv_ioda_free_pe (pe); pdn->pe_number =IODA_INVALID_PE ; pe->pdev =NULL ; -pci_dev_put (dev);return NULL ; }
@@ -1205,6 +1203,14 @@ static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)struct pci_controller *hose =pci_bus_to_host (npu_pdev->bus);struct pnv_phb *phb = hose->private_data; +/* +* Intentionally leak a reference on the npu device (for +* nvlink only; this is not an opencapi path) to make sure it +* never goes away, as it's been the case all along and some +* work is needed otherwise. +*/ +pci_dev_get (npu_pdev); +/* * Due to a hardware errata PE#0 on the NPU is reserved for * error handling. This means we only have three PEs remaining
@@ -1228,7 +1234,6 @@ static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev)*/ dev_info (&npu_pdev->dev,"Associating to existing PE %x\n" , pe_num); -pci_dev_get (npu_pdev); npu_pdn =pci_get_pdn (npu_pdev); rid = npu_pdev->bus->number <<8 | npu_pdn->devfn; npu_pdn->pe_number = pe_num;
Signature · 1 line
2.21.0
PATCH v2 2/11 powerpc/powernv/ioda: Protect PE list
In reply to Frederic BarratProtect the PHB's list of PE. Probably not needed as long as it was populated during PHB creation, but it feels right and will become required once we can add/remove opencapi devices on hotplug.
---
Changelog:
v2: no change
arch/powerpc/platforms/powernv/pci-ioda.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
arch/powerpc/platforms/powernv/pci-ioda.c +5 -1
@@ -1087,8 +1087,9 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev) }/* Put PE to the list */ +mutex_lock (&phb->ioda.pe_list_mutex);list_add_tail (&pe->list, &phb->ioda.pe_list); - +mutex_unlock (&phb->ioda.pe_list_mutex);return pe; }
@@ -3517,7 +3518,10 @@ static void pnv_ioda_release_pe(struct pnv_ioda_pe *pe)struct pnv_phb *phb = pe->phb;struct pnv_ioda_pe *slave, *tmp; +mutex_lock (&phb->ioda.pe_list_mutex);list_del (&pe->list); +mutex_unlock (&phb->ioda.pe_list_mutex); +switch (phb->type) {case PNV_PHB_IODA1 :pnv_pci_ioda1_release_pe_dma (pe);
Signature · 1 line
2.21.0
PATCH v2 4/11 powerpc/powernv/ioda: Release opencapi device
In reply to Frederic BarratWith hotplug, an opencapi device can now go away. It needs to be released, mostly to clean up its PE state. We were previously not defining any device callback. We can reuse the standard PCI release callback, it does a bit too much for an opencapi device, but it's harmless, and only needs minor tuning.
Also separate the undo of the PELT-V code in a separate function, it is not needed for NPU devices and it improves a bit the readability of the code.
---
Changelog:
v2:
- start using pe->device_count for NPU devices to match expectations in pnv_pci_release_device (and because it makes sense)
arch/powerpc/platforms/powernv/pci-ioda.c | 59 +++++++++++++++-------- 1 file changed, 40 insertions(+), 19 deletions(-)
arch/powerpc/platforms/powernv/pci-ioda.c +40 -19
@@ -188,7 +188,7 @@ static void pnv_ioda_free_pe(struct pnv_ioda_pe *pe)unsigned int pe_num = pe->pe_number;WARN_ON (pe->pdev); -WARN_ON (pe->npucomp);/* NPUs are not supposed to be freed */ +WARN_ON (pe->npucomp);/* NPUs for nvlink are not supposed to be freed */ kfree (pe->npucomp);memset (pe,0 ,sizeof (struct pnv_ioda_pe ));clear_bit (pe_num, phb->ioda.pe_alloc);
@@ -777,6 +777,34 @@ static int pnv_ioda_set_peltv(struct pnv_phb *phb,return 0 ; } +static void pnv_ioda_unset_peltv (struct pnv_phb *phb, +struct pnv_ioda_pe *pe, +struct pci_dev *parent) +{ +int64_t rc; + +while (parent) { +struct pci_dn *pdn =pci_get_pdn (parent); + +if (pdn && pdn->pe_number !=IODA_INVALID_PE ) { + rc =opal_pci_set_peltv (phb->opal_id, pdn->pe_number, + pe->pe_number, +OPAL_REMOVE_PE_FROM_DOMAIN ); +/* XXX What to do in case of error ? */ + } + parent = parent->bus->self; + } + +opal_pci_eeh_freeze_clear (phb->opal_id, pe->pe_number, +OPAL_EEH_ACTION_CLEAR_FREEZE_ALL ); + +/* Disassociate PE in PELT */ + rc =opal_pci_set_peltv (phb->opal_id, pe->pe_number, + pe->pe_number,OPAL_REMOVE_PE_FROM_DOMAIN ); +if (rc) +pe_warn (pe,"OPAL error %lld remove self from PELTV\n" , rc); +} +static int pnv_ioda_deconfigure_pe (struct pnv_phb *phb,struct pnv_ioda_pe *pe) {struct pci_dev *parent;
@@ -827,25 +855,13 @@ static int pnv_ioda_deconfigure_pe(struct pnv_phb *phb, struct pnv_ioda_pe *pe)for (rid = pe->rid; rid < rid_end; rid++) phb->ioda.pe_rmap[rid] =IODA_INVALID_PE ; -/* Release from all parents PELT-V */ -while (parent) { -struct pci_dn *pdn =pci_get_pdn (parent); -if (pdn && pdn->pe_number !=IODA_INVALID_PE ) { - rc =opal_pci_set_peltv (phb->opal_id, pdn->pe_number, - pe->pe_number,OPAL_REMOVE_PE_FROM_DOMAIN ); -/* XXX What to do in case of error ? */ - } - parent = parent->bus->self; - } - -opal_pci_eeh_freeze_clear (phb->opal_id, pe->pe_number, -OPAL_EEH_ACTION_CLEAR_FREEZE_ALL ); +/* +* Release from all parents PELT-V. NPUs don't have a PELTV +* table +*/ +if (phb->type !=PNV_PHB_NPU_NVLINK && phb->type !=PNV_PHB_NPU_OCAPI ) +pnv_ioda_unset_peltv (phb, pe, parent); -/* Disassociate PE in PELT */ - rc =opal_pci_set_peltv (phb->opal_id, pe->pe_number, - pe->pe_number,OPAL_REMOVE_PE_FROM_DOMAIN ); -if (rc) -pe_warn (pe,"OPAL error %lld remove self from PELTV\n" , rc); rc =opal_pci_set_pe (phb->opal_id, pe->pe_number, pe->rid, bcomp, dcomp, fcomp,OPAL_UNMAP_PE );if (rc)
@@ -1075,6 +1091,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_dev_PE(struct pci_dev *dev) pe->pbus =NULL ; pe->mve_number =-1 ; pe->rid = dev->bus->number <<8 | pdn->devfn; + pe->device_count++;pe_info (pe,"Associated device to PE\n" );
@@ -1239,6 +1256,7 @@ static struct pnv_ioda_pe *pnv_ioda_setup_npu_PE(struct pci_dev *npu_pdev) rid = npu_pdev->bus->number <<8 | npu_pdn->devfn; npu_pdn->pe_number = pe_num; phb->ioda.pe_rmap[rid] = pe->pe_number; + pe->device_count++;/* Map the PE to this link */ rc = opal_pci_set_pe(phb->opal_id, pe_num, rid,
@@ -3544,6 +3562,8 @@ static void pnv_ioda_release_pe(struct pnv_ioda_pe *pe)case PNV_PHB_IODA2 :pnv_pci_ioda2_release_pe_dma (pe);break ; +case PNV_PHB_NPU_OCAPI : +break ;default :WARN_ON (1 ); }
@@ -3645,6 +3665,7 @@ static const struct pci_controller_ops pnv_npu_ioda_controller_ops = {static const struct pci_controller_ops pnv_npu_ocapi_ioda_controller_ops = { .enable_device_hook = pnv_ocapi_enable_device_hook, + .release_device = pnv_pci_release_device, .window_alignment = pnv_pci_window_alignment, .reset_secondary_bus = pnv_pci_reset_secondary_bus, .shutdown = pnv_pci_ioda_shutdown,
Signature · 1 line
2.21.0
PATCH v2 6/11 pci/hotplug/pnv-php: Remove erroneous warning
In reply to Frederic BarratOn powernv, when removing a device through hotplug, the following warning is logged:
Invalid refcount <.> on <...>
It may be incorrect, the refcount may be set to a higher value than 1 and be valid. of_detach_node() can drop more than one reference. As it doesn't seem trivial to assert the correct value, let's remove the warning.
---
Changelog:
v2: no change
drivers/pci/hotplug/pnv_php.c | 6 ------ 1 file changed, 6 deletions(-)
drivers/pci/hotplug/pnv_php.c +0 -6
@@ -151,17 +151,11 @@ static void pnv_php_rmv_pdns(struct device_node *dn)static void pnv_php_detach_device_nodes (struct device_node *parent) {struct device_node *dn; -int refcount;for_each_child_of_node (parent, dn) {pnv_php_detach_device_nodes (dn);of_node_put (dn); - refcount =kref_read (&dn->kobj.kref); -if (refcount !=1 ) -pr_warn ("Invalid refcount %d on <%pOF>\n" , - refcount, dn); -of_detach_node (dn); } }
Signature · 1 line
2.21.0
PATCH v2 9/11 pci/hotplug/pnv-php: Relax check when disabling slot
In reply to Frederic BarratThe driver only allows to disable a slot in the POPULATED state. However, if an error occurs while enabling the slot, say because the link couldn't be trained, then the POPULATED state may not be reached, yet the power state of the slot is on. So allow to disable a slot in the REGISTERED state. Removing the devices will do nothing since it's not populated, and we'll set the power state of the slot back to off.
---
Changelog:
v2: no change
drivers/pci/hotplug/pnv_php.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/pci/hotplug/pnv_php.c +7 -1
@@ -566,7 +566,13 @@ static int pnv_php_disable_slot(struct hotplug_slot *slot)struct pnv_php_slot *php_slot =to_pnv_php_slot (slot);int ret; -if (php_slot->state !=PNV_PHP_STATE_POPULATED ) +/* +* Allow to disable a slot already in the registered state to +* cover cases where the slot couldn't be enabled and never +* reached the populated state +*/ +if (php_slot->state !=PNV_PHP_STATE_POPULATED && + php_slot->state !=PNV_PHP_STATE_REGISTERED )return 0 ;/* Remove all devices behind the slot */
Signature · 1 line
2.21.0
PATCH v2 3/11 powerpc/powernv/ioda: set up PE on opencapi device when enabling
In reply to Frederic BarratThe PE for an opencapi device was set as part of a late PHB fixup operation, when creating the PHB. To use the PCI hotplug framework, this is not going to work, as the PHB stays the same, it's only the devices underneath which are updated. For regular PCI devices, it is done as part of the reconfiguration of the bridge, but for opencapi PHBs, we don't have an intermediate bridge. So let's define the PE when the device is enabled. PEs are meaningless for opencapi, the NPU doesn't define them and opal is not doing anything with them.
---
Changelog:
v2: no change
arch/powerpc/platforms/powernv/pci-ioda.c | 31 +++++++++++++++++------ 1 file changed, 23 insertions(+), 8 deletions(-)
arch/powerpc/platforms/powernv/pci-ioda.c +23 -8
@@ -1274,8 +1274,6 @@ static void pnv_pci_ioda_setup_PEs(void) {struct pci_controller *hose;struct pnv_phb *phb; -struct pci_bus *bus; -struct pci_dev *pdev;struct pnv_ioda_pe *pe;list_for_each_entry (hose, &hose_list, list_node) {
@@ -1287,11 +1285,6 @@ static void pnv_pci_ioda_setup_PEs(void)if (phb->model ==PNV_PHB_MODEL_NPU2 )WARN_ON_ONCE (pnv_npu2_init (hose)); } -if (phb->type ==PNV_PHB_NPU_OCAPI ) { - bus = hose->bus; -list_for_each_entry (pdev, &bus->devices, bus_list) -pnv_ioda_setup_dev_PE (pdev); - } }list_for_each_entry (hose, &hose_list, list_node) { phb = hose->private_data;
@@ -3389,6 +3382,28 @@ static bool pnv_pci_enable_device_hook(struct pci_dev *dev)return true; } +static bool pnv_ocapi_enable_device_hook (struct pci_dev *dev) +{ +struct pci_controller *hose =pci_bus_to_host (dev->bus); +struct pnv_phb *phb = hose->private_data; +struct pci_dn *pdn; +struct pnv_ioda_pe *pe; + +if (!phb->initialized) +return true; + + pdn =pci_get_pdn (dev); +if (!pdn) +return false; + +if (pdn->pe_number ==IODA_INVALID_PE ) { + pe =pnv_ioda_setup_dev_PE (dev); +if (!pe) +return false; + } +return true; +} +static long pnv_pci_ioda1_unset_window (struct iommu_table_group *table_group,int num) {
@@ -3629,7 +3644,7 @@ static const struct pci_controller_ops pnv_npu_ioda_controller_ops = { };static const struct pci_controller_ops pnv_npu_ocapi_ioda_controller_ops = { - .enable_device_hook = pnv_pci_enable_device_hook, + .enable_device_hook = pnv_ocapi_enable_device_hook, .window_alignment = pnv_pci_window_alignment, .reset_secondary_bus = pnv_pci_reset_secondary_bus, .shutdown = pnv_pci_ioda_shutdown,
Signature · 1 line
2.21.0
PATCH v2 11/11 ocxl: Add PCI hotplug dependency to Kconfig
In reply to Frederic BarratThe PCI hotplug framework is used to update the devices when a new image is written to the FPGA.
---
Changelog:
v2: no change
drivers/misc/ocxl/Kconfig | 1 + 1 file changed, 1 insertion(+)
drivers/misc/ocxl/Kconfig +1 -0
@@ -11,6 +11,7 @@ config OCXL tristate "OpenCAPI coherent accelerator support" depends on PPC_POWERNV && PCI && EEH select OCXL_BASE + select HOTPLUG_PCI_POWERNV default m help Select this option to enable the ocxl driver for Open
Signature · 1 line
2.21.0
PATCH v2 8/11 pci/hotplug/pnv-php: Register opencapi slots
In reply to Frederic BarratAdd the opencapi PHBs to the list of PHBs being scanned to look for slots.
---
Changelog:
v2:
- fix unregistration (Andrew)
drivers/pci/hotplug/pnv_php.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/pci/hotplug/pnv_php.c +5 -0
@@ -1009,6 +1009,8 @@ static int __init pnv_php_init(void)for_each_compatible_node (dn,NULL ,"ibm,ioda3-phb" )pnv_php_register (dn); +for_each_compatible_node (dn,NULL ,"ibm,ioda2-npu2-opencapi-phb" ) +pnv_php_register_one (dn);/* slot directly under the PHB */ return 0 ; }
@@ -1021,6 +1023,9 @@ static void __exit pnv_php_exit(void)for_each_compatible_node (dn,NULL ,"ibm,ioda3-phb" )pnv_php_unregister (dn); + +for_each_compatible_node (dn,NULL ,"ibm,ioda2-npu2-opencapi-phb" ) +pnv_php_unregister_one (dn);/* slot directly under the PHB */ }module_init (pnv_php_init);
Signature · 1 line
2.21.0
PATCH v2 7/11 pci/hotplug/pnv-php: Improve error msg on power state change failure
In reply to Frederic BarratWhen changing the slot state, if opal hits an error and tells as such in the asynchronous reply, the warning "Wrong msg" is logged, which is rather confusing. Instead we can reuse the better message which is already used when we couldn't submit the asynchronous opal request initially.
---
Changelog:
v2: no change
drivers/pci/hotplug/pnv_php.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-)
drivers/pci/hotplug/pnv_php.c +11 -5
@@ -336,18 +336,19 @@ int pnv_php_set_slot_power_state(struct hotplug_slot *slot, ret =pnv_pci_set_power_state (php_slot->id, state, &msg);if (ret >0 ) {if (be64_to_cpu (msg.params[1 ]) != php_slot->dn->phandle || -be64_to_cpu (msg.params[2 ]) != state || -be64_to_cpu (msg.params[3 ]) !=OPAL_SUCCESS ) { +be64_to_cpu (msg.params[2 ]) != state) {pci_warn (php_slot->pdev,"Wrong msg (%lld, %lld, %lld)\n" ,be64_to_cpu (msg.params[1 ]),be64_to_cpu (msg.params[2 ]),be64_to_cpu (msg.params[3 ]));return -ENOMSG ; } +if (be64_to_cpu (msg.params[3 ]) !=OPAL_SUCCESS ) { + ret = -ENODEV ; + goto error; + } }else if (ret <0 ) { -pci_warn (php_slot->pdev,"Error %d powering %s\n" , - ret, (state ==OPAL_PCI_SLOT_POWER_ON ) ?"on" :"off" ); -return ret; + goto error; }if (state ==OPAL_PCI_SLOT_POWER_OFF || state ==OPAL_PCI_SLOT_OFFLINE )
@@ -356,6 +357,11 @@ int pnv_php_set_slot_power_state(struct hotplug_slot *slot, ret =pnv_php_add_devtree (php_slot);return ret; + +error: +pci_warn (php_slot->pdev,"Error %d powering %s\n" , + ret, (state ==OPAL_PCI_SLOT_POWER_ON ) ?"on" :"off" ); +return ret; }EXPORT_SYMBOL_GPL (pnv_php_set_slot_power_state);
Signature · 1 line
2.21.0
PATCH v2 5/11 powerpc/powernv/ioda: Find opencapi slot for a device node
In reply to Frederic BarratUnlike real PCI slots, opencapi slots are directly associated to the (virtual) opencapi PHB, there's no intermediate bridge. So when looking for a slot ID, we must start the search from the device node itself and not its parent.
Also, the slot ID is not attached to a specific bdfn, so let's build it from the PHB ID, like skiboot.
---
Changelog:
v2:
- rename variable used when looping (Andrew)
arch/powerpc/include/asm/pnv-pci.h | 1 + arch/powerpc/platforms/powernv/pci.c | 24 ++++++++++++++---------- 2 files changed, 15 insertions(+), 10 deletions(-)
arch/powerpc/include/asm/pnv-pci.h +1 -0
@@ -15,6 +15,7 @@#define PCI_SLOT_ID_PREFIX (1UL << 63)#define PCI_SLOT_ID (phb_id, bdfn) \ (PCI_SLOT_ID_PREFIX | ((uint64_t)(bdfn) << 16) | (phb_id)) +#define PCI_PHB_SLOT_ID (phb_id) (phb_id)extern int pnv_pci_get_slot_id (struct device_node *np,uint64_t *id);extern int pnv_pci_get_device_tree (uint32_t phandle,void *buf,uint64_t len);
arch/powerpc/platforms/powernv/pci.c +14 -10
@@ -38,7 +38,7 @@ static DEFINE_MUTEX(tunnel_mutex);int pnv_pci_get_slot_id (struct device_node *np,uint64_t *id) { -struct device_node *parent = np; +struct device_node *node = np;u32 bdfn;u64 phbid;int ret;
@@ -48,25 +48,29 @@ int pnv_pci_get_slot_id(struct device_node *np, uint64_t *id)return -ENXIO ; bdfn = ((bdfn &0x00ffff00 ) >>8 ); -while ((parent =of_get_parent (parent))) { -if (!PCI_DN (parent)) { -of_node_put (parent); +for (node = np; node; node =of_get_parent (node)) { +if (!PCI_DN (node)) { +of_node_put (node);break ; } -if (!of_device_is_compatible (parent,"ibm,ioda2-phb" ) && - !of_device_is_compatible (parent,"ibm,ioda3-phb" )) { -of_node_put (parent); +if (!of_device_is_compatible (node,"ibm,ioda2-phb" ) && + !of_device_is_compatible (node,"ibm,ioda3-phb" ) && + !of_device_is_compatible (node,"ibm,ioda2-npu2-opencapi-phb" )) { +of_node_put (node);continue ; } - ret =of_property_read_u64 (parent,"ibm,opal-phbid" , &phbid); + ret =of_property_read_u64 (node,"ibm,opal-phbid" , &phbid);if (ret) { -of_node_put (parent); +of_node_put (node);return -ENXIO ; } - *id =PCI_SLOT_ID (phbid, bdfn); +if (of_device_is_compatible (node,"ibm,ioda2-npu2-opencapi-phb" )) + *id =PCI_PHB_SLOT_ID (phbid); +else + *id =PCI_SLOT_ID (phbid, bdfn);return 0 ; }
Signature · 1 line
2.21.0
PATCH v2 10/11 pci/hotplug/pnv-php: Wrap warnings in macro
In reply to Frederic BarratAn opencapi slot doesn't have an associated bridge device. It's not needed for operation, but any warning is displayed through pci_warn() which uses the pci_dev struct of the assocated bridge device. So wrap those warning so that a different trace mechanism can be used if it's an opencapi slot.
---
Changelog:
v2: no change
drivers/pci/hotplug/pnv_php.c | 51 +++++++++++++++++++----------------
1 file changed, 28 insertions(+), 23 deletions(-)
drivers/pci/hotplug/pnv_php.c +28 -23
@@ -18,6 +18,9 @@#define DRIVER_AUTHOR "Gavin Shan, IBM Corporation"#define DRIVER_DESC "PowerPC PowerNV PCI Hotplug Driver" +#define SLOT_WARN(sl, x...) \ + ((sl)->pdev ? pci_warn((sl)->pdev, x) : dev_warn(&(sl)->bus->dev, x)) +struct pnv_php_event {bool added;struct pnv_php_slot *php_slot;
@@ -265,7 +268,7 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) ret =pnv_pci_get_device_tree (php_slot->dn->phandle, fdt1,0x10000 );if (ret) { -pci_warn (php_slot->pdev,"Error %d getting FDT blob\n" , ret); +SLOT_WARN (php_slot,"Error %d getting FDT blob\n" , ret); goto free_fdt1; }
@@ -279,7 +282,7 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) dt =of_fdt_unflatten_tree (fdt, php_slot->dn,NULL );if (!dt) { ret = -EINVAL ; -pci_warn (php_slot->pdev,"Cannot unflatten FDT\n" ); +SLOT_WARN (php_slot,"Cannot unflatten FDT\n" ); goto free_fdt; }
@@ -289,15 +292,15 @@ static int pnv_php_add_devtree(struct pnv_php_slot *php_slot) ret =pnv_php_populate_changeset (&php_slot->ocs, php_slot->dn);if (ret) {pnv_php_reverse_nodes (php_slot->dn); -pci_warn (php_slot->pdev,"Error %d populating changeset\n" , - ret); +SLOT_WARN (php_slot,"Error %d populating changeset\n" , + ret); goto free_dt; } php_slot->dn->child =NULL ; ret =of_changeset_apply (&php_slot->ocs);if (ret) { -pci_warn (php_slot->pdev,"Error %d applying changeset\n" , ret); +SLOT_WARN (php_slot,"Error %d applying changeset\n" , ret); goto destroy_changeset; }
@@ -337,10 +340,10 @@ int pnv_php_set_slot_power_state(struct hotplug_slot *slot,if (ret >0 ) {if (be64_to_cpu (msg.params[1 ]) != php_slot->dn->phandle ||be64_to_cpu (msg.params[2 ]) != state) { -pci_warn (php_slot->pdev,"Wrong msg (%lld, %lld, %lld)\n" , -be64_to_cpu (msg.params[1 ]), -be64_to_cpu (msg.params[2 ]), -be64_to_cpu (msg.params[3 ])); +SLOT_WARN (php_slot,"Wrong msg (%lld, %lld, %lld)\n" , +be64_to_cpu (msg.params[1 ]), +be64_to_cpu (msg.params[2 ]), +be64_to_cpu (msg.params[3 ]));return -ENOMSG ; }if (be64_to_cpu (msg.params[3 ]) !=OPAL_SUCCESS ) {
@@ -359,8 +362,8 @@ int pnv_php_set_slot_power_state(struct hotplug_slot *slot,return ret; error: -pci_warn (php_slot->pdev,"Error %d powering %s\n" , - ret, (state ==OPAL_PCI_SLOT_POWER_ON ) ?"on" :"off" ); +SLOT_WARN (php_slot,"Error %d powering %s\n" , + ret, (state ==OPAL_PCI_SLOT_POWER_ON ) ?"on" :"off" );return ret; }EXPORT_SYMBOL_GPL (pnv_php_set_slot_power_state);
@@ -378,8 +381,8 @@ static int pnv_php_get_power_state(struct hotplug_slot *slot, u8 *state)*/ ret =pnv_pci_get_power_state (php_slot->id, &power_state);if (ret) { -pci_warn (php_slot->pdev,"Error %d getting power status\n" , - ret); +SLOT_WARN (php_slot,"Error %d getting power status\n" , + ret); }else { *state = power_state; }
@@ -402,7 +405,7 @@ static int pnv_php_get_adapter_state(struct hotplug_slot *slot, u8 *state) *state = presence; ret =0 ; }else { -pci_warn (php_slot->pdev,"Error %d getting presence\n" , ret); +SLOT_WARN (php_slot,"Error %d getting presence\n" , ret); }return ret;
@@ -681,7 +684,7 @@ static int pnv_php_register_slot(struct pnv_php_slot *php_slot) ret =pci_hp_register (&php_slot->slot, php_slot->bus, php_slot->slot_no, php_slot->name);if (ret) { -pci_warn (php_slot->pdev,"Error %d registering slot\n" , ret); +SLOT_WARN (php_slot,"Error %d registering slot\n" , ret);return ret; }
@@ -734,7 +737,7 @@ static int pnv_php_enable_msix(struct pnv_php_slot *php_slot)/* Enable MSIx */ ret =pci_enable_msix_exact (pdev, &entry,1 );if (ret) { -pci_warn (pdev,"Error %d enabling MSIx\n" , ret); +SLOT_WARN (php_slot,"Error %d enabling MSIx\n" , ret);return ret; }
@@ -784,8 +787,9 @@ static irqreturn_t pnv_php_interrupt(int irq, void *data) (sts & PCI_EXP_SLTSTA_PDC)) { ret =pnv_pci_get_presence_state (php_slot->id, &presence);if (ret) { -pci_warn (pdev,"PCI slot [%s] error %d getting presence (0x%04x), to retry the operation.\n" , - php_slot->name, ret, sts); +SLOT_WARN (php_slot, +"PCI slot [%s] error %d getting presence (0x%04x), to retry the operation.\n" , + php_slot->name, ret, sts);return IRQ_HANDLED ; }
@@ -815,8 +819,9 @@ static irqreturn_t pnv_php_interrupt(int irq, void *data)*/ event =kzalloc (sizeof (*event),GFP_ATOMIC );if (!event) { -pci_warn (pdev,"PCI slot [%s] missed hotplug event 0x%04x\n" , - php_slot->name, sts); +SLOT_WARN (php_slot, +"PCI slot [%s] missed hotplug event 0x%04x\n" , + php_slot->name, sts);return IRQ_HANDLED ; }
@@ -840,7 +845,7 @@ static void pnv_php_init_irq(struct pnv_php_slot *php_slot, int irq)/* Allocate workqueue */ php_slot->wq =alloc_workqueue ("pciehp-%s" ,0 ,0 , php_slot->name);if (!php_slot->wq) { -pci_warn (pdev,"Cannot alloc workqueue\n" ); +SLOT_WARN (php_slot,"Cannot alloc workqueue\n" );pnv_php_disable_irq (php_slot, true);return ; }
@@ -864,7 +869,7 @@ static void pnv_php_init_irq(struct pnv_php_slot *php_slot, int irq) php_slot->name, php_slot);if (ret) {pnv_php_disable_irq (php_slot, true); -pci_warn (pdev,"Error %d enabling IRQ %d\n" , ret, irq); +SLOT_WARN (php_slot,"Error %d enabling IRQ %d\n" , ret, irq);return ; }
@@ -900,7 +905,7 @@ static void pnv_php_enable_irq(struct pnv_php_slot *php_slot) ret =pci_enable_device (pdev);if (ret) { -pci_warn (pdev,"Error %d enabling device\n" , ret); +SLOT_WARN (php_slot,"Error %d enabling device\n" , ret);return ; }
Signature · 1 line
2.21.0
Re: [PATCH v2 05/11] powerpc/powernv/ioda: Find opencapi slot for a device node
In reply to Frederic BarratQuoted 9 lines · Frederic Barrat
Unlike real PCI slots, opencapi slots are directly associated to the (virtual) opencapi PHB, there's no intermediate bridge. So when looking for a slot ID, we must start the search from the device node itself and not its parent.
Also, the slot ID is not attached to a specific bdfn, so let's build it from the PHB ID, like skiboot.
Signed-off-by Frederic Barrat <[email protected]>
Signature · 2 lines
Andrew Donnellan OzLabs, ADL Canberra [email protected] IBM Australia Limited
Re: [PATCH v2 08/11] pci/hotplug/pnv-php: Register opencapi slots
In reply to Frederic BarratQuoted 4 lines · Frederic Barrat
Add the opencapi PHBs to the list of PHBs being scanned to look for slots.
Signed-off-by Frederic Barrat <[email protected]>
Signature · 2 lines
Andrew Donnellan OzLabs, ADL Canberra [email protected] IBM Australia Limited
Re: [PATCH v2 04/11] powerpc/powernv/ioda: Release opencapi device
In reply to Frederic BarratQuoted 11 lines · Frederic Barrat
With hotplug, an opencapi device can now go away. It needs to be released, mostly to clean up its PE state. We were previously not defining any device callback. We can reuse the standard PCI release callback, it does a bit too much for an opencapi device, but it's harmless, and only needs minor tuning.
Also separate the undo of the PELT-V code in a separate function, it is not needed for NPU devices and it improves a bit the readability of the code.
Signed-off-by Frederic Barrat <[email protected]>
Signature · 2 lines
Andrew Donnellan OzLabs, ADL Canberra [email protected] IBM Australia Limited
Re: [PATCH v2 01/11] powerpc/powernv/ioda: Fix ref count for devices with their own PE
In reply to Frederic BarratQuoted 20 lines · Frederic Barrat
The
pci_dnstructure used to store a pointer to the structpci_dev, so taking a reference on the device was required. However, thepci_devpointer was later removed from thepci_dnstructure, but the reference was kept for the npu device.
See commit902bdc57451c("powerpc/powernv/idoa: Remove unnecessary pcidev frompci_dn").We don't need to take a reference on the device when assigning the PE as the struct
pnv_ioda_peis cleaned up at the same time as the (physical) device is released. Doing so prevents the device from being released, which is a problem for opencapi devices, since we want to be able to remove them through PCI hotplug.Now the ugly part: nvlink npu devices are not meant to be released. Because of the above, we've always leaked a reference and simply removing it now is dangerous and would likely require more work. There's currently no release device callback for nvlink devices for example. So to be safe, this patch leaks a reference on the npu device, but only for nvlink and not opencapi.
Quoted 3 lines
It took me a while to parse exactly what you're doing here.
- In pnv_ioda_setup_dev_PE(), we take a reference on the pci_dev, this is to protect a pointer in the pci_dn structure, but not to protect the pointer in the pci_dev structure (which doesn't need to be protected by taking a reference, because the lifetime of the pnv_ioda_pe is the same as the pci_dev).
- The pointer in the pci_dn structure has now been removed, so we should remove the pci_dev_get() accordingly.
This seems okay to me, though anything PE-related is irritatingly hairy so one can never be truly certain...
Signature · 2 lines
Andrew Donnellan OzLabs, ADL Canberra [email protected] IBM Australia Limited
Re: [PATCH v2 01/11] powerpc/powernv/ioda: Fix ref count for devices with their own PE
Le 08/01/2020 à 08:33, Andrew Donnellan a écrit :
Quoted 25 lines · Frederic Barrat
Quoted 20 lines
The
pci_dnstructure used to store a pointer to the structpci_dev, so taking a reference on the device was required. However, thepci_devpointer was later removed from thepci_dnstructure, but the reference was kept for the npu device.
See commit902bdc57451c("powerpc/powernv/idoa: Remove unnecessary pcidev frompci_dn").We don't need to take a reference on the device when assigning the PE as the struct
pnv_ioda_peis cleaned up at the same time as the (physical) device is released. Doing so prevents the device from being released, which is a problem for opencapi devices, since we want to be able to remove them through PCI hotplug.Now the ugly part: nvlink npu devices are not meant to be released. Because of the above, we've always leaked a reference and simply removing it now is dangerous and would likely require more work. There's currently no release device callback for nvlink devices for example. So to be safe, this patch leaks a reference on the npu device, but only for nvlink and not opencapi.
Quoted 3 lines
Quoted 7 lines
It took me a while to parse exactly what you're doing here.
- In
pnv_ioda_setup_dev_PE(), we take a reference on thepci_dev, this
is to protect a pointer in thepci_dnstructure, but not to protect the pointer in thepci_devstructure (which doesn't need to be protected by taking a reference, because the lifetime of thepnv_ioda_peis the same as thepci_dev).- The pointer in the
pci_dnstructure has now been removed, so we should remove thepci_dev_get()accordingly.
Correct. Did I do such a bad job explaining it in the commit message that I need to rephrase?
Fred
Quoted 4 lines
This seems okay to me, though anything PE-related is irritatingly hairy so one can never be truly certain...
Reviewed-by Andrew Donnellan <[email protected]>
Re: [PATCH v2 01/11] powerpc/powernv/ioda: Fix ref count for devices with their own PE
Quoted 10 lines · Frederic Barrat
Quoted 7 lines
It took me a while to parse exactly what you're doing here.
- In
pnv_ioda_setup_dev_PE(), we take a reference on thepci_dev, this
is to protect a pointer in thepci_dnstructure, but not to protect the pointer in thepci_devstructure (which doesn't need to be protected by taking a reference, because the lifetime of thepnv_ioda_peis the same as thepci_dev).- The pointer in the
pci_dnstructure has now been removed, so we should remove thepci_dev_get()accordingly.Correct. Did I do such a bad job explaining it in the commit message that I need to rephrase?
I might just be a bit slow :)
Signature · 2 lines
Andrew Donnellan OzLabs, ADL Canberra [email protected] IBM Australia Limited
Re: [PATCH v2 01/11] powerpc/powernv/ioda: Fix ref count for devices with their own PE
In reply to Frederic BarratQuoted 20 lines · Frederic Barrat
The
pci_dnstructure used to store a pointer to the structpci_dev, so taking a reference on the device was required. However, thepci_devpointer was later removed from thepci_dnstructure, but the reference was kept for the npu device.
See commit902bdc57451c("powerpc/powernv/idoa: Remove unnecessary pcidev frompci_dn").We don't need to take a reference on the device when assigning the PE as the struct
pnv_ioda_peis cleaned up at the same time as the (physical) device is released. Doing so prevents the device from being released, which is a problem for opencapi devices, since we want to be able to remove them through PCI hotplug.Now the ugly part: nvlink npu devices are not meant to be released. Because of the above, we've always leaked a reference and simply removing it now is dangerous and would likely require more work. There's currently no release device callback for nvlink devices for example. So to be safe, this patch leaks a reference on the npu device, but only for nvlink and not opencapi.
Quoted 3 lines
Series applied to powerpc next, thanks.
https://git.kernel.org/powerpc/c/05dd7da76986937fb288b4213b1fa10dbe0d1b33
cheers