cxl: Add support for disabling CAPP when unloading CXL
Subsystems (2)
cover cxl: Add support for disabling CAPP when unloading CXL
Recent updates to OPAL have implemented necessary infrastructure [1] to disable CAPP and switch PHB back to PCIE mode during fast reset. This small patch-set uses the same OPAL infrastructure to force disable of CAPP when CXL module is unloaded via rmmod.
References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
Vaibhav Jain (2):
powerpc/powernv: Add support for CXL mode switch that need PHB reset
cxl: Force a CAPP reset when unloading CXL module
arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++--- drivers/misc/cxl/cxl.h | 1 + drivers/misc/cxl/main.c | 3 + drivers/misc/cxl/pci.c | 25 ++++++++- 4 files changed, 91 insertions(+), 9 deletions(-)
Signature · 1 line
2.20.1
RFC PATCH 1/2 powerpc/powernv: Add support for CXL mode switch that need PHB reset
Recent updates to OPAL [1] have provided support for new CXL modes on PHB that need to force a cold reset on the bridge (CRESET). However PHB CRESET is a multi step process and cannot be completed synchronously as expected by current kernel implementation that issues opal call opal_pci_set_phb_cxl_mode().
Hence this patch updates pnv_phb_to_cxl_mode() to implement a polling loop that handles specific error codes (OPAL_BUSY) returned from opal_pci_set_phb_cxl_mode() and drive the OPAL pci-state machine, if the requested CXL mode needs a CRESET.
The patch also updates pnv_phb_to_cxl_mode() to convert and return OPAL error codes into kernel error codes. This removes a previous issue where callers to this function would have to include 'opal-api.h' to check for specific OPAL error codes.
References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.html
--- arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++--- 1 file changed, 63 insertions(+), 8 deletions(-)
arch/powerpc/platforms/powernv/pci-cxl.c +63 -8
@@ -10,6 +10,7 @@#include <linux/module.h> #include <asm/pnv-pci.h> #include <asm/opal.h> +#include <linux/delay.h> #include "pci.h"
@@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)struct pci_controller *hose =pci_bus_to_host (dev->bus);struct pnv_phb *phb = hose->private_data;struct pnv_ioda_pe *pe; +unsigned long starttime, endtime;int rc; pe =pnv_ioda_get_pe (dev);if (!pe) -return -ENODEV ; +return -ENOENT ; + +pe_info (pe,"Switching PHB to CXL mode=%d\n" , mode); + +/* +* Use a 15 second timeout for mode switch. Value arrived after +* limited testing and may need more tweaking. +*/ + starttime = jiffies; + endtime = starttime +HZ *15 ; + +do { + rc =opal_pci_set_phb_cxl_mode (phb->opal_id, mode, + pe->pe_number); + +/* Wait until mode transistion done */ +if (rc !=OPAL_BUSY && rc !=OPAL_BUSY_EVENT ) +break ; + +/* Check if we timedout */ +if (time_after (jiffies, endtime)) { + rc =OPAL_TIMEOUT ; +break ; + } -pe_info (pe,"Switching PHB to CXL\n" ); +/* Opal Busy with mode switch. Run pci state-machine */ + rc =opal_pci_poll (phb->opal_id); +if (rc >=0 ) { +/* wait for some time */ +if (rc >0 ) +msleep (rc); +opal_poll_events (NULL ); + rc =OPAL_BUSY ; +/* Continue with the mode switch */ + } + }while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ); + +pe_level_printk (pe,KERN_DEBUG ,"CXL mode switch finished in %u-msecs." , +jiffies_to_msecs (jiffies - starttime)); - rc =opal_pci_set_phb_cxl_mode (phb->opal_id, mode, pe->pe_number); -if (rc ==OPAL_UNSUPPORTED ) -dev_err (&dev->dev,"Required cxl mode not supported by firmware - update skiboot\n" ); -else if (rc) -dev_err (&dev->dev,"opal_pci_set_phb_cxl_mode failed: %i\n" , rc); +/* Check OPAL errors and convert them to kernel error codes */ +switch (rc) { +case OPAL_SUCCESS : +return 0 ; -return rc; +case OPAL_PARAMETER : +dev_err (&dev->dev,"CXL not supported on this PHB\n" ); +return -ENOENT ; + +case OPAL_UNSUPPORTED : +dev_err (&dev->dev, +"Required cxl mode not supported by firmware" +" - update skiboot\n" ); +return -ENODEV ; + +case OPAL_TIMEOUT : +dev_err (&dev->dev,"opal_pci_set_phb_cxl_mode Timedout\n" ); +return -ETIME ; + +default : +dev_err (&dev->dev, +"opal_pci_set_phb_cxl_mode failed: %i\n" , rc); +return -EIO ; + }; }EXPORT_SYMBOL (pnv_phb_to_cxl_mode);
Signature · 1 line
2.20.1
Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
Quoted 69 lines
Recent updates to OPAL [1] have provided support for new CXL modes on PHB that need to force a cold reset on the bridge (CRESET). However PHB CRESET is a multi step process and cannot be completed synchronously as expected by current kernel implementation that issues opal call
opal_pci_set_phb_cxl_mode().Hence this patch updates
pnv_phb_to_cxl_mode()to implement a polling loop that handles specific error codes (OPAL_BUSY) returned fromopal_pci_set_phb_cxl_mode()and drive the OPAL pci-state machine, if the requested CXL mode needs a CRESET.The patch also updates
pnv_phb_to_cxl_mode()to convert and return OPAL error codes into kernel error codes. This removes a previous issue where callers to this function would have to include 'opal-api.h' to check for specific OPAL error codes.References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.htmlSigned-off-by Vaibhav Jain <[email protected]>--- arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++--- 1 file changed, 63 insertions(+), 8 deletions(-)arch/powerpc/platforms/powernv/pci-cxl.c +25
-1@@ -10,6 +10,7 @@#include <linux/module.h> #include <asm/pnv-pci.h> #include <asm/opal.h> +#include <linux/delay.h> #include "pci.h" @@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)struct pci_controller *hose =pci_bus_to_host (dev->bus);struct pnv_phb *phb = hose->private_data;struct pnv_ioda_pe *pe; +unsigned long starttime, endtime;int rc; pe =pnv_ioda_get_pe (dev);if (!pe) -return -ENODEV ; +return -ENOENT ; + +pe_info (pe,"Switching PHB to CXL mode=%d\n" , mode); + +/* +* Use a 15 second timeout for mode switch. Value arrived after +* limited testing and may need more tweaking. +*/ + starttime = jiffies; + endtime = starttime +HZ *15 ; + + do { + rc = opal_pci_set_phb_cxl_mode(phb->opal_id, mode, + pe->pe_number); + + /* Wait until mode transistion done */ + if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT) + break; + + /* Check if we timedout */ + if (time_after(jiffies, endtime)) { + rc = OPAL_TIMEOUT; + break;
arch/powerpc/platforms/powernv/pci-cxl.c int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode) · line 58
Quoted 10 lines
arch/powerpc/platforms/powernv/pci-cxl.c +8
-1+ } -pe_info (pe,"Switching PHB to CXL\n" ); +/* Opal Busy with mode switch. Run pci state-machine */ + rc =opal_pci_poll (phb->opal_id); +if (rc >=0 ) { +/* wait for some time */ +if (rc >0 ) +msleep (rc); +opal_poll_events (NULL );
Why is a call to opal_poll_events() needed?
Fred
Quoted 42 lines
arch/powerpc/platforms/powernv/pci-cxl.c +30
-6+ rc =OPAL_BUSY ; +/* Continue with the mode switch */ + } + }while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ); + +pe_level_printk (pe,KERN_DEBUG ,"CXL mode switch finished in %u-msecs." , +jiffies_to_msecs (jiffies - starttime)); - rc =opal_pci_set_phb_cxl_mode (phb->opal_id, mode, pe->pe_number); -if (rc ==OPAL_UNSUPPORTED ) -dev_err (&dev->dev,"Required cxl mode not supported by firmware - update skiboot\n" ); -else if (rc) -dev_err (&dev->dev,"opal_pci_set_phb_cxl_mode failed: %i\n" , rc); +/* Check OPAL errors and convert them to kernel error codes */ +switch (rc) { +case OPAL_SUCCESS : +return 0 ; -return rc; +case OPAL_PARAMETER : +dev_err (&dev->dev,"CXL not supported on this PHB\n" ); +return -ENOENT ; + +case OPAL_UNSUPPORTED : +dev_err (&dev->dev, +"Required cxl mode not supported by firmware" +" - update skiboot\n" ); +return -ENODEV ; + +case OPAL_TIMEOUT : +dev_err (&dev->dev,"opal_pci_set_phb_cxl_mode Timedout\n" ); +return -ETIME ; + +default : +dev_err (&dev->dev, +"opal_pci_set_phb_cxl_mode failed: %i\n" , rc); +return -EIO ; + }; }EXPORT_SYMBOL (pnv_phb_to_cxl_mode);
Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
Quoted 3 lines · Frederic Barrat
arch/powerpc/platforms/powernv/pci-cxl.c int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode) · line 58
Quoted 1 line
arch/powerpc/platforms/powernv/pci-cxl.c +1
-0+opal_poll_events (NULL );Why is a call to
opal_poll_events()needed?
Trying to make sure that opal pollers are run on the current CPU and any opal timer are executed.
Signature · 2 lines
Vaibhav Jain <[email protected]> Linux Technology Center, IBM India Pvt. Ltd.
Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
In reply to Vaibhav JainQuoted 37 lines · Vaibhav Jain
Recent updates to OPAL [1] have provided support for new CXL modes on PHB that need to force a cold reset on the bridge (CRESET). However PHB CRESET is a multi step process and cannot be completed synchronously as expected by current kernel implementation that issues opal call
opal_pci_set_phb_cxl_mode().Hence this patch updates
pnv_phb_to_cxl_mode()to implement a polling loop that handles specific error codes (OPAL_BUSY) returned fromopal_pci_set_phb_cxl_mode()and drive the OPAL pci-state machine, if the requested CXL mode needs a CRESET.The patch also updates
pnv_phb_to_cxl_mode()to convert and return OPAL error codes into kernel error codes. This removes a previous issue where callers to this function would have to include 'opal-api.h' to check for specific OPAL error codes.References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.htmlSigned-off-by Vaibhav Jain <[email protected]>--- arch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++--- 1 file changed, 63 insertions(+), 8 deletions(-)arch/powerpc/platforms/powernv/pci-cxl.c +1
-0@@ -10,6 +10,7 @@#include <linux/module.h> #include <asm/pnv-pci.h> #include <asm/opal.h> +#include <linux/delay.h> #include "pci.h"
arch/powerpc/platforms/powernv/pci-cxl.c int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode) · line 27
Quoted 10 lines
arch/powerpc/platforms/powernv/pci-cxl.c +2
-1@@ -18,21 +19,75 @@ int pnv_phb_to_cxl_mode(struct pci_dev *dev, uint64_t mode)struct pci_controller *hose =pci_bus_to_host (dev->bus);struct pnv_phb *phb = hose->private_data;struct pnv_ioda_pe *pe; +unsigned long starttime, endtime;int rc; pe =pnv_ioda_get_pe (dev);if (!pe) -return -ENODEV ; +return -ENOENT ;
The return code of pnv_phb_to_cxl_mode() is also returned by an api in the cxllib librarie. So, hoping that nobody test the value !!
Quoted 73 lines
arch/powerpc/platforms/powernv/pci-cxl.c +60
-7+ +pe_info (pe,"Switching PHB to CXL mode=%d\n" , mode); + +/* +* Use a 15 second timeout for mode switch. Value arrived after +* limited testing and may need more tweaking. +*/ + starttime = jiffies; + endtime = starttime +HZ *15 ; + +do { + rc =opal_pci_set_phb_cxl_mode (phb->opal_id, mode, + pe->pe_number); + +/* Wait until mode transistion done */ +if (rc !=OPAL_BUSY && rc !=OPAL_BUSY_EVENT ) +break ; + +/* Check if we timedout */ +if (time_after (jiffies, endtime)) { + rc =OPAL_TIMEOUT ; +break ; + } -pe_info (pe,"Switching PHB to CXL\n" ); +/* Opal Busy with mode switch. Run pci state-machine */ + rc =opal_pci_poll (phb->opal_id); +if (rc >=0 ) { +/* wait for some time */ +if (rc >0 ) +msleep (rc); +opal_poll_events (NULL ); + rc =OPAL_BUSY ; +/* Continue with the mode switch */ + } + }while (rc ==OPAL_BUSY || rc ==OPAL_BUSY_EVENT ); + +pe_level_printk (pe,KERN_DEBUG ,"CXL mode switch finished in %u-msecs." , +jiffies_to_msecs (jiffies - starttime)); - rc =opal_pci_set_phb_cxl_mode (phb->opal_id, mode, pe->pe_number); -if (rc ==OPAL_UNSUPPORTED ) -dev_err (&dev->dev,"Required cxl mode not supported by firmware - update skiboot\n" ); -else if (rc) -dev_err (&dev->dev,"opal_pci_set_phb_cxl_mode failed: %i\n" , rc); +/* Check OPAL errors and convert them to kernel error codes */ +switch (rc) { +case OPAL_SUCCESS : +return 0 ; -return rc; +case OPAL_PARAMETER : +dev_err (&dev->dev,"CXL not supported on this PHB\n" ); +return -ENOENT ; + +case OPAL_UNSUPPORTED : +dev_err (&dev->dev, +"Required cxl mode not supported by firmware" +" - update skiboot\n" ); +return -ENODEV ; + +case OPAL_TIMEOUT : +dev_err (&dev->dev,"opal_pci_set_phb_cxl_mode Timedout\n" ); +return -ETIME ; + +default : +dev_err (&dev->dev, +"opal_pci_set_phb_cxl_mode failed: %i\n" , rc); +return -EIO ; + }; }EXPORT_SYMBOL (pnv_phb_to_cxl_mode);
Re: [RFC PATCH 1/2] powerpc/powernv: Add support for CXL mode switch that need PHB reset
Thanks for reviewing this patch Christophe,
Quoted 8 lines · christophe lombard
Review
Quoted 5 lines
pe = pnv_ioda_get_pe(dev); if (!pe)(quoted diff) +1
-1- return -ENODEV; + return -ENOENT;The return code of
pnv_phb_to_cxl_mode()is also returned by an api in the cxllib librarie. So, hoping that nobody test the value !!
Agreed. I did peek into cxllib_switch_phb_mode() before sending the patch and saw two conflicting cases. While switching to CXL_MODE_PCI we make sure that we return kernel error codes and while switching to CXL_MODE_CXL we return OPAL error codes.
I havent seen how CX5 handles return values from this function but I am betting thats its the usual zero & non-zero return value check, which then should work with the proposed change.
Signature · 2 lines
Vaibhav Jain <[email protected]> Linux Technology Center, IBM India Pvt. Ltd.
RFC PATCH 2/2 cxl: Force a CAPP reset when unloading CXL module
In reply to Vaibhav JainThis patch forces shutdown of CAPP when CXL module is unloaded. This is accomplished via a call to pnv_phb_to_cxl_mode() with mode == OPAL_PHB_CAPI_MODE_PCIE.
--- drivers/misc/cxl/cxl.h | 1 + drivers/misc/cxl/main.c | 3 +++ drivers/misc/cxl/pci.c | 25 ++++++++++++++++++++++++- 3 files changed, 28 insertions(+), 1 deletion(-)
drivers/misc/cxl/cxl.h +1 -0
@@ -1136,4 +1136,5 @@ void cxl_context_mm_count_get(struct cxl_context *ctx);/* Decrements the reference count to "struct mm_struct" */ void cxl_context_mm_count_put (struct cxl_context *ctx); +void cxl_pci_shutdown_capp (void );#endif
drivers/misc/cxl/main.c +3 -0
@@ -372,6 +372,9 @@ static void exit_cxl(void)if (cxl_is_power8 ())unregister_cxl_calls (&cxl_calls);idr_destroy (&cxl_adapter_idr); + +if (cpu_has_feature (CPU_FTR_HVMODE )) +cxl_pci_shutdown_capp (); }module_init (init_cxl);
drivers/misc/cxl/pci.c +24 -1
@@ -25,7 +25,7 @@#include "cxl.h" #include <misc/cxl.h> - +#include <misc/cxllib.h> #define CXL_PCI_VSEC_ID 0x1280#define CXL_VSEC_MIN_SIZE 0x80
@@ -2065,6 +2065,29 @@ static void cxl_pci_resume(struct pci_dev *pdev) } } +void cxl_pci_shutdown_capp (void ) +{ +struct pci_dev *pdev; +struct pci_bus *root_bus; +int rc; + +/* Iterate over all CAPP supported PHB's and force them to PCI mode */ +list_for_each_entry (root_bus, &pci_root_buses, node) { +for_each_pci_bridge (pdev, root_bus) { + +if (!cxllib_slot_is_supported (pdev,0 )) +continue ; + + rc =pnv_phb_to_cxl_mode (pdev, +OPAL_PHB_CAPI_MODE_PCIE ); +if (rc) +dev_err (&pdev->dev, +"cxl: Error resetting CAPP. Err=%d\n" , + rc); + } + } +} +static const struct pci_error_handlers cxl_err_handler = { .error_detected = cxl_pci_error_detected, .slot_reset = cxl_pci_slot_reset,
Signature · 1 line
2.20.1
Re: [RFC PATCH 2/2] cxl: Force a CAPP reset when unloading CXL module
Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
Quoted 62 lines
This patch forces shutdown of CAPP when CXL module is unloaded. This is accomplished via a call to
pnv_phb_to_cxl_mode()with mode ==OPAL_PHB_CAPI_MODE_PCIE.Signed-off-by Vaibhav Jain <[email protected]>--- drivers/misc/cxl/cxl.h | 1 + drivers/misc/cxl/main.c | 3 +++ drivers/misc/cxl/pci.c | 25 ++++++++++++++++++++++++- 3 files changed, 28 insertions(+), 1 deletion(-)drivers/misc/cxl/cxl.h +1
-0@@ -1136,4 +1136,5 @@ void cxl_context_mm_count_get(struct cxl_context *ctx);/* Decrements the reference count to "struct mm_struct" */ void cxl_context_mm_count_put (struct cxl_context *ctx); +void cxl_pci_shutdown_capp (void );#endif drivers/misc/cxl/main.c +3
-0@@ -372,6 +372,9 @@ static void exit_cxl(void)if (cxl_is_power8 ())unregister_cxl_calls (&cxl_calls);idr_destroy (&cxl_adapter_idr); + +if (cpu_has_feature (CPU_FTR_HVMODE )) +cxl_pci_shutdown_capp (); }module_init (init_cxl);drivers/misc/cxl/pci.c +11
-1@@ -25,7 +25,7 @@#include "cxl.h" #include <misc/cxl.h> - +#include <misc/cxllib.h> #define CXL_PCI_VSEC_ID 0x1280#define CXL_VSEC_MIN_SIZE 0x80@@ -2065,6 +2065,29 @@ static void cxl_pci_resume(struct pci_dev *pdev) } } +void cxl_pci_shutdown_capp (void ) +{ +struct pci_dev *pdev; +struct pci_bus *root_bus; +int rc; + +/* Iterate over all CAPP supported PHB's and force them to PCI mode */ +list_for_each_entry (root_bus, &pci_root_buses, node) { +for_each_pci_bridge (pdev, root_bus) { +
drivers/misc/cxl/pci.c static void cxl_pci_resume(struct pci_dev *pdev) · line 2087
Quoted 10 lines
drivers/misc/cxl/pci.c +10
-0+if (!cxllib_slot_is_supported (pdev,0 )) +continue ; + + rc =pnv_phb_to_cxl_mode (pdev, +OPAL_PHB_CAPI_MODE_PCIE ); +if (rc) +dev_err (&pdev->dev, +"cxl: Error resetting CAPP. Err=%d\n" , + rc); + }
That's the part I don't like. We're iterating over quite a few PCI devices, we basically don't know the ones we need to reset.
If we have a per-adapter property on /sys to activate the reset-on-unload, then we could move the call to pnv_phb_to_cxl_mode(OPAL_PHB_CAPI_MODE_PCIE) on the cxl_remove() callback, and only do it for the adapters we've been asked.
Fred
Quoted 7 lines
drivers/misc/cxl/pci.c +3
-0+ } +} +static const struct pci_error_handlers cxl_err_handler = { .error_detected = cxl_pci_error_detected, .slot_reset = cxl_pci_slot_reset,
Hi Vaibhav,
I think there's value in there, as I'm hearing Mellanox would prefer to start from a "clean" state, as in pci mode, when they load their driver after a card FW update.
However, I'm pretty reluctant to make it the default behaviour. It's not like we haven't had our share of problems on reset before :-) Furthermore, for non-mlx5 case, we don't really have a reason for this. I'm thinking we could activate it through a per-adapter property, on /sys. See comment in one of the patches.
Fred
Le 25/01/2019 à 06:11, Vaibhav Jain a écrit :
Quoted 18 lines
Recent updates to OPAL have implemented necessary infrastructure [1] to disable CAPP and switch PHB back to PCIE mode during fast reset. This small patch-set uses the same OPAL infrastructure to force disable of CAPP when CXL module is unloaded via rmmod.
References:
[1]: https://lists.ozlabs.org/pipermail/skiboot/2019-January/013063.htmlVaibhav Jain (2):
powerpc/powernv: Add support for CXL mode switch that need PHB reset
cxl: Force a CAPP reset when unloading CXL modulearch/powerpc/platforms/powernv/pci-cxl.c | 71 +++++++++++++++++++++--- drivers/misc/cxl/cxl.h | 1 + drivers/misc/cxl/main.c | 3 + drivers/misc/cxl/pci.c | 25 ++++++++- 4 files changed, 91 insertions(+), 9 deletions(-)