crypto: qat - Disable VFs through a sysfs interface
Subsystems (2)
cover crypto: qat - Disable VFs through a sysfs interface
The main goal of this patch series is to introduce the ability to disable SR-IOV VFs by writing zero to the sriov_numvfs sysfs file. Alongside this, a few additional enhancements and fixes have been implemented.
Summary of Changes:
Patch #1: Preserves the entire ADF_GENERAL_SEC section during device
shutdown.
Patch #2: Adjusts the order in adf_dev_stop() to disable IOV before
stopping the AEs.
Patch #3: Adds support for the ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE
message to ensure proper VF shutdown notification.
Patch #4: Fixes a race condition by setting the vf->restarting flag
before sending the restart message.
Patch #5: Enables SR-IOV VF disablement through the sysfs interface.
Adam Guerin (1):
crypto: qat - preserve ADF_GENERAL_SEC
Michal Witwicki (4):
crypto: qat - disable IOV in adf_dev_stop()
crypto: qat - fix recovery flow for VFs
crypto: qat - ensure correct order in VF restarting handler
crypto: qat - allow disabling SR-IOV VFs
drivers/crypto/intel/qat/qat_420xx/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_4xxx/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c3xxx/adf_drv.c | 4 +- .../crypto/intel/qat/qat_c3xxxvf/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c62x/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c62xvf/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_common/adf_aer.c | 2 +- drivers/crypto/intel/qat/qat_common/adf_cfg.c | 29 +++ drivers/crypto/intel/qat/qat_common/adf_cfg.h | 2 + .../intel/qat/qat_common/adf_common_drv.h | 2 +- .../crypto/intel/qat/qat_common/adf_ctl_drv.c | 6 +- .../crypto/intel/qat/qat_common/adf_init.c | 44 +--- .../intel/qat/qat_common/adf_pfvf_pf_msg.c | 9 +- .../intel/qat/qat_common/adf_pfvf_vf_msg.c | 14 ++ .../intel/qat/qat_common/adf_pfvf_vf_msg.h | 1 + .../crypto/intel/qat/qat_common/adf_sriov.c | 194 ++++++++++++------ .../crypto/intel/qat/qat_common/adf_sysfs.c | 4 +- .../crypto/intel/qat/qat_common/adf_vf_isr.c | 4 +- .../crypto/intel/qat/qat_dh895xcc/adf_drv.c | 4 +- .../crypto/intel/qat/qat_dh895xccvf/adf_drv.c | 4 +- 20 files changed, 212 insertions(+), 131 deletions(-)
base-commit: 64409cf846e03f8372654e3b50cd31644b277f8c
Signature · 1 line
2.44.0
PATCH 1/5 crypto: qat - preserve ADF_GENERAL_SEC
From: Adam Guerin <[email protected]>
The ADF_GENERAL_SEC configuration section contains values that must be preserved during state transitions (down -> up, up -> down). This patch modifies the logic in adf_dev_shutdown() to maintain all key values within this section, rather than selectively saving and restoring only the ADF_SERVICES_ENABLED attribute.
To achieve this, a new function has been introduced that deletes all configuration sections except for the one specified by name. This function is invoked during adf_dev_down(), with ADF_GENERAL_SEC as the argument. Consequently, the adf_dev_shutdown_cache_cfg() function has been removed as it is now redundant.
Additionally, this patch eliminates the cache_config parameter from the adf_dev_down() function since ADF_GENERAL_SEC should always be retained. This change does not cause any side effects because all entries in the key-value store are cleared when a module is unloaded.
--- drivers/crypto/intel/qat/qat_420xx/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_4xxx/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c3xxx/adf_drv.c | 4 +- .../crypto/intel/qat/qat_c3xxxvf/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c62x/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c62xvf/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_common/adf_aer.c | 2 +- drivers/crypto/intel/qat/qat_common/adf_cfg.c | 29 ++++++++++++++ drivers/crypto/intel/qat/qat_common/adf_cfg.h | 2 + .../intel/qat/qat_common/adf_common_drv.h | 2 +- .../crypto/intel/qat/qat_common/adf_ctl_drv.c | 6 +-- .../crypto/intel/qat/qat_common/adf_init.c | 40 ++----------------- .../crypto/intel/qat/qat_common/adf_sriov.c | 2 +- .../crypto/intel/qat/qat_common/adf_sysfs.c | 4 +- .../crypto/intel/qat/qat_common/adf_vf_isr.c | 2 +- .../crypto/intel/qat/qat_dh895xcc/adf_drv.c | 4 +- .../crypto/intel/qat/qat_dh895xccvf/adf_drv.c | 4 +- 17 files changed, 60 insertions(+), 61 deletions(-)
drivers/crypto/intel/qat/qat_420xx/adf_drv.c +2 -2
@@ -163,7 +163,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err:adf_cleanup_accel (accel_dev);return ret;
@@ -177,7 +177,7 @@ static void adf_remove(struct pci_dev *pdev)pr_err ("QAT: Driver removal failed\n" );return ; } -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev); }
drivers/crypto/intel/qat/qat_4xxx/adf_drv.c +2 -2
@@ -165,7 +165,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err:adf_cleanup_accel (accel_dev);return ret;
@@ -179,7 +179,7 @@ static void adf_remove(struct pci_dev *pdev)pr_err ("QAT: Driver removal failed\n" );return ; } -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev); }
drivers/crypto/intel/qat/qat_c3xxx/adf_drv.c +2 -2
@@ -202,7 +202,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err_free_reg:pci_release_regions (accel_pci_dev->pci_dev); out_err_disable:
@@ -221,7 +221,7 @@ static void adf_remove(struct pci_dev *pdev)pr_err ("QAT: Driver removal failed\n" );return ; } -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev);adf_cleanup_pci_dev (accel_dev);kfree (accel_dev);
drivers/crypto/intel/qat/qat_c3xxxvf/adf_drv.c +2 -2
@@ -176,7 +176,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err_free_reg:pci_release_regions (accel_pci_dev->pci_dev); out_err_disable:
@@ -196,7 +196,7 @@ static void adf_remove(struct pci_dev *pdev)return ; }adf_flush_vf_wq (accel_dev); -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev);adf_cleanup_pci_dev (accel_dev);kfree (accel_dev);
drivers/crypto/intel/qat/qat_c62x/adf_drv.c +2 -2
@@ -202,7 +202,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err_free_reg:pci_release_regions (accel_pci_dev->pci_dev); out_err_disable:
@@ -221,7 +221,7 @@ static void adf_remove(struct pci_dev *pdev)pr_err ("QAT: Driver removal failed\n" );return ; } -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev);adf_cleanup_pci_dev (accel_dev);kfree (accel_dev);
drivers/crypto/intel/qat/qat_c62xvf/adf_drv.c +2 -2
@@ -176,7 +176,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err_free_reg:pci_release_regions (accel_pci_dev->pci_dev); out_err_disable:
@@ -196,7 +196,7 @@ static void adf_remove(struct pci_dev *pdev)return ; }adf_flush_vf_wq (accel_dev); -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev);adf_cleanup_pci_dev (accel_dev);kfree (accel_dev);
drivers/crypto/intel/qat/qat_common/adf_aer.c +1 -1
@@ -44,7 +44,7 @@ static pci_ers_result_t adf_error_detected(struct pci_dev *pdev,adf_pf2vf_notify_restarting (accel_dev);adf_pf2vf_wait_for_restarting_complete (accel_dev);pci_clear_master (pdev); -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);return PCI_ERS_RESULT_NEED_RESET ; }
drivers/crypto/intel/qat/qat_common/adf_cfg.c +29 -0
@@ -100,6 +100,8 @@ void adf_cfg_dev_dbgfs_rm(struct adf_accel_dev *accel_dev) }static void adf_cfg_section_del_all (struct list_head *head); +static void adf_cfg_section_del_all_except (struct list_head *head, +const char *section_name);void adf_cfg_del_all (struct adf_accel_dev *accel_dev) {
@@ -111,6 +113,17 @@ void adf_cfg_del_all(struct adf_accel_dev *accel_dev)clear_bit (ADF_STATUS_CONFIGURED , &accel_dev->status); } +void adf_cfg_del_all_except (struct adf_accel_dev *accel_dev, +const char *section_name) +{ +struct adf_cfg_device_data *dev_cfg_data = accel_dev->cfg; + +down_write (&dev_cfg_data->lock); +adf_cfg_section_del_all_except (&dev_cfg_data->sec_list, section_name); +up_write (&dev_cfg_data->lock); +clear_bit (ADF_STATUS_CONFIGURED , &accel_dev->status); +} +/** * adf_cfg_dev_remove() - Clears acceleration device configuration table. * @accel_dev: Pointer to acceleration device.
@@ -185,6 +198,22 @@ static void adf_cfg_section_del_all(struct list_head *head) } } +static void adf_cfg_section_del_all_except (struct list_head *head, +const char *section_name) +{ +struct list_head *list, *tmp; +struct adf_cfg_section *ptr; + +list_for_each_prev_safe (list, tmp, head) { + ptr = list_entry(list, struct adf_cfg_section, list); +if (!strcmp (ptr->name, section_name)) +continue ; +adf_cfg_keyval_del_all (&ptr->param_head); +list_del (list); +kfree (ptr); + } +} +static struct adf_cfg_key_val *adf_cfg_key_value_find (struct adf_cfg_section *s,const char *key) {
drivers/crypto/intel/qat/qat_common/adf_cfg.h +2 -0
@@ -35,6 +35,8 @@ void adf_cfg_dev_dbgfs_add(struct adf_accel_dev *accel_dev);void adf_cfg_dev_dbgfs_rm (struct adf_accel_dev *accel_dev);int adf_cfg_section_add (struct adf_accel_dev *accel_dev,const char *name);void adf_cfg_del_all (struct adf_accel_dev *accel_dev); +void adf_cfg_del_all_except (struct adf_accel_dev *accel_dev, +const char *section_name); int adf_cfg_add_key_value_param(struct adf_accel_dev *accel_dev, const char *section_name, const char *key, const void *val,
drivers/crypto/intel/qat/qat_common/adf_common_drv.h +1 -1
@@ -56,7 +56,7 @@ int adf_service_register(struct service_hndl *service);int adf_service_unregister (struct service_hndl *service);int adf_dev_up (struct adf_accel_dev *accel_dev,bool init_config); -int adf_dev_down (struct adf_accel_dev *accel_dev,bool cache_config); +int adf_dev_down (struct adf_accel_dev *accel_dev);int adf_dev_restart (struct adf_accel_dev *accel_dev);void adf_devmgr_update_class_index (struct adf_hw_device_data *hw_data);
drivers/crypto/intel/qat/qat_common/adf_ctl_drv.c +3 -3
@@ -247,7 +247,7 @@ static void adf_ctl_stop_devices(u32 id)if (!accel_dev->is_vf)continue ; -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); } }
@@ -256,7 +256,7 @@ static void adf_ctl_stop_devices(u32 id)if (!adf_dev_started (accel_dev))continue ; -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); } } }
@@ -319,7 +319,7 @@ static int adf_ctl_ioctl_dev_start(struct file *fp, unsigned int cmd,if (ret) {dev_err (&GET_DEV (accel_dev),"Failed to start qat_dev%d\n" , ctl_data->device_id); -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); } out:kfree (ctl_data);
drivers/crypto/intel/qat/qat_common/adf_init.c +4 -36
@@ -393,9 +393,9 @@ static void adf_dev_shutdown(struct adf_accel_dev *accel_dev)clear_bit (ADF_STATUS_IRQ_ALLOCATED , &accel_dev->status); } -/* Delete configuration only if not restarting */ +/* If not restarting, delete all cfg sections except for GENERAL */ if (!test_bit (ADF_STATUS_RESTARTING , &accel_dev->status)) -adf_cfg_del_all (accel_dev); +adf_cfg_del_all_except (accel_dev,ADF_GENERAL_SEC );if (hw_data->exit_arb) hw_data->exit_arb (accel_dev);
@@ -445,33 +445,7 @@ void adf_error_notifier(struct adf_accel_dev *accel_dev) } } -static int adf_dev_shutdown_cache_cfg (struct adf_accel_dev *accel_dev) -{ -char services[ADF_CFG_MAX_VAL_LEN_IN_BYTES ] = {0 }; -int ret; - - ret =adf_cfg_get_param_value (accel_dev,ADF_GENERAL_SEC , -ADF_SERVICES_ENABLED , services); - -adf_dev_stop (accel_dev); -adf_dev_shutdown (accel_dev); - -if (!ret) { - ret =adf_cfg_section_add (accel_dev,ADF_GENERAL_SEC ); -if (ret) -return ret; - - ret =adf_cfg_add_key_value_param (accel_dev,ADF_GENERAL_SEC , -ADF_SERVICES_ENABLED , - services,ADF_STR ); -if (ret) -return ret; - } - -return 0 ; -} - -int adf_dev_down (struct adf_accel_dev *accel_dev,bool reconfig) +int adf_dev_down (struct adf_accel_dev *accel_dev) {int ret =0 ;
@@ -480,15 +454,9 @@ int adf_dev_down(struct adf_accel_dev *accel_dev, bool reconfig)mutex_lock (&accel_dev->state_lock); -if (reconfig) { - ret =adf_dev_shutdown_cache_cfg (accel_dev); - goto out; - } -adf_dev_stop (accel_dev);adf_dev_shutdown (accel_dev); -out:mutex_unlock (&accel_dev->state_lock);return ret; }
@@ -535,7 +503,7 @@ int adf_dev_restart(struct adf_accel_dev *accel_dev)if (!accel_dev)return -EFAULT ; -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); ret =adf_dev_up (accel_dev, false);/* if device is already up return success*/
drivers/crypto/intel/qat/qat_common/adf_sriov.c +1 -1
@@ -192,7 +192,7 @@ int adf_sriov_configure(struct pci_dev *pdev, int numvfs)return -EBUSY ; } - ret =adf_dev_down (accel_dev, true); + ret =adf_dev_down (accel_dev);if (ret)return ret; }
drivers/crypto/intel/qat/qat_common/adf_sysfs.c +2 -2
@@ -62,7 +62,7 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr,break ; } - ret =adf_dev_down (accel_dev, true); + ret =adf_dev_down (accel_dev);if (ret)return ret;
@@ -76,7 +76,7 @@ static ssize_t state_store(struct device *dev, struct device_attribute *attr, }else if (ret) {dev_err (dev,"Failed to start device qat_dev%d\n" , accel_id); -adf_dev_down (accel_dev, true); +adf_dev_down (accel_dev);return ret; }break ;
drivers/crypto/intel/qat/qat_common/adf_vf_isr.c +1 -1
@@ -71,7 +71,7 @@ static void adf_dev_stop_async(struct work_struct *work)struct adf_accel_dev *accel_dev = stop_data->accel_dev;adf_dev_restarting_notify (accel_dev); -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);/* Re-enable PF2VF interrupts */ adf_enable_pf2vf_interrupts (accel_dev);
drivers/crypto/intel/qat/qat_dh895xcc/adf_drv.c +2 -2
@@ -202,7 +202,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err_free_reg:pci_release_regions (accel_pci_dev->pci_dev); out_err_disable:
@@ -221,7 +221,7 @@ static void adf_remove(struct pci_dev *pdev)pr_err ("QAT: Driver removal failed\n" );return ; } -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev);adf_cleanup_pci_dev (accel_dev);kfree (accel_dev);
drivers/crypto/intel/qat/qat_dh895xccvf/adf_drv.c +2 -2
@@ -176,7 +176,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)return ret; out_err_dev_stop: -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev); out_err_free_reg:pci_release_regions (accel_pci_dev->pci_dev); out_err_disable:
@@ -196,7 +196,7 @@ static void adf_remove(struct pci_dev *pdev)return ; }adf_flush_vf_wq (accel_dev); -adf_dev_down (accel_dev, false); +adf_dev_down (accel_dev);adf_cleanup_accel (accel_dev);adf_cleanup_pci_dev (accel_dev);kfree (accel_dev);
Signature · 1 line
2.44.0
PATCH 2/5 crypto: qat - disable IOV in adf_dev_stop()
In reply to Michal WitwickiDisabling IOV has the side effect of re-enabling the AEs that might attempt to do DMAs into the heartbeat buffers.
Move the disable_iov() function in adf_dev_stop() before the AEs are stopped.
--- drivers/crypto/intel/qat/qat_common/adf_init.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_init.c +2 -2
@@ -323,6 +323,8 @@ static void adf_dev_stop(struct adf_accel_dev *accel_dev)if (hw_data->stop_timer) hw_data->stop_timer (accel_dev); + hw_data->disable_iov (accel_dev); +if (wait)msleep (100 );
@@ -386,8 +388,6 @@ static void adf_dev_shutdown(struct adf_accel_dev *accel_dev)adf_tl_shutdown (accel_dev); - hw_data->disable_iov (accel_dev); -if (test_bit (ADF_STATUS_IRQ_ALLOCATED , &accel_dev->status)) { hw_data->free_irq (accel_dev);clear_bit (ADF_STATUS_IRQ_ALLOCATED , &accel_dev->status);
Signature · 1 line
2.44.0
PATCH 3/5 crypto: qat - fix recovery flow for VFs
In reply to Michal WitwickiWhen the PFVF protocol was updated to support version 5, i.e. ADF_PFVF_COMPAT_FALLBACK, the compatibility version for the VF was updated without supporting the message RESTARTING_COMPLETE required for such version.
Add support for the ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE message in the VF drivers. This message is sent by the VF driver to the PF to notify the completion of the shutdown flow.
--- .../crypto/intel/qat/qat_common/adf_pfvf_vf_msg.c | 14 ++++++++++++++ .../crypto/intel/qat/qat_common/adf_pfvf_vf_msg.h | 1 + drivers/crypto/intel/qat/qat_common/adf_vf_isr.c | 2 ++ 3 files changed, 17 insertions(+)
drivers/crypto/intel/qat/qat_common/adf_pfvf_vf_msg.c +14 -0
@@ -48,6 +48,20 @@ void adf_vf2pf_notify_shutdown(struct adf_accel_dev *accel_dev) }EXPORT_SYMBOL_GPL (adf_vf2pf_notify_shutdown); +void adf_vf2pf_notify_restart_complete (struct adf_accel_dev *accel_dev) +{ +struct pfvf_message msg = { .type =ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE }; + +/* Check compatibility version */ +if (accel_dev->vf.pf_compat_ver <ADF_PFVF_COMPAT_FALLBACK ) +return ; + +if (adf_send_vf2pf_msg (accel_dev, msg)) +dev_err (&GET_DEV (accel_dev), +"Failed to send Restarting complete event to PF\n" ); +} +EXPORT_SYMBOL_GPL (adf_vf2pf_notify_restart_complete); +int adf_vf2pf_request_version (struct adf_accel_dev *accel_dev) {u8 pf_version;
drivers/crypto/intel/qat/qat_common/adf_pfvf_vf_msg.h +1 -0
@@ -6,6 +6,7 @@#if defined(CONFIG_PCI_IOV )int adf_vf2pf_notify_init (struct adf_accel_dev *accel_dev);void adf_vf2pf_notify_shutdown (struct adf_accel_dev *accel_dev); +void adf_vf2pf_notify_restart_complete (struct adf_accel_dev *accel_dev);int adf_vf2pf_request_version (struct adf_accel_dev *accel_dev);int adf_vf2pf_get_capabilities (struct adf_accel_dev *accel_dev);int adf_vf2pf_get_ring_to_svc (struct adf_accel_dev *accel_dev);
drivers/crypto/intel/qat/qat_common/adf_vf_isr.c +2 -0
@@ -13,6 +13,7 @@#include "adf_cfg.h" #include "adf_cfg_strings.h" #include "adf_cfg_common.h" +#include "adf_pfvf_vf_msg.h" #include "adf_transport_access_macros.h" #include "adf_transport_internal.h"
@@ -75,6 +76,7 @@ static void adf_dev_stop_async(struct work_struct *work)/* Re-enable PF2VF interrupts */ adf_enable_pf2vf_interrupts (accel_dev); +adf_vf2pf_notify_restart_complete (accel_dev);kfree (stop_data); }
Signature · 1 line
2.44.0
PATCH 4/5 crypto: qat - ensure correct order in VF restarting handler
In reply to Michal WitwickiIn the process of sending the ADF_PF2VF_MSGTYPE_RESTARTING message to Virtual Functions (VFs), the Physical Function (PF) should set the vf->restarting flag to true before dispatching the message. This change is necessary to prevent a race condition where the handling of the ADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE message (which sets the vf->restarting flag to false) runs immediately after the message is sent, but before the flag is set to true.
Set the vf->restarting to true before sending the message ADF_PF2VF_MSGTYPE_RESTARTING, if supported by the version of the protocol and if the VF is started.
--- drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_msg.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_msg.c +6 -3
@@ -18,14 +18,17 @@ void adf_pf2vf_notify_restarting(struct adf_accel_dev *accel_dev)dev_dbg (&GET_DEV (accel_dev),"pf2vf notify restarting\n" );for (i =0 , vf = accel_dev->pf.vf_info; i < num_vfs; i++, vf++) { - vf->restarting = false; +if (vf->init && vf->vf_compat_ver >=ADF_PFVF_COMPAT_FALLBACK ) + vf->restarting = true; +else + vf->restarting = false; +if (!vf->init)continue ; +if (adf_send_pf2vf_msg (accel_dev, i, msg))dev_err (&GET_DEV (accel_dev),"Failed to send restarting msg to VF%d\n" , i); -else if (vf->vf_compat_ver >=ADF_PFVF_COMPAT_FALLBACK ) - vf->restarting = true; } }
Signature · 1 line
2.44.0
PATCH 5/5 crypto: qat - allow disabling SR-IOV VFs
In reply to Michal WitwickiThe QAT driver allows enabling SR-IOV VFs but does not allow them to be disabled through a write to sysfs.
Disabling SR-IOV VFs can be only achieved by bringing down and up a device using the attribute /sys/bus/pci/devices/<BDF>/qat/state.
The documentation for the sysfs attribute sriov_numvfs specifies that "a userspace application wanting to disable the VFs would write a zero to this file".
Add support for disabling SR-IOV VFs by writing '0' to the 'sriov_numvfs' attribute in sysfs.
Enabling or disabling SR-IOV always requires adf_dev_down() to be called. This action subsequently leads to the deletion of the ADF_KERNEL_SEC configuration section. The keys ADF_NUM_CY and ADF_NUM_DC within that section must be set to '0', otherwise, the driver will register into the Linux Crypto Framework. Because of this, the configuration in the ADF_KERNEL_SEC section must be added before every sriov_enable.
--- .../crypto/intel/qat/qat_common/adf_sriov.c | 194 ++++++++++++------ 1 file changed, 128 insertions(+), 66 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_sriov.c +128 -66
@@ -86,11 +86,133 @@ static int adf_enable_sriov(struct adf_accel_dev *accel_dev)return pci_enable_sriov (pdev, totalvfs); } +static int adf_add_sriov_configuration (struct adf_accel_dev *accel_dev) +{ +unsigned long val =0 ; +int ret; + + ret =adf_cfg_section_add (accel_dev,ADF_KERNEL_SEC ); +if (ret) +return ret; + + ret =adf_cfg_add_key_value_param (accel_dev,ADF_KERNEL_SEC ,ADF_NUM_CY , + &val,ADF_DEC ); +if (ret) +return ret; + + ret =adf_cfg_add_key_value_param (accel_dev,ADF_KERNEL_SEC ,ADF_NUM_DC , + &val,ADF_DEC ); +if (ret) +return ret; + +set_bit (ADF_STATUS_CONFIGURED , &accel_dev->status); + +return ret; +} + +static int adf_do_disable_sriov (struct adf_accel_dev *accel_dev) +{ +int ret; + +if (adf_dev_in_use (accel_dev)) { +dev_err (&GET_DEV (accel_dev), +"Cannot disable SR-IOV, device in use\n" ); +return -EBUSY ; + } + +if (adf_dev_started (accel_dev)) { +if (adf_devmgr_in_reset (accel_dev)) { +dev_err (&GET_DEV (accel_dev), +"Cannot disable SR-IOV, device in reset\n" ); +return -EBUSY ; + } + + ret =adf_dev_down (accel_dev); +if (ret) + goto err_del_cfg; + } + +adf_disable_sriov (accel_dev); + + ret =adf_dev_up (accel_dev, true); +if (ret) + goto err_del_cfg; + +return 0 ; + +err_del_cfg: +adf_cfg_del_all_except (accel_dev,ADF_GENERAL_SEC ); +return ret; +} + +static int adf_do_enable_sriov (struct adf_accel_dev *accel_dev) +{ +struct pci_dev *pdev =accel_to_pci_dev (accel_dev); +int totalvfs =pci_sriov_get_totalvfs (pdev); +unsigned long val; +int ret; + +if (!device_iommu_mapped (&GET_DEV (accel_dev))) { +dev_warn (&GET_DEV (accel_dev), +"IOMMU should be enabled for SR-IOV to work correctly\n" ); +return -EINVAL ; + } + +if (adf_dev_started (accel_dev)) { +if (adf_devmgr_in_reset (accel_dev) ||adf_dev_in_use (accel_dev)) { +dev_err (&GET_DEV (accel_dev),"Device busy\n" ); +return -EBUSY ; + } + + ret =adf_dev_down (accel_dev); +if (ret) +return ret; + } + + ret =adf_add_sriov_configuration (accel_dev); +if (ret) + goto err_del_cfg; + +/* Allocate memory for VF info structs */ + accel_dev->pf.vf_info =kcalloc (totalvfs,sizeof (struct adf_accel_vf_info ), +GFP_KERNEL ); + ret = -ENOMEM ; +if (!accel_dev->pf.vf_info) + goto err_del_cfg; + + ret =adf_dev_up (accel_dev, false); +if (ret) { +dev_err (&GET_DEV (accel_dev),"Failed to start qat_dev%d\n" , + accel_dev->accel_id); + goto err_free_vf_info; + } + + ret =adf_enable_sriov (accel_dev); +if (ret) + goto err_free_vf_info; + + val =1 ; + ret =adf_cfg_add_key_value_param (accel_dev,ADF_GENERAL_SEC ,ADF_SRIOV_ENABLED , + &val,ADF_DEC ); +if (ret) + goto err_free_vf_info; + +return totalvfs; + +err_free_vf_info: +adf_dev_down (accel_dev); +kfree (accel_dev->pf.vf_info); + accel_dev->pf.vf_info =NULL ; +return ret; +err_del_cfg: +adf_cfg_del_all_except (accel_dev,ADF_GENERAL_SEC ); +return ret; +} +void adf_reenable_sriov (struct adf_accel_dev *accel_dev) {struct pci_dev *pdev =accel_to_pci_dev (accel_dev);char cfg[ADF_CFG_MAX_VAL_LEN_IN_BYTES ] = {0 }; -unsigned long val =0 ;if (adf_cfg_get_param_value (accel_dev,ADF_GENERAL_SEC ,ADF_SRIOV_ENABLED , cfg))
@@ -99,15 +221,9 @@ void adf_reenable_sriov(struct adf_accel_dev *accel_dev)if (!accel_dev->pf.vf_info)return ; -if (adf_cfg_add_key_value_param (accel_dev,ADF_KERNEL_SEC ,ADF_NUM_CY , - &val,ADF_DEC )) -return ; - -if (adf_cfg_add_key_value_param (accel_dev,ADF_KERNEL_SEC ,ADF_NUM_DC , - &val,ADF_DEC )) +if (adf_add_sriov_configuration (accel_dev))return ; -set_bit (ADF_STATUS_CONFIGURED , &accel_dev->status);dev_dbg (&pdev->dev,"Re-enabling SRIOV\n" );adf_enable_sriov (accel_dev); }
@@ -168,70 +284,16 @@ EXPORT_SYMBOL_GPL(adf_disable_sriov);int adf_sriov_configure (struct pci_dev *pdev,int numvfs) {struct adf_accel_dev *accel_dev =adf_devmgr_pci_to_accel_dev (pdev); -int totalvfs =pci_sriov_get_totalvfs (pdev); -unsigned long val; -int ret;if (!accel_dev) {dev_err (&pdev->dev,"Failed to find accel_dev\n" );return -EFAULT ; } -if (!device_iommu_mapped (&pdev->dev)) -dev_warn (&pdev->dev,"IOMMU should be enabled for SR-IOV to work correctly\n" ); - -if (accel_dev->pf.vf_info) { -dev_info (&pdev->dev,"Already enabled for this device\n" ); -return -EINVAL ; - } - -if (adf_dev_started (accel_dev)) { -if (adf_devmgr_in_reset (accel_dev) || -adf_dev_in_use (accel_dev)) { -dev_err (&GET_DEV (accel_dev),"Device busy\n" ); -return -EBUSY ; - } - - ret =adf_dev_down (accel_dev); -if (ret) -return ret; - } - -if (adf_cfg_section_add (accel_dev,ADF_KERNEL_SEC )) -return -EFAULT ; - val =0 ; -if (adf_cfg_add_key_value_param (accel_dev,ADF_KERNEL_SEC , -ADF_NUM_CY , (void *)&val,ADF_DEC )) -return -EFAULT ; - ret =adf_cfg_add_key_value_param (accel_dev,ADF_KERNEL_SEC ,ADF_NUM_DC , - &val,ADF_DEC ); -if (ret) -return ret; - -set_bit (ADF_STATUS_CONFIGURED , &accel_dev->status); - -/* Allocate memory for VF info structs */ - accel_dev->pf.vf_info =kcalloc (totalvfs, -sizeof (struct adf_accel_vf_info ), -GFP_KERNEL ); -if (!accel_dev->pf.vf_info) -return -ENOMEM ; - -if (adf_dev_up (accel_dev, false)) { -dev_err (&GET_DEV (accel_dev),"Failed to start qat_dev%d\n" , - accel_dev->accel_id); -return -EFAULT ; - } - - ret =adf_enable_sriov (accel_dev); -if (ret) -return ret; - - val =1 ; -adf_cfg_add_key_value_param (accel_dev,ADF_GENERAL_SEC ,ADF_SRIOV_ENABLED , - &val,ADF_DEC ); - -return numvfs; +if (numvfs) +return adf_do_enable_sriov (accel_dev); +else +return adf_do_disable_sriov (accel_dev); }EXPORT_SYMBOL_GPL (adf_sriov_configure);
Signature · 1 line
2.44.0
Herbert Xu
maintainerQuoted 48 lines · Michal Witwicki
The main goal of this patch series is to introduce the ability to disable SR-IOV VFs by writing zero to the
sriov_numvfssysfs file. Alongside this, a few additional enhancements and fixes have been implemented.Summary of Changes:
Patch #1: Preserves the entireADF_GENERAL_SECsection during device
shutdown.
Patch #2: Adjusts the order inadf_dev_stop()to disable IOV before
stopping the AEs.
Patch #3: Adds support for theADF_VF2PF_MSGTYPE_RESTARTING_COMPLETE
message to ensure proper VF shutdown notification.
Patch #4: Fixes a race condition by setting thevf->restartingflag
before sending the restart message.
Patch #5: Enables SR-IOV VF disablement through the sysfs interface.Adam Guerin (1):
crypto: qat - preserveADF_GENERAL_SECMichal Witwicki (4):
crypto: qat - disable IOV inadf_dev_stop()
crypto: qat - fix recovery flow for VFs
crypto: qat - ensure correct order in VF restarting handler
crypto: qat - allow disabling SR-IOV VFsdrivers/crypto/intel/qat/qat_420xx/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_4xxx/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c3xxx/adf_drv.c | 4 +- .../crypto/intel/qat/qat_c3xxxvf/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c62x/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_c62xvf/adf_drv.c | 4 +- drivers/crypto/intel/qat/qat_common/adf_aer.c | 2 +- drivers/crypto/intel/qat/qat_common/adf_cfg.c | 29 +++ drivers/crypto/intel/qat/qat_common/adf_cfg.h | 2 + .../intel/qat/qat_common/adf_common_drv.h | 2 +- .../crypto/intel/qat/qat_common/adf_ctl_drv.c | 6 +- .../crypto/intel/qat/qat_common/adf_init.c | 44 +--- .../intel/qat/qat_common/adf_pfvf_pf_msg.c | 9 +- .../intel/qat/qat_common/adf_pfvf_vf_msg.c | 14 ++ .../intel/qat/qat_common/adf_pfvf_vf_msg.h | 1 + .../crypto/intel/qat/qat_common/adf_sriov.c | 194 ++++++++++++------ .../crypto/intel/qat/qat_common/adf_sysfs.c | 4 +- .../crypto/intel/qat/qat_common/adf_vf_isr.c | 4 +- .../crypto/intel/qat/qat_dh895xcc/adf_drv.c | 4 +- .../crypto/intel/qat/qat_dh895xccvf/adf_drv.c | 4 +- 20 files changed, 212 insertions(+), 131 deletions(-)
Quoted 3 lines
base-commit:
64409cf846e03f8372654e3b50cd31644b277f8cSignature · 1 line
2.44.0
All applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt