crypto: qat - enable QAT GEN4 SRIOV VF live migration for QAT GEN4
Subsystems (6)
v5 cover crypto: qat - enable QAT GEN4 SRIOV VF live migration for QAT GEN4
This set enables live migration for Intel QAT GEN4 SRIOV Virtual Functions (VFs).
It is composed of 10 patches. Patch 1~6 refactor the original QAT PF driver implementation which will be reused by the following patches. Patch 7 introduces the logic to the QAT PF driver that allows to save and restore the state of a bank (a QAT VF is a wrapper around banks) and drain a ring pair. Patch 8 adds the QAT PF driver a set of interfaces to allow to save and restore the state of a VF that will be called by the module qat_vfio_pci which will be introduced in the last patch. Patch 9 implements the defined device interfaces. The last one adds a vfio pci extension specific for QAT which intercepts the vfio device operations for a QAT VF to allow live migration.
Here are the steps required to test the live migration of a QAT GEN4 VF:
1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko
2. Assign the VFs to the virtual machine and enable device live migration
3. Run a workload using a QAT VF inside the VM, for example using qatlib
(https://github.com/intel/qatlib)
4. Migrate the VM from the source node to a destination node
Changes in v5 since v4: https://lore.kernel.org/kvm/[email protected]
- Remove device ID recheck as no consensus has been reached yet (Kevin) - Add missing state PRE_COPY_P2P in precopy_iotcl (Kevin) - Rearrange the state transition flow for better readability (Kevin) - Remove unnecessary Reviewed-by in commit message (Kevin)
Changes in v4 since v3: https://lore.kernel.org/kvm/[email protected]
- Change the order of maintainer entry for QAT vfio pci driver in
MAINTAINERS to make it alphabetical (Alex)
- Put QAT VFIO PCI driver under vfio/pci directly instead of
vfio/pci/intel (Alex)
- Add id_table recheck during device probe (Alex)
Changes in v3 since v2: https://lore.kernel.org/kvm/[email protected]
- Use state_mutex directly instead of unnecessary deferred_reset mode
(Jason)
Changes in v2 since v1: https://lore.kernel.org/all/[email protected]
- Add VFIO_MIGRATION_PRE_COPY support (Alex) - Remove unnecessary module dependancy in Kconfig (Alex) - Use direct function calls instead of function pointers in qat vfio
variant driver (Jason)
- Address the comments including uncessary pointer check and kfree, missing lock and direct use of pci_iov_vf_id (Shameer)
- Change CHECK_STAT macro to avoid repeat comparison (Kamlesh)
Changes in v1 since RFC: https://lore.kernel.org/all/[email protected]
- Address comments including the right module dependancy in Kconfig,
source file name and module description (Alex)
- Added PCI error handler and P2P state handler (Suggested by Kevin) - Refactor the state check duing loading ring state (Kevin) - Fix missed call to vfio_put_device in the error case (Breet) - Migrate the shadow states in PF driver - Rebase on top of 6.8-rc1
Giovanni Cabiddu (2):
crypto: qat - adf_get_etr_base() helper
crypto: qat - relocate CSR access code
Siming Wan (3):
crypto: qat - rename get_sla_arr_of_type()
crypto: qat - expand CSR operations for QAT GEN4 devices
crypto: qat - add bank save and restore flows
Xin Zeng (5):
crypto: qat - relocate and rename 4xxx PF2VM definitions
crypto: qat - move PFVF compat checker to a function
crypto: qat - add interface for live migration
crypto: qat - implement interface for live migration vfio/qat: Add vfio_pci driver for Intel QAT VF devices
MAINTAINERS | 8 + .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 3 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 5 + .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c | 1 + .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c | 1 + .../intel/qat/qat_c62x/adf_c62x_hw_data.c | 1 + .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c | 1 + drivers/crypto/intel/qat/qat_common/Makefile | 6 +- .../intel/qat/qat_common/adf_accel_devices.h | 88 ++ .../intel/qat/qat_common/adf_common_drv.h | 10 + .../qat/qat_common/adf_gen2_hw_csr_data.c | 101 ++ .../qat/qat_common/adf_gen2_hw_csr_data.h | 86 ++ .../intel/qat/qat_common/adf_gen2_hw_data.c | 97 -- .../intel/qat/qat_common/adf_gen2_hw_data.h | 76 -- .../qat/qat_common/adf_gen4_hw_csr_data.c | 231 ++++ .../qat/qat_common/adf_gen4_hw_csr_data.h | 188 +++ .../intel/qat/qat_common/adf_gen4_hw_data.c | 380 +++++-- .../intel/qat/qat_common/adf_gen4_hw_data.h | 127 +-- .../intel/qat/qat_common/adf_gen4_pfvf.c | 8 +- .../intel/qat/qat_common/adf_gen4_vf_mig.c | 1010 +++++++++++++++++ .../intel/qat/qat_common/adf_gen4_vf_mig.h | 10 + .../intel/qat/qat_common/adf_mstate_mgr.c | 318 ++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 89 ++ .../intel/qat/qat_common/adf_pfvf_pf_proto.c | 8 +- .../intel/qat/qat_common/adf_pfvf_utils.h | 11 + drivers/crypto/intel/qat/qat_common/adf_rl.c | 10 +- drivers/crypto/intel/qat/qat_common/adf_rl.h | 2 + .../crypto/intel/qat/qat_common/adf_sriov.c | 7 +- .../intel/qat/qat_common/adf_transport.c | 4 +- .../crypto/intel/qat/qat_common/qat_mig_dev.c | 130 +++ .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c | 1 + .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c | 1 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/qat/Kconfig | 12 + drivers/vfio/pci/qat/Makefile | 3 + drivers/vfio/pci/qat/main.c | 662 +++++++++++ include/linux/qat/qat_mig_dev.h | 31 + 38 files changed, 3344 insertions(+), 387 deletions(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c create mode 100644 drivers/vfio/pci/qat/Kconfig create mode 100644 drivers/vfio/pci/qat/Makefile create mode 100644 drivers/vfio/pci/qat/main.c create mode 100644 include/linux/qat/qat_mig_dev.h
base-commit: 318407ed77e4140d02e43a001b1f4753e3ce6b5f
Signature · 1 line
2.18.2
PATCH v5 1/10 crypto: qat - adf_get_etr_base() helper
From: Giovanni Cabiddu <[email protected]>
Add and use the new helper function adf_get_etr_base() which retrieves the virtual address of the ring bar.
This will be used extensively when adding support for Live Migration.
--- drivers/crypto/intel/qat/qat_common/adf_common_drv.h | 10 ++++++++++ drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c | 4 +--- drivers/crypto/intel/qat/qat_common/adf_transport.c | 4 +--- 3 files changed, 12 insertions(+), 6 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_common_drv.h +10 -0
@@ -248,6 +248,16 @@ static inline void __iomem *adf_get_pmisc_base(struct adf_accel_dev *accel_dev)return pmisc->virt_addr; } +static inline void __iomem *adf_get_etr_base (struct adf_accel_dev *accel_dev) +{ +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_bar *etr; + + etr = &GET_BARS (accel_dev)[hw_data->get_etr_bar_id (hw_data)]; + +return etr->virt_addr; +} +static inline void __iomem *adf_get_aram_base (struct adf_accel_dev *accel_dev) {struct adf_hw_device_data *hw_data = accel_dev->hw_device;
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +1 -3
@@ -321,8 +321,7 @@ static int reset_ring_pair(void __iomem *csr, u32 bank_number)int adf_gen4_ring_pair_reset (struct adf_accel_dev *accel_dev,u32 bank_number) {struct adf_hw_device_data *hw_data = accel_dev->hw_device; -u32 etr_bar_id = hw_data->get_etr_bar_id (hw_data); -void __iomem *csr; +void __iomem *csr =adf_get_etr_base (accel_dev);int ret;if (bank_number >= hw_data->num_banks)
@@ -331,7 +330,6 @@ int adf_gen4_ring_pair_reset(struct adf_accel_dev *accel_dev, u32 bank_number)dev_dbg (&GET_DEV (accel_dev),"ring pair reset for bank:%d\n" , bank_number); - csr = (&GET_BARS (accel_dev)[etr_bar_id])->virt_addr; ret =reset_ring_pair (csr, bank_number);if (ret) dev_err(&GET_DEV(accel_dev),
drivers/crypto/intel/qat/qat_common/adf_transport.c +1 -3
@@ -474,7 +474,6 @@ static int adf_init_bank(struct adf_accel_dev *accel_dev,int adf_init_etr_data (struct adf_accel_dev *accel_dev) {struct adf_etr_data *etr_data; -struct adf_hw_device_data *hw_data = accel_dev->hw_device;void __iomem *csr_addr;u32 size;u32 num_banks =0 ;
@@ -495,8 +494,7 @@ int adf_init_etr_data(struct adf_accel_dev *accel_dev) } accel_dev->transport = etr_data; - i = hw_data->get_etr_bar_id (hw_data); - csr_addr = accel_dev->accel_pci_dev.pci_bars[i].virt_addr; + csr_addr =adf_get_etr_base (accel_dev);/* accel_dev->debugfs_dir should always be non-NULL here */ etr_data->debug = debugfs_create_dir("transport",
Signature · 1 line
2.18.2
PATCH v5 2/10 crypto: qat - relocate and rename 4xxx PF2VM definitions
In reply to Xin ZengMove and rename ADF_4XXX_PF2VM_OFFSET and ADF_4XXX_VM2PF_OFFSET to ADF_GEN4_PF2VM_OFFSET and ADF_GEN4_VM2PF_OFFSET respectively. These definitions are moved from adf_gen4_pfvf.c to adf_gen4_hw_data.h as they are specific to GEN4 and not just to qat_4xxx.
This change is made in anticipation of their use in live migration.
This does not introduce any functional change.
--- drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h | 4 ++++ drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.c | 8 +++----- 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +4 -0
@@ -197,6 +197,10 @@ do { \/* Arbiter threads mask with error value */ #define ADF_GEN4_ENA_THD_MASK_ERROR GENMASK(ADF_NUM_THREADS_PER_AE, 0) +/* PF2VM communication channel */ +#define ADF_GEN4_PF2VM_OFFSET (i) (0x40B010 + (i) * 0x20) +#define ADF_GEN4_VM2PF_OFFSET (i) (0x40B014 + (i) * 0x20) +void adf_gen4_set_ssm_wdtimer (struct adf_accel_dev *accel_dev);enum icp_qat_gen4_slice_mask {
drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.c +3 -5
@@ -6,12 +6,10 @@#include "adf_accel_devices.h" #include "adf_common_drv.h" #include "adf_gen4_pfvf.h" +#include "adf_gen4_hw_data.h" #include "adf_pfvf_pf_proto.h" #include "adf_pfvf_utils.h" -#define ADF_4XXX_PF2VM_OFFSET (i) (0x40B010 + ((i) * 0x20)) -#define ADF_4XXX_VM2PF_OFFSET (i) (0x40B014 + ((i) * 0x20)) -/* VF2PF interrupt source registers */ #define ADF_4XXX_VM2PF_SOU 0x41A180#define ADF_4XXX_VM2PF_MSK 0x41A1C0
@@ -29,12 +27,12 @@ static const struct pfvf_csr_format csr_gen4_fmt = {static u32 adf_gen4_pf_get_pf2vf_offset (u32 i) { -return ADF_4XXX_PF2VM_OFFSET (i); +return ADF_GEN4_PF2VM_OFFSET (i); }static u32 adf_gen4_pf_get_vf2pf_offset (u32 i) { -return ADF_4XXX_VM2PF_OFFSET (i); +return ADF_GEN4_VM2PF_OFFSET (i); }static void adf_gen4_enable_vf2pf_interrupts (void __iomem *pmisc_addr,u32 vf_mask)
Signature · 1 line
2.18.2
PATCH v5 3/10 crypto: qat - move PFVF compat checker to a function
In reply to Xin ZengMove the code that implements VF version compatibility on the PF side to a separate function so that it can be reused when doing VM live migration.
This does not introduce any functional change.
--- .../crypto/intel/qat/qat_common/adf_pfvf_pf_proto.c | 8 +------- drivers/crypto/intel/qat/qat_common/adf_pfvf_utils.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 7 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_proto.c +1 -7
@@ -242,13 +242,7 @@ static int adf_handle_vf2pf_msg(struct adf_accel_dev *accel_dev, u8 vf_nr,"VersionRequest received from VF%d (vers %d) to PF (vers %d)\n" , vf_nr, vf_compat_ver, ADF_PFVF_COMPAT_THIS_VERSION); -if (vf_compat_ver ==0 ) - compat =ADF_PF2VF_VF_INCOMPATIBLE ; -else if (vf_compat_ver <=ADF_PFVF_COMPAT_THIS_VERSION ) - compat =ADF_PF2VF_VF_COMPATIBLE ; -else - compat =ADF_PF2VF_VF_COMPAT_UNKNOWN ; - + compat =adf_vf_compat_checker (vf_compat_ver); vf_info->vf_compat_ver = vf_compat_ver; resp->type =ADF_PF2VF_MSGTYPE_VERSION_RESP ;
drivers/crypto/intel/qat/qat_common/adf_pfvf_utils.h +11 -0
@@ -28,4 +28,15 @@ u32 adf_pfvf_csr_msg_of(struct adf_accel_dev *accel_dev, struct pfvf_message msgstruct pfvf_message adf_pfvf_message_of (struct adf_accel_dev *accel_dev,u32 raw_msg,const struct pfvf_csr_format *fmt); +static inline u8 adf_vf_compat_checker (u8 vf_compat_ver) +{ +if (vf_compat_ver ==0 ) +return ADF_PF2VF_VF_INCOMPATIBLE ; + +if (vf_compat_ver <=ADF_PFVF_COMPAT_THIS_VERSION ) +return ADF_PF2VF_VF_COMPATIBLE ; + +return ADF_PF2VF_VF_COMPAT_UNKNOWN ; +} +#endif /* ADF_PFVF_UTILS_H */
Signature · 1 line
2.18.2
PATCH v5 4/10 crypto: qat - relocate CSR access code
In reply to Xin ZengFrom: Giovanni Cabiddu <[email protected]>
As the common hw_data files are growing and the adf_hw_csr_ops is going to be extended with new operations, move all logic related to ring CSRs to the newly created adf_gen[2|4]_hw_csr_data.[c|h] files.
This does not introduce any functional change.
--- .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 1 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 1 + .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c | 1 + .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c | 1 + .../intel/qat/qat_c62x/adf_c62x_hw_data.c | 1 + .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c | 1 + drivers/crypto/intel/qat/qat_common/Makefile | 2 + .../qat/qat_common/adf_gen2_hw_csr_data.c | 101 ++++++++++++++++++ .../qat/qat_common/adf_gen2_hw_csr_data.h | 86 +++++++++++++++ .../intel/qat/qat_common/adf_gen2_hw_data.c | 97 ----------------- .../intel/qat/qat_common/adf_gen2_hw_data.h | 76 ------------- .../qat/qat_common/adf_gen4_hw_csr_data.c | 101 ++++++++++++++++++ .../qat/qat_common/adf_gen4_hw_csr_data.h | 97 +++++++++++++++++ .../intel/qat/qat_common/adf_gen4_hw_data.c | 97 ----------------- .../intel/qat/qat_common/adf_gen4_hw_data.h | 94 +--------------- .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c | 1 + .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c | 1 + 17 files changed, 397 insertions(+), 362 deletions(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +1 -0
@@ -10,6 +10,7 @@#include <adf_fw_config.h> #include <adf_gen4_config.h> #include <adf_gen4_dc.h> +#include <adf_gen4_hw_csr_data.h> #include <adf_gen4_hw_data.h> #include <adf_gen4_pfvf.h> #include <adf_gen4_pm.h>
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +1 -0
@@ -10,6 +10,7 @@#include <adf_fw_config.h> #include <adf_gen4_config.h> #include <adf_gen4_dc.h> +#include <adf_gen4_hw_csr_data.h> #include <adf_gen4_hw_data.h> #include <adf_gen4_pfvf.h> #include <adf_gen4_pm.h>
drivers/crypto/intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c +1 -0
@@ -6,6 +6,7 @@#include <adf_common_drv.h> #include <adf_gen2_config.h> #include <adf_gen2_dc.h> +#include <adf_gen2_hw_csr_data.h> #include <adf_gen2_hw_data.h> #include <adf_gen2_pfvf.h> #include "adf_c3xxx_hw_data.h"
drivers/crypto/intel/qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c +1 -0
@@ -4,6 +4,7 @@#include <adf_common_drv.h> #include <adf_gen2_config.h> #include <adf_gen2_dc.h> +#include <adf_gen2_hw_csr_data.h> #include <adf_gen2_hw_data.h> #include <adf_gen2_pfvf.h> #include <adf_pfvf_vf_msg.h>
drivers/crypto/intel/qat/qat_c62x/adf_c62x_hw_data.c +1 -0
@@ -6,6 +6,7 @@#include <adf_common_drv.h> #include <adf_gen2_config.h> #include <adf_gen2_dc.h> +#include <adf_gen2_hw_csr_data.h> #include <adf_gen2_hw_data.h> #include <adf_gen2_pfvf.h> #include "adf_c62x_hw_data.h"
drivers/crypto/intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c +1 -0
@@ -4,6 +4,7 @@#include <adf_common_drv.h> #include <adf_gen2_config.h> #include <adf_gen2_dc.h> +#include <adf_gen2_hw_csr_data.h> #include <adf_gen2_hw_data.h> #include <adf_gen2_pfvf.h> #include <adf_pfvf_vf_msg.h>
drivers/crypto/intel/qat/qat_common/Makefile +2 -0
@@ -14,9 +14,11 @@ intel_qat-objs := adf_cfg.o \ adf_hw_arbiter.o \ adf_sysfs.o \ adf_sysfs_ras_counters.o \ + adf_gen2_hw_csr_data.o \ adf_gen2_hw_data.o \ adf_gen2_config.o \ adf_gen4_config.o \ + adf_gen4_hw_csr_data.o \ adf_gen4_hw_data.o \ adf_gen4_pm.o \ adf_gen2_dc.o \
drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c new file +101 -0
101 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h new file +86 -0
86 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen2_hw_data.c +0 -97
103 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen2_hw_data.h +0 -76
97 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c new file +101 -0
101 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h new file +97 -0
97 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +0 -97
103 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +2 -92
111 lines not on this page · raw message
drivers/crypto/intel/qat/qat_dh895xcc/adf_dh895xcc_hw_data.c +1 -0
@@ -5,6 +5,7 @@#include <adf_common_drv.h> #include <adf_gen2_config.h> #include <adf_gen2_dc.h> +#include <adf_gen2_hw_csr_data.h> #include <adf_gen2_hw_data.h> #include <adf_gen2_pfvf.h> #include "adf_dh895xcc_hw_data.h"
drivers/crypto/intel/qat/qat_dh895xccvf/adf_dh895xccvf_hw_data.c +1 -0
@@ -4,6 +4,7 @@#include <adf_common_drv.h> #include <adf_gen2_config.h> #include <adf_gen2_dc.h> +#include <adf_gen2_hw_csr_data.h> #include <adf_gen2_hw_data.h> #include <adf_gen2_pfvf.h> #include <adf_pfvf_vf_msg.h>
Signature · 1 line
2.18.2
PATCH v5 5/10 crypto: qat - rename get_sla_arr_of_type()
In reply to Xin ZengFrom: Siming Wan <[email protected]>
The function get_sla_arr_of_type() returns a pointer to an SLA type specific array.
Rename it and expose it as it will be used externally to this module.
This does not introduce any functional change.
--- drivers/crypto/intel/qat/qat_common/adf_rl.c | 10 +++++----- drivers/crypto/intel/qat/qat_common/adf_rl.h | 2 ++ 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_rl.c +5 -5
@@ -183,14 +183,14 @@ static enum adf_cfg_service_type srv_to_cfg_svc_type(enum adf_base_services rl_s }/** -* get_sla_arr_of_type() - Returns a pointer to SLA type specific array +* adf_rl_get_sla_arr_of_type() - Returns a pointer to SLA type specific array * @rl_data: pointer to ratelimiting data * @type: SLA type * @sla_arr: pointer to variable where requested pointer will be stored * * Return: Max number of elements allowed for the returned array */ -static u32 get_sla_arr_of_type (struct adf_rl *rl_data,enum rl_node_type type, +u32 adf_rl_get_sla_arr_of_type (struct adf_rl *rl_data,enum rl_node_type type,struct rl_sla ***sla_arr) {switch (type) {
@@ -778,7 +778,7 @@ static void clear_sla(struct adf_rl *rl_data, struct rl_sla *sla) rp_in_use[sla->ring_pairs_ids[i]] = false;update_budget (sla, old_cir, true); -get_sla_arr_of_type (rl_data, sla->type, &sla_type_arr); +adf_rl_get_sla_arr_of_type (rl_data, sla->type, &sla_type_arr);assign_node_to_parent (rl_data->accel_dev, sla, true);adf_rl_send_admin_delete_msg (rl_data->accel_dev, node_id, sla->type);mark_rps_usage (sla, rl_data->rp_in_use, false);
@@ -875,7 +875,7 @@ static int add_update_sla(struct adf_accel_dev *accel_dev,if (!is_update) {mark_rps_usage (sla, rl_data->rp_in_use, true); -get_sla_arr_of_type (rl_data, sla->type, &sla_type_arr); +adf_rl_get_sla_arr_of_type (rl_data, sla->type, &sla_type_arr); sla_type_arr[sla->node_id] = sla; rl_data->sla[sla->sla_id] = sla; }
@@ -1065,7 +1065,7 @@ void adf_rl_remove_sla_all(struct adf_accel_dev *accel_dev, bool incl_default)/* Unregister and remove all SLAs */ for (j =RL_LEAF ; j >= end_type; j--) { - max_id =get_sla_arr_of_type (rl_data, j, &sla_type_arr); + max_id =adf_rl_get_sla_arr_of_type (rl_data, j, &sla_type_arr);for (i =0 ; i < max_id; i++) {if (!sla_type_arr[i])
drivers/crypto/intel/qat/qat_common/adf_rl.h +2 -0
@@ -151,6 +151,8 @@ struct rl_sla {u16 ring_pairs_cnt; }; +u32 adf_rl_get_sla_arr_of_type (struct adf_rl *rl_data,enum rl_node_type type, +struct rl_sla ***sla_arr);int adf_rl_add_sla (struct adf_accel_dev *accel_dev,struct adf_rl_sla_input_data *sla_in); int adf_rl_update_sla(struct adf_accel_dev *accel_dev,
Signature · 1 line
2.18.2
PATCH v5 6/10 crypto: qat - expand CSR operations for QAT GEN4 devices
In reply to Xin ZengFrom: Siming Wan <[email protected]>
Extend the CSR operations for QAT GEN4 devices to allow saving and restoring the rings state.
The new operations will be used as a building block for implementing the state save and restore of Virtual Functions necessary for VM live migration.
This adds the following operations:
- read ring status register
- read ring underflow/overflow status register
- read ring nearly empty status register
- read ring nearly full status register
- read ring full status register
- read ring complete status register
- read ring exception status register
- read/write ring exception interrupt mask register
- read ring configuration register
- read ring base register
- read/write ring interrupt enable register
- read ring interrupt flag register
- read/write ring interrupt source select register
- read ring coalesced interrupt enable register
- read ring coalesced interrupt control register
- read ring flag and coalesced interrupt enable register
- read ring service arbiter enable register
- get ring coalesced interrupt control enable mask
--- .../intel/qat/qat_common/adf_accel_devices.h | 27 ++++ .../qat/qat_common/adf_gen4_hw_csr_data.c | 130 ++++++++++++++++++ .../qat/qat_common/adf_gen4_hw_csr_data.h | 93 ++++++++++++- 3 files changed, 249 insertions(+), 1 deletion(-)
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +27 -0
@@ -150,22 +150,49 @@ struct adf_hw_csr_ops { u32 ring);void (*write_csr_ring_tail)(void __iomem *csr_base_addr,u32 bank,u32 ring,u32 value); +u32 (*read_csr_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_uo_stat)(void __iomem *csr_base_addr,u32 bank);u32 (*read_csr_e_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_ne_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_nf_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_f_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_c_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_exp_stat)(void __iomem *csr_base_addr,u32 bank); +u32 (*read_csr_exp_int_en)(void __iomem *csr_base_addr,u32 bank); +void (*write_csr_exp_int_en)(void __iomem *csr_base_addr,u32 bank, +u32 value); +u32 (*read_csr_ring_config)(void __iomem *csr_base_addr,u32 bank, +u32 ring);void (*write_csr_ring_config)(void __iomem *csr_base_addr,u32 bank,u32 ring,u32 value); +dma_addr_t (*read_csr_ring_base)(void __iomem *csr_base_addr,u32 bank, +u32 ring);void (*write_csr_ring_base)(void __iomem *csr_base_addr,u32 bank,u32 ring,dma_addr_t addr); +u32 (*read_csr_int_en)(void __iomem *csr_base_addr,u32 bank); +void (*write_csr_int_en)(void __iomem *csr_base_addr,u32 bank, +u32 value); +u32 (*read_csr_int_flag)(void __iomem *csr_base_addr,u32 bank);void (*write_csr_int_flag)(void __iomem *csr_base_addr,u32 bank,u32 value); +u32 (*read_csr_int_srcsel)(void __iomem *csr_base_addr,u32 bank);void (*write_csr_int_srcsel)(void __iomem *csr_base_addr,u32 bank); +void (*write_csr_int_srcsel_w_val)(void __iomem *csr_base_addr, +u32 bank,u32 value); +u32 (*read_csr_int_col_en)(void __iomem *csr_base_addr,u32 bank);void (*write_csr_int_col_en)(void __iomem *csr_base_addr,u32 bank,u32 value); +u32 (*read_csr_int_col_ctl)(void __iomem *csr_base_addr,u32 bank);void (*write_csr_int_col_ctl)(void __iomem *csr_base_addr,u32 bank,u32 value); +u32 (*read_csr_int_flag_and_col)(void __iomem *csr_base_addr, +u32 bank);void (*write_csr_int_flag_and_col)(void __iomem *csr_base_addr,u32 bank,u32 value); +u32 (*read_csr_ring_srv_arb_en)(void __iomem *csr_base_addr,u32 bank);void (*write_csr_ring_srv_arb_en)(void __iomem *csr_base_addr,u32 bank,u32 value); +u32 (*get_int_col_ctl_enable_mask)(void); };struct adf_cfg_device_data ;
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c +130 -0
201 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h +92 -1
148 lines not on this page · raw message
Signature · 1 line
2.18.2
PATCH v5 7/10 crypto: qat - add bank save and restore flows
In reply to Xin ZengFrom: Siming Wan <[email protected]>
Add logic to save, restore, quiesce and drain a ring bank for QAT GEN4 devices.
This allows to save and restore the state of a Virtual Function (VF) and will be used to implement VM live migration.
--- .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 2 + .../intel/qat/qat_common/adf_accel_devices.h | 38 +++ .../intel/qat/qat_common/adf_gen4_hw_data.c | 279 ++++++++++++++++++ .../intel/qat/qat_common/adf_gen4_hw_data.h | 19 ++ 4 files changed, 338 insertions(+)
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +2 -0
@@ -455,6 +455,8 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data, u32 dev_id) hw_data->get_ring_to_svc_map = adf_gen4_get_ring_to_svc_map; hw_data->disable_iov = adf_disable_sriov; hw_data->ring_pair_reset = adf_gen4_ring_pair_reset; + hw_data->bank_state_save = adf_gen4_bank_state_save; + hw_data->bank_state_restore = adf_gen4_bank_state_restore; hw_data->enable_pm = adf_gen4_enable_pm; hw_data->handle_pm_interrupt = adf_gen4_handle_pm_interrupt; hw_data->dev_config = adf_gen4_dev_config;
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +38 -0
@@ -140,6 +140,40 @@ struct admin_info {u32 mailbox_offset; }; +struct ring_config { +u64 base; +u32 config; +u32 head; +u32 tail; +u32 reserved0; +}; + +struct bank_state { +u32 ringstat0; +u32 ringstat1; +u32 ringuostat; +u32 ringestat; +u32 ringnestat; +u32 ringnfstat; +u32 ringfstat; +u32 ringcstat0; +u32 ringcstat1; +u32 ringcstat2; +u32 ringcstat3; +u32 iaintflagen; +u32 iaintflagreg; +u32 iaintflagsrcsel0; +u32 iaintflagsrcsel1; +u32 iaintcolen; +u32 iaintcolctl; +u32 iaintflagandcolen; +u32 ringexpstat; +u32 ringexpintenable; +u32 ringsrvarben; +u32 reserved0; +struct ring_config rings[ADF_ETR_MAX_RINGS_PER_BANK ]; +}; +struct adf_hw_csr_ops {u64 (*build_csr_ring_base_addr)(dma_addr_t addr,u32 size); u32 (*read_csr_ring_head)(void __iomem *csr_base_addr, u32 bank,
@@ -271,6 +305,10 @@ struct adf_hw_device_data {void (*enable_ints)(struct adf_accel_dev *accel_dev);void (*set_ssm_wdtimer)(struct adf_accel_dev *accel_dev);int (*ring_pair_reset)(struct adf_accel_dev *accel_dev,u32 bank_nr); +int (*bank_state_save)(struct adf_accel_dev *accel_dev,u32 bank_number, +struct bank_state *state); +int (*bank_state_restore)(struct adf_accel_dev *accel_dev, +u32 bank_number,struct bank_state *state);void (*reset_device)(struct adf_accel_dev *accel_dev);void (*set_msix_rttable)(struct adf_accel_dev *accel_dev);const char *(*uof_get_name)(struct adf_accel_dev *accel_dev,u32 obj_num);
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +279 -0
288 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +19 -0
@@ -77,10 +77,19 @@#define ADF_RPRESET_POLL_TIMEOUT_US (5 * USEC_PER_SEC)#define ADF_RPRESET_POLL_DELAY_US 20#define ADF_WQM_CSR_RPRESETCTL_RESET BIT(0) +#define ADF_WQM_CSR_RPRESETCTL_DRAIN BIT(2)#define ADF_WQM_CSR_RPRESETCTL (bank) (0x6000 + ((bank) << 3))#define ADF_WQM_CSR_RPRESETSTS_STATUS BIT(0)#define ADF_WQM_CSR_RPRESETSTS (bank) (ADF_WQM_CSR_RPRESETCTL(bank) + 4) +/* Ring interrupt */ +#define ADF_RP_INT_SRC_SEL_F_RISE_MASK BIT(2) +#define ADF_RP_INT_SRC_SEL_F_FALL_MASK GENMASK(2, 0) +#define ADF_RP_INT_SRC_SEL_RANGE_WIDTH 4 +#define ADF_COALESCED_POLL_DELAY_US 1000 +#define ADF_WQM_CSR_RPINTSOU (bank) (0x200000 + ((bank) << 12)) +#define ADF_WQM_CSR_RP_IDX_RX 1 +/* Error source registers */ #define ADF_GEN4_ERRSOU0 (0x41A200)#define ADF_GEN4_ERRSOU1 (0x41A204)
@@ -150,5 +159,15 @@ void adf_gen4_set_msix_default_rttable(struct adf_accel_dev *accel_dev);void adf_gen4_set_ssm_wdtimer (struct adf_accel_dev *accel_dev);int adf_gen4_init_thd2arb_map (struct adf_accel_dev *accel_dev);u16 adf_gen4_get_ring_to_svc_map (struct adf_accel_dev *accel_dev); +int adf_gen4_bank_quiesce_coal_timer (struct adf_accel_dev *accel_dev, +u32 bank_idx,int timeout_ms); +int adf_gen4_bank_drain_start (struct adf_accel_dev *accel_dev, +u32 bank_number,int timeout_us); +void adf_gen4_bank_drain_finish (struct adf_accel_dev *accel_dev, +u32 bank_number); +int adf_gen4_bank_state_save (struct adf_accel_dev *accel_dev,u32 bank_number, +struct bank_state *state); +int adf_gen4_bank_state_restore (struct adf_accel_dev *accel_dev, +u32 bank_number,struct bank_state *state);#endif
Signature · 1 line
2.18.2
PATCH v5 8/10 crypto: qat - add interface for live migration
In reply to Xin ZengExtend the driver with a new interface to be used for VF live migration. This allows to create and destroy a qat_mig_dev object that contains a set of methods to allow to save and restore the state of QAT VF. This interface will be used by the qat-vfio-pci module.
--- drivers/crypto/intel/qat/qat_common/Makefile | 2 +- .../intel/qat/qat_common/adf_accel_devices.h | 17 +++ .../intel/qat/qat_common/adf_gen4_vf_mig.h | 10 ++ .../crypto/intel/qat/qat_common/qat_mig_dev.c | 130 ++++++++++++++++++ include/linux/qat/qat_mig_dev.h | 31 +++++ 5 files changed, 189 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c create mode 100644 include/linux/qat/qat_mig_dev.h
drivers/crypto/intel/qat/qat_common/Makefile +1 -1
@@ -54,6 +54,6 @@ intel_qat-$(CONFIG_DEBUG_FS) += adf_transport_debug.o \ intel_qat-$(CONFIG_PCI_IOV) += adf_sriov.o adf_vf_isr.o adf_pfvf_utils.o \ adf_pfvf_pf_msg.o adf_pfvf_pf_proto.o \ adf_pfvf_vf_msg.o adf_pfvf_vf_proto.o \ - adf_gen2_pfvf.o adf_gen4_pfvf.o + adf_gen2_pfvf.o adf_gen4_pfvf.o qat_mig_dev.o intel_qat-$(CONFIG_CRYPTO_DEV_QAT_ERROR_INJECTION) += adf_heartbeat_inject.o
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +17 -0
@@ -9,6 +9,7 @@#include <linux/pci.h> #include <linux/ratelimit.h> #include <linux/types.h> +#include <linux/qat/qat_mig_dev.h> #include "adf_cfg_common.h" #include "adf_rl.h" #include "adf_telemetry.h"
@@ -258,6 +259,20 @@ struct adf_dc_ops {void (*build_deflate_ctx)(void *ctx); }; +struct qat_migdev_ops { +int (*init)(struct qat_mig_dev *mdev); +void (*cleanup)(struct qat_mig_dev *mdev); +void (*reset)(struct qat_mig_dev *mdev); +int (*open)(struct qat_mig_dev *mdev); +void (*close)(struct qat_mig_dev *mdev); +int (*suspend)(struct qat_mig_dev *mdev); +int (*resume)(struct qat_mig_dev *mdev); +int (*save_state)(struct qat_mig_dev *mdev); +int (*save_setup)(struct qat_mig_dev *mdev); +int (*load_state)(struct qat_mig_dev *mdev); +int (*load_setup)(struct qat_mig_dev *mdev,int size); +}; +struct adf_dev_err_mask {u32 cppagentcmdpar_mask;u32 parerr_ath_cph_mask;
@@ -325,6 +340,7 @@ struct adf_hw_device_data {struct adf_dev_err_mask dev_err_mask;struct adf_rl_hw_data rl_data;struct adf_tl_hw_data tl_data; +struct qat_migdev_ops vfmig_ops;const char *fw_name;const char *fw_mmp_name;u32 fuses;
@@ -381,6 +397,7 @@ struct adf_hw_device_data {#define GET_CSR_OPS (accel_dev) (&(accel_dev)->hw_device->csr_ops)#define GET_PFVF_OPS (accel_dev) (&(accel_dev)->hw_device->pfvf_ops)#define GET_DC_OPS (accel_dev) (&(accel_dev)->hw_device->dc_ops) +#define GET_VFMIG_OPS (accel_dev) (&(accel_dev)->hw_device->vfmig_ops)#define GET_TL_DATA (accel_dev) GET_HW_DATA(accel_dev)->tl_data#define accel_to_pci_dev (accel_ptr) accel_ptr->accel_pci_dev.pci_dev
drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h new file +10 -0
@@ -0,0 +1,10 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2024 Intel Corporation */ +#ifndef ADF_GEN4_VF_MIG_H_ +#define ADF_GEN4_VF_MIG_H_ + +#include "adf_accel_devices.h" + +void adf_gen4_init_vf_mig_ops (struct qat_migdev_ops *vfmig_ops); + +#endif
drivers/crypto/intel/qat/qat_common/qat_mig_dev.c new file +130 -0
130 lines not on this page · raw message
include/linux/qat/qat_mig_dev.h new file +31 -0
@@ -0,0 +1,31 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2024 Intel Corporation */ +#ifndef QAT_MIG_DEV_H_ +#define QAT_MIG_DEV_H_ + +struct pci_dev ; + +struct qat_mig_dev { +void *parent_accel_dev; +u8 *state; +u32 setup_size; +u32 remote_setup_size; +u32 state_size; +s32 vf_id; +}; + +struct qat_mig_dev *qat_vfmig_create (struct pci_dev *pdev,int vf_id); +int qat_vfmig_init (struct qat_mig_dev *mdev); +void qat_vfmig_cleanup (struct qat_mig_dev *mdev); +void qat_vfmig_reset (struct qat_mig_dev *mdev); +int qat_vfmig_open (struct qat_mig_dev *mdev); +void qat_vfmig_close (struct qat_mig_dev *mdev); +int qat_vfmig_suspend (struct qat_mig_dev *mdev); +int qat_vfmig_resume (struct qat_mig_dev *mdev); +int qat_vfmig_save_state (struct qat_mig_dev *mdev); +int qat_vfmig_save_setup (struct qat_mig_dev *mdev); +int qat_vfmig_load_state (struct qat_mig_dev *mdev); +int qat_vfmig_load_setup (struct qat_mig_dev *mdev,int size); +void qat_vfmig_destroy (struct qat_mig_dev *mdev); + +#endif /*QAT_MIG_DEV_H_*/
Signature · 1 line
2.18.2
Jason Gunthorpe
reviewerRe: [PATCH v5 08/10] crypto: qat - add interface for live migration
Quoted 7 lines · Xin Zeng
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +3
-0@@ -258,6 +259,20 @@ struct adf_dc_ops {void (*build_deflate_ctx)(void *ctx); }; +struct qat_migdev_ops { +int (*init)(struct qat_mig_dev *mdev); +void (*cleanup)(struct qat_mig_dev *mdev);
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h struct adf_dc_ops { · line 274
Quoted 10 lines
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +10
-0+void (*reset)(struct qat_mig_dev *mdev); +int (*open)(struct qat_mig_dev *mdev); +void (*close)(struct qat_mig_dev *mdev); +int (*suspend)(struct qat_mig_dev *mdev); +int (*resume)(struct qat_mig_dev *mdev); +int (*save_state)(struct qat_mig_dev *mdev); +int (*save_setup)(struct qat_mig_dev *mdev); +int (*load_state)(struct qat_mig_dev *mdev); +int (*load_setup)(struct qat_mig_dev *mdev,int size); +};
Why do we still have these ops? There is only one implementation
Jason
Cabiddu, Giovanni
maintainerRe: [PATCH v5 08/10] crypto: qat - add interface for live migration
Hi Jason,
Quoted 5 lines · Jason Gunthorpe
Quoted 4 lines · Xin Zeng
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +0
-0@@ -258,6 +259,20 @@ struct adf_dc_ops {void (*build_deflate_ctx)(void *ctx); };
Quoted 15 lines
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h struct adf_dc_ops { · line 274
Quoted 13 lines
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +13
-0+struct qat_migdev_ops { +int (*init)(struct qat_mig_dev *mdev); +void (*cleanup)(struct qat_mig_dev *mdev); +void (*reset)(struct qat_mig_dev *mdev); +int (*open)(struct qat_mig_dev *mdev); +void (*close)(struct qat_mig_dev *mdev); +int (*suspend)(struct qat_mig_dev *mdev); +int (*resume)(struct qat_mig_dev *mdev); +int (*save_state)(struct qat_mig_dev *mdev); +int (*save_setup)(struct qat_mig_dev *mdev); +int (*load_state)(struct qat_mig_dev *mdev); +int (*load_setup)(struct qat_mig_dev *mdev,int size); +};Why do we still have these ops? There is only one implementation
This is related to the architecture of the QAT driver. The core QAT module (intel_qat.ko), which exposes the functions required for live migration, supports multiple devices. Each QAT specific driver registers the functions that it supports.
Even if live migration is currently supported only by QAT GEN4 devices, my preference is to keep the qat_vfio_pci module to invoke GEN agnostic functions. Also, I would prefer to avoid having such functions calling directly the GEN4 specific implementation.
Regards,
Signature · 1 line
Giovanni
PATCH v5 9/10 crypto: qat - implement interface for live migration
In reply to Xin ZengAdd logic to implement the interface for live migration defined in qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions (VFs).
This introduces a migration data manager which is used to handle the device state during migration. The manager ensures that the device state is stored in a format that can be restored in the destination node.
The VF state is organized into a hierarchical structure that includes a preamble, a general state section, a MISC bar section and an ETR bar section. The latter contains the state of the 4 ring pairs contained on a VF. Here is a graphical representation of the state:
preamble | general state section | leaf state
| MISC bar state section| leaf state
| ETR bar state section | bank0 state section | leaf state
| bank1 state section | leaf state
| bank2 state section | leaf state
| bank3 state section | leaf stateIn addition to the implementation of the qat_migdev_ops interface and the state manager framework, add a mutex in pfvf to avoid pf2vf messages during migration.
--- .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 2 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 2 + drivers/crypto/intel/qat/qat_common/Makefile | 2 + .../intel/qat/qat_common/adf_accel_devices.h | 6 + .../intel/qat/qat_common/adf_gen4_hw_data.h | 10 + .../intel/qat/qat_common/adf_gen4_vf_mig.c | 1010 +++++++++++++++++ .../intel/qat/qat_common/adf_mstate_mgr.c | 318 ++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 89 ++ .../crypto/intel/qat/qat_common/adf_sriov.c | 7 +- 9 files changed, 1445 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +2 -0
@@ -17,6 +17,7 @@#include <adf_gen4_ras.h> #include <adf_gen4_timer.h> #include <adf_gen4_tl.h> +#include <adf_gen4_vf_mig.h> #include "adf_420xx_hw_data.h" #include "icp_qat_hw.h"
@@ -488,6 +489,7 @@ void adf_init_hw_data_420xx(struct adf_hw_device_data *hw_data, u32 dev_id)adf_gen4_init_dc_ops (&hw_data->dc_ops);adf_gen4_init_ras_ops (&hw_data->ras_ops);adf_gen4_init_tl_data (&hw_data->tl_data); +adf_gen4_init_vf_mig_ops (&hw_data->vfmig_ops);adf_init_rl_data (&hw_data->rl_data); }
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +2 -0
@@ -17,6 +17,7 @@#include "adf_gen4_ras.h" #include <adf_gen4_timer.h> #include <adf_gen4_tl.h> +#include <adf_gen4_vf_mig.h> #include "adf_4xxx_hw_data.h" #include "icp_qat_hw.h"
@@ -472,6 +473,7 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data, u32 dev_id)adf_gen4_init_dc_ops (&hw_data->dc_ops);adf_gen4_init_ras_ops (&hw_data->ras_ops);adf_gen4_init_tl_data (&hw_data->tl_data); +adf_gen4_init_vf_mig_ops (&hw_data->vfmig_ops);adf_init_rl_data (&hw_data->rl_data); }
drivers/crypto/intel/qat/qat_common/Makefile +2 -0
@@ -20,12 +20,14 @@ intel_qat-objs := adf_cfg.o \ adf_gen4_config.o \ adf_gen4_hw_csr_data.o \ adf_gen4_hw_data.o \ + adf_gen4_vf_mig.o \ adf_gen4_pm.o \ adf_gen2_dc.o \ adf_gen4_dc.o \ adf_gen4_ras.o \ adf_gen4_timer.o \ adf_clock.o \ + adf_mstate_mgr.o \ qat_crypto.o \ qat_compression.o \ qat_comp_algs.o \
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +6 -0
@@ -412,11 +412,17 @@ struct adf_fw_loader_data {struct adf_accel_vf_info {struct adf_accel_dev *accel_dev;struct mutex pf2vf_lock;/* protect CSR access for PF2VF messages */ +struct mutex pfvf_mig_lock;/* protects PFVF state for migration */ struct ratelimit_state vf2pf_ratelimit;u32 vf_nr;bool init;bool restarting;u8 vf_compat_ver; +/* +* Private area used for device migration. +* Memory allocation and free is managed by migration driver. +*/ +void *mig_priv; };struct adf_dc_data {
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +10 -0
@@ -86,6 +86,7 @@#define ADF_RP_INT_SRC_SEL_F_RISE_MASK BIT(2)#define ADF_RP_INT_SRC_SEL_F_FALL_MASK GENMASK(2, 0)#define ADF_RP_INT_SRC_SEL_RANGE_WIDTH 4 +#define ADF_COALESCED_POLL_TIMEOUT_US (1 * USEC_PER_SEC)#define ADF_COALESCED_POLL_DELAY_US 1000#define ADF_WQM_CSR_RPINTSOU (bank) (0x200000 + ((bank) << 12))#define ADF_WQM_CSR_RP_IDX_RX 1
@@ -120,6 +121,15 @@/* PF2VM communication channel */ #define ADF_GEN4_PF2VM_OFFSET (i) (0x40B010 + (i) * 0x20)#define ADF_GEN4_VM2PF_OFFSET (i) (0x40B014 + (i) * 0x20) +#define ADF_GEN4_VINTMSKPF2VM_OFFSET (i) (0x40B00C + (i) * 0x20) +#define ADF_GEN4_VINTSOUPF2VM_OFFSET (i) (0x40B008 + (i) * 0x20) +#define ADF_GEN4_VINTMSK_OFFSET (i) (0x40B004 + (i) * 0x20) +#define ADF_GEN4_VINTSOU_OFFSET (i) (0x40B000 + (i) * 0x20) + +struct adf_gen4_vfmig { +struct adf_mstate_mgr *mstate_mgr; +bool bank_stopped[ADF_GEN4_NUM_BANKS_PER_VF ]; +};void adf_gen4_set_ssm_wdtimer (struct adf_accel_dev *accel_dev);
drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c new file +1010 -0
1,010 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c new file +318 -0
318 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h new file +89 -0
89 lines not on this page · raw message
drivers/crypto/intel/qat/qat_common/adf_sriov.c +6 -1
@@ -26,10 +26,12 @@ static void adf_iov_send_resp(struct work_struct *work)u32 vf_nr = vf_info->vf_nr;bool ret; +mutex_lock (&vf_info->pfvf_mig_lock); ret =adf_recv_and_handle_vf2pf_msg (accel_dev, vf_nr);if (ret)/* re-enable interrupt on PF from this VF */ adf_enable_vf2pf_interrupts (accel_dev,1 << vf_nr); +mutex_unlock (&vf_info->pfvf_mig_lock);kfree (pf2vf_resp); }
@@ -62,6 +64,7 @@ static int adf_enable_sriov(struct adf_accel_dev *accel_dev) vf_info->vf_nr = i;mutex_init (&vf_info->pf2vf_lock); +mutex_init (&vf_info->pfvf_mig_lock);ratelimit_state_init (&vf_info->vf2pf_ratelimit,ADF_VF2PF_RATELIMIT_INTERVAL ,ADF_VF2PF_RATELIMIT_BURST );
@@ -138,8 +141,10 @@ void adf_disable_sriov(struct adf_accel_dev *accel_dev)if (hw_data->configure_iov_threads) hw_data->configure_iov_threads (accel_dev, false); -for (i =0 , vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) +for (i =0 , vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) {mutex_destroy (&vf->pf2vf_lock); +mutex_destroy (&vf->pfvf_mig_lock); + }if (!test_bit (ADF_STATUS_RESTARTING , &accel_dev->status)) {kfree (accel_dev->pf.vf_info);
Signature · 1 line
2.18.2
Re: [PATCH v5 09/10] crypto: qat - implement interface for live migration
Quoted 1,156 lines · Xin Zeng
Add logic to implement the interface for live migration defined in
qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions (VFs).This introduces a migration data manager which is used to handle the device state during migration. The manager ensures that the device state is stored in a format that can be restored in the destination node.
The VF state is organized into a hierarchical structure that includes a preamble, a general state section, a MISC bar section and an ETR bar section. The latter contains the state of the 4 ring pairs contained on a VF. Here is a graphical representation of the state:
preamble | general state section | leaf state | MISC bar state section| leaf state | ETR bar state section | bank0 state section | leaf state | bank1 state section | leaf state | bank2 state section | leaf state | bank3 state section | leaf stateIn addition to the implementation of the
qat_migdev_opsinterface and the state manager framework, add a mutex in pfvf to avoid pf2vf messages during migration.Signed-off-by Xin Zeng <[email protected]>Reviewed-by Giovanni Cabiddu <[email protected]>--- .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 2 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 2 + drivers/crypto/intel/qat/qat_common/Makefile | 2 + .../intel/qat/qat_common/adf_accel_devices.h | 6 + .../intel/qat/qat_common/adf_gen4_hw_data.h | 10 + .../intel/qat/qat_common/adf_gen4_vf_mig.c | 1010 +++++++++++++++++ .../intel/qat/qat_common/adf_mstate_mgr.c | 318 ++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 89 ++ .../crypto/intel/qat/qat_common/adf_sriov.c | 7 +- 9 files changed, 1445 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.hdrivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +2
-0@@ -17,6 +17,7 @@#include <adf_gen4_ras.h> #include <adf_gen4_timer.h> #include <adf_gen4_tl.h> +#include <adf_gen4_vf_mig.h> #include "adf_420xx_hw_data.h" #include "icp_qat_hw.h" @@ -488,6 +489,7 @@ void adf_init_hw_data_420xx(struct adf_hw_device_data *hw_data, u32 dev_id)adf_gen4_init_dc_ops (&hw_data->dc_ops);adf_gen4_init_ras_ops (&hw_data->ras_ops);adf_gen4_init_tl_data (&hw_data->tl_data); +adf_gen4_init_vf_mig_ops (&hw_data->vfmig_ops);adf_init_rl_data (&hw_data->rl_data); }drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +2
-0@@ -17,6 +17,7 @@#include "adf_gen4_ras.h" #include <adf_gen4_timer.h> #include <adf_gen4_tl.h> +#include <adf_gen4_vf_mig.h> #include "adf_4xxx_hw_data.h" #include "icp_qat_hw.h" @@ -472,6 +473,7 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data, u32 dev_id)adf_gen4_init_dc_ops (&hw_data->dc_ops);adf_gen4_init_ras_ops (&hw_data->ras_ops);adf_gen4_init_tl_data (&hw_data->tl_data); +adf_gen4_init_vf_mig_ops (&hw_data->vfmig_ops);adf_init_rl_data (&hw_data->rl_data); }drivers/crypto/intel/qat/qat_common/Makefile +2
-0@@ -20,12 +20,14 @@ intel_qat-objs := adf_cfg.o \ adf_gen4_config.o \ adf_gen4_hw_csr_data.o \ adf_gen4_hw_data.o \ + adf_gen4_vf_mig.o \ adf_gen4_pm.o \ adf_gen2_dc.o \ adf_gen4_dc.o \ adf_gen4_ras.o \ adf_gen4_timer.o \ adf_clock.o \ + adf_mstate_mgr.o \ qat_crypto.o \ qat_compression.o \ qat_comp_algs.o \drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +6
-0@@ -412,11 +412,17 @@ struct adf_fw_loader_data {struct adf_accel_vf_info {struct adf_accel_dev *accel_dev;struct mutex pf2vf_lock;/* protect CSR access for PF2VF messages */ +struct mutex pfvf_mig_lock;/* protects PFVF state for migration */ struct ratelimit_state vf2pf_ratelimit;u32 vf_nr;bool init;bool restarting;u8 vf_compat_ver; +/* +* Private area used for device migration. +* Memory allocation and free is managed by migration driver. +*/ +void *mig_priv; };struct adf_dc_data {drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +10
-0@@ -86,6 +86,7 @@#define ADF_RP_INT_SRC_SEL_F_RISE_MASK BIT(2)#define ADF_RP_INT_SRC_SEL_F_FALL_MASK GENMASK(2, 0)#define ADF_RP_INT_SRC_SEL_RANGE_WIDTH 4 +#define ADF_COALESCED_POLL_TIMEOUT_US (1 * USEC_PER_SEC)#define ADF_COALESCED_POLL_DELAY_US 1000#define ADF_WQM_CSR_RPINTSOU (bank) (0x200000 + ((bank) << 12))#define ADF_WQM_CSR_RP_IDX_RX 1@@ -120,6 +121,15 @@/* PF2VM communication channel */ #define ADF_GEN4_PF2VM_OFFSET (i) (0x40B010 + (i) * 0x20)#define ADF_GEN4_VM2PF_OFFSET (i) (0x40B014 + (i) * 0x20) +#define ADF_GEN4_VINTMSKPF2VM_OFFSET (i) (0x40B00C + (i) * 0x20) +#define ADF_GEN4_VINTSOUPF2VM_OFFSET (i) (0x40B008 + (i) * 0x20) +#define ADF_GEN4_VINTMSK_OFFSET (i) (0x40B004 + (i) * 0x20) +#define ADF_GEN4_VINTSOU_OFFSET (i) (0x40B000 + (i) * 0x20) + +struct adf_gen4_vfmig { +struct adf_mstate_mgr *mstate_mgr; +bool bank_stopped[ADF_GEN4_NUM_BANKS_PER_VF ]; +};void adf_gen4_set_ssm_wdtimer (struct adf_accel_dev *accel_dev);drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c new file +1000
-0@@ -0,0 +1,1010 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2024 Intel Corporation */ +#include <linux/delay.h> +#include <linux/dev_printk.h> +#include <linux/kernel.h> +#include <linux/slab.h> +#include <linux/string.h> +#include <linux/types.h> +#include <asm/errno.h> + +#include "adf_accel_devices.h" +#include "adf_common_drv.h" +#include "adf_gen4_hw_data.h" +#include "adf_gen4_pfvf.h" +#include "adf_pfvf_utils.h" +#include "adf_mstate_mgr.h" +#include "adf_gen4_vf_mig.h" + +#define ADF_GEN4_VF_MSTATE_SIZE 4096 +#define ADF_GEN4_PFVF_RSP_TIMEOUT_US 5000 + +static int adf_gen4_vfmig_save_setup (struct qat_mig_dev *mdev); +static int adf_gen4_vfmig_load_setup (struct qat_mig_dev *mdev,int len); + +static int adf_gen4_vfmig_init_device (struct qat_mig_dev *mdev) +{ +u8 *state; + + state =kmalloc (ADF_GEN4_VF_MSTATE_SIZE ,GFP_KERNEL ); +if (!state) +return -ENOMEM ; + + mdev->state = state; + mdev->state_size =ADF_GEN4_VF_MSTATE_SIZE ; + mdev->setup_size =0 ; + mdev->remote_setup_size =0 ; + +return 0 ; +} + +static void adf_gen4_vfmig_cleanup_device (struct qat_mig_dev *mdev) +{ +kfree (mdev->state); + mdev->state =NULL ; +} + +static void adf_gen4_vfmig_reset_device (struct qat_mig_dev *mdev) +{ + mdev->setup_size =0 ; + mdev->remote_setup_size =0 ; +} + +static int adf_gen4_vfmig_open_device (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vfmig; + + vf_info = &accel_dev->pf.vf_info[mdev->vf_id]; + + vfmig =kzalloc (sizeof (*vfmig),GFP_KERNEL ); +if (!vfmig) +return -ENOMEM ; + + vfmig->mstate_mgr =adf_mstate_mgr_new (mdev->state, mdev->state_size); +if (!vfmig->mstate_mgr) { +kfree (vfmig); +return -ENOMEM ; + } + vf_info->mig_priv = vfmig; + mdev->setup_size =0 ; + mdev->remote_setup_size =0 ; + +return 0 ; +} + +static void adf_gen4_vfmig_close_device (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vfmig; + + vf_info = &accel_dev->pf.vf_info[mdev->vf_id]; +if (vf_info->mig_priv) { + vfmig = vf_info->mig_priv; +adf_mstate_mgr_destroy (vfmig->mstate_mgr); +kfree (vfmig); + vf_info->mig_priv =NULL ; + } +} + +static int adf_gen4_vfmig_suspend_device (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vf_mig; +u32 vf_nr = mdev->vf_id; +int ret, i; + + vf_info = &accel_dev->pf.vf_info[vf_nr]; + vf_mig = vf_info->mig_priv; + +/* Stop all inflight jobs */ +for (i =0 ; i < hw_data->num_banks_per_vf; i++) { +u32 pf_bank_nr = i + vf_nr * hw_data->num_banks_per_vf; + + ret =adf_gen4_bank_drain_start (accel_dev, pf_bank_nr, +ADF_RPRESET_POLL_TIMEOUT_US ); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to drain bank %d for vf_nr %d\n" , i, + vf_nr); +return ret; + } + vf_mig->bank_stopped[i] = true; + +adf_gen4_bank_quiesce_coal_timer (accel_dev, pf_bank_nr, +ADF_COALESCED_POLL_TIMEOUT_US ); + } + +return 0 ; +} + +static int adf_gen4_vfmig_resume_device (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vf_mig; +u32 vf_nr = mdev->vf_id; +int i; + + vf_info = &accel_dev->pf.vf_info[vf_nr]; + vf_mig = vf_info->mig_priv; + +for (i =0 ; i < hw_data->num_banks_per_vf; i++) { +u32 pf_bank_nr = i + vf_nr * hw_data->num_banks_per_vf; + +if (vf_mig->bank_stopped[i]) { +adf_gen4_bank_drain_finish (accel_dev, pf_bank_nr); + vf_mig->bank_stopped[i] = false; + } + } + +return 0 ; +} + +struct adf_vf_bank_info { +struct adf_accel_dev *accel_dev; +u32 vf_nr; +u32 bank_nr; +}; + +struct mig_user_sla { +enum adf_base_services srv; +u64 rp_mask; +u32 cir; +u32 pir; +}; + +static int adf_mstate_sla_check (struct adf_mstate_mgr *sub_mgr,u8 *src_buf, +u32 src_size,void *opaque) +{ +struct adf_mstate_vreginfo _sinfo = { src_buf, src_size }; +struct adf_mstate_vreginfo *sinfo = &_sinfo, *dinfo = opaque; +u32 src_sla_cnt = sinfo->size /sizeof (struct mig_user_sla ); +u32 dst_sla_cnt = dinfo->size /sizeof (struct mig_user_sla ); +struct mig_user_sla *src_slas = sinfo->addr; +struct mig_user_sla *dst_slas = dinfo->addr; +int i, j; + +for (i =0 ; i < src_sla_cnt; i++) { +for (j =0 ; j < dst_sla_cnt; j++) { +if (src_slas[i].srv != dst_slas[j].srv || + src_slas[i].rp_mask != dst_slas[j].rp_mask) +continue ; + +if (src_slas[i].cir > dst_slas[j].cir || + src_slas[i].pir > dst_slas[j].pir) { +pr_err ("QAT: DST VF rate limiting mismatch.\n" ); +return -EINVAL ; + } +break ; + } + +if (j == dst_sla_cnt) { +pr_err ("QAT: SRC VF rate limiting mismatch - SRC srv %d and rp_mask 0x%llx.\n" , + src_slas[i].srv, src_slas[i].rp_mask); +return -EINVAL ; + } + } + +return 0 ; +} + +static inline int adf_mstate_check_cap_size (u32 src_sz,u32 dst_sz,u32 max_sz) +{ +if (src_sz > max_sz || dst_sz > max_sz) +return -EINVAL ; +else +return 0 ; +} + +static int adf_mstate_compatver_check (struct adf_mstate_mgr *sub_mgr, +u8 *src_buf,u32 src_sz,void *opaque) +{ +struct adf_mstate_vreginfo *info = opaque; +u8 compat =0 ; +u8 *pcompat; + +if (src_sz != info->size) { +pr_debug ("QAT: State mismatch (compat version size), current %u, expected %u\n" , + src_sz, info->size); +return -EINVAL ; + } + +memcpy (info->addr, src_buf, info->size); + pcompat = info->addr; +if (*pcompat ==0 ) { +pr_warn ("QAT: Unable to determine the version of VF\n" ); +return 0 ; + } + + compat =adf_vf_compat_checker (*pcompat); +if (compat ==ADF_PF2VF_VF_INCOMPATIBLE ) { +pr_debug ("QAT: SRC VF driver (ver=%u) is incompatible with DST PF driver (ver=%u)\n" , + *pcompat,ADF_PFVF_COMPAT_THIS_VERSION ); +return -EINVAL ; + } + +if (compat ==ADF_PF2VF_VF_COMPAT_UNKNOWN ) +pr_debug ("QAT: SRC VF driver (ver=%u) is newer than DST PF driver (ver=%u)\n" , + *pcompat,ADF_PFVF_COMPAT_THIS_VERSION ); + +return 0 ; +} + +/* +* adf_mstate_capmask_compare() - compare QAT device capability mask +* @sinfo: Pointer to source capability info +* @dinfo: Pointer to target capability info +* +* This function compares the capability mask between source VF and target VF +* +* Returns: 0 if target capability mask is identical to source capability mask, +* 1 if target mask can represent all the capabilities represented by source mask, +* -1 if target mask can't represent all the capabilities represented by source +* mask. +*/ +static int adf_mstate_capmask_compare (struct adf_mstate_vreginfo *sinfo, +struct adf_mstate_vreginfo *dinfo) +{ +u64 src =0 , dst =0 ; + +if (adf_mstate_check_cap_size (sinfo->size, dinfo->size,sizeof (u64))) { +pr_debug ("QAT: Unexpected capability size %u %u %zu\n" , + sinfo->size, dinfo->size,sizeof (u64)); +return -1 ; + } + +memcpy (&src, sinfo->addr, sinfo->size); +memcpy (&dst, dinfo->addr, dinfo->size); + +pr_debug ("QAT: Check cap compatibility of cap %llu %llu\n" , src, dst); + +if (src == dst) +return 0 ; + +if ((src | dst) == dst) +return 1 ; + +return -1 ; +} + +static int adf_mstate_capmask_superset (struct adf_mstate_mgr *sub_mgr,u8 *buf, +u32 size,void *opa) +{ +struct adf_mstate_vreginfo sinfo = { buf, size }; + +if (adf_mstate_capmask_compare (&sinfo, opa) >=0 ) +return 0 ; + +return -EINVAL ; +} + +static int adf_mstate_capmask_equal (struct adf_mstate_mgr *sub_mgr,u8 *buf, +u32 size,void *opa) +{ +struct adf_mstate_vreginfo sinfo = { buf, size }; + +if (adf_mstate_capmask_compare (&sinfo, opa) ==0 ) +return 0 ; + +return -EINVAL ; +} + +static int adf_mstate_set_vreg (struct adf_mstate_mgr *sub_mgr,u8 *buf, +u32 size,void *opa) +{ +struct adf_mstate_vreginfo *info = opa; + +if (size != info->size) { +pr_debug ("QAT: Unexpected cap size %u %u\n" , size, info->size); +return -EINVAL ; + } +memcpy (info->addr, buf, info->size); + +return 0 ; +} + +static u32 adf_gen4_vfmig_get_slas (struct adf_accel_dev *accel_dev,u32 vf_nr, +struct mig_user_sla *pmig_slas) +{ +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_rl *rl_data = accel_dev->rate_limiting; +struct rl_sla **sla_type_arr =NULL ; +u64 rp_mask, rp_index; +u32 max_num_sla; +u32 sla_cnt =0 ; +int i, j; + +if (!accel_dev->rate_limiting) +return 0 ; + + rp_index = vf_nr * hw_data->num_banks_per_vf; + max_num_sla =adf_rl_get_sla_arr_of_type (rl_data,RL_LEAF , &sla_type_arr); + +for (i =0 ; i < max_num_sla; i++) { +if (!sla_type_arr[i]) +continue ; + + rp_mask =0 ; +for (j =0 ; j < sla_type_arr[i]->ring_pairs_cnt; j++) + rp_mask |=BIT (sla_type_arr[i]->ring_pairs_ids[j]); + +if (rp_mask &GENMASK_ULL (rp_index +3 , rp_index)) { + pmig_slas->rp_mask = rp_mask; + pmig_slas->cir = sla_type_arr[i]->cir; + pmig_slas->pir = sla_type_arr[i]->pir; + pmig_slas->srv = sla_type_arr[i]->srv; + pmig_slas++; + sla_cnt++; + } + } + +return sla_cnt; +} + +static int adf_gen4_vfmig_load_etr_regs (struct adf_mstate_mgr *sub_mgr, +u8 *state,u32 size,void *opa) +{ +struct adf_vf_bank_info *vf_bank_info = opa; +struct adf_accel_dev *accel_dev = vf_bank_info->accel_dev; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +u32 pf_bank_nr; +int ret; + + pf_bank_nr = vf_bank_info->bank_nr + vf_bank_info->vf_nr * hw_data->num_banks_per_vf; + ret = hw_data->bank_state_restore (accel_dev, pf_bank_nr, + (struct bank_state *)state); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to load regs for vf%d bank%d\n" , + vf_bank_info->vf_nr, vf_bank_info->bank_nr); +return ret; + } + +return 0 ; +} + +static int adf_gen4_vfmig_load_etr_bank (struct adf_accel_dev *accel_dev, +u32 vf_nr,u32 bank_nr, +struct adf_mstate_mgr *mstate_mgr) +{ +struct adf_vf_bank_info vf_bank_info = {accel_dev, vf_nr, bank_nr}; +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct adf_mstate_mgr sub_sects_mgr; +char bank_ids[ADF_MSTATE_ID_LEN ]; + +snprintf (bank_ids,sizeof (bank_ids),ADF_MSTATE_BANK_IDX_IDS "%x" , bank_nr); + subsec =adf_mstate_sect_lookup (mstate_mgr, bank_ids,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev), +"Failed to lookup sec %s for vf%d bank%d\n" , +ADF_MSTATE_BANK_IDX_IDS , vf_nr, bank_nr); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_psect (&sub_sects_mgr, subsec); + l2_subsec =adf_mstate_sect_lookup (&sub_sects_mgr,ADF_MSTATE_ETR_REGS_IDS , + adf_gen4_vfmig_load_etr_regs, + &vf_bank_info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev), +"Failed to add sec %s for vf%d bank%d\n" , +ADF_MSTATE_ETR_REGS_IDS , vf_nr, bank_nr); +return -EINVAL ; + } + +return 0 ; +} + +static int adf_gen4_vfmig_load_etr (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_mgr sub_sects_mgr; +struct adf_mstate_sect_h *subsec; +int ret, i; + + subsec =adf_mstate_sect_lookup (mstate_mgr,ADF_MSTATE_ETRB_IDS ,NULL , +NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to load sec %s\n" , +ADF_MSTATE_ETRB_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_psect (&sub_sects_mgr, subsec); +for (i =0 ; i < hw_data->num_banks_per_vf; i++) { + ret =adf_gen4_vfmig_load_etr_bank (accel_dev, vf_nr, i, + &sub_sects_mgr); +if (ret) +return ret; + } + +return 0 ; +} + +static int adf_gen4_vfmig_load_misc (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +void __iomem *csr =adf_get_pmisc_base (accel_dev); +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct adf_mstate_mgr sub_sects_mgr; +struct { +char *id; +u64 ofs; + } misc_states[] = { + {ADF_MSTATE_VINTMSK_IDS ,ADF_GEN4_VINTMSK_OFFSET (vf_nr)}, + {ADF_MSTATE_VINTMSK_PF2VM_IDS ,ADF_GEN4_VINTMSKPF2VM_OFFSET (vf_nr)}, + {ADF_MSTATE_PF2VM_IDS ,ADF_GEN4_PF2VM_OFFSET (vf_nr)}, + {ADF_MSTATE_VM2PF_IDS ,ADF_GEN4_VM2PF_OFFSET (vf_nr)}, + }; +int i; + + subsec =adf_mstate_sect_lookup (mstate_mgr,ADF_MSTATE_MISCB_IDS ,NULL , +NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to load sec %s\n" , +ADF_MSTATE_MISCB_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_psect (&sub_sects_mgr, subsec); +for (i =0 ; i <ARRAY_SIZE (misc_states); i++) { +struct adf_mstate_vreginfo info; +u32 regv; + + info.addr = ®v; + info.size =sizeof (regv); + l2_subsec =adf_mstate_sect_lookup (&sub_sects_mgr, + misc_states[i].id, + adf_mstate_set_vreg, + &info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev), +"Failed to load sec %s\n" , misc_states[i].id); +return -EINVAL ; + } +ADF_CSR_WR (csr, misc_states[i].ofs, regv); + } + +return 0 ; +} + +static int adf_gen4_vfmig_load_generic (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct mig_user_sla dst_slas[RL_RP_CNT_PER_LEAF_MAX ] = { }; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct adf_mstate_mgr sub_sects_mgr; +u32 dst_sla_cnt; +struct { +char *id; +int (*action)(struct adf_mstate_mgr *sub_mgr,u8 *buf,u32 size,void *opa); +struct adf_mstate_vreginfo info; + } gen_states[] = { + {ADF_MSTATE_IOV_INIT_IDS , adf_mstate_set_vreg, + {&vf_info->init,sizeof (vf_info->init)}}, + {ADF_MSTATE_COMPAT_VER_IDS , adf_mstate_compatver_check, + {&vf_info->vf_compat_ver,sizeof (vf_info->vf_compat_ver)}}, + {ADF_MSTATE_SLA_IDS , adf_mstate_sla_check, {dst_slas,0 }}, + }; +int i; + + subsec =adf_mstate_sect_lookup (mstate_mgr,ADF_MSTATE_GEN_IDS ,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to load sec %s\n" , +ADF_MSTATE_GEN_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_psect (&sub_sects_mgr, subsec); +for (i =0 ; i <ARRAY_SIZE (gen_states); i++) { +if (gen_states[i].info.addr == dst_slas) { + dst_sla_cnt =adf_gen4_vfmig_get_slas (accel_dev, vf_nr, dst_slas); + gen_states[i].info.size = dst_sla_cnt *sizeof (struct mig_user_sla ); + } + + l2_subsec =adf_mstate_sect_lookup (&sub_sects_mgr, + gen_states[i].id, + gen_states[i].action, + &gen_states[i].info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to load sec %s\n" , + gen_states[i].id); +return -EINVAL ; + } + } + +return 0 ; +} + +static int adf_gen4_vfmig_load_config (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct adf_mstate_mgr sub_sects_mgr; +struct { +char *id; +int (*action)(struct adf_mstate_mgr *sub_mgr,u8 *buf,u32 size,void *opa); +struct adf_mstate_vreginfo info; + } setups[] = { + {ADF_MSTATE_GEN_CAP_IDS , adf_mstate_capmask_superset, + {&hw_data->accel_capabilities_mask,sizeof (hw_data->accel_capabilities_mask)}}, + {ADF_MSTATE_GEN_SVCMAP_IDS , adf_mstate_capmask_equal, + {&hw_data->ring_to_svc_map,sizeof (hw_data->ring_to_svc_map)}}, + {ADF_MSTATE_GEN_EXTDC_IDS , adf_mstate_capmask_superset, + {&hw_data->extended_dc_capabilities,sizeof (hw_data->extended_dc_capabilities)}}, + }; +int i; + + subsec =adf_mstate_sect_lookup (mstate_mgr,ADF_MSTATE_CONFIG_IDS ,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to load sec %s\n" , +ADF_MSTATE_CONFIG_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_psect (&sub_sects_mgr, subsec); +for (i =0 ; i <ARRAY_SIZE (setups); i++) { + l2_subsec =adf_mstate_sect_lookup (&sub_sects_mgr, setups[i].id, + setups[i].action, &setups[i].info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to load sec %s\n" , + setups[i].id); +return -EINVAL ; + } + } + +return 0 ; +} + +static int adf_gen4_vfmig_save_etr_regs (struct adf_mstate_mgr *subs,u8 *state, +u32 size,void *opa) +{ +struct adf_vf_bank_info *vf_bank_info = opa; +struct adf_accel_dev *accel_dev = vf_bank_info->accel_dev; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +u32 pf_bank_nr; +int ret; + + pf_bank_nr = vf_bank_info->bank_nr; + pf_bank_nr += vf_bank_info->vf_nr * hw_data->num_banks_per_vf; + + ret = hw_data->bank_state_save (accel_dev, pf_bank_nr, + (struct bank_state *)state); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to save regs for vf%d bank%d\n" , + vf_bank_info->vf_nr, vf_bank_info->bank_nr); +return ret; + } + +return sizeof (struct bank_state ); +} + +static int adf_gen4_vfmig_save_etr_bank (struct adf_accel_dev *accel_dev, +u32 vf_nr,u32 bank_nr, +struct adf_mstate_mgr *mstate_mgr) +{ +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct adf_vf_bank_info vf_bank_info; +struct adf_mstate_mgr sub_sects_mgr; +char bank_ids[ADF_MSTATE_ID_LEN ]; + +snprintf (bank_ids,sizeof (bank_ids),ADF_MSTATE_BANK_IDX_IDS "%x" , bank_nr); + + subsec =adf_mstate_sect_add (mstate_mgr, bank_ids,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev), +"Failed to add sec %s for vf%d bank%d\n" , +ADF_MSTATE_BANK_IDX_IDS , vf_nr, bank_nr); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_parent (&sub_sects_mgr, mstate_mgr); + vf_bank_info.accel_dev = accel_dev; + vf_bank_info.vf_nr = vf_nr; + vf_bank_info.bank_nr = bank_nr; + l2_subsec =adf_mstate_sect_add (&sub_sects_mgr,ADF_MSTATE_ETR_REGS_IDS , + adf_gen4_vfmig_save_etr_regs, + &vf_bank_info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev), +"Failed to add sec %s for vf%d bank%d\n" , +ADF_MSTATE_ETR_REGS_IDS , vf_nr, bank_nr); +return -EINVAL ; + } +adf_mstate_sect_update (mstate_mgr, &sub_sects_mgr, subsec); + +return 0 ; +} + +static int adf_gen4_vfmig_save_etr (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_mgr sub_sects_mgr; +struct adf_mstate_sect_h *subsec; +int ret, i; + + subsec =adf_mstate_sect_add (mstate_mgr,ADF_MSTATE_ETRB_IDS ,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , +ADF_MSTATE_ETRB_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_parent (&sub_sects_mgr, mstate_mgr); +for (i =0 ; i < hw_data->num_banks_per_vf; i++) { + ret =adf_gen4_vfmig_save_etr_bank (accel_dev, vf_nr, i, + &sub_sects_mgr); +if (ret) +return ret; + } +adf_mstate_sect_update (mstate_mgr, &sub_sects_mgr, subsec); + +return 0 ; +} + +static int adf_gen4_vfmig_save_misc (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +void __iomem *csr =adf_get_pmisc_base (accel_dev); +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct adf_mstate_mgr sub_sects_mgr; +struct { +char *id; +u64 offset; + } misc_states[] = { + {ADF_MSTATE_VINTSRC_IDS ,ADF_GEN4_VINTSOU_OFFSET (vf_nr)}, + {ADF_MSTATE_VINTMSK_IDS ,ADF_GEN4_VINTMSK_OFFSET (vf_nr)}, + {ADF_MSTATE_VINTSRC_PF2VM_IDS ,ADF_GEN4_VINTSOUPF2VM_OFFSET (vf_nr)}, + {ADF_MSTATE_VINTMSK_PF2VM_IDS ,ADF_GEN4_VINTMSKPF2VM_OFFSET (vf_nr)}, + {ADF_MSTATE_PF2VM_IDS ,ADF_GEN4_PF2VM_OFFSET (vf_nr)}, + {ADF_MSTATE_VM2PF_IDS ,ADF_GEN4_VM2PF_OFFSET (vf_nr)}, + }; +ktime_t time_exp; +int i; + + subsec =adf_mstate_sect_add (mstate_mgr,ADF_MSTATE_MISCB_IDS ,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , +ADF_MSTATE_MISCB_IDS ); +return -EINVAL ; + } + + time_exp =ktime_add_us (ktime_get (),ADF_GEN4_PFVF_RSP_TIMEOUT_US ); +while (!mutex_trylock (&vf_info->pfvf_mig_lock)) { +if (ktime_after (ktime_get (), time_exp)) { +dev_err (&GET_DEV (accel_dev),"Failed to get pfvf mig lock\n" ); +return -ETIMEDOUT ; + } +usleep_range (500 ,1000 ); + } + +adf_mstate_mgr_init_from_parent (&sub_sects_mgr, mstate_mgr); +for (i =0 ; i <ARRAY_SIZE (misc_states); i++) { +struct adf_mstate_vreginfo info; +u32 regv; + + info.addr = ®v; + info.size =sizeof (regv); + regv =ADF_CSR_RD (csr, misc_states[i].offset); + + l2_subsec =adf_mstate_sect_add_vreg (&sub_sects_mgr, + misc_states[i].id, + &info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , + misc_states[i].id); +mutex_unlock (&vf_info->pfvf_mig_lock); +return -EINVAL ; + } + } + +mutex_unlock (&vf_info->pfvf_mig_lock); +adf_mstate_sect_update (mstate_mgr, &sub_sects_mgr, subsec); + +return 0 ; +} + +static int adf_gen4_vfmig_save_generic (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_mgr sub_sects_mgr; +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct mig_user_sla src_slas[RL_RP_CNT_PER_LEAF_MAX ] = { }; +u32 src_sla_cnt; +struct { +char *id; +struct adf_mstate_vreginfo info; + } gen_states[] = { + {ADF_MSTATE_IOV_INIT_IDS , + {&vf_info->init,sizeof (vf_info->init)}}, + {ADF_MSTATE_COMPAT_VER_IDS , + {&vf_info->vf_compat_ver,sizeof (vf_info->vf_compat_ver)}}, + {ADF_MSTATE_SLA_IDS , {src_slas,0 }}, + }; +int i; + + subsec =adf_mstate_sect_add (mstate_mgr,ADF_MSTATE_GEN_IDS ,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , +ADF_MSTATE_GEN_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_parent (&sub_sects_mgr, mstate_mgr); +for (i =0 ; i <ARRAY_SIZE (gen_states); i++) { +if (gen_states[i].info.addr == src_slas) { + src_sla_cnt =adf_gen4_vfmig_get_slas (accel_dev, vf_nr, src_slas); + gen_states[i].info.size = src_sla_cnt *sizeof (struct mig_user_sla ); + } + + l2_subsec =adf_mstate_sect_add_vreg (&sub_sects_mgr, + gen_states[i].id, + &gen_states[i].info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , + gen_states[i].id); +return -EINVAL ; + } + } +adf_mstate_sect_update (mstate_mgr, &sub_sects_mgr, subsec); + +return 0 ; +} + +static int adf_gen4_vfmig_save_config (struct adf_accel_dev *accel_dev,u32 vf_nr) +{ +struct adf_accel_vf_info *vf_info = &accel_dev->pf.vf_info[vf_nr]; +struct adf_hw_device_data *hw_data = accel_dev->hw_device; +struct adf_gen4_vfmig *vfmig = vf_info->mig_priv; +struct adf_mstate_mgr *mstate_mgr = vfmig->mstate_mgr; +struct adf_mstate_mgr sub_sects_mgr; +struct adf_mstate_sect_h *subsec, *l2_subsec; +struct { +char *id; +struct adf_mstate_vreginfo info; + } setups[] = { + {ADF_MSTATE_GEN_CAP_IDS , + {&hw_data->accel_capabilities_mask,sizeof (hw_data->accel_capabilities_mask)}}, + {ADF_MSTATE_GEN_SVCMAP_IDS , + {&hw_data->ring_to_svc_map,sizeof (hw_data->ring_to_svc_map)}}, + {ADF_MSTATE_GEN_EXTDC_IDS , + {&hw_data->extended_dc_capabilities,sizeof (hw_data->extended_dc_capabilities)}}, + }; +int i; + + subsec =adf_mstate_sect_add (mstate_mgr,ADF_MSTATE_CONFIG_IDS ,NULL ,NULL ); +if (!subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , +ADF_MSTATE_CONFIG_IDS ); +return -EINVAL ; + } + +adf_mstate_mgr_init_from_parent (&sub_sects_mgr, mstate_mgr); +for (i =0 ; i <ARRAY_SIZE (setups); i++) { + l2_subsec =adf_mstate_sect_add_vreg (&sub_sects_mgr, setups[i].id, + &setups[i].info); +if (!l2_subsec) { +dev_err (&GET_DEV (accel_dev),"Failed to add sec %s\n" , + setups[i].id); +return -EINVAL ; + } + } +adf_mstate_sect_update (mstate_mgr, &sub_sects_mgr, subsec); + +return 0 ; +} + +static int adf_gen4_vfmig_save_state (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vfmig; +u32 vf_nr = mdev->vf_id; +int ret; + + vf_info = &accel_dev->pf.vf_info[vf_nr]; + vfmig = vf_info->mig_priv; + + ret =adf_gen4_vfmig_save_setup (mdev); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to save setup for vf_nr %d\n" , vf_nr); +return ret; + } + +adf_mstate_mgr_init (vfmig->mstate_mgr, mdev->state + mdev->setup_size, + mdev->state_size - mdev->setup_size); +if (!adf_mstate_preamble_add (vfmig->mstate_mgr)) +return -EINVAL ; + + ret =adf_gen4_vfmig_save_generic (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to save generic state for vf_nr %d\n" , vf_nr); +return ret; + } + + ret =adf_gen4_vfmig_save_misc (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to save misc bar state for vf_nr %d\n" , vf_nr); +return ret; + } + + ret =adf_gen4_vfmig_save_etr (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to save etr bar state for vf_nr %d\n" , vf_nr); +return ret; + } + +adf_mstate_preamble_update (vfmig->mstate_mgr); + +return 0 ; +} + +static int adf_gen4_vfmig_load_state (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vfmig; +u32 vf_nr = mdev->vf_id; +int ret; + + vf_info = &accel_dev->pf.vf_info[vf_nr]; + vfmig = vf_info->mig_priv; + + ret =adf_gen4_vfmig_load_setup (mdev, mdev->state_size); +if (ret) { +dev_err (&GET_DEV (accel_dev),"Failed to load setup for vf_nr %d\n" , + vf_nr); +return ret; + } + + ret =adf_mstate_mgr_init_from_remote (vfmig->mstate_mgr, + mdev->state + mdev->remote_setup_size, + mdev->state_size - mdev->remote_setup_size, +NULL ,NULL ); +if (ret) { +dev_err (&GET_DEV (accel_dev),"Invalid state for vf_nr %d\n" , + vf_nr); +return ret; + } + + ret =adf_gen4_vfmig_load_generic (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to load general state for vf_nr %d\n" , vf_nr); +return ret; + } + + ret =adf_gen4_vfmig_load_misc (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to load misc bar state for vf_nr %d\n" , vf_nr); +return ret; + } + + ret =adf_gen4_vfmig_load_etr (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to load etr bar state for vf_nr %d\n" , vf_nr); +return ret; + } + +return 0 ; +} + +static int adf_gen4_vfmig_save_setup (struct qat_mig_dev *mdev) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vfmig; +u32 vf_nr = mdev->vf_id; +int ret; + + vf_info = &accel_dev->pf.vf_info[vf_nr]; + vfmig = vf_info->mig_priv; + +if (mdev->setup_size) +return 0 ; + +adf_mstate_mgr_init (vfmig->mstate_mgr, mdev->state, mdev->state_size); +if (!adf_mstate_preamble_add (vfmig->mstate_mgr)) +return -EINVAL ; + + ret =adf_gen4_vfmig_save_config (accel_dev, mdev->vf_id); +if (ret) +return ret; + +adf_mstate_preamble_update (vfmig->mstate_mgr); + mdev->setup_size =adf_mstate_state_size (vfmig->mstate_mgr); + +return 0 ; +} + +static int adf_gen4_vfmig_load_setup (struct qat_mig_dev *mdev,int len) +{ +struct adf_accel_dev *accel_dev = mdev->parent_accel_dev; +struct adf_accel_vf_info *vf_info; +struct adf_gen4_vfmig *vfmig; +u32 vf_nr = mdev->vf_id; +u32 setup_size; +int ret; + + vf_info = &accel_dev->pf.vf_info[vf_nr]; + vfmig = vf_info->mig_priv; + +if (mdev->remote_setup_size) +return 0 ; + +if (len <sizeof (struct adf_mstate_preh )) +return -EAGAIN ; + +adf_mstate_mgr_init (vfmig->mstate_mgr, mdev->state, mdev->state_size); + setup_size =adf_mstate_state_size_from_remote (vfmig->mstate_mgr); +if (setup_size > mdev->state_size) +return -EINVAL ; + +if (len < setup_size) +return -EAGAIN ; + + ret =adf_mstate_mgr_init_from_remote (vfmig->mstate_mgr, mdev->state, + setup_size,NULL ,NULL ); +if (ret) { +dev_err (&GET_DEV (accel_dev),"Invalide setup for vf_nr %d\n" , + vf_nr); +return ret; + } + + mdev->remote_setup_size = setup_size; + + ret =adf_gen4_vfmig_load_config (accel_dev, vf_nr); +if (ret) { +dev_err (&GET_DEV (accel_dev), +"Failed to load config for vf_nr %d\n" , vf_nr); +return ret; + } + +return 0 ; +} + +void adf_gen4_init_vf_mig_ops (struct qat_migdev_ops *vfmig_ops) +{ + vfmig_ops->init = adf_gen4_vfmig_init_device; + vfmig_ops->cleanup = adf_gen4_vfmig_cleanup_device; + vfmig_ops->reset = adf_gen4_vfmig_reset_device;
drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c line 1010
Quoted 10 lines
drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c new file +10
-0+ vfmig_ops->open = adf_gen4_vfmig_open_device; + vfmig_ops->close = adf_gen4_vfmig_close_device; + vfmig_ops->suspend = adf_gen4_vfmig_suspend_device; + vfmig_ops->resume = adf_gen4_vfmig_resume_device; + vfmig_ops->save_state = adf_gen4_vfmig_save_state; + vfmig_ops->load_state = adf_gen4_vfmig_load_state; + vfmig_ops->load_setup = adf_gen4_vfmig_load_setup; + vfmig_ops->save_setup = adf_gen4_vfmig_save_setup; +} +EXPORT_SYMBOL_GPL (adf_gen4_init_vf_mig_ops);
This GEN4 device supports live migration functionality. The above part of the code supports the live migration function and has nothing to do with crypto.
Therefore, these should be moved to the vfio/pci/qat directory.
Thanks,
Longfang.
Quoted 457 lines
drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c new file +318
-0@@ -0,0 +1,318 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2024 Intel Corporation */ + +#include <linux/slab.h> +#include <linux/types.h> +#include "adf_mstate_mgr.h" + +#define ADF_MSTATE_MAGIC 0xADF5CAEA +#define ADF_MSTATE_VERSION 0x1 + +struct adf_mstate_sect_h { +u8 id[ADF_MSTATE_ID_LEN ]; +u32 size; +u32 sub_sects; +u8 state[]; +}; + +u32 adf_mstate_state_size (struct adf_mstate_mgr *mgr) +{ +return mgr->state - mgr->buf; +} + +static inline u32 adf_mstate_avail_room (struct adf_mstate_mgr *mgr) +{ +return mgr->buf + mgr->size - mgr->state; +} + +void adf_mstate_mgr_init (struct adf_mstate_mgr *mgr,u8 *buf,u32 size) +{ + mgr->buf = buf; + mgr->state = buf; + mgr->size = size; + mgr->n_sects =0 ; +}; + +struct adf_mstate_mgr *adf_mstate_mgr_new (u8 *buf,u32 size) +{ +struct adf_mstate_mgr *mgr; + + mgr =kzalloc (sizeof (*mgr),GFP_KERNEL ); +if (!mgr) +return NULL ; + +adf_mstate_mgr_init (mgr, buf, size); + +return mgr; +} + +void adf_mstate_mgr_destroy (struct adf_mstate_mgr *mgr) +{ +kfree (mgr); +} + +void adf_mstate_mgr_init_from_parent (struct adf_mstate_mgr *mgr, +struct adf_mstate_mgr *p_mgr) +{ +adf_mstate_mgr_init (mgr, p_mgr->state, + p_mgr->size -adf_mstate_state_size (p_mgr)); +} + +void adf_mstate_mgr_init_from_psect (struct adf_mstate_mgr *mgr, +struct adf_mstate_sect_h *p_sect) +{ +adf_mstate_mgr_init (mgr, p_sect->state, p_sect->size); + mgr->n_sects = p_sect->sub_sects; +} + +static void adf_mstate_preamble_init (struct adf_mstate_preh *preamble) +{ + preamble->magic =ADF_MSTATE_MAGIC ; + preamble->version =ADF_MSTATE_VERSION ; + preamble->preh_len =sizeof (*preamble); + preamble->size =0 ; + preamble->n_sects =0 ; +} + +/* default preambles checker */ +static int adf_mstate_preamble_def_checker (struct adf_mstate_preh *preamble, +void *opaque) +{ +struct adf_mstate_mgr *mgr = opaque; + +if (preamble->magic !=ADF_MSTATE_MAGIC || + preamble->version >ADF_MSTATE_VERSION || + preamble->preh_len > mgr->size) { +pr_debug ("QAT: LM - Invalid state (magic=%#x, version=%#x, hlen=%u), state_size=%u\n" , + preamble->magic, preamble->version, preamble->preh_len, + mgr->size); +return -EINVAL ; + } + +return 0 ; +} + +struct adf_mstate_preh *adf_mstate_preamble_add (struct adf_mstate_mgr *mgr) +{ +struct adf_mstate_preh *pre = (struct adf_mstate_preh *)mgr->buf; + +if (adf_mstate_avail_room (mgr) <sizeof (*pre)) { +pr_err ("QAT: LM - Not enough space for preamble\n" ); +return NULL ; + } + +adf_mstate_preamble_init (pre); + mgr->state += pre->preh_len; + +return pre; +} + +int adf_mstate_preamble_update (struct adf_mstate_mgr *mgr) +{ +struct adf_mstate_preh *preamble = (struct adf_mstate_preh *)mgr->buf; + + preamble->size =adf_mstate_state_size (mgr) - preamble->preh_len; + preamble->n_sects = mgr->n_sects; + +return 0 ; +} + +static void adf_mstate_dump_sect (struct adf_mstate_sect_h *sect, +const char *prefix) +{ +pr_debug ("QAT: LM - %s QAT state section %s\n" , prefix, sect->id); +print_hex_dump_debug ("h-" ,DUMP_PREFIX_OFFSET ,16 ,2 , sect, +sizeof (*sect), true); +print_hex_dump_debug ("s-" ,DUMP_PREFIX_OFFSET ,16 ,2 , sect->state, + sect->size, true); +} + +static inline void __adf_mstate_sect_update (struct adf_mstate_mgr *mgr, +struct adf_mstate_sect_h *sect, +u32 size, +u32 n_subsects) +{ + sect->size += size; + sect->sub_sects += n_subsects; + mgr->n_sects++; + mgr->state += sect->size; + +adf_mstate_dump_sect (sect,"Add" ); +} + +void adf_mstate_sect_update (struct adf_mstate_mgr *p_mgr, +struct adf_mstate_mgr *curr_mgr, +struct adf_mstate_sect_h *sect) +{ +__adf_mstate_sect_update (p_mgr, sect,adf_mstate_state_size (curr_mgr), + curr_mgr->n_sects); +} + +static struct adf_mstate_sect_h *adf_mstate_sect_add_header (struct adf_mstate_mgr *mgr, +const char *id) +{ +struct adf_mstate_sect_h *sect = (struct adf_mstate_sect_h *)(mgr->state); + +if (adf_mstate_avail_room (mgr) <sizeof (*sect)) { +pr_debug ("QAT: LM - Not enough space for header of QAT state sect %s\n" , id); +return NULL ; + } + +strscpy (sect->id, id,sizeof (sect->id)); + sect->size =0 ; + sect->sub_sects =0 ; + mgr->state +=sizeof (*sect); + +return sect; +} + +struct adf_mstate_sect_h *adf_mstate_sect_add_vreg (struct adf_mstate_mgr *mgr, +const char *id, +struct adf_mstate_vreginfo *info) +{ +struct adf_mstate_sect_h *sect; + + sect =adf_mstate_sect_add_header (mgr, id); +if (!sect) +return NULL ; + +if (adf_mstate_avail_room (mgr) < info->size) { +pr_debug ("QAT: LM - Not enough space for QAT state sect %s, requires %u\n" , + id, info->size); +return NULL ; + } + +memcpy (sect->state, info->addr, info->size); +__adf_mstate_sect_update (mgr, sect, info->size,0 ); + +return sect; +} + +struct adf_mstate_sect_h *adf_mstate_sect_add (struct adf_mstate_mgr *mgr, +const char *id, +adf_mstate_populate populate, +void *opaque) +{ +struct adf_mstate_mgr sub_sects_mgr; +struct adf_mstate_sect_h *sect; +int avail_room, size; + + sect =adf_mstate_sect_add_header (mgr, id); +if (!sect) +return NULL ; + +if (!populate) +return sect; + + avail_room =adf_mstate_avail_room (mgr); +adf_mstate_mgr_init_from_parent (&sub_sects_mgr, mgr); + + size = (*populate)(&sub_sects_mgr, sect->state, avail_room, opaque); +if (size <0 ) +return NULL ; + + size +=adf_mstate_state_size (&sub_sects_mgr); +if (avail_room < size) { +pr_debug ("QAT: LM - Not enough space for QAT state sect %s, requires %u\n" , + id, size); +return NULL ; + } +__adf_mstate_sect_update (mgr, sect, size, sub_sects_mgr.n_sects); + +return sect; +} + +static int adf_mstate_sect_validate (struct adf_mstate_mgr *mgr) +{ +struct adf_mstate_sect_h *start = (struct adf_mstate_sect_h *)mgr->state; +struct adf_mstate_sect_h *sect = start; +u64 end; +int i; + + end = (uintptr_t )mgr->buf + mgr->size; +for (i =0 ; i < mgr->n_sects; i++) { +uintptr_t s_start = (uintptr_t )sect->state; +uintptr_t s_end = s_start + sect->size; + +if (s_end < s_start || s_end > end) { +pr_debug ("QAT: LM - Corrupted state section (index=%u, size=%u) in state_mgr (size=%u, secs=%u)\n" , + i, sect->size, mgr->size, mgr->n_sects); +return -EINVAL ; + } + sect = (struct adf_mstate_sect_h *)s_end; + } + +pr_debug ("QAT: LM - Scanned section (last child=%s, size=%lu) in state_mgr (size=%u, secs=%u)\n" , + start->id,sizeof (struct adf_mstate_sect_h ) * (ulong)(sect - start), + mgr->size, mgr->n_sects); + +return 0 ; +} + +u32 adf_mstate_state_size_from_remote (struct adf_mstate_mgr *mgr) +{ +struct adf_mstate_preh *preh = (struct adf_mstate_preh *)mgr->buf; + +return preh->preh_len + preh->size; +} + +int adf_mstate_mgr_init_from_remote (struct adf_mstate_mgr *mgr,u8 *buf,u32 size, +adf_mstate_preamble_checker pre_checker, +void *opaque) +{ +struct adf_mstate_preh *pre; +int ret; + +adf_mstate_mgr_init (mgr, buf, size); + pre = (struct adf_mstate_preh *)(mgr->buf); + +pr_debug ("QAT: LM - Dump state preambles\n" ); +print_hex_dump_debug ("" ,DUMP_PREFIX_OFFSET ,16 ,2 , pre, pre->preh_len,0 ); + +if (pre_checker) + ret = (*pre_checker)(pre, opaque); +else + ret =adf_mstate_preamble_def_checker (pre, mgr); +if (ret) +return ret; + + mgr->state = mgr->buf + pre->preh_len; + mgr->n_sects = pre->n_sects; + +return adf_mstate_sect_validate (mgr); +} + +struct adf_mstate_sect_h *adf_mstate_sect_lookup (struct adf_mstate_mgr *mgr, +const char *id, +adf_mstate_action action, +void *opaque) +{ +struct adf_mstate_sect_h *sect = (struct adf_mstate_sect_h *)mgr->state; +struct adf_mstate_mgr sub_sects_mgr; +int i, ret; + +for (i =0 ; i < mgr->n_sects; i++) { +if (!strncmp (sect->id, id,sizeof (sect->id))) + goto found; + + sect = (struct adf_mstate_sect_h *)(sect->state + sect->size); + } + +return NULL ; + +found: +adf_mstate_dump_sect (sect,"Found" ); + +adf_mstate_mgr_init_from_psect (&sub_sects_mgr, sect); +if (sect->sub_sects &&adf_mstate_sect_validate (&sub_sects_mgr)) +return NULL ; + +if (!action) +return sect; + + ret = (*action)(&sub_sects_mgr, sect->state, sect->size, opaque); +if (ret) +return NULL ; + +return sect; +}drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h new file +89
-0@@ -0,0 +1,89 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2024 Intel Corporation */ + +#ifndef ADF_MSTATE_MGR_H +#define ADF_MSTATE_MGR_H + +#define ADF_MSTATE_ID_LEN 8 + +#define ADF_MSTATE_ETRB_IDS "ETRBAR" +#define ADF_MSTATE_MISCB_IDS "MISCBAR" +#define ADF_MSTATE_EXTB_IDS "EXTBAR" +#define ADF_MSTATE_GEN_IDS "GENER" +#define ADF_MSTATE_CONFIG_IDS "CONFIG" +#define ADF_MSTATE_SECTION_NUM 5 + +#define ADF_MSTATE_BANK_IDX_IDS "bnk" + +#define ADF_MSTATE_ETR_REGS_IDS "mregs" +#define ADF_MSTATE_VINTSRC_IDS "visrc" +#define ADF_MSTATE_VINTMSK_IDS "vimsk" +#define ADF_MSTATE_SLA_IDS "sla" +#define ADF_MSTATE_IOV_INIT_IDS "iovinit" +#define ADF_MSTATE_COMPAT_VER_IDS "compver" +#define ADF_MSTATE_GEN_CAP_IDS "gencap" +#define ADF_MSTATE_GEN_SVCMAP_IDS "svcmap" +#define ADF_MSTATE_GEN_EXTDC_IDS "extdc" +#define ADF_MSTATE_VINTSRC_PF2VM_IDS "vispv" +#define ADF_MSTATE_VINTMSK_PF2VM_IDS "vimpv" +#define ADF_MSTATE_VM2PF_IDS "vm2pf" +#define ADF_MSTATE_PF2VM_IDS "pf2vm" + +struct adf_mstate_mgr { +u8 *buf; +u8 *state; +u32 size; +u32 n_sects; +}; + +struct adf_mstate_preh { +u32 magic; +u32 version; +u16 preh_len; +u16 n_sects; +u32 size; +}; + +struct adf_mstate_vreginfo { +void *addr; +u32 size; +}; + +struct adf_mstate_sect_h ; + +typedef int (*adf_mstate_preamble_checker )(struct adf_mstate_preh *preamble,void *opa); +typedef int (*adf_mstate_populate )(struct adf_mstate_mgr *sub_mgr,u8 *buf, +u32 size,void *opa); +typedef int (*adf_mstate_action )(struct adf_mstate_mgr *sub_mgr,u8 *buf,u32 size, +void *opa); + +struct adf_mstate_mgr *adf_mstate_mgr_new (u8 *buf,u32 size); +void adf_mstate_mgr_destroy (struct adf_mstate_mgr *mgr); +void adf_mstate_mgr_init (struct adf_mstate_mgr *mgr,u8 *buf,u32 size); +void adf_mstate_mgr_init_from_parent (struct adf_mstate_mgr *mgr, +struct adf_mstate_mgr *p_mgr); +void adf_mstate_mgr_init_from_psect (struct adf_mstate_mgr *mgr, +struct adf_mstate_sect_h *p_sect); +int adf_mstate_mgr_init_from_remote (struct adf_mstate_mgr *mgr, +u8 *buf,u32 size, +adf_mstate_preamble_checker checker, +void *opaque); +struct adf_mstate_preh *adf_mstate_preamble_add (struct adf_mstate_mgr *mgr); +int adf_mstate_preamble_update (struct adf_mstate_mgr *mgr); +u32 adf_mstate_state_size (struct adf_mstate_mgr *mgr); +u32 adf_mstate_state_size_from_remote (struct adf_mstate_mgr *mgr); +void adf_mstate_sect_update (struct adf_mstate_mgr *p_mgr, +struct adf_mstate_mgr *curr_mgr, +struct adf_mstate_sect_h *sect); +struct adf_mstate_sect_h *adf_mstate_sect_add_vreg (struct adf_mstate_mgr *mgr, +const char *id, +struct adf_mstate_vreginfo *info); +struct adf_mstate_sect_h *adf_mstate_sect_add (struct adf_mstate_mgr *mgr, +const char *id, +adf_mstate_populate populate, +void *opaque); +struct adf_mstate_sect_h *adf_mstate_sect_lookup (struct adf_mstate_mgr *mgr, +const char *id, +adf_mstate_action action, +void *opaque); +#endif drivers/crypto/intel/qat/qat_common/adf_sriov.c +6
-1@@ -26,10 +26,12 @@ static void adf_iov_send_resp(struct work_struct *work)u32 vf_nr = vf_info->vf_nr;bool ret; +mutex_lock (&vf_info->pfvf_mig_lock); ret =adf_recv_and_handle_vf2pf_msg (accel_dev, vf_nr);if (ret)/* re-enable interrupt on PF from this VF */ adf_enable_vf2pf_interrupts (accel_dev,1 << vf_nr); +mutex_unlock (&vf_info->pfvf_mig_lock);kfree (pf2vf_resp); }@@ -62,6 +64,7 @@ static int adf_enable_sriov(struct adf_accel_dev *accel_dev) vf_info->vf_nr = i;mutex_init (&vf_info->pf2vf_lock); +mutex_init (&vf_info->pfvf_mig_lock);ratelimit_state_init (&vf_info->vf2pf_ratelimit,ADF_VF2PF_RATELIMIT_INTERVAL ,ADF_VF2PF_RATELIMIT_BURST );@@ -138,8 +141,10 @@ void adf_disable_sriov(struct adf_accel_dev *accel_dev)if (hw_data->configure_iov_threads) hw_data->configure_iov_threads (accel_dev, false); -for (i =0 , vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) +for (i =0 , vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) {mutex_destroy (&vf->pf2vf_lock); +mutex_destroy (&vf->pfvf_mig_lock); + }if (!test_bit (ADF_STATUS_RESTARTING , &accel_dev->status)) {kfree (accel_dev->pf.vf_info);
RE: [PATCH v5 09/10] crypto: qat - implement interface for live migration
Quoted 79 lines · liulongfang
To: Zeng, Xin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Tian, Kevin <[email protected]>
Cc: [email protected]; [email protected]; qat-linux <qat- [email protected]>
Subject: Re: [PATCH v5 09/10] crypto: qat - implement interface for live migrationQuoted 26 lines · Xin Zeng
Add logic to implement the interface for live migration defined in
qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions (VFs).This introduces a migration data manager which is used to handle the device state during migration. The manager ensures that the device state is stored in a format that can be restored in the destination node.
The VF state is organized into a hierarchical structure that includes a preamble, a general state section, a MISC bar section and an ETR bar section. The latter contains the state of the 4 ring pairs contained on a VF. Here is a graphical representation of the state:
preamble | general state section | leaf state | MISC bar state section| leaf state | ETR bar state section | bank0 state section | leaf state | bank1 state section | leaf state | bank2 state section | leaf state | bank3 state section | leaf stateIn addition to the implementation of the
qat_migdev_opsinterface and the state manager framework, add a mutex in pfvf to avoid pf2vf messages during migration.Signed-off-by Xin Zeng <[email protected]>Reviewed-by Giovanni Cabiddu <[email protected]>Quoted 12 lines
--- .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 2 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 2 + drivers/crypto/intel/qat/qat_common/Makefile | 2 + .../intel/qat/qat_common/adf_accel_devices.h | 6 + .../intel/qat/qat_common/adf_gen4_hw_data.h | 10 + .../intel/qat/qat_common/adf_gen4_vf_mig.c | 1010 +++++++++++++++++ .../intel/qat/qat_common/adf_mstate_mgr.c | 318 ++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 89 ++ .../crypto/intel/qat/qat_common/adf_sriov.c | 7 +- 9 files changed, 1445 insertions(+), 1 deletion(-) create mode 100644
drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.cQuoted 1 line
create mode 100644
drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.cQuoted 1 line
create mode 100644
drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.hReview
Quoted 2 lines
(quoted diff) +0
-0
b/drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.cQuoted 9 lines
index
9ccbf5998d5c..d255cb3ebd9c100644drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +5
-0@@ -17,6 +17,7 @@ +void adf_gen4_init_vf_mig_ops (struct qat_migdev_ops *vfmig_ops) +{ + vfmig_ops->init = adf_gen4_vfmig_init_device; + vfmig_ops->cleanup = adf_gen4_vfmig_cleanup_device; + vfmig_ops->reset = adf_gen4_vfmig_reset_device;drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c
Quoted 10 lines
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +10
-0+ vfmig_ops->open = adf_gen4_vfmig_open_device; + vfmig_ops->close = adf_gen4_vfmig_close_device; + vfmig_ops->suspend = adf_gen4_vfmig_suspend_device; + vfmig_ops->resume = adf_gen4_vfmig_resume_device; + vfmig_ops->save_state = adf_gen4_vfmig_save_state; + vfmig_ops->load_state = adf_gen4_vfmig_load_state; + vfmig_ops->load_setup = adf_gen4_vfmig_load_setup; + vfmig_ops->save_setup = adf_gen4_vfmig_save_setup; +} +EXPORT_SYMBOL_GPL (adf_gen4_init_vf_mig_ops);This GEN4 device supports live migration functionality. The above part of the code supports the live migration function and has nothing to do with crypto.
Quoted 2 lines
Therefore, these should be moved to the vfio/pci/qat directory.
Thanks for the suggestion, but
1. The migration operations of QAT VF rely on QAT PF driver sitting in crypto tree to handle. Some of the states can only be accessed from PF by PF driver. For each generation of PF, we will have a PF driver. It is obvious more clear and nature to make these operations part QAT PF driver rather than the variant VF driver.
2. The interfaces are defined clear enough to understand the dependency of the variant migration VF driver to QAT PF driver.
3. A device driver sitting in crypto tree usually not only provides crypto stuff, but also provide helpers to support other functionalities such as the non vfio use space process access logic, it does make sense to provide migration helpers in device driver as well.
Those why I prefer to put these helpers into QAT PF driver.
Thanks
PATCH v5 10/10 vfio/qat: Add vfio_pci driver for Intel QAT VF devices
In reply to Xin ZengAdd vfio pci driver for Intel QAT VF devices.
This driver uses vfio_pci_core to register to the VFIO subsystem. It acts as a vfio agent and interacts with the QAT PF driver to implement VF live migration.
--- MAINTAINERS | 8 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/qat/Kconfig | 12 + drivers/vfio/pci/qat/Makefile | 3 + drivers/vfio/pci/qat/main.c | 662 ++++++++++++++++++++++++++++++++++ 6 files changed, 689 insertions(+) create mode 100644 drivers/vfio/pci/qat/Kconfig create mode 100644 drivers/vfio/pci/qat/Makefile create mode 100644 drivers/vfio/pci/qat/main.c
MAINTAINERS +8 -0
@@ -23105,6 +23105,14 @@ L: [email protected] S: Maintained F: drivers/vfio/platform/ +VFIO QAT PCI DRIVER +M: Xin Zeng <[email protected]> +M: Giovanni Cabiddu <[email protected]> +L: [email protected] +L: [email protected] +S: Supported +F: drivers/vfio/pci/qat/ + VGA_SWITCHEROO R: Lukas Wunner <[email protected]> S: Maintained
drivers/vfio/pci/Kconfig +2 -0
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig" source "drivers/vfio/pci/virtio/Kconfig" +source "drivers/vfio/pci/qat/Kconfig" + endmenu
drivers/vfio/pci/Makefile +2 -0
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/ obj-$(CONFIG_PDS_VFIO_PCI) += pds/ obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/ + +obj-$(CONFIG_QAT_VFIO_PCI) += qat/
drivers/vfio/pci/qat/Kconfig new file +12 -0
@@ -0,0 +1,12 @@ +# SPDX-License-Identifier: GPL-2.0-only +config QAT_VFIO_PCI + tristate "VFIO support for QAT VF PCI devices" + select VFIO_PCI_CORE + depends on CRYPTO_DEV_QAT_4XXX + help + This provides migration support for Intel(R) QAT Virtual Function + using the VFIO framework. + + To compile this as a module, choose M here: the module + will be called qat_vfio_pci. If you don't know what to do here, + say N.
drivers/vfio/pci/qat/Makefile new file +3 -0
@@ -0,0 +1,3 @@ +# SPDX-License-Identifier: GPL-2.0-only +obj-$(CONFIG_QAT_VFIO_PCI) += qat_vfio_pci.o +qat_vfio_pci-y := main.o
drivers/vfio/pci/qat/main.c new file +662 -0
662 lines not on this page · raw message
Signature · 1 line
2.18.2
RE: [PATCH v5 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices
Quoted 5 lines
From: Zeng, Xin <[email protected]>
Sent: Wednesday, March 6, 2024 9:59 PMAdd vfio pci driver for Intel QAT VF devices.
Quoted 8 lines
This driver uses
vfio_pci_coreto register to the VFIO subsystem. It acts as a vfio agent and interacts with the QAT PF driver to implement VF live migration.Co-developed-by Yahui Cao <[email protected]>Signed-off-by Yahui Cao <[email protected]>Signed-off-by Xin Zeng <[email protected]>Reviewed-by Giovanni Cabiddu <[email protected]>
If Alex doesn't require closing the ID matching open before merging this:
Herbert Xu
maintainerQuoted 126 lines · Xin Zeng
This set enables live migration for Intel QAT GEN4 SRIOV Virtual Functions (VFs).
It is composed of 10 patches. Patch 1~6 refactor the original QAT PF driver implementation which will be reused by the following patches. Patch 7 introduces the logic to the QAT PF driver that allows to save and restore the state of a bank (a QAT VF is a wrapper around banks) and drain a ring pair. Patch 8 adds the QAT PF driver a set of interfaces to allow to save and restore the state of a VF that will be called by the moduleqat_vfio_pciwhich will be introduced in the last patch. Patch 9 implements the defined device interfaces. The last one adds a vfio pci extension specific for QAT which intercepts the vfio device operations for a QAT VF to allow live migration.Here are the steps required to test the live migration of a QAT GEN4 VF:
1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko
2. Assign the VFs to the virtual machine and enable device live migration
3. Run a workload using a QAT VF inside the VM, for example using qatlib
(https://github.com/intel/qatlib)
4. Migrate the VM from the source node to a destination nodeChanges in v5 since v4: https://lore.kernel.org/kvm/[email protected]
(quoted diff) +0
-4- Remove device ID recheck as no consensus has been reached yet (Kevin) - Add missing state PRE_COPY_P2P in precopy_iotcl (Kevin) - Rearrange the state transition flow for better readability (Kevin) - Remove unnecessary Reviewed-by in commit message (Kevin)Changes in v4 since v3: https://lore.kernel.org/kvm/[email protected]
(quoted diff) +0
-3- Change the order of maintainer entry for QAT vfio pci driver in MAINTAINERS to make it alphabetical (Alex) - Put QAT VFIO PCI driver under vfio/pci directly instead of vfio/pci/intel (Alex) - Add id_table recheck during device probe (Alex)Changes in v3 since v2: https://lore.kernel.org/kvm/[email protected]
- Usestate_mutexdirectly instead of unnecessarydeferred_resetmode
(Jason)Changes in v2 since v1: https://lore.kernel.org/all/[email protected]
(quoted diff) +0
-5- Add VFIO_MIGRATION_PRE_COPY support (Alex) - Remove unnecessary module dependancy in Kconfig (Alex) - Use direct function calls instead of function pointers in qat vfio variant driver (Jason) - Address the comments including uncessary pointer check and kfree, missing lock and direct use of pci_iov_vf_id (Shameer) - Change CHECK_STAT macro to avoid repeat comparison (Kamlesh)Changes in v1 since RFC: https://lore.kernel.org/all/[email protected]
(quoted diff) +0
-6- Address comments including the right module dependancy in Kconfig, source file name and module description (Alex) - Added PCI error handler and P2P state handler (Suggested by Kevin) - Refactor the state check duing loading ring state (Kevin) - Fix missed call to vfio_put_device in the error case (Breet) - Migrate the shadow states in PF driver - Rebase on top of 6.8-rc1Giovanni Cabiddu (2):
crypto: qat -adf_get_etr_base()helper
crypto: qat - relocate CSR access codeSiming Wan (3):
crypto: qat - renameget_sla_arr_of_type()
crypto: qat - expand CSR operations for QAT GEN4 devices
crypto: qat - add bank save and restore flowsXin Zeng (5):
crypto: qat - relocate and rename 4xxx PF2VM definitions
crypto: qat - move PFVF compat checker to a function
crypto: qat - add interface for live migration
crypto: qat - implement interface for live migration vfio/qat: Addvfio_pcidriver for Intel QAT VF devicesMAINTAINERS | 8 + .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 3 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 5 + .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c | 1 + .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c | 1 + .../intel/qat/qat_c62x/adf_c62x_hw_data.c | 1 + .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c | 1 + drivers/crypto/intel/qat/qat_common/Makefile | 6 +- .../intel/qat/qat_common/adf_accel_devices.h | 88 ++ .../intel/qat/qat_common/adf_common_drv.h | 10 + .../qat/qat_common/adf_gen2_hw_csr_data.c | 101 ++ .../qat/qat_common/adf_gen2_hw_csr_data.h | 86 ++ .../intel/qat/qat_common/adf_gen2_hw_data.c | 97 -- .../intel/qat/qat_common/adf_gen2_hw_data.h | 76 -- .../qat/qat_common/adf_gen4_hw_csr_data.c | 231 ++++ .../qat/qat_common/adf_gen4_hw_csr_data.h | 188 +++ .../intel/qat/qat_common/adf_gen4_hw_data.c | 380 +++++-- .../intel/qat/qat_common/adf_gen4_hw_data.h | 127 +-- .../intel/qat/qat_common/adf_gen4_pfvf.c | 8 +- .../intel/qat/qat_common/adf_gen4_vf_mig.c | 1010 +++++++++++++++++ .../intel/qat/qat_common/adf_gen4_vf_mig.h | 10 + .../intel/qat/qat_common/adf_mstate_mgr.c | 318 ++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 89 ++ .../intel/qat/qat_common/adf_pfvf_pf_proto.c | 8 +- .../intel/qat/qat_common/adf_pfvf_utils.h | 11 + drivers/crypto/intel/qat/qat_common/adf_rl.c | 10 +- drivers/crypto/intel/qat/qat_common/adf_rl.h | 2 + .../crypto/intel/qat/qat_common/adf_sriov.c | 7 +- .../intel/qat/qat_common/adf_transport.c | 4 +- .../crypto/intel/qat/qat_common/qat_mig_dev.c | 130 +++ .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c | 1 + .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c | 1 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/qat/Kconfig | 12 + drivers/vfio/pci/qat/Makefile | 3 + drivers/vfio/pci/qat/main.c | 662 +++++++++++ include/linux/qat/qat_mig_dev.h | 31 + 38 files changed, 3344 insertions(+), 387 deletions(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c create mode 100644 drivers/vfio/pci/qat/Kconfig create mode 100644 drivers/vfio/pci/qat/Makefile create mode 100644 drivers/vfio/pci/qat/main.c create mode 100644 include/linux/qat/qat_mig_dev.h
Quoted 3 lines
base-commit:
318407ed77e4140d02e43a001b1f4753e3ce6b5fSignature · 1 line
2.18.2
Patches 1-9 applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Quoted 127 lines · On Thu, 28 Mar 2024 18:51:41 +0800 Herbert Xu <[email protected]>…
Quoted 126 lines · Xin Zeng
This set enables live migration for Intel QAT GEN4 SRIOV Virtual Functions (VFs).
It is composed of 10 patches. Patch 1~6 refactor the original QAT PF driver implementation which will be reused by the following patches. Patch 7 introduces the logic to the QAT PF driver that allows to save and restore the state of a bank (a QAT VF is a wrapper around banks) and drain a ring pair. Patch 8 adds the QAT PF driver a set of interfaces to allow to save and restore the state of a VF that will be called by the moduleqat_vfio_pciwhich will be introduced in the last patch. Patch 9 implements the defined device interfaces. The last one adds a vfio pci extension specific for QAT which intercepts the vfio device operations for a QAT VF to allow live migration.Here are the steps required to test the live migration of a QAT GEN4 VF:
1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko
2. Assign the VFs to the virtual machine and enable device live migration
3. Run a workload using a QAT VF inside the VM, for example using qatlib
(https://github.com/intel/qatlib)
4. Migrate the VM from the source node to a destination nodeChanges in v5 since v4: https://lore.kernel.org/kvm/[email protected]
(quoted diff) +0
-4- Remove device ID recheck as no consensus has been reached yet (Kevin) - Add missing state PRE_COPY_P2P in precopy_iotcl (Kevin) - Rearrange the state transition flow for better readability (Kevin) - Remove unnecessary Reviewed-by in commit message (Kevin)Changes in v4 since v3: https://lore.kernel.org/kvm/[email protected]
(quoted diff) +0
-3- Change the order of maintainer entry for QAT vfio pci driver in MAINTAINERS to make it alphabetical (Alex) - Put QAT VFIO PCI driver under vfio/pci directly instead of vfio/pci/intel (Alex) - Add id_table recheck during device probe (Alex)Changes in v3 since v2: https://lore.kernel.org/kvm/[email protected]
- Usestate_mutexdirectly instead of unnecessarydeferred_resetmode
(Jason)Changes in v2 since v1: https://lore.kernel.org/all/[email protected]
(quoted diff) +0
-5- Add VFIO_MIGRATION_PRE_COPY support (Alex) - Remove unnecessary module dependancy in Kconfig (Alex) - Use direct function calls instead of function pointers in qat vfio variant driver (Jason) - Address the comments including uncessary pointer check and kfree, missing lock and direct use of pci_iov_vf_id (Shameer) - Change CHECK_STAT macro to avoid repeat comparison (Kamlesh)Changes in v1 since RFC: https://lore.kernel.org/all/[email protected]
(quoted diff) +0
-6- Address comments including the right module dependancy in Kconfig, source file name and module description (Alex) - Added PCI error handler and P2P state handler (Suggested by Kevin) - Refactor the state check duing loading ring state (Kevin) - Fix missed call to vfio_put_device in the error case (Breet) - Migrate the shadow states in PF driver - Rebase on top of 6.8-rc1Giovanni Cabiddu (2):
crypto: qat -adf_get_etr_base()helper
crypto: qat - relocate CSR access codeSiming Wan (3):
crypto: qat - renameget_sla_arr_of_type()
crypto: qat - expand CSR operations for QAT GEN4 devices
crypto: qat - add bank save and restore flowsXin Zeng (5):
crypto: qat - relocate and rename 4xxx PF2VM definitions
crypto: qat - move PFVF compat checker to a function
crypto: qat - add interface for live migration
crypto: qat - implement interface for live migration vfio/qat: Addvfio_pcidriver for Intel QAT VF devicesMAINTAINERS | 8 + .../intel/qat/qat_420xx/adf_420xx_hw_data.c | 3 + .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c | 5 + .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c | 1 + .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c | 1 + .../intel/qat/qat_c62x/adf_c62x_hw_data.c | 1 + .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c | 1 + drivers/crypto/intel/qat/qat_common/Makefile | 6 +- .../intel/qat/qat_common/adf_accel_devices.h | 88 ++ .../intel/qat/qat_common/adf_common_drv.h | 10 + .../qat/qat_common/adf_gen2_hw_csr_data.c | 101 ++ .../qat/qat_common/adf_gen2_hw_csr_data.h | 86 ++ .../intel/qat/qat_common/adf_gen2_hw_data.c | 97 -- .../intel/qat/qat_common/adf_gen2_hw_data.h | 76 -- .../qat/qat_common/adf_gen4_hw_csr_data.c | 231 ++++ .../qat/qat_common/adf_gen4_hw_csr_data.h | 188 +++ .../intel/qat/qat_common/adf_gen4_hw_data.c | 380 +++++-- .../intel/qat/qat_common/adf_gen4_hw_data.h | 127 +-- .../intel/qat/qat_common/adf_gen4_pfvf.c | 8 +- .../intel/qat/qat_common/adf_gen4_vf_mig.c | 1010 +++++++++++++++++ .../intel/qat/qat_common/adf_gen4_vf_mig.h | 10 + .../intel/qat/qat_common/adf_mstate_mgr.c | 318 ++++++ .../intel/qat/qat_common/adf_mstate_mgr.h | 89 ++ .../intel/qat/qat_common/adf_pfvf_pf_proto.c | 8 +- .../intel/qat/qat_common/adf_pfvf_utils.h | 11 + drivers/crypto/intel/qat/qat_common/adf_rl.c | 10 +- drivers/crypto/intel/qat/qat_common/adf_rl.h | 2 + .../crypto/intel/qat/qat_common/adf_sriov.c | 7 +- .../intel/qat/qat_common/adf_transport.c | 4 +- .../crypto/intel/qat/qat_common/qat_mig_dev.c | 130 +++ .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c | 1 + .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c | 1 + drivers/vfio/pci/Kconfig | 2 + drivers/vfio/pci/Makefile | 2 + drivers/vfio/pci/qat/Kconfig | 12 + drivers/vfio/pci/qat/Makefile | 3 + drivers/vfio/pci/qat/main.c | 662 +++++++++++ include/linux/qat/qat_mig_dev.h | 31 + 38 files changed, 3344 insertions(+), 387 deletions(-) create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c create mode 100644 drivers/vfio/pci/qat/Kconfig create mode 100644 drivers/vfio/pci/qat/Makefile create mode 100644 drivers/vfio/pci/qat/main.c create mode 100644 include/linux/qat/qat_mig_dev.h
Quoted 5 lines
Quoted 3 lines
base-commit:
318407ed77e4140d02e43a001b1f4753e3ce6b5fSignature · 1 line
2.18.2Patches 1-9 applied. Thanks.
Hi Herbert,
Would you mind making a branch available for those in anticipation of the qat vfio variant driver itself being merged through the vfio tree? Thanks,
Alex
Herbert Xu
maintainerQuoted 4 lines · Alex Williamson
Would you mind making a branch available for those in anticipation of the qat vfio variant driver itself being merged through the vfio tree? Thanks,
OK, I've just pushed out a vfio branch. Please take a look to see if I messed anything up.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Cabiddu, Giovanni
maintainerHi Alex,
Quoted 8 lines · Herbert Xu
Quoted 4 lines · Alex Williamson
Would you mind making a branch available for those in anticipation of the qat vfio variant driver itself being merged through the vfio tree? Thanks,
OK, I've just pushed out a vfio branch. Please take a look to see if I messed anything up.
What are the next steps here?
Shall we re-send the patch vfio/qat: Add vfio_pci driver for Intel QAT VF devices rebased against vfio-next?
Or, wait for you to merge the branch from Herbert, then rebase and re-send? Or, are you going to take the patch that was sent to the mailing list as is and handle the rebase? (There is only a small conflict to sort on the makefiles).
Thanks,
Signature · 1 line
Giovanni
Quoted 11 lines · Giovanni
Hi Alex,
Quoted 8 lines · Herbert Xu
Quoted 4 lines · Alex Williamson
Would you mind making a branch available for those in anticipation of the qat vfio variant driver itself being merged through the vfio tree? Thanks,
OK, I've just pushed out a vfio branch. Please take a look to see if I messed anything up.
Quoted 8 lines
What are the next steps here?
Shall we re-send the patch
vfio/qat: Add vfio_pci driver for Intel QAT VF devicesrebased against vfio-next?
Or, wait for you to merge the branch from Herbert, then rebase and re-send? Or, are you going to take the patch that was sent to the mailing list as is and handle the rebase? (There is only a small conflict to sort on the makefiles).
Hi Giovanni,
The code itself looks fine to me, the Makefile conflict is trivial, MAINTAINERS also requires a trivial re-ordering to keep it alphabetical now that virtio-vfio-pci is merged. The only thing I spot that could use some attention is the documentation, where our acceptance criteria requests:
Additionally, drivers should make an attempt to provide sufficient documentation for reviewers to understand the device specific extensions, for example in the case of migration data, how is the device state composed and consumed, which portions are not otherwise available to the user via vfio-pci, what safeguards exist to validate the data, etc.
A lot of the code here is very similar in flow to the other migration drivers, but I think it would be good to address some of the topics above in comments throughout the driver. For example, how does the driver address P2P states, what information is provided in PRE_COPY, how is versioning handled, is user sensitive data included in the device migration data, typical ranges of device migration data size, etc.
Kevin might have an edge in understanding the theory of operation here already and documenting the interesting aspects of the driver in comments might drive a little more engagement. Thanks,
Alex