Skip to content

crypto: qat - enable QAT GEN4 SRIOV VF live migration for QAT GEN4

Xin Zeng kvm linux-crypto 5 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v5 Latest10 patches 10 repliesIn review
  2. v410 patches 6 repliesSuperseded
  3. v310 patches 12 repliesSuperseded
Earlier versions · 2
VersionPostedPatchesRepliesStatus
v210 patches0 repliesSuperseded
v110 patches19 repliesSuperseded

Patches in v5 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10crypto: qat - adf_get_etr_base() helperdrivers/crypto/intel/qat/qat_common/adf_common_drv.h drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c drivers/crypto/intel/qat/qat_common/adf_transport.c+12 -60 replies
2/10crypto: qat - relocate and rename 4xxx PF2VM definitionsdrivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.c+7 -50 replies
3/10crypto: qat - move PFVF compat checker to a functiondrivers/crypto/intel/qat/qat_common/adf_pfvf_pf_proto.c drivers/crypto/intel/qat/qat_common/adf_pfvf_utils.h+12 -70 replies
4/10crypto: qat - relocate CSR access codedrivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c drivers/crypto/intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c +14 more+397 -3620 replies
5/10crypto: qat - rename get_sla_arr_of_type()drivers/crypto/intel/qat/qat_common/adf_rl.c drivers/crypto/intel/qat/qat_common/adf_rl.h+7 -50 replies
6/10crypto: qat - expand CSR operations for QAT GEN4 devicesdrivers/crypto/intel/qat/qat_common/adf_accel_devices.h drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h+249 -10 replies
7/10crypto: qat - add bank save and restore flowsdrivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c drivers/crypto/intel/qat/qat_common/adf_accel_devices.h drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +1 more+338 -00 replies
8/10crypto: qat - add interface for live migrationdrivers/crypto/intel/qat/qat_common/Makefile drivers/crypto/intel/qat/qat_common/adf_accel_devices.h drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h +2 more+189 -12 replies
9/10crypto: qat - implement interface for live migrationdrivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c drivers/crypto/intel/qat/qat_common/Makefile +6 more+1445 -12 replies
10/10vfio/qat: Add vfio_pci driver for Intel QAT VF devicesMAINTAINERS drivers/vfio/pci/Kconfig drivers/vfio/pci/Makefile +3 more+689 -01 reply

Reviews trailers per patch

8 of 10 patches have a Reviewed-by or Acked-by.

PatchGiovanni CabidduDamian MuszynskiKevin Tian
1/10
2/10R-b
3/10R-b
4/10
5/10R-b R-b
6/10R-b
7/10R-b
8/10R-b
9/10R-b
10/10R-b R-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 3 reviewers.

Read the v5 thread

Cover letter [PATCH v5 00/10] crypto: qat - enable QAT GEN4 SRIOV VF live migration for QAT GEN4

This set enables live migration for Intel QAT GEN4 SRIOV Virtual Functions (VFs).
It is composed of 10 patches. Patch 1~6 refactor the original QAT PF driver implementation which will be reused by the following patches. Patch 7 introduces the logic to the QAT PF driver that allows to save and restore the state of a bank (a QAT VF is a wrapper around banks) and drain a ring pair. Patch 8 adds the QAT PF driver a set of interfaces to allow to save and restore the state of a VF that will be called by the module qat_vfio_pci which will be introduced in the last patch. Patch 9 implements the defined device interfaces. The last one adds a vfio pci extension specific for QAT which intercepts the vfio device operations for a QAT VF to allow live migration.

Here are the steps required to test the live migration of a QAT GEN4 VF:
1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko
2. Assign the VFs to the virtual machine and enable device live migration
3. Run a workload using a QAT VF inside the VM, for example using qatlib
(https://github.com/intel/qatlib)
4. Migrate the VM from the source node to a destination node

Changes in v5 since v4: https://lore.kernel.org/kvm/[email protected]

-  Remove device ID recheck as no consensus has been reached yet (Kevin)
-  Add missing state PRE_COPY_P2P in precopy_iotcl (Kevin)
-  Rearrange the state transition flow for better readability (Kevin)
-  Remove unnecessary Reviewed-by in commit message (Kevin)

Changes in v4 since v3: https://lore.kernel.org/kvm/[email protected]
- Change the order of maintainer entry for QAT vfio pci driver in
MAINTAINERS to make it alphabetical (Alex)
- Put QAT VFIO PCI driver under vfio/pci directly instead of
vfio/pci/intel (Alex)
- Add id_table recheck during device probe (Alex)

Changes in v3 since v2: https://lore.kernel.org/kvm/[email protected]
- Use state_mutex directly instead of unnecessary deferred_reset mode
(Jason)

Changes in v2 since v1: https://lore.kernel.org/all/[email protected]

-  Add VFIO_MIGRATION_PRE_COPY support (Alex)
-  Remove unnecessary module dependancy in Kconfig (Alex)
-  Use direct function calls instead of function pointers in qat vfio

variant driver (Jason)
- Address the comments including uncessary pointer check and kfree, missing lock and direct use of pci_iov_vf_id (Shameer)
- Change CHECK_STAT macro to avoid repeat comparison (Kamlesh)

Changes in v1 since RFC: https://lore.kernel.org/all/[email protected]
- Address comments including the right module dependancy in Kconfig,
source file name and module description (Alex)

-  Added PCI error handler and P2P state handler (Suggested by Kevin)
-  Refactor the state check duing loading ring state (Kevin) 
-  Fix missed call to vfio_put_device in the error case (Breet)
-  Migrate the shadow states in PF driver
-  Rebase on top of 6.8-rc1

Giovanni Cabiddu (2):
crypto: qat - adf_get_etr_base() helper
crypto: qat - relocate CSR access code

Siming Wan (3):
crypto: qat - rename get_sla_arr_of_type()
crypto: qat - expand CSR operations for QAT GEN4 devices
crypto: qat - add bank save and restore flows

Xin Zeng (5):
crypto: qat - relocate and rename 4xxx PF2VM definitions
crypto: qat - move PFVF compat checker to a function
crypto: qat - add interface for live migration
crypto: qat - implement interface for live migration vfio/qat: Add vfio_pci driver for Intel QAT VF devices

 MAINTAINERS                                   |    8 +
 .../intel/qat/qat_420xx/adf_420xx_hw_data.c   |    3 +
 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |    5 +
 .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c   |    1 +
 .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c     |    1 +
 .../intel/qat/qat_c62x/adf_c62x_hw_data.c     |    1 +
 .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c |    1 +
 drivers/crypto/intel/qat/qat_common/Makefile  |    6 +-
 .../intel/qat/qat_common/adf_accel_devices.h  |   88 ++
 .../intel/qat/qat_common/adf_common_drv.h     |   10 +
 .../qat/qat_common/adf_gen2_hw_csr_data.c     |  101 ++
 .../qat/qat_common/adf_gen2_hw_csr_data.h     |   86 ++
 .../intel/qat/qat_common/adf_gen2_hw_data.c   |   97 --
 .../intel/qat/qat_common/adf_gen2_hw_data.h   |   76 --
 .../qat/qat_common/adf_gen4_hw_csr_data.c     |  231 ++++
 .../qat/qat_common/adf_gen4_hw_csr_data.h     |  188 +++
 .../intel/qat/qat_common/adf_gen4_hw_data.c   |  380 +++++--
 .../intel/qat/qat_common/adf_gen4_hw_data.h   |  127 +--
 .../intel/qat/qat_common/adf_gen4_pfvf.c      |    8 +-
 .../intel/qat/qat_common/adf_gen4_vf_mig.c    | 1010 +++++++++++++++++
 .../intel/qat/qat_common/adf_gen4_vf_mig.h    |   10 +
 .../intel/qat/qat_common/adf_mstate_mgr.c     |  318 ++++++
 .../intel/qat/qat_common/adf_mstate_mgr.h     |   89 ++
 .../intel/qat/qat_common/adf_pfvf_pf_proto.c  |    8 +-
 .../intel/qat/qat_common/adf_pfvf_utils.h     |   11 +
 drivers/crypto/intel/qat/qat_common/adf_rl.c  |   10 +-
 drivers/crypto/intel/qat/qat_common/adf_rl.h  |    2 +
 .../crypto/intel/qat/qat_common/adf_sriov.c   |    7 +-
 .../intel/qat/qat_common/adf_transport.c      |    4 +-
 .../crypto/intel/qat/qat_common/qat_mig_dev.c |  130 +++
 .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c   |    1 +
 .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c   |    1 +
 drivers/vfio/pci/Kconfig                      |    2 +
 drivers/vfio/pci/Makefile                     |    2 +
 drivers/vfio/pci/qat/Kconfig                  |   12 +
 drivers/vfio/pci/qat/Makefile                 |    3 +
 drivers/vfio/pci/qat/main.c                   |  662 +++++++++++
 include/linux/qat/qat_mig_dev.h               |   31 +
 38 files changed, 3344 insertions(+), 387 deletions(-)
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c
 create mode 100644 drivers/vfio/pci/qat/Kconfig
 create mode 100644 drivers/vfio/pci/qat/Makefile
 create mode 100644 drivers/vfio/pci/qat/main.c
 create mode 100644 include/linux/qat/qat_mig_dev.h

base-commit: 318407ed77e4140d02e43a001b1f4753e3ce6b5f

Signature · 1 line
2.18.2