Skip to contentSkip to conversation

crypto: qat - enable SRIOV VF live migration for

PATCH Superseded Xin Zeng 7 participants 30 messages Feb 1, 2024 – Feb 22, 2024

Subsystems (5)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Xin Zeng

author
#0 lore raw

cover crypto: qat - enable SRIOV VF live migration for

This set enables live migration for Intel QAT GEN4 SRIOV Virtual Functions (VFs).
It is composed of 10 patches. Patch 1~6 refactor the original QAT PF driver implementation which will be reused by the following patches. Patch 7 introduces the logic to the QAT PF driver that allows to save and restore the state of a bank (a QAT VF is a wrapper around banks) and drain a ring pair. Patch 8 adds the QAT PF driver a set of interfaces to allow to save and restore the state of a VF that will be called by the module qat_vfio_pci which will be introduced in the last patch. Patch 9 implements the defined device interfaces. The last one adds a vfio pci extension specific for QAT which intercepts the vfio device operations for a QAT VF to allow live migration.

Here are the steps required to test the live migration of a QAT GEN4 VF:
1. Bind one or more QAT GEN4 VF devices to the module qat_vfio_pci.ko
2. Assign the VFs to the virtual machine and enable device live migration
3. Run a workload using a QAT VF inside the VM, for example using qatlib
(https://github.com/intel/qatlib)
4. Migrate the VM from the source node to a destination node

RFC:
https://lore.kernel.org/all/[email protected]/

Change logs:
RFC -> v1:
- Address comments including the right module dependancy in Kconfig, source file name and module description (Alex)

-  Added PCI error handler and P2P state handler (Suggested by Kevin)
-  Refactor the state check duing loading ring state (Kevin) 
-  Fix missed call to vfio_put_device in the error case (Breet)
-  Migrate the shadow states in PF driver
-  Rebase on top of 6.8-rc1

Giovanni Cabiddu (2):
crypto: qat - adf_get_etr_base() helper
crypto: qat - relocate CSR access code

Siming Wan (3):
crypto: qat - rename get_sla_arr_of_type()
crypto: qat - expand CSR operations for QAT GEN4 devices
crypto: qat - add bank save and restore flows

Xin Zeng (5):
crypto: qat - relocate and rename 4xxx PF2VM definitions
crypto: qat - move PFVF compat checker to a function
crypto: qat - add interface for live migration
crypto: qat - implement interface for live migration vfio/qat: Add vfio_pci driver for Intel QAT VF devices

 MAINTAINERS                                   |   8 +
 .../intel/qat/qat_420xx/adf_420xx_hw_data.c   |   3 +
 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |   5 +
 .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c   |   1 +
 .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c     |   1 +
 .../intel/qat/qat_c62x/adf_c62x_hw_data.c     |   1 +
 .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c |   1 +
 drivers/crypto/intel/qat/qat_common/Makefile  |   6 +-
 .../intel/qat/qat_common/adf_accel_devices.h  |  74 ++
 .../intel/qat/qat_common/adf_common_drv.h     |  10 +
 .../qat/qat_common/adf_gen2_hw_csr_data.c     | 101 +++
 .../qat/qat_common/adf_gen2_hw_csr_data.h     |  86 ++
 .../intel/qat/qat_common/adf_gen2_hw_data.c   |  97 --
 .../intel/qat/qat_common/adf_gen2_hw_data.h   |  76 --
 .../qat/qat_common/adf_gen4_hw_csr_data.c     | 231 +++++
 .../qat/qat_common/adf_gen4_hw_csr_data.h     | 188 ++++
 .../intel/qat/qat_common/adf_gen4_hw_data.c   | 382 +++++---
 .../intel/qat/qat_common/adf_gen4_hw_data.h   | 127 +--
 .../intel/qat/qat_common/adf_gen4_pfvf.c      |   8 +-
 .../intel/qat/qat_common/adf_gen4_vf_mig.c    | 834 ++++++++++++++++++
 .../intel/qat/qat_common/adf_gen4_vf_mig.h    |  10 +
 .../intel/qat/qat_common/adf_mstate_mgr.c     | 310 +++++++
 .../intel/qat/qat_common/adf_mstate_mgr.h     |  87 ++
 .../intel/qat/qat_common/adf_pfvf_pf_proto.c  |   8 +-
 .../intel/qat/qat_common/adf_pfvf_utils.h     |  11 +
 drivers/crypto/intel/qat/qat_common/adf_rl.c  |  10 +-
 drivers/crypto/intel/qat/qat_common/adf_rl.h  |   2 +
 .../crypto/intel/qat/qat_common/adf_sriov.c   |   7 +-
 .../intel/qat/qat_common/adf_transport.c      |   4 +-
 .../crypto/intel/qat/qat_common/qat_mig_dev.c |  35 +
 .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c   |   1 +
 .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c   |   1 +
 drivers/vfio/pci/Kconfig                      |   2 +
 drivers/vfio/pci/Makefile                     |   2 +
 drivers/vfio/pci/intel/qat/Kconfig            |  13 +
 drivers/vfio/pci/intel/qat/Makefile           |   4 +
 drivers/vfio/pci/intel/qat/main.c             | 572 ++++++++++++
 include/linux/qat/qat_mig_dev.h               |  31 +
 38 files changed, 2963 insertions(+), 387 deletions(-)
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c
 create mode 100644 drivers/vfio/pci/intel/qat/Kconfig
 create mode 100644 drivers/vfio/pci/intel/qat/Makefile
 create mode 100644 drivers/vfio/pci/intel/qat/main.c
 create mode 100644 include/linux/qat/qat_mig_dev.h

base-commit: 57f39c1d3cff476b5ce091bbf3df520488a682e5

Signature · 1 line
2.18.2
author
#1 lore raw

PATCH 1/10 crypto: qat - adf_get_etr_base() helper

From: Giovanni Cabiddu <[email protected]>

Add and use the new helper function adf_get_etr_base() which retrieves the virtual address of the ring bar.

This will be used extensively when adding support for Live Migration.

---
 drivers/crypto/intel/qat/qat_common/adf_common_drv.h   | 10 ++++++++++
 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c |  4 +---
 drivers/crypto/intel/qat/qat_common/adf_transport.c    |  4 +---
 3 files changed, 12 insertions(+), 6 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_common_drv.h +10 -0
index f06188033a93..cf1164a27f6f 100644
@@ -238,6 +238,16 @@ static inline void __iomem *adf_get_pmisc_base(struct adf_accel_dev *accel_dev)
 	return pmisc->virt_addr;
 }
 
+static inline void __iomem *adf_get_etr_base(struct adf_accel_dev *accel_dev)
+{
+	struct adf_hw_device_data *hw_data = accel_dev->hw_device;
+	struct adf_bar *etr;
+
+	etr = &GET_BARS(accel_dev)[hw_data->get_etr_bar_id(hw_data)];
+
+	return etr->virt_addr;
+}
+
 static inline void __iomem *adf_get_aram_base(struct adf_accel_dev *accel_dev)
 {
 	struct adf_hw_device_data *hw_data = accel_dev->hw_device;
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +1 -3
index f752653ccb47..a0d1326ac73d 100644
@@ -320,8 +320,7 @@ static int reset_ring_pair(void __iomem *csr, u32 bank_number)
 int adf_gen4_ring_pair_reset(struct adf_accel_dev *accel_dev, u32 bank_number)
 {
 	struct adf_hw_device_data *hw_data = accel_dev->hw_device;
-	u32 etr_bar_id = hw_data->get_etr_bar_id(hw_data);
-	void __iomem *csr;
+	void __iomem *csr = adf_get_etr_base(accel_dev);
 	int ret;
 
 	if (bank_number >= hw_data->num_banks)
@@ -330,7 +329,6 @@ int adf_gen4_ring_pair_reset(struct adf_accel_dev *accel_dev, u32 bank_number)
 	dev_dbg(&GET_DEV(accel_dev),
 		"ring pair reset for bank:%d\n", bank_number);
 
-	csr = (&GET_BARS(accel_dev)[etr_bar_id])->virt_addr;
 	ret = reset_ring_pair(csr, bank_number);
 	if (ret)
 		dev_err(&GET_DEV(accel_dev),
drivers/crypto/intel/qat/qat_common/adf_transport.c +1 -3
index 630d0483c4e0..1efdf46490f1 100644
@@ -474,7 +474,6 @@ static int adf_init_bank(struct adf_accel_dev *accel_dev,
 int adf_init_etr_data(struct adf_accel_dev *accel_dev)
 {
 	struct adf_etr_data *etr_data;
-	struct adf_hw_device_data *hw_data = accel_dev->hw_device;
 	void __iomem *csr_addr;
 	u32 size;
 	u32 num_banks = 0;
@@ -495,8 +494,7 @@ int adf_init_etr_data(struct adf_accel_dev *accel_dev)
 	}
 
 	accel_dev->transport = etr_data;
-	i = hw_data->get_etr_bar_id(hw_data);
-	csr_addr = accel_dev->accel_pci_dev.pci_bars[i].virt_addr;
+	csr_addr = adf_get_etr_base(accel_dev);
 
 	/* accel_dev->debugfs_dir should always be non-NULL here */
 	etr_data->debug = debugfs_create_dir("transport",
Signature · 1 line
2.18.2
author
#2 lore raw

PATCH 2/10 crypto: qat - relocate and rename 4xxx PF2VM definitions

In reply to Xin Zeng

Move and rename ADF_4XXX_PF2VM_OFFSET and ADF_4XXX_VM2PF_OFFSET to ADF_GEN4_PF2VM_OFFSET and ADF_GEN4_VM2PF_OFFSET respectively. These definitions are moved from adf_gen4_pfvf.c to adf_gen4_hw_data.h as they are specific to GEN4 and not just to qat_4xxx.

This change is made in anticipation of their use in live migration.

This does not introduce any functional change.

---
 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h | 4 ++++
 drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.c    | 8 +++-----
 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +4 -0
index 7d8a774cadc8..9cabbb5ce96b 100644
@@ -197,6 +197,10 @@ do { \
 /* Arbiter threads mask with error value */
 #define ADF_GEN4_ENA_THD_MASK_ERROR	GENMASK(ADF_NUM_THREADS_PER_AE, 0)
 
+/* PF2VM communication channel */
+#define ADF_GEN4_PF2VM_OFFSET(i)	(0x40B010 + (i) * 0x20)
+#define ADF_GEN4_VM2PF_OFFSET(i)	(0x40B014 + (i) * 0x20)
+
 void adf_gen4_set_ssm_wdtimer(struct adf_accel_dev *accel_dev);
 
 enum icp_qat_gen4_slice_mask {
drivers/crypto/intel/qat/qat_common/adf_gen4_pfvf.c +3 -5
index 8e8efe93f3ee..21474d402d09 100644
@@ -6,12 +6,10 @@
 #include "adf_accel_devices.h"
 #include "adf_common_drv.h"
 #include "adf_gen4_pfvf.h"
+#include "adf_gen4_hw_data.h"
 #include "adf_pfvf_pf_proto.h"
 #include "adf_pfvf_utils.h"
 
-#define ADF_4XXX_PF2VM_OFFSET(i)	(0x40B010 + ((i) * 0x20))
-#define ADF_4XXX_VM2PF_OFFSET(i)	(0x40B014 + ((i) * 0x20))
-
 /* VF2PF interrupt source registers */
 #define ADF_4XXX_VM2PF_SOU		0x41A180
 #define ADF_4XXX_VM2PF_MSK		0x41A1C0
@@ -29,12 +27,12 @@ static const struct pfvf_csr_format csr_gen4_fmt = {
 
 static u32 adf_gen4_pf_get_pf2vf_offset(u32 i)
 {
-	return ADF_4XXX_PF2VM_OFFSET(i);
+	return ADF_GEN4_PF2VM_OFFSET(i);
 }
 
 static u32 adf_gen4_pf_get_vf2pf_offset(u32 i)
 {
-	return ADF_4XXX_VM2PF_OFFSET(i);
+	return ADF_GEN4_VM2PF_OFFSET(i);
 }
 
 static void adf_gen4_enable_vf2pf_interrupts(void __iomem *pmisc_addr, u32 vf_mask)
Signature · 1 line
2.18.2
author
#3 lore raw

PATCH 3/10 crypto: qat - move PFVF compat checker to a function

In reply to Xin Zeng

Move the code that implements VF version compatibility on the PF side to a separate function so that it can be reused when doing VM live migration.

This does not introduce any functional change.

---
 .../crypto/intel/qat/qat_common/adf_pfvf_pf_proto.c   |  8 +-------
 drivers/crypto/intel/qat/qat_common/adf_pfvf_utils.h  | 11 +++++++++++
 2 files changed, 12 insertions(+), 7 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_pfvf_pf_proto.c +1 -7
index 388e58bcbcaf..d737d07937c7 100644
@@ -242,13 +242,7 @@ static int adf_handle_vf2pf_msg(struct adf_accel_dev *accel_dev, u8 vf_nr,
 			"VersionRequest received from VF%d (vers %d) to PF (vers %d)\n",
 			vf_nr, vf_compat_ver, ADF_PFVF_COMPAT_THIS_VERSION);
 
-		if (vf_compat_ver == 0)
-			compat = ADF_PF2VF_VF_INCOMPATIBLE;
-		else if (vf_compat_ver <= ADF_PFVF_COMPAT_THIS_VERSION)
-			compat = ADF_PF2VF_VF_COMPATIBLE;
-		else
-			compat = ADF_PF2VF_VF_COMPAT_UNKNOWN;
-
+		compat = adf_vf_compat_checker(vf_compat_ver);
 		vf_info->vf_compat_ver = vf_compat_ver;
 
 		resp->type = ADF_PF2VF_MSGTYPE_VERSION_RESP;
drivers/crypto/intel/qat/qat_common/adf_pfvf_utils.h +11 -0
index 2be048e2287b..1a044297d873 100644
@@ -28,4 +28,15 @@ u32 adf_pfvf_csr_msg_of(struct adf_accel_dev *accel_dev, struct pfvf_message msg
 struct pfvf_message adf_pfvf_message_of(struct adf_accel_dev *accel_dev, u32 raw_msg,
 					const struct pfvf_csr_format *fmt);
 
+static inline u8 adf_vf_compat_checker(u8 vf_compat_ver)
+{
+	if (vf_compat_ver == 0)
+		return ADF_PF2VF_VF_INCOMPATIBLE;
+
+	if (vf_compat_ver <= ADF_PFVF_COMPAT_THIS_VERSION)
+		return ADF_PF2VF_VF_COMPATIBLE;
+
+	return ADF_PF2VF_VF_COMPAT_UNKNOWN;
+}
+
 #endif /* ADF_PFVF_UTILS_H */
Signature · 1 line
2.18.2
author
#4 lore raw

PATCH 4/10 crypto: qat - relocate CSR access code

In reply to Xin Zeng

From: Giovanni Cabiddu <[email protected]>

As the common hw_data files are growing and the adf_hw_csr_ops is going to be extended with new operations, move all logic related to ring CSRs to the newly created adf_gen[2|4]_hw_csr_data.[c|h] files.

This does not introduce any functional change.

---
 .../intel/qat/qat_420xx/adf_420xx_hw_data.c   |   1 +
 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |   1 +
 .../intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c   |   1 +
 .../qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c     |   1 +
 .../intel/qat/qat_c62x/adf_c62x_hw_data.c     |   1 +
 .../intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c |   1 +
 drivers/crypto/intel/qat/qat_common/Makefile  |   2 +
 .../qat/qat_common/adf_gen2_hw_csr_data.c     | 101 ++++++++++++++++++
 .../qat/qat_common/adf_gen2_hw_csr_data.h     |  86 +++++++++++++++
 .../intel/qat/qat_common/adf_gen2_hw_data.c   |  97 -----------------
 .../intel/qat/qat_common/adf_gen2_hw_data.h   |  76 -------------
 .../qat/qat_common/adf_gen4_hw_csr_data.c     | 101 ++++++++++++++++++
 .../qat/qat_common/adf_gen4_hw_csr_data.h     |  97 +++++++++++++++++
 .../intel/qat/qat_common/adf_gen4_hw_data.c   |  97 -----------------
 .../intel/qat/qat_common/adf_gen4_hw_data.h   |  94 +---------------
 .../qat/qat_dh895xcc/adf_dh895xcc_hw_data.c   |   1 +
 .../qat_dh895xccvf/adf_dh895xccvf_hw_data.c   |   1 +
 17 files changed, 397 insertions(+), 362 deletions(-)
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +1 -0
index a87d29ae724f..bd36b536f663 100644
@@ -10,6 +10,7 @@
 #include <adf_fw_config.h>
 #include <adf_gen4_config.h>
 #include <adf_gen4_dc.h>
+#include <adf_gen4_hw_csr_data.h>
 #include <adf_gen4_hw_data.h>
 #include <adf_gen4_pfvf.h>
 #include <adf_gen4_pm.h>
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +1 -0
index 94a0ebb03d8c..054e48d194cd 100644
@@ -10,6 +10,7 @@
 #include <adf_fw_config.h>
 #include <adf_gen4_config.h>
 #include <adf_gen4_dc.h>
+#include <adf_gen4_hw_csr_data.h>
 #include <adf_gen4_hw_data.h>
 #include <adf_gen4_pfvf.h>
 #include <adf_gen4_pm.h>
drivers/crypto/intel/qat/qat_c3xxx/adf_c3xxx_hw_data.c +1 -0
index a882e0ea2279..201f9412c582 100644
@@ -6,6 +6,7 @@
 #include <adf_common_drv.h>
 #include <adf_gen2_config.h>
 #include <adf_gen2_dc.h>
+#include <adf_gen2_hw_csr_data.h>
 #include <adf_gen2_hw_data.h>
 #include <adf_gen2_pfvf.h>
 #include "adf_c3xxx_hw_data.h"
drivers/crypto/intel/qat/qat_c3xxxvf/adf_c3xxxvf_hw_data.c +1 -0
index 84d9486e04de..a512ca4efd3f 100644
@@ -4,6 +4,7 @@
 #include <adf_common_drv.h>
 #include <adf_gen2_config.h>
 #include <adf_gen2_dc.h>
+#include <adf_gen2_hw_csr_data.h>
 #include <adf_gen2_hw_data.h>
 #include <adf_gen2_pfvf.h>
 #include <adf_pfvf_vf_msg.h>
drivers/crypto/intel/qat/qat_c62x/adf_c62x_hw_data.c +1 -0
index 48cf3eb7c734..6b5b0cf9c7c7 100644
@@ -6,6 +6,7 @@
 #include <adf_common_drv.h>
 #include <adf_gen2_config.h>
 #include <adf_gen2_dc.h>
+#include <adf_gen2_hw_csr_data.h>
 #include <adf_gen2_hw_data.h>
 #include <adf_gen2_pfvf.h>
 #include "adf_c62x_hw_data.h"
drivers/crypto/intel/qat/qat_c62xvf/adf_c62xvf_hw_data.c +1 -0
index 751d7aa57fc7..4aaaaf921734 100644
@@ -4,6 +4,7 @@
 #include <adf_common_drv.h>
 #include <adf_gen2_config.h>
 #include <adf_gen2_dc.h>
+#include <adf_gen2_hw_csr_data.h>
 #include <adf_gen2_hw_data.h>
 #include <adf_gen2_pfvf.h>
 #include <adf_pfvf_vf_msg.h>
drivers/crypto/intel/qat/qat_common/Makefile +2 -0
index 6908727bff3b..da4d806a9287 100644
@@ -14,9 +14,11 @@ intel_qat-objs := adf_cfg.o \
 	adf_hw_arbiter.o \
 	adf_sysfs.o \
 	adf_sysfs_ras_counters.o \
+	adf_gen2_hw_csr_data.o \
 	adf_gen2_hw_data.o \
 	adf_gen2_config.o \
 	adf_gen4_config.o \
+	adf_gen4_hw_csr_data.o \
 	adf_gen4_hw_data.o \
 	adf_gen4_pm.o \
 	adf_gen2_dc.o \
drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.c new file +101 -0

101 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen2_hw_csr_data.h new file +86 -0

86 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen2_hw_data.c +0 -97

103 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen2_hw_data.h +0 -76

97 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c new file +101 -0

101 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h new file +97 -0

97 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +0 -97

103 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +2 -92

111 lines not on this page · raw message

drivers/crypto/intel/qat/qat_dh895xcc/adf_dh895xcc_hw_data.c +1 -0
index af14090cc4be..6e24d57e6b98 100644
@@ -5,6 +5,7 @@
 #include <adf_common_drv.h>
 #include <adf_gen2_config.h>
 #include <adf_gen2_dc.h>
+#include <adf_gen2_hw_csr_data.h>
 #include <adf_gen2_hw_data.h>
 #include <adf_gen2_pfvf.h>
 #include "adf_dh895xcc_hw_data.h"
drivers/crypto/intel/qat/qat_dh895xccvf/adf_dh895xccvf_hw_data.c +1 -0
index 70e56cc16ece..f4ee4c2e00da 100644
@@ -4,6 +4,7 @@
 #include <adf_common_drv.h>
 #include <adf_gen2_config.h>
 #include <adf_gen2_dc.h>
+#include <adf_gen2_hw_csr_data.h>
 #include <adf_gen2_hw_data.h>
 #include <adf_gen2_pfvf.h>
 #include <adf_pfvf_vf_msg.h>
Signature · 1 line
2.18.2
author
#5 lore raw

PATCH 5/10 crypto: qat - rename get_sla_arr_of_type()

In reply to Xin Zeng

From: Siming Wan <[email protected]>

The function get_sla_arr_of_type() returns a pointer to an SLA type specific array.
Rename it and expose it as it will be used externally to this module.

This does not introduce any functional change.

---
 drivers/crypto/intel/qat/qat_common/adf_rl.c | 10 +++++-----
 drivers/crypto/intel/qat/qat_common/adf_rl.h |  2 ++
 2 files changed, 7 insertions(+), 5 deletions(-)
drivers/crypto/intel/qat/qat_common/adf_rl.c +5 -5
index de1b214dba1f..41017af1be68 100644
@@ -183,14 +183,14 @@ static enum adf_cfg_service_type srv_to_cfg_svc_type(enum adf_base_services rl_s
 }
 
 /**
- * get_sla_arr_of_type() - Returns a pointer to SLA type specific array
+ * adf_rl_get_sla_arr_of_type() - Returns a pointer to SLA type specific array
  * @rl_data: pointer to ratelimiting data
  * @type: SLA type
  * @sla_arr: pointer to variable where requested pointer will be stored
  *
  * Return: Max number of elements allowed for the returned array
  */
-static u32 get_sla_arr_of_type(struct adf_rl *rl_data, enum rl_node_type type,
+u32 adf_rl_get_sla_arr_of_type(struct adf_rl *rl_data, enum rl_node_type type,
 			       struct rl_sla ***sla_arr)
 {
 	switch (type) {
@@ -778,7 +778,7 @@ static void clear_sla(struct adf_rl *rl_data, struct rl_sla *sla)
 		rp_in_use[sla->ring_pairs_ids[i]] = false;
 
 	update_budget(sla, old_cir, true);
-	get_sla_arr_of_type(rl_data, sla->type, &sla_type_arr);
+	adf_rl_get_sla_arr_of_type(rl_data, sla->type, &sla_type_arr);
 	assign_node_to_parent(rl_data->accel_dev, sla, true);
 	adf_rl_send_admin_delete_msg(rl_data->accel_dev, node_id, sla->type);
 	mark_rps_usage(sla, rl_data->rp_in_use, false);
@@ -857,7 +857,7 @@ static int add_update_sla(struct adf_accel_dev *accel_dev,
 
 	if (!is_update) {
 		mark_rps_usage(sla, rl_data->rp_in_use, true);
-		get_sla_arr_of_type(rl_data, sla->type, &sla_type_arr);
+		adf_rl_get_sla_arr_of_type(rl_data, sla->type, &sla_type_arr);
 		sla_type_arr[sla->node_id] = sla;
 		rl_data->sla[sla->sla_id] = sla;
 	}
@@ -1047,7 +1047,7 @@ void adf_rl_remove_sla_all(struct adf_accel_dev *accel_dev, bool incl_default)
 
 	/* Unregister and remove all SLAs */
 	for (j = RL_LEAF; j >= end_type; j--) {
-		max_id = get_sla_arr_of_type(rl_data, j, &sla_type_arr);
+		max_id = adf_rl_get_sla_arr_of_type(rl_data, j, &sla_type_arr);
 
 		for (i = 0; i < max_id; i++) {
 			if (!sla_type_arr[i])
drivers/crypto/intel/qat/qat_common/adf_rl.h +2 -0
index 269c6656fb90..bfe750ea0e83 100644
@@ -151,6 +151,8 @@ struct rl_sla {
 	u16 ring_pairs_cnt;
 };
 
+u32 adf_rl_get_sla_arr_of_type(struct adf_rl *rl_data, enum rl_node_type type,
+			       struct rl_sla ***sla_arr);
 int adf_rl_add_sla(struct adf_accel_dev *accel_dev,
 		   struct adf_rl_sla_input_data *sla_in);
 int adf_rl_update_sla(struct adf_accel_dev *accel_dev,
Signature · 1 line
2.18.2
author
#6 lore raw

PATCH 6/10 crypto: qat - expand CSR operations for QAT GEN4 devices

In reply to Xin Zeng

From: Siming Wan <[email protected]>

Extend the CSR operations for QAT GEN4 devices to allow saving and restoring the rings state.

The new operations will be used as a building block for implementing the state save and restore of Virtual Functions necessary for VM live migration.

This adds the following operations:
- read ring status register
- read ring underflow/overflow status register
- read ring nearly empty status register
- read ring nearly full status register
- read ring full status register
- read ring complete status register
- read ring exception status register
- read/write ring exception interrupt mask register
- read ring configuration register
- read ring base register
- read/write ring interrupt enable register
- read ring interrupt flag register
- read/write ring interrupt source select register
- read ring coalesced interrupt enable register
- read ring coalesced interrupt control register
- read ring flag and coalesced interrupt enable register
- read ring service arbiter enable register
- get ring coalesced interrupt control enable mask

---
 .../intel/qat/qat_common/adf_accel_devices.h  |  27 ++++
 .../qat/qat_common/adf_gen4_hw_csr_data.c     | 130 ++++++++++++++++++
 .../qat/qat_common/adf_gen4_hw_csr_data.h     |  93 ++++++++++++-
 3 files changed, 249 insertions(+), 1 deletion(-)
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +27 -0
index a16c7e6edc65..5cc94bea176a 100644
@@ -150,22 +150,49 @@ struct adf_hw_csr_ops {
 				  u32 ring);
 	void (*write_csr_ring_tail)(void __iomem *csr_base_addr, u32 bank,
 				    u32 ring, u32 value);
+	u32 (*read_csr_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_uo_stat)(void __iomem *csr_base_addr, u32 bank);
 	u32 (*read_csr_e_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_ne_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_nf_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_f_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_c_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_exp_stat)(void __iomem *csr_base_addr, u32 bank);
+	u32 (*read_csr_exp_int_en)(void __iomem *csr_base_addr, u32 bank);
+	void (*write_csr_exp_int_en)(void __iomem *csr_base_addr, u32 bank,
+				     u32 value);
+	u32 (*read_csr_ring_config)(void __iomem *csr_base_addr, u32 bank,
+				    u32 ring);
 	void (*write_csr_ring_config)(void __iomem *csr_base_addr, u32 bank,
 				      u32 ring, u32 value);
+	dma_addr_t (*read_csr_ring_base)(void __iomem *csr_base_addr, u32 bank,
+					 u32 ring);
 	void (*write_csr_ring_base)(void __iomem *csr_base_addr, u32 bank,
 				    u32 ring, dma_addr_t addr);
+	u32 (*read_csr_int_en)(void __iomem *csr_base_addr, u32 bank);
+	void (*write_csr_int_en)(void __iomem *csr_base_addr, u32 bank,
+				 u32 value);
+	u32 (*read_csr_int_flag)(void __iomem *csr_base_addr, u32 bank);
 	void (*write_csr_int_flag)(void __iomem *csr_base_addr, u32 bank,
 				   u32 value);
+	u32 (*read_csr_int_srcsel)(void __iomem *csr_base_addr, u32 bank);
 	void (*write_csr_int_srcsel)(void __iomem *csr_base_addr, u32 bank);
+	void (*write_csr_int_srcsel_w_val)(void __iomem *csr_base_addr,
+					   u32 bank, u32 value);
+	u32 (*read_csr_int_col_en)(void __iomem *csr_base_addr, u32 bank);
 	void (*write_csr_int_col_en)(void __iomem *csr_base_addr, u32 bank,
 				     u32 value);
+	u32 (*read_csr_int_col_ctl)(void __iomem *csr_base_addr, u32 bank);
 	void (*write_csr_int_col_ctl)(void __iomem *csr_base_addr, u32 bank,
 				      u32 value);
+	u32 (*read_csr_int_flag_and_col)(void __iomem *csr_base_addr,
+					 u32 bank);
 	void (*write_csr_int_flag_and_col)(void __iomem *csr_base_addr,
 					   u32 bank, u32 value);
+	u32 (*read_csr_ring_srv_arb_en)(void __iomem *csr_base_addr, u32 bank);
 	void (*write_csr_ring_srv_arb_en)(void __iomem *csr_base_addr, u32 bank,
 					  u32 value);
+	u32 (*get_int_col_ctl_enable_mask)(void);
 };
 
 struct adf_cfg_device_data;
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.c +130 -0

201 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_csr_data.h +92 -1

148 lines not on this page · raw message

Signature · 1 line
2.18.2
author
#7 lore raw

PATCH 7/10 crypto: qat - add bank save and restore flows

In reply to Xin Zeng

From: Siming Wan <[email protected]>

Add logic to save, restore, quiesce and drain a ring bank for QAT GEN4 devices.
This allows to save and restore the state of a Virtual Function (VF) and will be used to implement VM live migration.

---
 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |   2 +
 .../intel/qat/qat_common/adf_accel_devices.h  |  39 +++
 .../intel/qat/qat_common/adf_gen4_hw_data.c   | 281 ++++++++++++++++++
 .../intel/qat/qat_common/adf_gen4_hw_data.h   |  19 ++
 4 files changed, 341 insertions(+)
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +2 -0
index 054e48d194cd..a04488ada49b 100644
@@ -487,6 +487,8 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data, u32 dev_id)
 	hw_data->get_ring_to_svc_map = get_ring_to_svc_map;
 	hw_data->disable_iov = adf_disable_sriov;
 	hw_data->ring_pair_reset = adf_gen4_ring_pair_reset;
+	hw_data->bank_state_save = adf_gen4_bank_state_save;
+	hw_data->bank_state_restore = adf_gen4_bank_state_restore;
 	hw_data->enable_pm = adf_gen4_enable_pm;
 	hw_data->handle_pm_interrupt = adf_gen4_handle_pm_interrupt;
 	hw_data->dev_config = adf_gen4_dev_config;
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +39 -0
index 5cc94bea176a..bdb026155f64 100644
@@ -140,6 +140,41 @@ struct admin_info {
 	u32 mailbox_offset;
 };
 
+struct ring_config {
+	u32 config;
+	u64 base;
+	u32 head;
+	u32 tail;
+};
+
+struct bank_state {
+	u32 reservd0;
+	u32 reservd1;
+	u32 num_rings;
+	u32 ringstat0;
+	u32 ringstat1;
+	u32 ringuostat;
+	u32 ringestat;
+	u32 ringnestat;
+	u32 ringnfstat;
+	u32 ringfstat;
+	u32 ringcstat0;
+	u32 ringcstat1;
+	u32 ringcstat2;
+	u32 ringcstat3;
+	u32 iaintflagen;
+	u32 iaintflagreg;
+	u32 iaintflagsrcsel0;
+	u32 iaintflagsrcsel1;
+	u32 iaintcolen;
+	u32 iaintcolctl;
+	u32 iaintflagandcolen;
+	u32 ringexpstat;
+	u32 ringexpintenable;
+	u32 ringsrvarben;
+	struct ring_config rings[ADF_ETR_MAX_RINGS_PER_BANK];
+};
+
 struct adf_hw_csr_ops {
 	u64 (*build_csr_ring_base_addr)(dma_addr_t addr, u32 size);
 	u32 (*read_csr_ring_head)(void __iomem *csr_base_addr, u32 bank,
@@ -271,6 +306,10 @@ struct adf_hw_device_data {
 	void (*enable_ints)(struct adf_accel_dev *accel_dev);
 	void (*set_ssm_wdtimer)(struct adf_accel_dev *accel_dev);
 	int (*ring_pair_reset)(struct adf_accel_dev *accel_dev, u32 bank_nr);
+	int (*bank_state_save)(struct adf_accel_dev *accel_dev, u32 bank_number,
+			       struct bank_state *state);
+	int (*bank_state_restore)(struct adf_accel_dev *accel_dev,
+				  u32 bank_number, struct bank_state *state);
 	void (*reset_device)(struct adf_accel_dev *accel_dev);
 	void (*set_msix_rttable)(struct adf_accel_dev *accel_dev);
 	const char *(*uof_get_name)(struct adf_accel_dev *accel_dev, u32 obj_num);
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +281 -0

290 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +19 -0
index f94ccf64e55d..788fe9d345c0 100644
@@ -77,10 +77,19 @@
 #define ADF_RPRESET_POLL_TIMEOUT_US	(5 * USEC_PER_SEC)
 #define ADF_RPRESET_POLL_DELAY_US	20
 #define ADF_WQM_CSR_RPRESETCTL_RESET	BIT(0)
+#define ADF_WQM_CSR_RPRESETCTL_DRAIN	BIT(2)
 #define ADF_WQM_CSR_RPRESETCTL(bank)	(0x6000 + ((bank) << 3))
 #define ADF_WQM_CSR_RPRESETSTS_STATUS	BIT(0)
 #define ADF_WQM_CSR_RPRESETSTS(bank)	(ADF_WQM_CSR_RPRESETCTL(bank) + 4)
 
+/* Ring interrupt */
+#define ADF_RP_INT_SRC_SEL_F_RISE_MASK	BIT(2)
+#define ADF_RP_INT_SRC_SEL_F_FALL_MASK	GENMASK(2, 0)
+#define ADF_RP_INT_SRC_SEL_RANGE_WIDTH	4
+#define ADF_COALESCED_POLL_DELAY_US	1000
+#define ADF_WQM_CSR_RPINTSOU(bank)	(0x200000 + ((bank) << 12))
+#define ADF_WQM_CSR_RP_IDX_RX		1
+
 /* Error source registers */
 #define ADF_GEN4_ERRSOU0	(0x41A200)
 #define ADF_GEN4_ERRSOU1	(0x41A204)
@@ -149,5 +158,15 @@ int adf_gen4_ring_pair_reset(struct adf_accel_dev *accel_dev, u32 bank_number);
 void adf_gen4_set_msix_default_rttable(struct adf_accel_dev *accel_dev);
 void adf_gen4_set_ssm_wdtimer(struct adf_accel_dev *accel_dev);
 int adf_gen4_init_thd2arb_map(struct adf_accel_dev *accel_dev);
+int adf_gen4_bank_quiesce_coal_timer(struct adf_accel_dev *accel_dev,
+				     u32 bank_idx, int timeout_ms);
+int adf_gen4_bank_drain_start(struct adf_accel_dev *accel_dev,
+			      u32 bank_number, int timeout_us);
+void adf_gen4_bank_drain_finish(struct adf_accel_dev *accel_dev,
+				u32 bank_number);
+int adf_gen4_bank_state_save(struct adf_accel_dev *accel_dev, u32 bank_number,
+			     struct bank_state *state);
+int adf_gen4_bank_state_restore(struct adf_accel_dev *accel_dev,
+				u32 bank_number, struct bank_state *state);
 
 #endif
Signature · 1 line
2.18.2
#8 lore raw

Re: [EXTERNAL] [PATCH 07/10] crypto: qat - add bank save and restore flows

Quoted 14 lines · Xin Zeng

This message was sent from outside of Texas Instruments.
Do not click links or open attachments unless you recognize the source of this email and know the content is safe.

From: Siming Wan <[email protected]>

Add logic to save, restore, quiesce and drain a ring bank for QAT GEN4 devices.
This allows to save and restore the state of a Virtual Function (VF) and will be used to implement VM live migration.

...

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c int adf_gen4_init_thd2arb_map(struct adf_accel_dev *accel_dev) · line 474

Quoted 8 lines
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +8 -0
+#define CHECK_STAT(op, expect_val, name, args...) \
+({ \
+	u32 __expect_val = (expect_val); \
+	u32 actual_val = op(args); \
+	if (__expect_val != actual_val) \
+		pr_err("QAT: Fail to restore %s register. Expected 0x%x, but actual is 0x%x\n", \
+			name, __expect_val, actual_val); \
+	(__expect_val == actual_val) ? 0 : -EINVAL; \

I was wondering if this can be done like following, saves repeat comparison.

(__expect_val == actual_val) ? 0 : (pr_err("QAT: Fail to restore %s \
                                          register. Expected 0x%x, \
                                          but actual is 0x%x\n", \
                 			  name, __expect_val, \
                                          actual_val), -EINVAL); \

Regards,
Kamlesh

Quoted 1 line

+})

...

#9 lore raw

RE: [EXTERNAL] [PATCH 07/10] crypto: qat - add bank save and restore flows

From: Kamlesh Gurudasani <[email protected]>
Sent: Sunday, February 4, 2024 3:50 PM
To: Zeng, Xin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Tian, Kevin <[email protected]>
Cc: [email protected]; [email protected]; qat-linux <[email protected]>; Wan, Siming <[email protected]>; Zeng, Xin <[email protected]>
Subject: Re: [EXTERNAL] [PATCH 07/10] crypto: qat - add bank save and restore flows

Quoted 14 lines · Xin Zeng

This message was sent from outside of Texas Instruments.
Do not click links or open attachments unless you recognize the source of this email and know the content is safe.

From: Siming Wan <[email protected]>

Add logic to save, restore, quiesce and drain a ring bank for QAT GEN4 devices.
This allows to save and restore the state of a Virtual Function (VF) and will be used to implement VM live migration.

...

drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c int adf_gen4_init_thd2arb_map(struct adf_accel_dev *accel_dev) · line 474

Quoted 7 lines
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.c +7 -0
+#define CHECK_STAT(op, expect_val, name, args...) \ ({ \
+	u32 __expect_val = (expect_val); \
+	u32 actual_val = op(args); \
+	if (__expect_val != actual_val) \
+		pr_err("QAT: Fail to restore %s register. Expected 0x%x, but actual is 0x%x\n", \
+			name, __expect_val, actual_val); \
+	(__expect_val == actual_val) ? 0 : -EINVAL; \

I was wondering if this can be done like following, saves repeat comparison.

(__expect_val == actual_val) ? 0 : (pr_err("QAT: Fail to restore %s \
                                          register. Expected 0x%x, \
                                          but actual is 0x%x\n", \
                 			  name, __expect_val, \
                                          actual_val), -EINVAL); \ Regards, Kamlesh
Quoted 1 line

+})

...

Thanks for your comments! Will update.

author
#10 lore raw

PATCH 8/10 crypto: qat - add interface for live migration

In reply to Xin Zeng

Extend the driver with a new interface to be used for VF live migration. This allows to create and destroy a qat_mig_dev object that contains a set of methods to allow to save and restore the state of QAT VF. This interface will be used by the qat-vfio-pci module.

---
 drivers/crypto/intel/qat/qat_common/Makefile  |  2 +-
 .../intel/qat/qat_common/adf_accel_devices.h  |  2 ++
 .../crypto/intel/qat/qat_common/qat_mig_dev.c | 35 +++++++++++++++++++
 include/linux/qat/qat_mig_dev.h               | 31 ++++++++++++++++
 4 files changed, 69 insertions(+), 1 deletion(-)
 create mode 100644 drivers/crypto/intel/qat/qat_common/qat_mig_dev.c
 create mode 100644 include/linux/qat/qat_mig_dev.h
drivers/crypto/intel/qat/qat_common/Makefile +1 -1
index da4d806a9287..05cac8a9e463 100644
@@ -54,4 +54,4 @@ intel_qat-$(CONFIG_DEBUG_FS) += adf_transport_debug.o \
 intel_qat-$(CONFIG_PCI_IOV) += adf_sriov.o adf_vf_isr.o adf_pfvf_utils.o \
 			       adf_pfvf_pf_msg.o adf_pfvf_pf_proto.o \
 			       adf_pfvf_vf_msg.o adf_pfvf_vf_proto.o \
-			       adf_gen2_pfvf.o adf_gen4_pfvf.o
+			       adf_gen2_pfvf.o adf_gen4_pfvf.o qat_mig_dev.o
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +2 -0
index bdb026155f64..0fa52d20c131 100644
@@ -9,6 +9,7 @@
 #include <linux/pci.h>
 #include <linux/ratelimit.h>
 #include <linux/types.h>
+#include <linux/qat/qat_mig_dev.h>
 #include "adf_cfg_common.h"
 #include "adf_rl.h"
 #include "adf_telemetry.h"
@@ -325,6 +326,7 @@ struct adf_hw_device_data {
 	struct adf_dev_err_mask dev_err_mask;
 	struct adf_rl_hw_data rl_data;
 	struct adf_tl_hw_data tl_data;
+	struct qat_migdev_ops vfmig_ops;
 	const char *fw_name;
 	const char *fw_mmp_name;
 	u32 fuses;
drivers/crypto/intel/qat/qat_common/qat_mig_dev.c new file +35 -0
new file mode 100644
index 000000000000..45520a715544
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2024 Intel Corporation */
+#include <linux/dev_printk.h>
+#include <linux/export.h>
+#include <linux/pci.h>
+#include <linux/types.h>
+#include <linux/qat/qat_mig_dev.h>
+#include "adf_common_drv.h"
+
+struct qat_mig_dev *qat_vfmig_create(struct pci_dev *pdev, int vf_id)
+{
+	struct adf_accel_dev *accel_dev;
+	struct qat_mig_dev *mdev;
+
+	accel_dev = adf_devmgr_pci_to_accel_dev(pdev);
+	if (!accel_dev)
+		return ERR_PTR(-ENODEV);
+
+	mdev = kmalloc(sizeof(*mdev), GFP_KERNEL);
+	if (!mdev)
+		return ERR_PTR(-ENOMEM);
+
+	mdev->vf_id = vf_id;
+	mdev->parent_accel_dev = accel_dev;
+	mdev->ops = &accel_dev->hw_device->vfmig_ops;
+
+	return mdev;
+}
+EXPORT_SYMBOL_GPL(qat_vfmig_create);
+
+void qat_vfmig_destroy(struct qat_mig_dev *mdev)
+{
+	kfree(mdev);
+}
+EXPORT_SYMBOL_GPL(qat_vfmig_destroy);
include/linux/qat/qat_mig_dev.h new file +31 -0
new file mode 100644
index 000000000000..229454efe3ba
@@ -0,0 +1,31 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2024 Intel Corporation */
+#ifndef QAT_MIG_DEV_H_
+#define QAT_MIG_DEV_H_
+
+struct pci_dev;
+struct qat_migdev_ops;
+
+struct qat_mig_dev {
+	void *parent_accel_dev;
+	struct qat_migdev_ops *ops;
+	u8 *state;
+	u32 state_size;
+	s32 vf_id;
+};
+
+struct qat_migdev_ops {
+	int (*init)(struct qat_mig_dev *mdev);
+	void (*cleanup)(struct qat_mig_dev *mdev);
+	int (*open)(struct qat_mig_dev *mdev);
+	void (*close)(struct qat_mig_dev *mdev);
+	int (*suspend)(struct qat_mig_dev *mdev);
+	int (*resume)(struct qat_mig_dev *mdev);
+	int (*save_state)(struct qat_mig_dev *mdev);
+	int (*load_state)(struct qat_mig_dev *mdev);
+};
+
+struct qat_mig_dev *qat_vfmig_create(struct pci_dev *pdev, int vf_id);
+void qat_vfmig_destroy(struct qat_mig_dev *mdev);
+
+#endif /*QAT_MIG_DEV_H_*/
Signature · 1 line
2.18.2
author
#11 lore raw

PATCH 9/10 crypto: qat - implement interface for live migration

In reply to Xin Zeng

Add logic to implement the interface for live migration defined in qat/qat_mig_dev.h. This is specific for QAT GEN4 Virtual Functions (VFs).

This introduces a migration data manager which is used to handle the device state during migration. The manager ensures that the device state is stored in a format that can be restored in the destination node.

The VF state is organized into a hierarchical structure that includes a preamble, a general state section, a MISC bar section and an ETR bar section. The latter contains the state of the 4 ring pairs contained on a VF. Here is a graphical representation of the state:

    preamble | general state section | leaf state
             | MISC bar state section| leaf state
             | ETR bar state section | bank0 state section | leaf state
                                     | bank1 state section | leaf state
                                     | bank2 state section | leaf state
                                     | bank3 state section | leaf state

In addition to the implementation of the qat_migdev_ops interface and the state manager framework, add a mutex in pfvf to avoid pf2vf messages during migration.

---
 .../intel/qat/qat_420xx/adf_420xx_hw_data.c   |   2 +
 .../intel/qat/qat_4xxx/adf_4xxx_hw_data.c     |   2 +
 drivers/crypto/intel/qat/qat_common/Makefile  |   2 +
 .../intel/qat/qat_common/adf_accel_devices.h  |   6 +
 .../intel/qat/qat_common/adf_gen4_hw_data.h   |  10 +
 .../intel/qat/qat_common/adf_gen4_vf_mig.c    | 834 ++++++++++++++++++
 .../intel/qat/qat_common/adf_gen4_vf_mig.h    |  10 +
 .../intel/qat/qat_common/adf_mstate_mgr.c     | 310 +++++++
 .../intel/qat/qat_common/adf_mstate_mgr.h     |  87 ++
 .../crypto/intel/qat/qat_common/adf_sriov.c   |   7 +-
 10 files changed, 1269 insertions(+), 1 deletion(-)
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c
 create mode 100644 drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h
drivers/crypto/intel/qat/qat_420xx/adf_420xx_hw_data.c +2 -0
index bd36b536f663..a48bb83d3e3d 100644
@@ -17,6 +17,7 @@
 #include <adf_gen4_ras.h>
 #include <adf_gen4_timer.h>
 #include <adf_gen4_tl.h>
+#include <adf_gen4_vf_mig.h>
 #include "adf_420xx_hw_data.h"
 #include "icp_qat_hw.h"
 
@@ -520,6 +521,7 @@ void adf_init_hw_data_420xx(struct adf_hw_device_data *hw_data, u32 dev_id)
 	adf_gen4_init_dc_ops(&hw_data->dc_ops);
 	adf_gen4_init_ras_ops(&hw_data->ras_ops);
 	adf_gen4_init_tl_data(&hw_data->tl_data);
+	adf_gen4_init_vf_mig_ops(&hw_data->vfmig_ops);
 	adf_init_rl_data(&hw_data->rl_data);
 }
 
drivers/crypto/intel/qat/qat_4xxx/adf_4xxx_hw_data.c +2 -0
index a04488ada49b..5dce7dfd06c4 100644
@@ -17,6 +17,7 @@
 #include "adf_gen4_ras.h"
 #include <adf_gen4_timer.h>
 #include <adf_gen4_tl.h>
+#include <adf_gen4_vf_mig.h>
 #include "adf_4xxx_hw_data.h"
 #include "icp_qat_hw.h"
 
@@ -504,6 +505,7 @@ void adf_init_hw_data_4xxx(struct adf_hw_device_data *hw_data, u32 dev_id)
 	adf_gen4_init_dc_ops(&hw_data->dc_ops);
 	adf_gen4_init_ras_ops(&hw_data->ras_ops);
 	adf_gen4_init_tl_data(&hw_data->tl_data);
+	adf_gen4_init_vf_mig_ops(&hw_data->vfmig_ops);
 	adf_init_rl_data(&hw_data->rl_data);
 }
 
drivers/crypto/intel/qat/qat_common/Makefile +2 -0
index 05cac8a9e463..061f7d04a0bb 100644
@@ -20,12 +20,14 @@ intel_qat-objs := adf_cfg.o \
 	adf_gen4_config.o \
 	adf_gen4_hw_csr_data.o \
 	adf_gen4_hw_data.o \
+	adf_gen4_vf_mig.o \
 	adf_gen4_pm.o \
 	adf_gen2_dc.o \
 	adf_gen4_dc.o \
 	adf_gen4_ras.o \
 	adf_gen4_timer.o \
 	adf_clock.o \
+	adf_mstate_mgr.o \
 	qat_crypto.o \
 	qat_compression.o \
 	qat_comp_algs.o \
drivers/crypto/intel/qat/qat_common/adf_accel_devices.h +6 -0
index 0fa52d20c131..4bb50e00c210 100644
@@ -397,10 +397,16 @@ struct adf_fw_loader_data {
 struct adf_accel_vf_info {
 	struct adf_accel_dev *accel_dev;
 	struct mutex pf2vf_lock; /* protect CSR access for PF2VF messages */
+	struct mutex pfvf_mig_lock; /* protects PFVF state for migration */
 	struct ratelimit_state vf2pf_ratelimit;
 	u32 vf_nr;
 	bool init;
 	u8 vf_compat_ver;
+	/*
+	 * Private area used for device migration.
+	 * Memory allocation and free is managed by migration driver.
+	 */
+	void *mig_priv;
 };
 
 struct adf_dc_data {
drivers/crypto/intel/qat/qat_common/adf_gen4_hw_data.h +10 -0
index 788fe9d345c0..fdff987902ed 100644
@@ -86,6 +86,7 @@
 #define ADF_RP_INT_SRC_SEL_F_RISE_MASK	BIT(2)
 #define ADF_RP_INT_SRC_SEL_F_FALL_MASK	GENMASK(2, 0)
 #define ADF_RP_INT_SRC_SEL_RANGE_WIDTH	4
+#define ADF_COALESCED_POLL_TIMEOUT_US	(1 * USEC_PER_SEC)
 #define ADF_COALESCED_POLL_DELAY_US	1000
 #define ADF_WQM_CSR_RPINTSOU(bank)	(0x200000 + ((bank) << 12))
 #define ADF_WQM_CSR_RP_IDX_RX		1
@@ -120,6 +121,15 @@
 /* PF2VM communication channel */
 #define ADF_GEN4_PF2VM_OFFSET(i)	(0x40B010 + (i) * 0x20)
 #define ADF_GEN4_VM2PF_OFFSET(i)	(0x40B014 + (i) * 0x20)
+#define ADF_GEN4_VINTMSKPF2VM_OFFSET(i)	(0x40B00C + (i) * 0x20)
+#define ADF_GEN4_VINTSOUPF2VM_OFFSET(i)	(0x40B008 + (i) * 0x20)
+#define ADF_GEN4_VINTMSK_OFFSET(i)	(0x40B004 + (i) * 0x20)
+#define ADF_GEN4_VINTSOU_OFFSET(i)	(0x40B000 + (i) * 0x20)
+
+struct adf_gen4_vfmig {
+	struct adf_mstate_mgr *mstate_mgr;
+	bool bank_stopped[ADF_GEN4_NUM_BANKS_PER_VF];
+};
 
 void adf_gen4_set_ssm_wdtimer(struct adf_accel_dev *accel_dev);
 
drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.c new file +834 -0

834 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_gen4_vf_mig.h new file +10 -0
new file mode 100644
index 000000000000..dbb1f26b35f1
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2024 Intel Corporation */
+#ifndef ADF_GEN4_VF_MIG_H_
+#define ADF_GEN4_VF_MIG_H_
+
+#include <linux/qat/qat_mig_dev.h>
+
+void adf_gen4_init_vf_mig_ops(struct qat_migdev_ops *vfmig_ops);
+
+#endif
drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.c new file +310 -0

310 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_mstate_mgr.h new file +87 -0

87 lines not on this page · raw message

drivers/crypto/intel/qat/qat_common/adf_sriov.c +6 -1
index f44025bb6f99..bc0966f3c7b2 100644
@@ -26,10 +26,12 @@ static void adf_iov_send_resp(struct work_struct *work)
 	u32 vf_nr = vf_info->vf_nr;
 	bool ret;
 
+	mutex_lock(&vf_info->pfvf_mig_lock);
 	ret = adf_recv_and_handle_vf2pf_msg(accel_dev, vf_nr);
 	if (ret)
 		/* re-enable interrupt on PF from this VF */
 		adf_enable_vf2pf_interrupts(accel_dev, 1 << vf_nr);
+	mutex_unlock(&vf_info->pfvf_mig_lock);
 
 	kfree(pf2vf_resp);
 }
@@ -63,6 +65,7 @@ static int adf_enable_sriov(struct adf_accel_dev *accel_dev)
 		vf_info->vf_compat_ver = 0;
 
 		mutex_init(&vf_info->pf2vf_lock);
+		mutex_init(&vf_info->pfvf_mig_lock);
 		ratelimit_state_init(&vf_info->vf2pf_ratelimit,
 				     ADF_VF2PF_RATELIMIT_INTERVAL,
 				     ADF_VF2PF_RATELIMIT_BURST);
@@ -112,8 +115,10 @@ void adf_disable_sriov(struct adf_accel_dev *accel_dev)
 	if (hw_data->configure_iov_threads)
 		hw_data->configure_iov_threads(accel_dev, false);
 
-	for (i = 0, vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++)
+	for (i = 0, vf = accel_dev->pf.vf_info; i < totalvfs; i++, vf++) {
 		mutex_destroy(&vf->pf2vf_lock);
+		mutex_destroy(&vf->pfvf_mig_lock);
+	}
 
 	kfree(accel_dev->pf.vf_info);
 	accel_dev->pf.vf_info = NULL;
Signature · 1 line
2.18.2
author
#12 lore raw

PATCH 10/10 vfio/qat: Add vfio_pci driver for Intel QAT VF devices

In reply to Xin Zeng

Add vfio pci driver for Intel QAT VF devices.

This driver uses vfio_pci_core to register to the VFIO subsystem. It acts as a vfio agent and interacts with the QAT PF driver to implement VF live migration.

---
 MAINTAINERS                         |   8 +
 drivers/vfio/pci/Kconfig            |   2 +
 drivers/vfio/pci/Makefile           |   2 +
 drivers/vfio/pci/intel/qat/Kconfig  |  13 +
 drivers/vfio/pci/intel/qat/Makefile |   4 +
 drivers/vfio/pci/intel/qat/main.c   | 572 ++++++++++++++++++++++++++++
 6 files changed, 601 insertions(+)
 create mode 100644 drivers/vfio/pci/intel/qat/Kconfig
 create mode 100644 drivers/vfio/pci/intel/qat/Makefile
 create mode 100644 drivers/vfio/pci/intel/qat/main.c
MAINTAINERS +8 -0
index 8d1052fa6a69..c1d3e4cb3892 100644
@@ -23095,6 +23095,14 @@ S:	Maintained
 F:	Documentation/networking/device_drivers/ethernet/amd/pds_vfio_pci.rst
 F:	drivers/vfio/pci/pds/
 
+VFIO QAT PCI DRIVER
+M:	Xin Zeng <[email protected]>
+M:	Giovanni Cabiddu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/intel/qat/
+
 VFIO PLATFORM DRIVER
 M:	Eric Auger <[email protected]>
 L:	[email protected]
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..329d25c53274 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/intel/qat/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..a87b6b43ce1c 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_QAT_VFIO_PCI) += intel/qat/
drivers/vfio/pci/intel/qat/Kconfig new file +13 -0
new file mode 100644
index 000000000000..71b28ac0bf6a
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config QAT_VFIO_PCI
+	tristate "VFIO support for QAT VF PCI devices"
+	select VFIO_PCI_CORE
+	depends on CRYPTO_DEV_QAT
+	depends on CRYPTO_DEV_QAT_4XXX
+	help
+	  This provides migration support for Intel(R) QAT Virtual Function
+	  using the VFIO framework.
+
+	  To compile this as a module, choose M here: the module
+	  will be called qat_vfio_pci. If you don't know what to do here,
+	  say N.
drivers/vfio/pci/intel/qat/Makefile new file +4 -0
new file mode 100644
index 000000000000..9289ae4c51bf
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_QAT_VFIO_PCI) += qat_vfio_pci.o
+qat_vfio_pci-y := main.o
+
drivers/vfio/pci/intel/qat/main.c new file +572 -0

572 lines not on this page · raw message

Signature · 1 line
2.18.2
#13 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 31 lines · Xin Zeng
drivers/vfio/pci/intel/qat/main.c new file +31 -0
+static int qat_vf_pci_init_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+	struct qat_migdev_ops *ops;
+	struct qat_mig_dev *mdev;
+	struct pci_dev *parent;
+	int ret, vf_id;
+
+	core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;
+	core_vdev->mig_ops = &qat_vf_pci_mig_ops;
+
+	ret = vfio_pci_core_init_dev(core_vdev);
+	if (ret)
+		return ret;
+
+	mutex_init(&qat_vdev->state_mutex);
+	spin_lock_init(&qat_vdev->reset_lock);
+
+	parent = qat_vdev->core_device.pdev->physfn;
+	vf_id = pci_iov_vf_id(qat_vdev->core_device.pdev);
+	if (!parent || vf_id < 0) {
+		ret = -ENODEV;
+		goto err_rel;
+	}
+
+	mdev = qat_vfmig_create(parent, vf_id);
+	if (IS_ERR(mdev)) {
+		ret = PTR_ERR(mdev);
+		goto err_rel;
+	}

drivers/vfio/pci/intel/qat/main.c line 449

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
+		dev_err(&parent->dev, "Incomplete device migration ops structure!");
+		goto err_destroy;
+	}

Why are there ops pointers here? I would expect this should just be direct function calls to the PF QAT driver.

Quoted 13 lines
drivers/vfio/pci/intel/qat/main.c new file +13 -0
+static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	if (!qat_vdev->core_device.vdev.mig_ops)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA deadlock
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup work
+	 * to the unlock flow of the other running context.

drivers/vfio/pci/intel/qat/main.c line 512

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+	 */
+	spin_lock(&qat_vdev->reset_lock);
+	qat_vdev->deferred_reset = true;
+	if (!mutex_trylock(&qat_vdev->state_mutex)) {
+		spin_unlock(&qat_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&qat_vdev->reset_lock);
+	qat_vf_state_mutex_unlock(qat_vdev);
+}

Do you really need this? I thought this ugly thing was going to be a uniquely mlx5 thing..

Jason

author
#14 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Thanks for your comments, Jason.

Quoted 9 lines · Jason Gunthorpe

drivers/vfio/pci/intel/qat/main.c

Quoted 8 lines
drivers/vfio/pci/intel/qat/main.c new file +8 -0
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
+		dev_err(&parent->dev, "Incomplete device migration ops

structure!");

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		goto err_destroy;
+	}

Why are there ops pointers here? I would expect this should just be direct function calls to the PF QAT driver.

I indeed had a version where the direct function calls are Implemented in QAT driver, while when I look at the functions, most of them only translate the interface to the ops pointer. That's why I put ops pointers directly into vfio variant driver.

Quoted 12 lines

drivers/vfio/pci/intel/qat/main.c

Quoted 11 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	if (!qat_vdev->core_device.vdev.mig_ops)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA

deadlock

Quoted 16 lines

drivers/vfio/pci/intel/qat/main.c line 512

Quoted 13 lines
drivers/vfio/pci/intel/qat/main.c new file +13 -0
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup work
+	 * to the unlock flow of the other running context.
+	 */
+	spin_lock(&qat_vdev->reset_lock);
+	qat_vdev->deferred_reset = true;
+	if (!mutex_trylock(&qat_vdev->state_mutex)) {
+		spin_unlock(&qat_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&qat_vdev->reset_lock);
+	qat_vf_state_mutex_unlock(qat_vdev);
+}

Do you really need this? I thought this ugly thing was going to be a uniquely mlx5 thing..

I think that's still required to make the migration state synchronized if the VF is reset by other VFIO emulation paths. Is it the case? BTW, this implementation is not only in mlx5 driver, but also in other Vfio pci variant drivers such as hisilicon acc driver and pds driver. Thanks,
Xin

#15 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 17 lines · Xin

Thanks for your comments, Jason.

Quoted 9 lines · Jason Gunthorpe

drivers/vfio/pci/intel/qat/main.c

Quoted 8 lines
drivers/vfio/pci/intel/qat/main.c new file +8 -0
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
+		dev_err(&parent->dev, "Incomplete device migration ops

structure!");

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		goto err_destroy;
+	}

Why are there ops pointers here? I would expect this should just be direct function calls to the PF QAT driver.

Quoted 4 lines

I indeed had a version where the direct function calls are Implemented in QAT driver, while when I look at the functions, most of them only translate the interface to the ops pointer. That's why I put ops pointers directly into vfio variant driver.

But why is there an ops indirection at all? Are there more than one ops?

Quoted 28 lines
Quoted 11 lines

drivers/vfio/pci/intel/qat/main.c

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	if (!qat_vdev->core_device.vdev.mig_ops)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA

deadlock

Quoted 16 lines

drivers/vfio/pci/intel/qat/main.c line 512

Quoted 13 lines
drivers/vfio/pci/intel/qat/main.c new file +13 -0
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup work
+	 * to the unlock flow of the other running context.
+	 */
+	spin_lock(&qat_vdev->reset_lock);
+	qat_vdev->deferred_reset = true;
+	if (!mutex_trylock(&qat_vdev->state_mutex)) {
+		spin_unlock(&qat_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&qat_vdev->reset_lock);
+	qat_vf_state_mutex_unlock(qat_vdev);
+}

Do you really need this? I thought this ugly thing was going to be a uniquely mlx5 thing..

Quoted 5 lines

I think that's still required to make the migration state synchronized if the VF is reset by other VFIO emulation paths. Is it the case? BTW, this implementation is not only in mlx5 driver, but also in other Vfio pci variant drivers such as hisilicon acc driver and pds driver.

It had to specifically do with the mm lock interaction that, I thought, was going to be unique to the mlx driver. Otherwise you could just directly hold the state_mutex here.

Yishai do you remember the exact trace for the mmlock entanglement?

Jason

reviewer
#16 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 60 lines · Jason Gunthorpe
Quoted 17 lines · Xin

Thanks for your comments, Jason.

Quoted 9 lines · Jason Gunthorpe

drivers/vfio/pci/intel/qat/main.c

Quoted 8 lines
drivers/vfio/pci/intel/qat/main.c new file +8 -0
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
+		dev_err(&parent->dev, "Incomplete device migration ops

structure!");

Quoted 5 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+		goto err_destroy;
+	}

Why are there ops pointers here? I would expect this should just be direct function calls to the PF QAT driver.

Quoted 4 lines

I indeed had a version where the direct function calls are Implemented in QAT driver, while when I look at the functions, most of them only translate the interface to the ops pointer. That's why I put ops pointers directly into vfio variant driver.

But why is there an ops indirection at all? Are there more than one ops?

Quoted 28 lines
Quoted 11 lines

drivers/vfio/pci/intel/qat/main.c

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	if (!qat_vdev->core_device.vdev.mig_ops)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA

deadlock

Quoted 16 lines

drivers/vfio/pci/intel/qat/main.c line 512

Quoted 13 lines
drivers/vfio/pci/intel/qat/main.c new file +13 -0
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup work
+	 * to the unlock flow of the other running context.
+	 */
+	spin_lock(&qat_vdev->reset_lock);
+	qat_vdev->deferred_reset = true;
+	if (!mutex_trylock(&qat_vdev->state_mutex)) {
+		spin_unlock(&qat_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&qat_vdev->reset_lock);
+	qat_vf_state_mutex_unlock(qat_vdev);
+}

Do you really need this? I thought this ugly thing was going to be a uniquely mlx5 thing..

Quoted 5 lines

I think that's still required to make the migration state synchronized if the VF is reset by other VFIO emulation paths. Is it the case? BTW, this implementation is not only in mlx5 driver, but also in other Vfio pci variant drivers such as hisilicon acc driver and pds driver.

Quoted 6 lines

It had to specifically do with the mm lock interaction that, I thought, was going to be unique to the mlx driver. Otherwise you could just directly hold the state_mutex here.

Yishai do you remember the exact trace for the mmlock entanglement?

I found in my in-box (from more than 2.5 years ago) the below [1] lockdep WARNING when the state_mutex was used directly.

Once we moved to the 'deferred_reset' mode it solved that.

[1]

[  +1.063822] ======================================================
[  +0.000732] WARNING: possible circular locking dependency detected
[  +0.000747] 5.15.0-rc3 #236 Not tainted
[  +0.000556] ------------------------------------------------------
[  +0.000714] qemu-system-x86/7731 is trying to acquire lock:
[  +0.000659] ffff888126c64b78 (&vdev->vma_lock){+.+.}-{3:3}, at: vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.001127]

but task is already holding lock:
[ +0.000803] ffff888105f4c5d8 (&mm->mmap_lock#2){++++}-{3:3}, at: vaddr_get_pfns+0x64/0x240 [vfio_iommu_type1] [ +0.001119]
which lock already depends on the new lock.

[ +0.001086]
the existing dependency chain (in reverse order) is:
[ +0.000910]

              -> #3 (&mm->mmap_lock#2){++++}-{3:3}:
[  +0.000844]        __might_fault+0x56/0x80
[  +0.000572]        _copy_to_user+0x1e/0x80
[  +0.000556]        mlx5vf_pci_mig_rw.cold+0xa1/0x24f [mlx5_vfio_pci]
[  +0.000732]        vfs_read+0xa8/0x1c0
[  +0.000547]        __x64_sys_pread64+0x8c/0xc0
[  +0.000580]        do_syscall_64+0x3d/0x90
[  +0.000566]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000682]
              -> #2 (&mvdev->state_mutex){+.+.}-{3:3}:
[  +0.000899]        __mutex_lock+0x80/0x9d0
[  +0.000566]        mlx5vf_reset_done+0x2c/0x40 [mlx5_vfio_pci]
[  +0.000697]        vfio_pci_core_ioctl+0x585/0x1020 [vfio_pci_core]
[  +0.000721]        __x64_sys_ioctl+0x436/0x9a0
[  +0.000588]        do_syscall_64+0x3d/0x90
[  +0.000584]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000674]
              -> #1 (&vdev->memory_lock){+.+.}-{3:3}:
[  +0.000843]        down_write+0x38/0x70
[  +0.000544]        vfio_pci_zap_and_down_write_memory_lock+0x1c/0x30 [vfio_pci_core]
[  +0.003705]        vfio_basic_config_write+0x1e7/0x280 [vfio_pci_core]
[  +0.000744]        vfio_pci_config_rw+0x1b7/0x3af [vfio_pci_core]
[  +0.000716]        vfs_write+0xe6/0x390
[  +0.000539]        __x64_sys_pwrite64+0x8c/0xc0
[  +0.000603]        do_syscall_64+0x3d/0x90
[  +0.000572]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000661]
              -> #0 (&vdev->vma_lock){+.+.}-{3:3}:
[  +0.000828]        __lock_acquire+0x1244/0x2280
[  +0.000596]        lock_acquire+0xc2/0x2c0
[  +0.000556]        __mutex_lock+0x80/0x9d0
[  +0.000580]        vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000709]        __do_fault+0x32/0xa0
[  +0.000556]        __handle_mm_fault+0xbe8/0x1450
[  +0.000606]        handle_mm_fault+0x6c/0x140
[  +0.000624]        fixup_user_fault+0x6b/0x100
[  +0.000600]        vaddr_get_pfns+0x108/0x240 [vfio_iommu_type1]
[  +0.000726]        vfio_pin_pages_remote+0x326/0x460 [vfio_iommu_type1]
[  +0.000736]        vfio_iommu_type1_ioctl+0x43b/0x15a0 [vfio_iommu_type1]
[  +0.000752]        __x64_sys_ioctl+0x436/0x9a0
[  +0.000588]        do_syscall_64+0x3d/0x90
[  +0.000571]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000677]

other info that might help us debug this:

[ +0.001073] Chain exists of:

                &vdev->vma_lock --> &mvdev->state_mutex --> 

&mm->mmap_lock#2

[ +0.001285] Possible unsafe locking scenario:

[  +0.000808]        CPU0                    CPU1
[  +0.000599]        ----                    ----
[  +0.000593]   lock(&mm->mmap_lock#2);
[  +0.000530]                                lock(&mvdev->state_mutex);
[  +0.000725]                                lock(&mm->mmap_lock#2);
[  +0.000712]   lock(&vdev->vma_lock);
[  +0.000532]
               *** DEADLOCK ***
[  +0.000922] 2 locks held by qemu-system-x86/7731:
[  +0.000597]  #0: ffff88810c9bec88 (&iommu->lock#2){+.+.}-{3:3}, at: vfio_iommu_type1_ioctl+0x189/0x15a0 [vfio_iommu_type1]
[  +0.001177]  #1: ffff888105f4c5d8 (&mm->mmap_lock#2){++++}-{3:3}, at: vaddr_get_pfns+0x64/0x240 [vfio_iommu_type1]
[  +0.001153]
              stack backtrace:
[  +0.000689] CPU: 1 PID: 7731 Comm: qemu-system-x86 Not tainted 5.15.0-rc3 #236
[  +0.000932] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
[  +0.001192] Call Trace:
[  +0.000454]  dump_stack_lvl+0x45/0x59
[  +0.000527]  check_noncircular+0xf2/0x110
[  +0.000557]  __lock_acquire+0x1244/0x2280
[  +0.002462]  lock_acquire+0xc2/0x2c0
[  +0.000534]  ? vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000684]  ? lock_is_held_type+0x98/0x110
[  +0.000565]  __mutex_lock+0x80/0x9d0
[  +0.000542]  ? vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000676]  ? vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000681]  ? mark_held_locks+0x49/0x70
[  +0.000551]  ? lock_is_held_type+0x98/0x110
[  +0.000575]  vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000679]  __do_fault+0x32/0xa0
[  +0.000700]  __handle_mm_fault+0xbe8/0x1450
[  +0.000571]  handle_mm_fault+0x6c/0x140
[  +0.000564]  fixup_user_fault+0x6b/0x100
[  +0.000556]  vaddr_get_pfns+0x108/0x240 [vfio_iommu_type1]
[  +0.000666]  ? lock_is_held_type+0x98/0x110
[  +0.000572]  vfio_pin_pages_remote+0x326/0x460 [vfio_iommu_type1]
[  +0.000710]  vfio_iommu_type1_ioctl+0x43b/0x15a0 [vfio_iommu_type1]
[  +0.001050]  ? find_held_lock+0x2b/0x80
[  +0.000561]  ? lock_release+0xc2/0x2a0
[  +0.000537]  ? __fget_files+0xdc/0x1d0
[  +0.000541]  __x64_sys_ioctl+0x436/0x9a0
[  +0.000556]  ? lockdep_hardirqs_on_prepare+0xd4/0x170
[  +0.000641]  do_syscall_64+0x3d/0x90
[  +0.000529]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000670] RIP: 0033:0x7f54255bb45b
[  +0.000541] Code: 0f 1e fa 48 8b 05 2d aa 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fd a9 2c 00 f7 d8 64 89 01 48
[  +0.001843] RSP: 002b:00007f5415ca3d38 EFLAGS: 00000206 ORIG_RAX: 0000000000000010
[  +0.000929] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f54255bb45b
[  +0.000781] RDX: 00007f5415ca3d70 RSI: 0000000000003b71 RDI: 0000000000000012
[  +0.000775] RBP: 00007f5415ca3da0 R08: 0000000000000000 R09: 0000000000000000
[  +0.000777] R10: 00000000fe002000 R11: 0000000000000206 R12: 0000000000000004
[  +0.000775] R13: 0000000000000000 R14: 00000000fe000000 R15: 00007f5415ca47c0

Yishai

author
#17 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 116 lines · Yishai Hadas
Quoted 34 lines
Quoted 28 lines
Quoted 11 lines

drivers/vfio/pci/intel/qat/main.c

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	if (!qat_vdev->core_device.vdev.mig_ops)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA

deadlock

Quoted 16 lines

drivers/vfio/pci/intel/qat/main.c line 512

Quoted 13 lines
drivers/vfio/pci/intel/qat/main.c new file +13 -0
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup work
+	 * to the unlock flow of the other running context.
+	 */
+	spin_lock(&qat_vdev->reset_lock);
+	qat_vdev->deferred_reset = true;
+	if (!mutex_trylock(&qat_vdev->state_mutex)) {
+		spin_unlock(&qat_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&qat_vdev->reset_lock);
+	qat_vf_state_mutex_unlock(qat_vdev);
+}

Do you really need this? I thought this ugly thing was going to be a uniquely mlx5 thing..

Quoted 5 lines

I think that's still required to make the migration state synchronized if the VF is reset by other VFIO emulation paths. Is it the case? BTW, this implementation is not only in mlx5 driver, but also in other Vfio pci variant drivers such as hisilicon acc driver and pds driver.

Quoted 6 lines

It had to specifically do with the mm lock interaction that, I thought, was going to be unique to the mlx driver. Otherwise you could just directly hold the state_mutex here.

Yishai do you remember the exact trace for the mmlock entanglement?

I found in my in-box (from more than 2.5 years ago) the below [1] lockdep WARNING when the state_mutex was used directly.

Once we moved to the 'deferred_reset' mode it solved that.

[1]

[  +1.063822] ======================================================
[  +0.000732] WARNING: possible circular locking dependency detected
[  +0.000747] 5.15.0-rc3 #236 Not tainted
[  +0.000556] ------------------------------------------------------
[  +0.000714] qemu-system-x86/7731 is trying to acquire lock:
[  +0.000659] ffff888126c64b78 (&vdev->vma_lock){+.+.}-{3:3}, at:

vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[ +0.001127]
but task is already holding lock:
[ +0.000803] ffff888105f4c5d8 (&mm->mmap_lock#2){++++}-{3:3}, at:
vaddr_get_pfns+0x64/0x240 [vfio_iommu_type1]
[ +0.001119]
which lock already depends on the new lock.

[ +0.001086]
the existing dependency chain (in reverse order) is:
[ +0.000910]

               -> #3 (&mm->mmap_lock#2){++++}-{3:3}:
[  +0.000844]        __might_fault+0x56/0x80
[  +0.000572]        _copy_to_user+0x1e/0x80
[  +0.000556]        mlx5vf_pci_mig_rw.cold+0xa1/0x24f [mlx5_vfio_pci]
[  +0.000732]        vfs_read+0xa8/0x1c0
[  +0.000547]        __x64_sys_pread64+0x8c/0xc0
[  +0.000580]        do_syscall_64+0x3d/0x90
[  +0.000566]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000682]
               -> #2 (&mvdev->state_mutex){+.+.}-{3:3}:
[  +0.000899]        __mutex_lock+0x80/0x9d0
[  +0.000566]        mlx5vf_reset_done+0x2c/0x40 [mlx5_vfio_pci]
[  +0.000697]        vfio_pci_core_ioctl+0x585/0x1020 [vfio_pci_core]
[  +0.000721]        __x64_sys_ioctl+0x436/0x9a0
[  +0.000588]        do_syscall_64+0x3d/0x90
[  +0.000584]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000674]
               -> #1 (&vdev->memory_lock){+.+.}-{3:3}:
[  +0.000843]        down_write+0x38/0x70
[  +0.000544]        vfio_pci_zap_and_down_write_memory_lock+0x1c/0x30
[vfio_pci_core]
[  +0.003705]        vfio_basic_config_write+0x1e7/0x280 [vfio_pci_core]
[  +0.000744]        vfio_pci_config_rw+0x1b7/0x3af [vfio_pci_core]
[  +0.000716]        vfs_write+0xe6/0x390
[  +0.000539]        __x64_sys_pwrite64+0x8c/0xc0
[  +0.000603]        do_syscall_64+0x3d/0x90
[  +0.000572]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000661]
               -> #0 (&vdev->vma_lock){+.+.}-{3:3}:
[  +0.000828]        __lock_acquire+0x1244/0x2280
[  +0.000596]        lock_acquire+0xc2/0x2c0
[  +0.000556]        __mutex_lock+0x80/0x9d0
[  +0.000580]        vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000709]        __do_fault+0x32/0xa0
[  +0.000556]        __handle_mm_fault+0xbe8/0x1450
[  +0.000606]        handle_mm_fault+0x6c/0x140
[  +0.000624]        fixup_user_fault+0x6b/0x100
[  +0.000600]        vaddr_get_pfns+0x108/0x240 [vfio_iommu_type1]
[  +0.000726]        vfio_pin_pages_remote+0x326/0x460 [vfio_iommu_type1]
[  +0.000736]        vfio_iommu_type1_ioctl+0x43b/0x15a0 [vfio_iommu_type1]
[  +0.000752]        __x64_sys_ioctl+0x436/0x9a0
[  +0.000588]        do_syscall_64+0x3d/0x90
[  +0.000571]        entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000677]

other info that might help us debug this:

[ +0.001073] Chain exists of:

                 &vdev->vma_lock --> &mvdev->state_mutex -->

&mm->mmap_lock#2

[ +0.001285] Possible unsafe locking scenario:

Quoted 8 lines
[  +0.000808]        CPU0                    CPU1
[  +0.000599]        ----                    ----
[  +0.000593]   lock(&mm->mmap_lock#2);
[  +0.000530]                                lock(&mvdev->state_mutex);
[  +0.000725]                                lock(&mm->mmap_lock#2);
[  +0.000712]   lock(&vdev->vma_lock);
[  +0.000532]
                *** DEADLOCK ***

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?
Meanwhile, it seems the trace above is triggered with migration protocol v1, the context of CPU1 listed also seems protocol v1 specific. Is it the case?
Thanks,
Xin

Quoted 58 lines

[ +0.000922] 2 locks held by qemu-system-x86/7731:
[ +0.000597] #0: ffff88810c9bec88 (&iommu->lock#2){+.+.}-{3:3}, at:
vfio_iommu_type1_ioctl+0x189/0x15a0 [vfio_iommu_type1]
[ +0.001177] #1: ffff888105f4c5d8 (&mm->mmap_lock#2){++++}-{3:3}, at:
vaddr_get_pfns+0x64/0x240 [vfio_iommu_type1]
[ +0.001153]

               stack backtrace:

[ +0.000689] CPU: 1 PID: 7731 Comm: qemu-system-x86 Not tainted 5.15.0-rc3 #236
[ +0.000932] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014

[  +0.001192] Call Trace:
[  +0.000454]  dump_stack_lvl+0x45/0x59
[  +0.000527]  check_noncircular+0xf2/0x110
[  +0.000557]  __lock_acquire+0x1244/0x2280
[  +0.002462]  lock_acquire+0xc2/0x2c0
[  +0.000534]  ? vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000684]  ? lock_is_held_type+0x98/0x110
[  +0.000565]  __mutex_lock+0x80/0x9d0
[  +0.000542]  ? vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000676]  ? vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000681]  ? mark_held_locks+0x49/0x70
[  +0.000551]  ? lock_is_held_type+0x98/0x110
[  +0.000575]  vfio_pci_mmap_fault+0x35/0x140 [vfio_pci_core]
[  +0.000679]  __do_fault+0x32/0xa0
[  +0.000700]  __handle_mm_fault+0xbe8/0x1450
[  +0.000571]  handle_mm_fault+0x6c/0x140
[  +0.000564]  fixup_user_fault+0x6b/0x100
[  +0.000556]  vaddr_get_pfns+0x108/0x240 [vfio_iommu_type1]
[  +0.000666]  ? lock_is_held_type+0x98/0x110
[  +0.000572]  vfio_pin_pages_remote+0x326/0x460 [vfio_iommu_type1]
[  +0.000710]  vfio_iommu_type1_ioctl+0x43b/0x15a0 [vfio_iommu_type1]
[  +0.001050]  ? find_held_lock+0x2b/0x80
[  +0.000561]  ? lock_release+0xc2/0x2a0
[  +0.000537]  ? __fget_files+0xdc/0x1d0
[  +0.000541]  __x64_sys_ioctl+0x436/0x9a0
[  +0.000556]  ? lockdep_hardirqs_on_prepare+0xd4/0x170
[  +0.000641]  do_syscall_64+0x3d/0x90
[  +0.000529]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  +0.000670] RIP: 0033:0x7f54255bb45b
[  +0.000541] Code: 0f 1e fa 48 8b 05 2d aa 2c 00 64 c7 00 26 00 00 00

48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fd a9 2c 00 f7 d8 64 89 01 48 [ +0.001843] RSP: 002b:00007f5415ca3d38 EFLAGS: 00000206 ORIG_RAX:
0000000000000010
[ +0.000929] RAX: ffffffffffffffda RBX: 0000000000000000 RCX:
00007f54255bb45b
[ +0.000781] RDX: 00007f5415ca3d70 RSI: 0000000000003b71 RDI:
0000000000000012
[ +0.000775] RBP: 00007f5415ca3da0 R08: 0000000000000000 R09:
0000000000000000
[ +0.000777] R10: 00000000fe002000 R11: 0000000000000206 R12:
0000000000000004
[ +0.000775] R13: 0000000000000000 R14: 00000000fe000000 R15:
00007f5415ca47c0

Yishai

#18 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 4 lines · Xin

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

copy_to_user() would nest the mm_lock under the state mutex which is a locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

Jason

author
#19 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 15 lines · Jason Gunthorpe

To: Zeng, Xin <[email protected]>
Cc: Yishai Hadas <[email protected]>; [email protected]; [email protected]; [email protected]; Tian, Kevin <[email protected]>; [email protected]; [email protected]; qat-linux <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 4 lines · Xin

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

Quoted 8 lines

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

copy_to_user() would nest the mm_lock under the state mutex which is a locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

In protocol v2, we still have the scenario in precopy_ioctl where copy_to_user is called under state_mutex.

Thanks,
Xin

#20 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 19 lines · Xin
Quoted 18 lines · Jason Gunthorpe

To: Zeng, Xin <[email protected]>
Cc: Yishai Hadas <[email protected]>; [email protected]; [email protected]; [email protected]; Tian, Kevin <[email protected]>; [email protected]; [email protected]; qat-linux <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 4 lines · Xin

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

Quoted 8 lines
Quoted 5 lines

copy_to_user() would nest the mm_lock under the state mutex which is a locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

In protocol v2, we still have the scenario in precopy_ioctl where copy_to_user is called under state_mutex.

Why? Does mlx5 do that? It looked Ok to me:

        mlx5vf_state_mutex_unlock(mvdev);
        if (copy_to_user((void __user *)arg, &info, minsz))
                return -EFAULT;

Jason

author
#21 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 30 lines · Jason Gunthorpe
Quoted 4 lines · Xin
Quoted 3 lines · Jason Gunthorpe

To: Zeng, Xin <[email protected]>
Cc: Yishai Hadas <[email protected]>; [email protected]; [email protected]; [email protected];

Tian,

Quoted 4 lines
Quoted 4 lines

Kevin <[email protected]>; [email protected]; [email protected]; qat-linux <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF

devices

Quoted 8 lines
Quoted 8 lines · Xin
Quoted 5 lines

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

Quoted 4 lines
Quoted 4 lines

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

copy_to_user() would nest the mm_lock under the state mutex which is

a

Quoted 6 lines
Quoted 4 lines

locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

In protocol v2, we still have the scenario in precopy_ioctl where

Quoted 8 lines

copy_to_user is

Quoted 1 line

called under state_mutex.

Why? Does mlx5 do that? It looked Ok to me:

        mlx5vf_state_mutex_unlock(mvdev);
        if (copy_to_user((void __user *)arg, &info, minsz))
                return -EFAULT;

Indeed, thanks, Jason. BTW, is there any reason why was "deferred_reset" mode still implemented in mlx5 driver given this deadlock condition has been avoided with migration protocol v2 implementation.
Anyway, I'll use state_mutex directly instead of the "deferred_reset" mode in qat variant driver and update it in next version soon, please help to review. Thanks,
Xin

#22 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 36 lines · Xin
Quoted 35 lines · Jason Gunthorpe
Quoted 4 lines · Xin
Quoted 3 lines · Jason Gunthorpe

To: Zeng, Xin <[email protected]>
Cc: Yishai Hadas <[email protected]>; [email protected]; [email protected]; [email protected];

Tian,

Quoted 4 lines
Quoted 4 lines

Kevin <[email protected]>; [email protected]; [email protected]; qat-linux <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF

devices

Quoted 8 lines
Quoted 8 lines · Xin
Quoted 5 lines

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

Quoted 4 lines
Quoted 4 lines

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

copy_to_user() would nest the mm_lock under the state mutex which is

a

Quoted 6 lines
Quoted 4 lines

locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

In protocol v2, we still have the scenario in precopy_ioctl where

copy_to_user is

Quoted 1 line

called under state_mutex.

Why? Does mlx5 do that? It looked Ok to me:

Quoted 7 lines
Quoted 3 lines
        mlx5vf_state_mutex_unlock(mvdev);
        if (copy_to_user((void __user *)arg, &info, minsz))
                return -EFAULT;

Indeed, thanks, Jason. BTW, is there any reason why was "deferred_reset" mode still implemented in mlx5 driver given this deadlock condition has been avoided with migration protocol v2 implementation.

I do not remember. Yishai?

Jason

reviewer
#23 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 41 lines · Jason Gunthorpe
Quoted 40 lines · Xin
Quoted 38 lines · Jason Gunthorpe
Quoted 4 lines · Xin
Quoted 3 lines · Jason Gunthorpe

To: Zeng, Xin <[email protected]>
Cc: Yishai Hadas <[email protected]>; [email protected]; [email protected]; [email protected];

Tian,

Quoted 4 lines
Quoted 4 lines

Kevin <[email protected]>; [email protected]; [email protected]; qat-linux <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF

devices

Quoted 8 lines
Quoted 8 lines · Xin
Quoted 5 lines

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

Quoted 4 lines
Quoted 4 lines

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

copy_to_user() would nest the mm_lock under the state mutex which is

a

Quoted 6 lines
Quoted 4 lines

locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

In protocol v2, we still have the scenario in precopy_ioctl where

copy_to_user is

Quoted 1 line

called under state_mutex.

Why? Does mlx5 do that? It looked Ok to me:

         mlx5vf_state_mutex_unlock(mvdev);
         if (copy_to_user((void __user *)arg, &info, minsz))
                 return -EFAULT;
Quoted 6 lines
Quoted 3 lines

Indeed, thanks, Jason. BTW, is there any reason why was "deferred_reset" mode still implemented in mlx5 driver given this deadlock condition has been avoided with migration protocol v2 implementation.

I do not remember. Yishai?

Long time passed.., I also don't fully remember whether this was the only potential problem here, maybe Yes.

My plan is to prepare a cleanup patch for mlx5 and put it into our regression for a while, if all will be good, I may send it for the next kernel cycle.

By the way, there are other drivers around (i.e. hisi and mtty) that still run copy_to_user() under the state mutex and might hit the problem without the 'deferred_rest', see here[1].

If we'll reach to the conclusion that the only reason for that mechanism was the copy_to_user() under the state_mutex, those drivers can change their code easily and also send a patch to cleanup the 'deferred_reset'.

[1] https://elixir.bootlin.com/linux/v6.8-rc5/source/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c#L808
[2] https://elixir.bootlin.com/linux/v6.8-rc5/source/samples/vfio-mdev/mtty.c#L878

Yishai

reviewer
#24 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 53 lines · Yishai Hadas
Quoted 41 lines · Jason Gunthorpe
Quoted 40 lines · Xin
Quoted 38 lines · Jason Gunthorpe
Quoted 4 lines · Xin
Quoted 3 lines · Jason Gunthorpe

To: Zeng, Xin <[email protected]>
Cc: Yishai Hadas <[email protected]>; [email protected]; [email protected]; [email protected];

Tian,

Quoted 4 lines
Quoted 4 lines

Kevin <[email protected]>; [email protected]; [email protected]; qat-linux <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF

devices

Quoted 8 lines
Quoted 8 lines · Xin
Quoted 5 lines

Thanks for this information, but this flow is not clear to me why it cause deadlock. From this flow, CPU0 is not waiting for any resource held by CPU1, so after CPU0 releases mmap_lock, CPU1 can continue to run. Am I missing something?

Quoted 4 lines
Quoted 4 lines

At some point it was calling copy_to_user() under the state mutex. These days it doesn't.

copy_to_user() would nest the mm_lock under the state mutex which is

a

Quoted 6 lines
Quoted 4 lines

locking inversion.

So I wonder if we still have this problem now that the copy_to_user() is not under the mutex?

In protocol v2, we still have the scenario in precopy_ioctl where

copy_to_user is

Quoted 1 line

called under state_mutex.

Why? Does mlx5 do that? It looked Ok to me:

         mlx5vf_state_mutex_unlock(mvdev);
         if (copy_to_user((void __user *)arg, &info, minsz)) return -EFAULT;

Quoted 6 lines
Quoted 3 lines

Indeed, thanks, Jason. BTW, is there any reason why was "deferred_reset" mode still implemented in mlx5 driver given this deadlock condition has been avoided with migration protocol v2 implementation.

I do not remember. Yishai?

Long time passed.., I also don't fully remember whether this was the only potential problem here, maybe Yes.

My plan is to prepare a cleanup patch for mlx5 and put it into our regression for a while, if all will be good, I may send it for the next kernel cycle.

Quoted 9 lines

By the way, there are other drivers around (i.e. hisi and mtty) that still run copy_to_user() under the state mutex and might hit the problem without the 'deferred_rest', see here[1].

If we'll reach to the conclusion that the only reason for that mechanism was the copy_to_user() under the state_mutex, those drivers can change their code easily and also send a patch to cleanup the 'deferred_reset'.

[1] https://elixir.bootlin.com/linux/v6.8-rc5/source/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c#L808
[2] https://elixir.bootlin.com/linux/v6.8-rc5/source/samples/vfio-mdev/mtty.c#L878

Yishai

From an extra code review around, it looks as we still need the 'deferred_reset' flow in mlx5.

For example, we call copy_from_user() as part of mlx5vf_resume_read_header() which is called by mlx5vf_resume_write() under the state mutex.

So, no change is expected in mlx5.

Yishai

#25 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

In reply to Xin Zeng
Quoted 68 lines · On Thu, 1 Feb 2024 23:33:37 +0800 Xin Zeng <[email protected]> wrote:

Add vfio pci driver for Intel QAT VF devices.

This driver uses vfio_pci_core to register to the VFIO subsystem. It acts as a vfio agent and interacts with the QAT PF driver to implement VF live migration.

---
 MAINTAINERS                         |   8 +
 drivers/vfio/pci/Kconfig            |   2 +
 drivers/vfio/pci/Makefile           |   2 +
 drivers/vfio/pci/intel/qat/Kconfig  |  13 +
 drivers/vfio/pci/intel/qat/Makefile |   4 +
 drivers/vfio/pci/intel/qat/main.c   | 572 ++++++++++++++++++++++++++++
 6 files changed, 601 insertions(+)
 create mode 100644 drivers/vfio/pci/intel/qat/Kconfig
 create mode 100644 drivers/vfio/pci/intel/qat/Makefile
 create mode 100644 drivers/vfio/pci/intel/qat/main.c
MAINTAINERS +8 -0
index 8d1052fa6a69..c1d3e4cb3892 100644
@@ -23095,6 +23095,14 @@ S:	Maintained
 F:	Documentation/networking/device_drivers/ethernet/amd/pds_vfio_pci.rst
 F:	drivers/vfio/pci/pds/
 
+VFIO QAT PCI DRIVER
+M:	Xin Zeng <[email protected]>
+M:	Giovanni Cabiddu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/intel/qat/
+
 VFIO PLATFORM DRIVER
 M:	Eric Auger <[email protected]>
 L:	[email protected]
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..329d25c53274 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/intel/qat/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..a87b6b43ce1c 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_QAT_VFIO_PCI) += intel/qat/
drivers/vfio/pci/intel/qat/Kconfig new file +0 -0
new file mode 100644
index 000000000000..71b28ac0bf6a

drivers/vfio/pci/intel/qat/Kconfig line 6

Quoted 6 lines
drivers/vfio/pci/intel/qat/Kconfig new file +6 -0
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config QAT_VFIO_PCI
+	tristate "VFIO support for QAT VF PCI devices"
+	select VFIO_PCI_CORE
+	depends on CRYPTO_DEV_QAT
+	depends on CRYPTO_DEV_QAT_4XXX

CRYPTO_DEV_QAT_4XXX selects CRYPTO_DEV_QAT, is it necessary to depend on CRYPTO_DEV_QAT here?

Quoted 13 lines
drivers/vfio/pci/intel/qat/Kconfig new file +7 -0
+	help
+	  This provides migration support for Intel(R) QAT Virtual Function
+	  using the VFIO framework.
+
+	  To compile this as a module, choose M here: the module
+	  will be called qat_vfio_pci. If you don't know what to do here,
+	  say N.
drivers/vfio/pci/intel/qat/Makefile new file +0 -0
new file mode 100644
index 000000000000..9289ae4c51bf

drivers/vfio/pci/intel/qat/Makefile line 4

Quoted 4 lines
drivers/vfio/pci/intel/qat/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_QAT_VFIO_PCI) += qat_vfio_pci.o
+qat_vfio_pci-y := main.o
+

Blank line at end of file.

Quoted 6 lines
drivers/vfio/pci/intel/qat/main.c new file +0 -0
new file mode 100644
index 000000000000..85d0ed701397
@@ -0,0 +1,572 @@

...

drivers/vfio/pci/intel/qat/main.c line 418

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+static int qat_vf_pci_init_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+	struct qat_migdev_ops *ops;
+	struct qat_mig_dev *mdev;
+	struct pci_dev *parent;
+	int ret, vf_id;
+
+	core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY | VFIO_MIGRATION_P2P;

AFAICT it's always good practice to support VFIO_MIGRATION_PRE_COPY, even if only to leave yourself an option to mark an incompatible ABI change in the data stream that can be checked before the stop-copy phase of migration. See for example the mtty sample driver. Thanks,

Alex

Quoted 45 lines
drivers/vfio/pci/intel/qat/main.c new file +45 -0
+	core_vdev->mig_ops = &qat_vf_pci_mig_ops;
+
+	ret = vfio_pci_core_init_dev(core_vdev);
+	if (ret)
+		return ret;
+
+	mutex_init(&qat_vdev->state_mutex);
+	spin_lock_init(&qat_vdev->reset_lock);
+
+	parent = qat_vdev->core_device.pdev->physfn;
+	vf_id = pci_iov_vf_id(qat_vdev->core_device.pdev);
+	if (!parent || vf_id < 0) {
+		ret = -ENODEV;
+		goto err_rel;
+	}
+
+	mdev = qat_vfmig_create(parent, vf_id);
+	if (IS_ERR(mdev)) {
+		ret = PTR_ERR(mdev);
+		goto err_rel;
+	}
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
+		dev_err(&parent->dev, "Incomplete device migration ops structure!");
+		goto err_destroy;
+	}
+	ret = ops->init(mdev);
+	if (ret)
+		goto err_destroy;
+
+	qat_vdev->mdev = mdev;
+
+	return 0;
+
+err_destroy:
+	qat_vfmig_destroy(mdev);
+err_rel:
+	vfio_pci_core_release_dev(core_vdev);
+	return ret;
+}
author
#26 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Thanks for your comments, Alex.

Quoted 16 lines · Alex Williamson

Review

Quoted 1 line
(quoted diff) +0 -0

b/drivers/vfio/pci/intel/qat/Kconfig

drivers/vfio/pci/intel/qat/Kconfig line 6

Quoted 11 lines

new file mode 100644
index 000000000000..71b28ac0bf6a

drivers/vfio/pci/intel/qat/Kconfig new file +6 -0
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config QAT_VFIO_PCI
+	tristate "VFIO support for QAT VF PCI devices"
+	select VFIO_PCI_CORE
+	depends on CRYPTO_DEV_QAT
+	depends on CRYPTO_DEV_QAT_4XXX

CRYPTO_DEV_QAT_4XXX selects CRYPTO_DEV_QAT, is it necessary to depend on CRYPTO_DEV_QAT here?

You are right, we don't need it, will update it in next version.

Quoted 10 lines

Review

Quoted 9 lines
drivers/vfio/pci/intel/qat/Kconfig new file +7 -0
+	help
+	  This provides migration support for Intel(R) QAT Virtual Function
+	  using the VFIO framework.
+
+	  To compile this as a module, choose M here: the module
+	  will be called qat_vfio_pci. If you don't know what to do here,
+	  say N.
(quoted diff) +0 -0

b/drivers/vfio/pci/intel/qat/Makefile

Quoted 11 lines

drivers/vfio/pci/intel/qat/Makefile line 4

Quoted 9 lines

new file mode 100644
index 000000000000..9289ae4c51bf

drivers/vfio/pci/intel/qat/Makefile new file +4 -0
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_QAT_VFIO_PCI) += qat_vfio_pci.o
+qat_vfio_pci-y := main.o
+

Blank line at end of file.

Good catch, will update it in next version.

Quoted 21 lines

Review

Quoted 2 lines
(quoted diff) +0 -0

b/drivers/vfio/pci/intel/qat/main.c

Quoted 5 lines

new file mode 100644
index 000000000000..85d0ed701397

drivers/vfio/pci/intel/qat/main.c new file +0 -0
@@ -0,0 +1,572 @@

...

drivers/vfio/pci/intel/qat/main.c

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+static int qat_vf_pci_init_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+	struct qat_migdev_ops *ops;
+	struct qat_mig_dev *mdev;
+	struct pci_dev *parent;
+	int ret, vf_id;
+
+	core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY |

VFIO_MIGRATION_P2P;

Quoted 5 lines

AFAICT it's always good practice to support VFIO_MIGRATION_PRE_COPY, even if only to leave yourself an option to mark an incompatible ABI change in the data stream that can be checked before the stop-copy phase of migration. See for example the mtty sample driver. Thanks, Alex

If the good practice is to check migration stream compatibility, then we do incorporate additional information as part of device state for ABI compatibility testing such as magic, version and device capability in stop copy phase. Can we ignore the optional VFIO_MIGRATION_PRE_COPY support? Thanks,
Xin

#27 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

In reply to Xin Zeng
Quoted 283 lines

-----Original Message-----
From: Xin Zeng <[email protected]>
Sent: Thursday, February 1, 2024 3:34 PM
To: [email protected]; [email protected]; [email protected]; [email protected]; Shameerali Kolothum Thodi <[email protected]>; [email protected]
Cc: [email protected]; [email protected]; qat- [email protected]; Xin Zeng <[email protected]>; Yahui Cao <[email protected]>
Subject: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Add vfio pci driver for Intel QAT VF devices.

This driver uses vfio_pci_core to register to the VFIO subsystem. It acts as a vfio agent and interacts with the QAT PF driver to implement VF live migration.

---
 MAINTAINERS                         |   8 +
 drivers/vfio/pci/Kconfig            |   2 +
 drivers/vfio/pci/Makefile           |   2 +
 drivers/vfio/pci/intel/qat/Kconfig  |  13 +
 drivers/vfio/pci/intel/qat/Makefile |   4 +
 drivers/vfio/pci/intel/qat/main.c   | 572 ++++++++++++++++++++++++++++
 6 files changed, 601 insertions(+)
 create mode 100644 drivers/vfio/pci/intel/qat/Kconfig
 create mode 100644 drivers/vfio/pci/intel/qat/Makefile
 create mode 100644 drivers/vfio/pci/intel/qat/main.c
MAINTAINERS +0 -0
index 8d1052fa6a69..c1d3e4cb3892 100644
@@ -23095,6 +23095,14 @@ S:	Maintained
 F:
 	Documentation/networking/device_drivers/ethernet/amd/pds_vfio_

pci.rst
F: drivers/vfio/pci/pds/

MAINTAINERS +8 -0
+VFIO QAT PCI DRIVER
+M:	Xin Zeng <[email protected]>
+M:	Giovanni Cabiddu <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/intel/qat/
+
 VFIO PLATFORM DRIVER
 M:	Eric Auger <[email protected]>
 L:	[email protected]
drivers/vfio/pci/Kconfig +2 -0
index 18c397df566d..329d25c53274 100644
@@ -67,4 +67,6 @@ source "drivers/vfio/pci/pds/Kconfig"
 
 source "drivers/vfio/pci/virtio/Kconfig"
 
+source "drivers/vfio/pci/intel/qat/Kconfig"
+
 endmenu
drivers/vfio/pci/Makefile +2 -0
index 046139a4eca5..a87b6b43ce1c 100644
@@ -15,3 +15,5 @@ obj-$(CONFIG_HISI_ACC_VFIO_PCI) += hisilicon/
 obj-$(CONFIG_PDS_VFIO_PCI) += pds/
 
 obj-$(CONFIG_VIRTIO_VFIO_PCI) += virtio/
+
+obj-$(CONFIG_QAT_VFIO_PCI) += intel/qat/
drivers/vfio/pci/intel/qat/Kconfig new file +13 -0
new file mode 100644
index 000000000000..71b28ac0bf6a
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: GPL-2.0-only
+config QAT_VFIO_PCI
+	tristate "VFIO support for QAT VF PCI devices"
+	select VFIO_PCI_CORE
+	depends on CRYPTO_DEV_QAT
+	depends on CRYPTO_DEV_QAT_4XXX
+	help
+	  This provides migration support for Intel(R) QAT Virtual Function
+	  using the VFIO framework.
+
+	  To compile this as a module, choose M here: the module
+	  will be called qat_vfio_pci. If you don't know what to do here,
+	  say N.
drivers/vfio/pci/intel/qat/Makefile new file +4 -0
new file mode 100644
index 000000000000..9289ae4c51bf
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_QAT_VFIO_PCI) += qat_vfio_pci.o
+qat_vfio_pci-y := main.o
+
drivers/vfio/pci/intel/qat/main.c new file +164 -0
new file mode 100644
index 000000000000..85d0ed701397
@@ -0,0 +1,572 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2024 Intel Corporation */
+
+#include <linux/anon_inodes.h>
+#include <linux/container_of.h>
+#include <linux/device.h>
+#include <linux/file.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/pci.h>
+#include <linux/sizes.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+#include <linux/vfio_pci_core.h>
+#include <linux/qat/qat_mig_dev.h>
+
+struct qat_vf_migration_file {
+	struct file *filp;
+	/* protects migration region context */
+	struct mutex lock;
+	bool disabled;
+	struct qat_mig_dev *mdev;
+};
+
+struct qat_vf_core_device {
+	struct vfio_pci_core_device core_device;
+	struct qat_mig_dev *mdev;
+	/* protects migration state */
+	struct mutex state_mutex;
+	enum vfio_device_mig_state mig_state;
+	/* protects reset down flow */
+	spinlock_t reset_lock;
+	bool deferred_reset;
+	struct qat_vf_migration_file *resuming_migf;
+	struct qat_vf_migration_file *saving_migf;
+};
+
+static int qat_vf_pci_open_device(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev =
+		container_of(core_vdev, struct qat_vf_core_device,
+			     core_device.vdev);
+	struct vfio_pci_core_device *vdev = &qat_vdev->core_device;
+	int ret;
+
+	ret = vfio_pci_core_enable(vdev);
+	if (ret)
+		return ret;
+
+	ret = qat_vdev->mdev->ops->open(qat_vdev->mdev);
+	if (ret) {
+		vfio_pci_core_disable(vdev);
+		return ret;
+	}
+	qat_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
+
+	vfio_pci_core_finish_enable(vdev);
+
+	return 0;
+}
+
+static void qat_vf_disable_fd(struct qat_vf_migration_file *migf)
+{
+	mutex_lock(&migf->lock);
+	migf->disabled = true;
+	migf->filp->f_pos = 0;
+	mutex_unlock(&migf->lock);
+}
+
+static void qat_vf_disable_fds(struct qat_vf_core_device *qat_vdev)
+{
+	if (qat_vdev->resuming_migf) {
+		qat_vf_disable_fd(qat_vdev->resuming_migf);
+		fput(qat_vdev->resuming_migf->filp);
+		qat_vdev->resuming_migf = NULL;
+	}
+
+	if (qat_vdev->saving_migf) {
+		qat_vf_disable_fd(qat_vdev->saving_migf);
+		fput(qat_vdev->saving_migf->filp);
+		qat_vdev->saving_migf = NULL;
+	}
+}
+
+static void qat_vf_pci_close_device(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+
+	qat_vdev->mdev->ops->close(qat_vdev->mdev);
+	qat_vf_disable_fds(qat_vdev);
+	vfio_pci_core_close_device(core_vdev);
+}
+
+static ssize_t qat_vf_save_read(struct file *filp, char __user *buf,
+				size_t len, loff_t *pos)
+{
+	struct qat_vf_migration_file *migf = filp->private_data;
+	ssize_t done = 0;
+	loff_t *offs;
+	int ret;
+
+	if (pos)
+		return -ESPIPE;
+	offs = &filp->f_pos;
+
+	mutex_lock(&migf->lock);
+	if (*offs > migf->mdev->state_size || *offs < 0) {
+		done = -EINVAL;
+		goto out_unlock;
+	}
+
+	if (migf->disabled) {
+		done = -ENODEV;
+		goto out_unlock;
+	}
+
+	len = min_t(size_t, migf->mdev->state_size - *offs, len);
+	if (len) {
+		ret = copy_to_user(buf, migf->mdev->state + *offs, len);
+		if (ret) {
+			done = -EFAULT;
+			goto out_unlock;
+		}
+		*offs += len;
+		done = len;
+	}
+
+out_unlock:
+	mutex_unlock(&migf->lock);
+	return done;
+}
+
+static int qat_vf_release_file(struct inode *inode, struct file *filp)
+{
+	struct qat_vf_migration_file *migf = filp->private_data;
+
+	qat_vf_disable_fd(migf);
+	mutex_destroy(&migf->lock);
+	kfree(migf);
+
+	return 0;
+}
+
+static const struct file_operations qat_vf_save_fops = {
+	.owner = THIS_MODULE,
+	.read = qat_vf_save_read,
+	.release = qat_vf_release_file,
+	.llseek = no_llseek,
+};
+
+static struct qat_vf_migration_file *
+qat_vf_save_device_data(struct qat_vf_core_device *qat_vdev)
+{
+	struct qat_vf_migration_file *migf;
+	int ret;
+
+	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	migf->filp = anon_inode_getfile("qat_vf_mig", &qat_vf_save_fops,

migf, O_RDONLY);

drivers/vfio/pci/intel/qat/main.c new file +3 -0
+	ret = PTR_ERR_OR_ZERO(migf->filp);
+	if (ret) {
+		kfree(migf);

drivers/vfio/pci/intel/qat/main.c line 177

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+		return ERR_PTR(ret);
+	}
+
+	stream_open(migf->filp->f_inode, migf->filp);
+	mutex_init(&migf->lock);
+
+	ret = qat_vdev->mdev->ops->save_state(qat_vdev->mdev);
+	if (ret) {
+		fput(migf->filp);
+		kfree(migf);

Probably don't need that kfree(migf) here as fput() --> qat_vf_release_file () will do that.

Quoted 214 lines
drivers/vfio/pci/intel/qat/main.c new file +64 -0
+		return ERR_PTR(ret);
+	}
+
+	migf->mdev = qat_vdev->mdev;
+
+	return migf;
+}
+
+static ssize_t qat_vf_resume_write(struct file *filp, const char __user *buf,
+				   size_t len, loff_t *pos)
+{
+	struct qat_vf_migration_file *migf = filp->private_data;
+	loff_t end, *offs;
+	ssize_t done = 0;
+	int ret;
+
+	if (pos)
+		return -ESPIPE;
+	offs = &filp->f_pos;
+
+	if (*offs < 0 ||
+	    check_add_overflow((loff_t)len, *offs, &end))
+		return -EOVERFLOW;
+
+	if (end > migf->mdev->state_size)
+		return -ENOMEM;
+
+	mutex_lock(&migf->lock);
+	if (migf->disabled) {
+		done = -ENODEV;
+		goto out_unlock;
+	}
+
+	ret = copy_from_user(migf->mdev->state + *offs, buf, len);
+	if (ret) {
+		done = -EFAULT;
+		goto out_unlock;
+	}
+	*offs += len;
+	done = len;
+
+out_unlock:
+	mutex_unlock(&migf->lock);
+	return done;
+}
+
+static const struct file_operations qat_vf_resume_fops = {
+	.owner = THIS_MODULE,
+	.write = qat_vf_resume_write,
+	.release = qat_vf_release_file,
+	.llseek = no_llseek,
+};
+
+static struct qat_vf_migration_file *
+qat_vf_resume_device_data(struct qat_vf_core_device *qat_vdev)
+{
+	struct qat_vf_migration_file *migf;
+	int ret;
+
+	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	migf->filp = anon_inode_getfile("qat_vf_mig", &qat_vf_resume_fops,

migf, O_WRONLY);

drivers/vfio/pci/intel/qat/main.c new file +19 -0
+	ret = PTR_ERR_OR_ZERO(migf->filp);
+	if (ret) {
+		kfree(migf);
+		return ERR_PTR(ret);
+	}
+
+	migf->mdev = qat_vdev->mdev;
+	stream_open(migf->filp->f_inode, migf->filp);
+	mutex_init(&migf->lock);
+
+	return migf;
+}
+
+static int qat_vf_load_device_data(struct qat_vf_core_device *qat_vdev)
+{
+	return qat_vdev->mdev->ops->load_state(qat_vdev->mdev);
+}
+
+static struct file *qat_vf_pci_step_device_state(struct qat_vf_core_device

*qat_vdev, u32 new)

drivers/vfio/pci/intel/qat/main.c new file +5 -0
+{
+	u32 cur = qat_vdev->mig_state;
+	int ret;
+
+	if ((cur == VFIO_DEVICE_STATE_RUNNING && new ==

VFIO_DEVICE_STATE_RUNNING_P2P)) {

drivers/vfio/pci/intel/qat/main.c new file +7 -0
+		ret = qat_vdev->mdev->ops->suspend(qat_vdev->mdev);
+		if (ret)
+			return ERR_PTR(ret);
+		return NULL;
+	}
+
+	if ((cur == VFIO_DEVICE_STATE_RUNNING_P2P && new ==

VFIO_DEVICE_STATE_STOP) ||

drivers/vfio/pci/intel/qat/main.c new file +1 -0
+	    (cur == VFIO_DEVICE_STATE_STOP && new ==

VFIO_DEVICE_STATE_RUNNING_P2P))

drivers/vfio/pci/intel/qat/main.c new file +3 -0
+		return NULL;
+
+	if (cur == VFIO_DEVICE_STATE_STOP && new ==

VFIO_DEVICE_STATE_STOP_COPY) {

drivers/vfio/pci/intel/qat/main.c new file +11 -0
+		struct qat_vf_migration_file *migf;
+
+		migf = qat_vf_save_device_data(qat_vdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+		get_file(migf->filp);
+		qat_vdev->saving_migf = migf;
+		return migf->filp;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_STOP_COPY && new ==

VFIO_DEVICE_STATE_STOP) {

drivers/vfio/pci/intel/qat/main.c new file +5 -0
+		qat_vf_disable_fds(qat_vdev);
+		return NULL;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_STOP && new ==

VFIO_DEVICE_STATE_RESUMING) {

drivers/vfio/pci/intel/qat/main.c new file +11 -0
+		struct qat_vf_migration_file *migf;
+
+		migf = qat_vf_resume_device_data(qat_vdev);
+		if (IS_ERR(migf))
+			return ERR_CAST(migf);
+		get_file(migf->filp);
+		qat_vdev->resuming_migf = migf;
+		return migf->filp;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_RESUMING && new ==

VFIO_DEVICE_STATE_STOP) {

drivers/vfio/pci/intel/qat/main.c new file +9 -0
+		ret = qat_vf_load_device_data(qat_vdev);
+		if (ret)
+			return ERR_PTR(ret);
+
+		qat_vf_disable_fds(qat_vdev);
+		return NULL;
+	}
+
+	if (cur == VFIO_DEVICE_STATE_RUNNING_P2P && new ==

VFIO_DEVICE_STATE_RUNNING) {

drivers/vfio/pci/intel/qat/main.c new file +5 -0
+		qat_vdev->mdev->ops->resume(qat_vdev->mdev);
+		return NULL;
+	}
+
+	/* vfio_mig_get_next_state() does not use arcs other than the above

*/

drivers/vfio/pci/intel/qat/main.c new file +5 -0
+	WARN_ON(true);
+	return ERR_PTR(-EINVAL);
+}
+
+static void qat_vf_state_mutex_unlock(struct qat_vf_core_device

*qat_vdev)

drivers/vfio/pci/intel/qat/main.c new file +16 -0
+{
+again:
+	spin_lock(&qat_vdev->reset_lock);
+	if (qat_vdev->deferred_reset) {
+		qat_vdev->deferred_reset = false;
+		spin_unlock(&qat_vdev->reset_lock);
+		qat_vdev->mig_state = VFIO_DEVICE_STATE_RUNNING;
+		qat_vf_disable_fds(qat_vdev);
+		goto again;
+	}
+	mutex_unlock(&qat_vdev->state_mutex);
+	spin_unlock(&qat_vdev->reset_lock);
+}
+
+static struct file *qat_vf_pci_set_device_state(struct vfio_device *vdev,
+						enum vfio_device_mig_state

new_state)

drivers/vfio/pci/intel/qat/main.c new file +40 -0
+{
+	struct qat_vf_core_device *qat_vdev = container_of(vdev,
+			struct qat_vf_core_device, core_device.vdev);
+	enum vfio_device_mig_state next_state;
+	struct file *res = NULL;
+	int ret;
+
+	mutex_lock(&qat_vdev->state_mutex);
+	while (new_state != qat_vdev->mig_state) {
+		ret = vfio_mig_get_next_state(vdev, qat_vdev->mig_state,
+					      new_state, &next_state);
+		if (ret) {
+			res = ERR_PTR(ret);
+			break;
+		}
+		res = qat_vf_pci_step_device_state(qat_vdev, next_state);
+		if (IS_ERR(res))
+			break;
+		qat_vdev->mig_state = next_state;
+		if (WARN_ON(res && new_state != qat_vdev->mig_state)) {
+			fput(res);
+			res = ERR_PTR(-EINVAL);
+			break;
+		}
+	}
+	qat_vf_state_mutex_unlock(qat_vdev);
+
+	return res;
+}
+
+static int qat_vf_pci_get_device_state(struct vfio_device *vdev,
+				       enum vfio_device_mig_state *curr_state)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(vdev,
+			struct qat_vf_core_device, core_device.vdev);
+
+	mutex_lock(&qat_vdev->state_mutex);
+	*curr_state = qat_vdev->mig_state;
+	qat_vf_state_mutex_unlock(qat_vdev);
+

drivers/vfio/pci/intel/qat/main.c line 388

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+	return 0;
+}
+
+static int qat_vf_pci_get_data_size(struct vfio_device *vdev,
+				    unsigned long *stop_copy_length)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(vdev,
+			struct qat_vf_core_device, core_device.vdev);
+
+	*stop_copy_length = qat_vdev->mdev->state_size;

Do we need a lock here or this is not changing?

Quoted 31 lines
drivers/vfio/pci/intel/qat/main.c new file +30 -0
+	return 0;
+}
+
+static const struct vfio_migration_ops qat_vf_pci_mig_ops = {
+	.migration_set_state = qat_vf_pci_set_device_state,
+	.migration_get_state = qat_vf_pci_get_device_state,
+	.migration_get_data_size = qat_vf_pci_get_data_size,
+};
+
+static void qat_vf_pci_release_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+
+	qat_vdev->mdev->ops->cleanup(qat_vdev->mdev);
+	qat_vfmig_destroy(qat_vdev->mdev);
+	mutex_destroy(&qat_vdev->state_mutex);
+	vfio_pci_core_release_dev(core_vdev);
+}
+
+static int qat_vf_pci_init_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+	struct qat_migdev_ops *ops;
+	struct qat_mig_dev *mdev;
+	struct pci_dev *parent;
+	int ret, vf_id;
+
+	core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY |

VFIO_MIGRATION_P2P;

drivers/vfio/pci/intel/qat/main.c line 428

Quoted 10 lines
drivers/vfio/pci/intel/qat/main.c new file +10 -0
+	core_vdev->mig_ops = &qat_vf_pci_mig_ops;
+
+	ret = vfio_pci_core_init_dev(core_vdev);
+	if (ret)
+		return ret;
+
+	mutex_init(&qat_vdev->state_mutex);
+	spin_lock_init(&qat_vdev->reset_lock);
+
+	parent = qat_vdev->core_device.pdev->physfn;

Can we use pci_physfn() here?

drivers/vfio/pci/intel/qat/main.c line 430

Quoted 2 lines
drivers/vfio/pci/intel/qat/main.c new file +2 -0
+	vf_id = pci_iov_vf_id(qat_vdev->core_device.pdev);
+	if (!parent || vf_id < 0) {

Also if the pci_iov_vf_id() return success I don't think you need to check for parent and can use directly below.

Quoted 18 lines
drivers/vfio/pci/intel/qat/main.c new file +17 -0
+		ret = -ENODEV;
+		goto err_rel;
+	}
+
+	mdev = qat_vfmig_create(parent, vf_id);
+	if (IS_ERR(mdev)) {
+		ret = PTR_ERR(mdev);
+		goto err_rel;
+	}
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
+		dev_err(&parent->dev, "Incomplete device migration ops

structure!");

drivers/vfio/pci/intel/qat/main.c

Quoted 2 lines
drivers/vfio/pci/intel/qat/main.c new file +2 -0
+		goto err_destroy;
+	}

If all these ops are a must why cant we move the check inside the qat_vfmig_create()? Or rather call them explicitly as suggested by Jason.

Thanks,
Shameer

Quoted 133 lines
drivers/vfio/pci/intel/qat/main.c new file +38 -0
+	ret = ops->init(mdev);
+	if (ret)
+		goto err_destroy;
+
+	qat_vdev->mdev = mdev;
+
+	return 0;
+
+err_destroy:
+	qat_vfmig_destroy(mdev);
+err_rel:
+	vfio_pci_core_release_dev(core_vdev);
+	return ret;
+}
+
+static const struct vfio_device_ops qat_vf_pci_ops = {
+	.name = "qat-vf-vfio-pci",
+	.init = qat_vf_pci_init_dev,
+	.release = qat_vf_pci_release_dev,
+	.open_device = qat_vf_pci_open_device,
+	.close_device = qat_vf_pci_close_device,
+	.ioctl = vfio_pci_core_ioctl,
+	.read = vfio_pci_core_read,
+	.write = vfio_pci_core_write,
+	.mmap = vfio_pci_core_mmap,
+	.request = vfio_pci_core_request,
+	.match = vfio_pci_core_match,
+	.bind_iommufd = vfio_iommufd_physical_bind,
+	.unbind_iommufd = vfio_iommufd_physical_unbind,
+	.attach_ioas = vfio_iommufd_physical_attach_ioas,
+	.detach_ioas = vfio_iommufd_physical_detach_ioas,
+};
+
+static struct qat_vf_core_device *qat_vf_drvdata(struct pci_dev *pdev)
+{
+	struct vfio_pci_core_device *core_device = pci_get_drvdata(pdev);
+
+	return container_of(core_device, struct qat_vf_core_device,

core_device);

drivers/vfio/pci/intel/qat/main.c new file +12 -0
+}
+
+static void qat_vf_pci_aer_reset_done(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	if (!qat_vdev->core_device.vdev.mig_ops)
+		return;
+
+	/*
+	 * As the higher VFIO layers are holding locks across reset and using
+	 * those same locks with the mm_lock we need to prevent ABBA

deadlock

drivers/vfio/pci/intel/qat/main.c new file +2 -0
+	 * with the state_mutex and mm_lock.
+	 * In case the state_mutex was taken already we defer the cleanup

work

drivers/vfio/pci/intel/qat/main.c new file +20 -0
+	 * to the unlock flow of the other running context.
+	 */
+	spin_lock(&qat_vdev->reset_lock);
+	qat_vdev->deferred_reset = true;
+	if (!mutex_trylock(&qat_vdev->state_mutex)) {
+		spin_unlock(&qat_vdev->reset_lock);
+		return;
+	}
+	spin_unlock(&qat_vdev->reset_lock);
+	qat_vf_state_mutex_unlock(qat_vdev);
+}
+
+static int
+qat_vf_vfio_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct device *dev = &pdev->dev;
+	struct qat_vf_core_device *qat_vdev;
+	int ret;
+
+	qat_vdev = vfio_alloc_device(qat_vf_core_device, core_device.vdev,

dev, &qat_vf_pci_ops);

drivers/vfio/pci/intel/qat/main.c new file +26 -0
+	if (IS_ERR(qat_vdev))
+		return PTR_ERR(qat_vdev);
+
+	pci_set_drvdata(pdev, &qat_vdev->core_device);
+	ret = vfio_pci_core_register_device(&qat_vdev->core_device);
+	if (ret)
+		goto out_put_device;
+
+	return 0;
+
+out_put_device:
+	vfio_put_device(&qat_vdev->core_device.vdev);
+	return ret;
+}
+
+static void qat_vf_vfio_pci_remove(struct pci_dev *pdev)
+{
+	struct qat_vf_core_device *qat_vdev = qat_vf_drvdata(pdev);
+
+	vfio_pci_core_unregister_device(&qat_vdev->core_device);
+	vfio_put_device(&qat_vdev->core_device.vdev);
+}
+
+static const struct pci_device_id qat_vf_vfio_pci_table[] = {
+	/* Intel QAT GEN4 4xxx VF device */
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_INTEL,

0x4941) },

drivers/vfio/pci/intel/qat/main.c new file +1 -0
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_INTEL,

0x4943) },

drivers/vfio/pci/intel/qat/main.c new file +1 -0
+	{ PCI_DRIVER_OVERRIDE_DEVICE_VFIO(PCI_VENDOR_ID_INTEL,

0x4945) },

drivers/vfio/pci/intel/qat/main.c new file +22 -0
+	{}
+};
+MODULE_DEVICE_TABLE(pci, qat_vf_vfio_pci_table);
+
+static const struct pci_error_handlers qat_vf_err_handlers = {
+	.reset_done = qat_vf_pci_aer_reset_done,
+	.error_detected = vfio_pci_core_aer_err_detected,
+};
+
+static struct pci_driver qat_vf_vfio_pci_driver = {
+	.name = "qat_vfio_pci",
+	.id_table = qat_vf_vfio_pci_table,
+	.probe = qat_vf_vfio_pci_probe,
+	.remove = qat_vf_vfio_pci_remove,
+	.err_handler = &qat_vf_err_handlers,
+	.driver_managed_dma = true,
+};
+module_pci_driver(qat_vf_vfio_pci_driver)
+
+MODULE_LICENSE("GPL");
+MODULE_AUTHOR("Intel Corporation");
+MODULE_DESCRIPTION("QAT VFIO PCI - VFIO PCI driver with live migration

support for Intel(R) QAT GEN4 device family");
+MODULE_IMPORT_NS(CRYPTO_QAT);
--
2.18.2

author
#28 lore raw

RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Thanks for the comments, Shameer.

Quoted 31 lines

-----Original Message-----
From: Shameerali Kolothum Thodi <[email protected]>
Sent: Thursday, February 8, 2024 8:17 PM
To: Zeng, Xin <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Tian, Kevin <[email protected]>
Cc: [email protected]; [email protected]; qat-linux <qat- [email protected]>; Cao, Yahui <[email protected]>
Subject: RE: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 20 lines
drivers/vfio/pci/intel/qat/main.c new file +11 -0
+static struct qat_vf_migration_file *
+qat_vf_save_device_data(struct qat_vf_core_device *qat_vdev)
+{
+	struct qat_vf_migration_file *migf;
+	int ret;
+
+	migf = kzalloc(sizeof(*migf), GFP_KERNEL);
+	if (!migf)
+		return ERR_PTR(-ENOMEM);
+
+	migf->filp = anon_inode_getfile("qat_vf_mig", &qat_vf_save_fops,

migf, O_RDONLY);

drivers/vfio/pci/intel/qat/main.c new file +8 -0
+	ret = PTR_ERR_OR_ZERO(migf->filp);
+	if (ret) {
+		kfree(migf);
+		return ERR_PTR(ret);
+	}
+
+	stream_open(migf->filp->f_inode, migf->filp);
+	mutex_init(&migf->lock);
Quoted 8 lines

drivers/vfio/pci/intel/qat/main.c line 177

Quoted 5 lines
drivers/vfio/pci/intel/qat/main.c new file +5 -0
+
+	ret = qat_vdev->mdev->ops->save_state(qat_vdev->mdev);
+	if (ret) {
+		fput(migf->filp);
+		kfree(migf);

Probably don't need that kfree(migf) here as fput() --> qat_vf_release_file () will do that.

Thanks, it's redundant, will update it in next version.

Quoted 10 lines

drivers/vfio/pci/intel/qat/main.c line 388

Quoted 8 lines
drivers/vfio/pci/intel/qat/main.c new file +7 -0
+static int qat_vf_pci_get_data_size(struct vfio_device *vdev,
+				    unsigned long *stop_copy_length)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(vdev,
+			struct qat_vf_core_device, core_device.vdev);
+
+	*stop_copy_length = qat_vdev->mdev->state_size;

Do we need a lock here or this is not changing?

Yes, will update it in next version.

Quoted 36 lines
Quoted 36 lines
drivers/vfio/pci/intel/qat/main.c new file +30 -0
+	return 0;
+}
+
+static const struct vfio_migration_ops qat_vf_pci_mig_ops = {
+	.migration_set_state = qat_vf_pci_set_device_state,
+	.migration_get_state = qat_vf_pci_get_device_state,
+	.migration_get_data_size = qat_vf_pci_get_data_size,
+};
+
+static void qat_vf_pci_release_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+
+	qat_vdev->mdev->ops->cleanup(qat_vdev->mdev);
+	qat_vfmig_destroy(qat_vdev->mdev);
+	mutex_destroy(&qat_vdev->state_mutex);
+	vfio_pci_core_release_dev(core_vdev);
+}
+
+static int qat_vf_pci_init_dev(struct vfio_device *core_vdev)
+{
+	struct qat_vf_core_device *qat_vdev = container_of(core_vdev,
+			struct qat_vf_core_device, core_device.vdev);
+	struct qat_migdev_ops *ops;
+	struct qat_mig_dev *mdev;
+	struct pci_dev *parent;
+	int ret, vf_id;
+
+	core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY |

VFIO_MIGRATION_P2P;

drivers/vfio/pci/intel/qat/main.c new file +4 -0
+	core_vdev->mig_ops = &qat_vf_pci_mig_ops;
+
+	ret = vfio_pci_core_init_dev(core_vdev);
+	if (ret)
Quoted 8 lines

drivers/vfio/pci/intel/qat/main.c line 428

Quoted 6 lines
drivers/vfio/pci/intel/qat/main.c new file +6 -0
+		return ret;
+
+	mutex_init(&qat_vdev->state_mutex);
+	spin_lock_init(&qat_vdev->reset_lock);
+
+	parent = qat_vdev->core_device.pdev->physfn;

Can we use pci_physfn() here?

Sure, will update it in next version.

Quoted 6 lines

drivers/vfio/pci/intel/qat/main.c line 430

Quoted 3 lines
drivers/vfio/pci/intel/qat/main.c new file +2 -0
+	vf_id = pci_iov_vf_id(qat_vdev->core_device.pdev);
+	if (!parent || vf_id < 0) {

Also if the pci_iov_vf_id() return success I don't think you need to check for parent and can use directly below.

OK, will update it in next version.

Quoted 17 lines
Quoted 17 lines
drivers/vfio/pci/intel/qat/main.c new file +16 -0
+		ret = -ENODEV;
+		goto err_rel;
+	}
+
+	mdev = qat_vfmig_create(parent, vf_id);
+	if (IS_ERR(mdev)) {
+		ret = PTR_ERR(mdev);
+		goto err_rel;
+	}
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +1 -0
+		dev_err(&parent->dev, "Incomplete device migration ops

structure!");

(quoted diff) +2 -0
+		goto err_destroy;
+	}

If all these ops are a must why cant we move the check inside the qat_vfmig_create()?
Or rather call them explicitly as suggested by Jason.

We can do it, but it might make sense to leave the check to the APIs' user as some of these ops interfaces might be optional for other QAT variant driver in future.

Thanks,
Xin

#29 lore raw

Re: [PATCH 10/10] vfio/qat: Add vfio_pci driver for Intel QAT VF devices

Quoted 25 lines · Xin
Quoted 16 lines
Quoted 16 lines
drivers/vfio/pci/intel/qat/main.c new file +16 -0
+		ret = -ENODEV;
+		goto err_rel;
+	}
+
+	mdev = qat_vfmig_create(parent, vf_id);
+	if (IS_ERR(mdev)) {
+		ret = PTR_ERR(mdev);
+		goto err_rel;
+	}
+
+	ops = mdev->ops;
+	if (!ops || !ops->init || !ops->cleanup ||
+	    !ops->open || !ops->close ||
+	    !ops->save_state || !ops->load_state ||
+	    !ops->suspend || !ops->resume) {
+		ret = -EIO;
Quoted 8 lines

Review

Quoted 4 lines
(quoted diff) +1 -0
+		dev_err(&parent->dev, "Incomplete device migration ops

structure!");

(quoted diff) +2 -0
+		goto err_destroy;
+	}

If all these ops are a must why cant we move the check inside the qat_vfmig_create()?
Or rather call them explicitly as suggested by Jason.

Quoted 3 lines

We can do it, but it might make sense to leave the check to the APIs' user as some of these ops interfaces might be optional for other QAT variant driver in future.

If you have patches already that need ops then you can leave them, but otherwise they should be removed and added back if you come with future patches that require another implementation.

Jason