Skip to contentSkip to conversation

vfio/qat: add PCI_IOV dependency

PATCH In review Arnd Bergmann 6 participants 10 messages May 28, 2024 – Jun 11, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfio/qat: add PCI_IOV dependency

From: Arnd Bergmann <[email protected]>

The newly added driver depends on the crypto driver, but it uses exported symbols that are only available when IOV is also turned on:

x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'

Add this as a second dependency.

Fixes bb208810b1ab ("vfio/qat: Add vfio_pci driver for Intel QAT SR-IOV VF devices")
---
 drivers/vfio/pci/qat/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/vfio/pci/qat/Kconfig +2 -1
index bf52cfa4b595..fae9d6cb8ccb 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0-only
 config QAT_VFIO_PCI
 	tristate "VFIO support for QAT VF PCI devices"
-	select VFIO_PCI_CORE
 	depends on CRYPTO_DEV_QAT_4XXX
+	depends on PCI_IOV
+	select VFIO_PCI_CORE
 	help
 	  This provides migration support for Intel(R) QAT Virtual Function
 	  using the VFIO framework.
Signature · 1 line
2.39.2
reviewer
#1 lore raw
Quoted 31 lines

From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PM

From: Arnd Bergmann <[email protected]>

The newly added driver depends on the crypto driver, but it uses exported symbols that are only available when IOV is also turned on:

x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function

Quoted 8 lines

`qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'

at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.

Probably they'd better be moved out of CONFIG_PCI_IOV in "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.

Quoted 27 lines

Add this as a second dependency.

Fixes bb208810b1ab ("vfio/qat: Add vfio_pci driver for Intel QAT SR-IOV VF

devices")

---
 drivers/vfio/pci/qat/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/vfio/pci/qat/Kconfig +2 -2
index bf52cfa4b595..fae9d6cb8ccb 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0-only
 config QAT_VFIO_PCI
 	tristate "VFIO support for QAT VF PCI devices"
-	select VFIO_PCI_CORE
 	depends on CRYPTO_DEV_QAT_4XXX
+	depends on PCI_IOV
+	select VFIO_PCI_CORE
 	help
 	  This provides migration support for Intel(R) QAT Virtual Function
 	  using the VFIO framework.
--

2.39.2

#2 lore raw

On Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>

Quoted 49 lines

To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: add PCI_IOV dependency

Quoted 31 lines

From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PM

From: Arnd Bergmann <[email protected]>

The newly added driver depends on the crypto driver, but it uses exported symbols that are only available when IOV is also turned on:

x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function

Quoted 8 lines

`qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'

Quoted 8 lines

at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.

Probably they'd better be moved out of CONFIG_PCI_IOV in "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.

Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio variant driver and macro CRYPTO_DEV_QAT_4XXX. The variant driver doesn't directly reference the symbols exported by module qat_4xxx which is protected by CRYPTO_DEV_QAT_4XXX, but requires the module qat_4xxx at runtime so far. Alex suggested to put CRYPTO_DEV_QAT_4XXX as the dependency of this variant driver.
For CONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as well since no VFs will be created by QAT PF driver. To keep the consistency, it might be right to make it as the dependency of this variant driver as Arnd pointed out. What do you think?

Thanks,
Xin

Quoted 27 lines
Quoted 27 lines

Add this as a second dependency.

Fixes bb208810b1ab ("vfio/qat: Add vfio_pci driver for Intel QAT SR-IOV VF

devices")

---
 drivers/vfio/pci/qat/Kconfig | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/vfio/pci/qat/Kconfig +2 -2
index bf52cfa4b595..fae9d6cb8ccb 100644
@@ -1,8 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0-only
 config QAT_VFIO_PCI
 	tristate "VFIO support for QAT VF PCI devices"
-	select VFIO_PCI_CORE
 	depends on CRYPTO_DEV_QAT_4XXX
+	depends on PCI_IOV
+	select VFIO_PCI_CORE
 	help
 	  This provides migration support for Intel(R) QAT Virtual Function
 	  using the VFIO framework.
--

2.39.2

reviewer
#3 lore raw
Quoted 72 lines

From: Zeng, Xin <[email protected]>
Sent: Wednesday, May 29, 2024 11:11 AM

On Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>

Quoted 15 lines

To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: add PCI_IOV dependency

Quoted 6 lines

From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PM

From: Arnd Bergmann <[email protected]>

The newly added driver depends on the crypto driver, but it uses

exported

Quoted 34 lines
Quoted 25 lines

symbols that are only available when IOV is also turned on:

x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function

Quoted 8 lines

`qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'

Quoted 8 lines

at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.

Probably they'd better be moved out of CONFIG_PCI_IOV in "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.

Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio
variant driver and macro CRYPTO_DEV_QAT_4XXX. The variant driver doesn't directly reference the symbols exported by module qat_4xxx which is protected
by CRYPTO_DEV_QAT_4XXX, but requires the module qat_4xxx at runtime so far.
Alex suggested to put CRYPTO_DEV_QAT_4XXX as the dependency of this variant

Quoted 9 lines

driver.
For CONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as
well since no VFs will be created by QAT PF driver. To keep the consistency, it might
be right to make it as the dependency of this variant driver as Arnd pointed out.
What do you think?

Following this rationale then we need also make PCI_IOV a dependency for mlx5 and hisilicon given they are for VF migration too?

#4 lore raw
Quoted 56 lines · Kevin

To: Zeng, Xin <[email protected]>; Arnd Bergmann <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: add PCI_IOV dependency

Quoted 26 lines

From: Zeng, Xin <[email protected]>
Sent: Wednesday, May 29, 2024 11:11 AM

On Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>

Quoted 15 lines

To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: add PCI_IOV dependency

Quoted 6 lines

From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PM

From: Arnd Bergmann <[email protected]>

The newly added driver depends on the crypto driver, but it uses

exported

Quoted 6 lines
Quoted 1 line

symbols that are only available when IOV is also turned on:

at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.

Quoted 8 lines
Quoted 4 lines

Probably they'd better be moved out of CONFIG_PCI_IOV in "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.

Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio
variant driver and macro CRYPTO_DEV_QAT_4XXX. The variant driver

doesn't

Quoted 3 lines

directly reference the symbols exported by module qat_4xxx which is protected
by CRYPTO_DEV_QAT_4XXX, but requires the module qat_4xxx at runtime

so

Quoted 7 lines

far.
Alex suggested to put CRYPTO_DEV_QAT_4XXX as the dependency of this variant
driver.
For CONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as
well since no VFs will be created by QAT PF driver. To keep the consistency,

it

Quoted 8 lines
Quoted 5 lines

might
be right to make it as the dependency of this variant driver as Arnd pointed out.
What do you think?

Following this rationale then we need also make PCI_IOV a dependency for mlx5 and hisilicon given they are for VF migration too?

After more thoughts about this, I would agree with your first point that PCI_IOV should not be the dependency of the variant driver if we consider passthrough VFs in a nested virtualized environment.
So decoupling PCI_IOV from migration helpers in QAT PF driver sounds a better option.

Thanks,
Xin

#5 lore raw
Quoted 63 lines · Xin
Quoted 62 lines · Kevin

To: Zeng, Xin <[email protected]>; Arnd Bergmann <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: add PCI_IOV dependency

Quoted 26 lines

From: Zeng, Xin <[email protected]>
Sent: Wednesday, May 29, 2024 11:11 AM

On Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>

Quoted 15 lines

To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: add PCI_IOV dependency

Quoted 6 lines

From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PM

From: Arnd Bergmann <[email protected]>

The newly added driver depends on the crypto driver, but it uses

exported

Quoted 6 lines
Quoted 1 line

symbols that are only available when IOV is also turned on:

at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.

Quoted 8 lines
Quoted 4 lines

Probably they'd better be moved out of CONFIG_PCI_IOV in "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.

Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio
variant driver and macro CRYPTO_DEV_QAT_4XXX. The variant driver

doesn't

Quoted 3 lines

directly reference the symbols exported by module qat_4xxx which is protected
by CRYPTO_DEV_QAT_4XXX, but requires the module qat_4xxx at runtime

so

Quoted 7 lines

far.
Alex suggested to put CRYPTO_DEV_QAT_4XXX as the dependency of this variant
driver.
For CONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as
well since no VFs will be created by QAT PF driver. To keep the consistency,

it

Quoted 5 lines

might
be right to make it as the dependency of this variant driver as Arnd pointed out.
What do you think?

Quoted 8 lines
Quoted 2 lines

Following this rationale then we need also make PCI_IOV a dependency for mlx5 and hisilicon given they are for VF migration too?

After more thoughts about this, I would agree with your first point that PCI_IOV should not be the dependency of the variant driver if we consider passthrough VFs in a nested virtualized environment.
So decoupling PCI_IOV from migration helpers in QAT PF driver sounds a better option.

Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10? Is there an archive of qat-linux list somewhere? I can't find any relevant postings on linux-crypto@vger and can't find a public archive of qat-linux(??). Thanks,

Alex

#6 lore raw
Quoted 2 lines · Alex Williamson

Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10?

Yes. This is being taken care in the QAT PF driver. Xin just sent a fix for it [1].
@Herbert, can this be sent to stable after the review?

Quoted 3 lines

Is there an archive of qat-linux list somewhere?
I can't find any relevant postings on linux-crypto@vger and can't find a public archive of qat-linux(??). Thanks,

There isn't a public archive of qat-linux. All patches to the QAT driver are sent to [email protected].

[1] https://lore.kernel.org/all/[email protected]/

Regards,

Signature · 1 line
Giovanni
#7 lore raw
Quoted 6 lines · Giovanni
Quoted 2 lines · Alex Williamson

Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10?

Yes. This is being taken care in the QAT PF driver. Xin just sent a fix for it [1].
@Herbert, can this be sent to stable after the review?

This patch wasn't sent to linux-crypto so I cannot apply it.

In any case, is there any reason why the fix can't go through the vfio tree?

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#8 lore raw

Hi Herbert,

Quoted 9 lines · Herbert Xu
Quoted 6 lines · Giovanni
Quoted 2 lines · Alex Williamson

Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10?

Yes. This is being taken care in the QAT PF driver. Xin just sent a fix for it [1].
@Herbert, can this be sent to stable after the review?

This patch wasn't sent to linux-crypto so I cannot apply it.

Was sent yesterday to linux-crypto:
https://patchwork.kernel.org/project/linux-crypto/patch/[email protected]/

Quoted 2 lines

In any case, is there any reason why the fix can't go through the vfio tree?

It is a change in the QAT driver. That's why I asked.

Anyway, it should be fixed as part of v6.10 - I was wrong saying it should be sent to stable.

Regards,

Signature · 1 line
Giovanni
#9 lore raw
Quoted 4 lines · Giovanni
Quoted 3 lines

In any case, is there any reason why the fix can't go through the vfio tree?

It is a change in the QAT driver. That's why I asked.

Sorry, I got confused by the subject line. I'll get onto this patch.

Thanks,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt