vfio/qat: add PCI_IOV dependency
Subsystems (4)
PATCH 1/1 vfio/qat: add PCI_IOV dependency
From: Arnd Bergmann <[email protected]>
The newly added driver depends on the crypto driver, but it uses exported symbols that are only available when IOV is also turned on:
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld: drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'
Add this as a second dependency.
--- drivers/vfio/pci/qat/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/vfio/pci/qat/Kconfig +2 -1
@@ -1,8 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only config QAT_VFIO_PCI tristate "VFIO support for QAT VF PCI devices" - select VFIO_PCI_CORE depends on CRYPTO_DEV_QAT_4XXX + depends on PCI_IOV + select VFIO_PCI_CORE help This provides migration support for Intel(R) QAT Virtual Function using the VFIO framework.
Signature · 1 line
2.39.2
Tian, Kevin
reviewerQuoted 31 lines
From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PMFrom: Arnd Bergmann <[email protected]>
The newly added driver depends on the crypto driver, but it uses exported symbols that are only available when IOV is also turned on:
x86_64-linux-ld:
drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function
Quoted 8 lines
`qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'
at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.
Probably they'd better be moved out of CONFIG_PCI_IOV in "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.
Quoted 27 lines
Add this as a second dependency.
Fixes bb208810b1ab ("vfio/qat: Add vfio_pci driver for Intel QAT SR-IOV VFdevices")
Signed-off-by Arnd Bergmann <[email protected]>--- drivers/vfio/pci/qat/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/vfio/pci/qat/Kconfig +2
-2@@ -1,8 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only config QAT_VFIO_PCI tristate "VFIO support for QAT VF PCI devices" - select VFIO_PCI_CORE depends on CRYPTO_DEV_QAT_4XXX + depends on PCI_IOV + select VFIO_PCI_CORE help This provides migration support for Intel(R) QAT Virtual Function using the VFIO framework. --2.39.2
On Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>
Quoted 49 lines
To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: addPCI_IOVdependencyQuoted 31 lines
From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PMFrom: Arnd Bergmann <[email protected]>
The newly added driver depends on the crypto driver, but it uses exported symbols that are only available when IOV is also turned on:
x86_64-linux-ld:
drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in functionQuoted 8 lines
`qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'
Quoted 8 lines
at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.
Probably they'd better be moved out of
CONFIG_PCI_IOVin "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.
Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio variant driver and macro CRYPTO_DEV_QAT_4XXX. The variant driver doesn't directly reference the symbols exported by module qat_4xxx which is protected by CRYPTO_DEV_QAT_4XXX, but requires the module qat_4xxx at runtime so far. Alex suggested to put CRYPTO_DEV_QAT_4XXX as the dependency of this variant driver.
For CONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as well since no VFs will be created by QAT PF driver. To keep the consistency, it might be right to make it as the dependency of this variant driver as Arnd pointed out. What do you think?
Thanks,
Xin
Quoted 27 lines
Quoted 27 lines
Add this as a second dependency.
Fixes bb208810b1ab ("vfio/qat: Add vfio_pci driver for Intel QAT SR-IOV VFdevices")
Signed-off-by Arnd Bergmann <[email protected]>--- drivers/vfio/pci/qat/Kconfig | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)drivers/vfio/pci/qat/Kconfig +2
-2@@ -1,8 +1,9 @@ # SPDX-License-Identifier: GPL-2.0-only config QAT_VFIO_PCI tristate "VFIO support for QAT VF PCI devices" - select VFIO_PCI_CORE depends on CRYPTO_DEV_QAT_4XXX + depends on PCI_IOV + select VFIO_PCI_CORE help This provides migration support for Intel(R) QAT Virtual Function using the VFIO framework. --2.39.2
Tian, Kevin
reviewerQuoted 72 lines
From: Zeng, Xin <[email protected]>
Sent: Wednesday, May 29, 2024 11:11 AMOn Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>
Quoted 15 lines
To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: addPCI_IOVdependencyQuoted 6 lines
From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PMFrom: Arnd Bergmann <[email protected]>
The newly added driver depends on the crypto driver, but it uses
exported
Quoted 34 lines
Quoted 25 lines
symbols that are only available when IOV is also turned on:
x86_64-linux-ld:
drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_open_device':
main.c:(.text+0xd7): undefined reference to `qat_vfmig_open'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_release_dev':
main.c:(.text+0x122): undefined reference to `qat_vfmig_cleanup'
x86_64-linux-ld: main.c:(.text+0x12d): undefined reference to `qat_vfmig_destroy'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_resume_write':
main.c:(.text+0x308): undefined reference to `qat_vfmig_load_setup'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_save_device_data':
main.c:(.text+0x64c): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0x677): undefined reference to `qat_vfmig_save_setup'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_aer_reset_done':
main.c:(.text+0x82d): undefined reference to `qat_vfmig_reset'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in function `qat_vf_pci_close_device':
main.c:(.text+0x862): undefined reference to `qat_vfmig_close'
x86_64-linux-ld:drivers/vfio/pci/qat/main.o: in functionQuoted 8 lines
`qat_vf_pci_set_device_state':
main.c:(.text+0x9af): undefined reference to `qat_vfmig_suspend'
x86_64-linux-ld: main.c:(.text+0xa14): undefined reference to `qat_vfmig_save_state'
x86_64-linux-ld: main.c:(.text+0xb37): undefined reference to `qat_vfmig_resume'
x86_64-linux-ld: main.c:(.text+0xbc7): undefined reference to `qat_vfmig_load_state'Quoted 8 lines
at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.
Probably they'd better be moved out of
CONFIG_PCI_IOVin "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio
variant driver and macroCRYPTO_DEV_QAT_4XXX. The variant driver doesn't directly reference the symbols exported by moduleqat_4xxxwhich is protected
byCRYPTO_DEV_QAT_4XXX, but requires the moduleqat_4xxxat runtime so far.
Alex suggested to putCRYPTO_DEV_QAT_4XXXas the dependency of this variant
Quoted 9 lines
driver.
ForCONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as
well since no VFs will be created by QAT PF driver. To keep the consistency, it might
be right to make it as the dependency of this variant driver as Arnd pointed out.
What do you think?
Following this rationale then we need also make PCI_IOV a dependency for mlx5 and hisilicon given they are for VF migration too?
Quoted 56 lines · Kevin
To: Zeng, Xin <[email protected]>; Arnd Bergmann <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: addPCI_IOVdependencyQuoted 26 lines
From: Zeng, Xin <[email protected]>
Sent: Wednesday, May 29, 2024 11:11 AMOn Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>
Quoted 15 lines
To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: addPCI_IOVdependencyQuoted 6 lines
From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PMFrom: Arnd Bergmann <[email protected]>
The newly added driver depends on the crypto driver, but it uses
exported
Quoted 6 lines
Quoted 1 line
symbols that are only available when IOV is also turned on:
at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.
Quoted 8 lines
Quoted 4 lines
Probably they'd better be moved out of
CONFIG_PCI_IOVin "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio
variant driver and macroCRYPTO_DEV_QAT_4XXX. The variant driverdoesn't
Quoted 3 lines
directly reference the symbols exported by module
qat_4xxxwhich is protected
byCRYPTO_DEV_QAT_4XXX, but requires the moduleqat_4xxxat runtimeso
Quoted 7 lines
far.
Alex suggested to putCRYPTO_DEV_QAT_4XXXas the dependency of this variant
driver.
ForCONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as
well since no VFs will be created by QAT PF driver. To keep the consistency,it
Quoted 8 lines
Quoted 5 lines
might
be right to make it as the dependency of this variant driver as Arnd pointed out.
What do you think?Following this rationale then we need also make
PCI_IOVa dependency for mlx5 and hisilicon given they are for VF migration too?
After more thoughts about this, I would agree with your first point that PCI_IOV should not be the dependency of the variant driver if we consider passthrough VFs in a nested virtualized environment.
So decoupling PCI_IOV from migration helpers in QAT PF driver sounds a better option.
Thanks,
Xin
Quoted 63 lines · Xin
Quoted 62 lines · Kevin
To: Zeng, Xin <[email protected]>; Arnd Bergmann <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: addPCI_IOVdependencyQuoted 26 lines
From: Zeng, Xin <[email protected]>
Sent: Wednesday, May 29, 2024 11:11 AMOn Wednesday, May 29, 2024 10:25 AM, Tian, Kevin <[email protected]>
Quoted 15 lines
To: Arnd Bergmann <[email protected]>; Zeng, Xin <[email protected]>; Cabiddu, Giovanni <[email protected]>; Alex Williamson <[email protected]>; Cao, Yahui <[email protected]>
Cc: Arnd Bergmann <[email protected]>; Jason Gunthorpe <[email protected]>; Yishai Hadas <[email protected]>; Shameer Kolothum <[email protected]>; [email protected]; qat- linux <[email protected]>; [email protected]
Subject: RE: [PATCH] vfio/qat: addPCI_IOVdependencyQuoted 6 lines
From: Arnd Bergmann <[email protected]>
Sent: Tuesday, May 28, 2024 8:05 PMFrom: Arnd Bergmann <[email protected]>
The newly added driver depends on the crypto driver, but it uses
exported
Quoted 6 lines
Quoted 1 line
symbols that are only available when IOV is also turned on:
at a glance those undefined symbols don't use any symbol under IOV. They are just wrappers to certain callbacks registered by by respective qat drivers which support migration.
Quoted 8 lines
Quoted 4 lines
Probably they'd better be moved out of
CONFIG_PCI_IOVin "drivers/crypto/intel/qat/qat_common/Makefile" to remove this dependency in vfio variant driver.Thanks, Kevin :-). This dependency is like the relationship between the QAT vfio
variant driver and macroCRYPTO_DEV_QAT_4XXX. The variant driverdoesn't
Quoted 3 lines
directly reference the symbols exported by module
qat_4xxxwhich is protected
byCRYPTO_DEV_QAT_4XXX, but requires the moduleqat_4xxxat runtimeso
Quoted 7 lines
far.
Alex suggested to putCRYPTO_DEV_QAT_4XXXas the dependency of this variant
driver.
ForCONFIG_PCI_IOV, if it is disabled, this variant driver doesn't serve the user as
well since no VFs will be created by QAT PF driver. To keep the consistency,it
Quoted 5 lines
might
be right to make it as the dependency of this variant driver as Arnd pointed out.
What do you think?
Quoted 8 lines
Quoted 2 lines
Following this rationale then we need also make
PCI_IOVa dependency for mlx5 and hisilicon given they are for VF migration too?After more thoughts about this, I would agree with your first point that
PCI_IOVshould not be the dependency of the variant driver if we consider passthrough VFs in a nested virtualized environment.
So decouplingPCI_IOVfrom migration helpers in QAT PF driver sounds a better option.
Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10? Is there an archive of qat-linux list somewhere? I can't find any relevant postings on linux-crypto@vger and can't find a public archive of qat-linux(??). Thanks,
Alex
Cabiddu, Giovanni
maintainerQuoted 2 lines · Alex Williamson
Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10?
Yes. This is being taken care in the QAT PF driver. Xin just sent a fix for it [1].
@Herbert, can this be sent to stable after the review?
Quoted 3 lines
Is there an archive of qat-linux list somewhere?
I can't find any relevant postings on linux-crypto@vger and can't find a public archive of qat-linux(??). Thanks,
There isn't a public archive of qat-linux. All patches to the QAT driver are sent to [email protected].
[1] https://lore.kernel.org/all/[email protected]/
Regards,
Signature · 1 line
Giovanni
Quoted 6 lines · Giovanni
Quoted 2 lines · Alex Williamson
Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10?
Yes. This is being taken care in the QAT PF driver. Xin just sent a fix for it [1].
@Herbert, can this be sent to stable after the review?
This patch wasn't sent to linux-crypto so I cannot apply it.
In any case, is there any reason why the fix can't go through the vfio tree?
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Cabiddu, Giovanni
maintainerHi Herbert,
Quoted 9 lines · Herbert Xu
Quoted 6 lines · Giovanni
Quoted 2 lines · Alex Williamson
Is this then being taken care of in the QAT PF driver? Are there patches posted targeting v6.10?
Yes. This is being taken care in the QAT PF driver. Xin just sent a fix for it [1].
@Herbert, can this be sent to stable after the review?This patch wasn't sent to linux-crypto so I cannot apply it.
Was sent yesterday to linux-crypto:
https://patchwork.kernel.org/project/linux-crypto/patch/[email protected]/
Quoted 2 lines
In any case, is there any reason why the fix can't go through the vfio tree?
It is a change in the QAT driver. That's why I asked.
Anyway, it should be fixed as part of v6.10 - I was wrong saying it should be sent to stable.
Regards,
Signature · 1 line
Giovanni
Quoted 4 lines · Giovanni
Quoted 3 lines
In any case, is there any reason why the fix can't go through the vfio tree?
It is a change in the QAT driver. That's why I asked.
Sorry, I got confused by the subject line. I'll get onto this patch.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt