Skip to contentSkip to conversation

vfio-pci/zdev: require KVM to be built-in

PATCH In review Randy Dunlap 4 participants 17 messages Aug 14, 2022 – Aug 18, 2022

Subsystems (6)
KVM s390 VFIO driver KVM s390 +2 more subsystems
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 vfio-pci/zdev: require KVM to be built-in

Fix build errors when CONFIG_KVM=m:

s390-linux-ld: drivers/vfio/pci/vfio_pci_zdev.o: in function `vfio_pci_zdev_open_device':
vfio_pci_zdev.c:(.text+0x242): undefined reference to `kvm_s390_pci_register_kvm'
s390-linux-ld: drivers/vfio/pci/vfio_pci_zdev.o: in function `vfio_pci_zdev_close_device':
vfio_pci_zdev.c:(.text+0x296): undefined reference to `kvm_s390_pci_unregister_kvm'

Having a bool Kconfig symbol depend on a tristate symbol can often lead to problems like this.

Fixes 8061d1c31f1a ("vfio-pci/zdev: add open/close device hooks")
Cc Matthew Rosato <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Pierre Morel <[email protected]>
Cc Eric Farman <[email protected]>
---
 drivers/vfio/pci/Kconfig |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/pci/Kconfig +1 -1
@@ -46,7 +46,7 @@ endif
 
 config VFIO_PCI_ZDEV_KVM
 	bool "VFIO PCI extensions for s390x KVM passthrough"
-	depends on S390 && KVM
+	depends on S390 && KVM=y
 	default y
 	help
 	  Support s390x-specific extensions to enable support for enhancements
#1 lore raw

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +1 -1
index 6287a843e8bc..1733339cc4eb 100644
@@ -1038,7 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)
 int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
 void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
 #else
drivers/vfio/pci/Kconfig +2 -2
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD
 endif
 
 config VFIO_PCI_ZDEV_KVM
-       bool "VFIO PCI extensions for s390x KVM passthrough"
+       def_tristate y
+       prompt "VFIO PCI extensions for s390x KVM passthrough"
        depends on S390 && KVM
-       default y
        help
          Support s390x-specific extensions to enable support for 

enhancements
to KVM passthrough capabilities, such as interpretive
execution of

----

What do you think? It seems to me it solves the problem, what do you think?

Regards,
Pierre

Quoted 35 lines · Randy Dunlap

Fix build errors when CONFIG_KVM=m:

s390-linux-ld: drivers/vfio/pci/vfio_pci_zdev.o: in function `vfio_pci_zdev_open_device':
vfio_pci_zdev.c:(.text+0x242): undefined reference to `kvm_s390_pci_register_kvm'
s390-linux-ld: drivers/vfio/pci/vfio_pci_zdev.o: in function `vfio_pci_zdev_close_device':
vfio_pci_zdev.c:(.text+0x296): undefined reference to `kvm_s390_pci_unregister_kvm'

Having a bool Kconfig symbol depend on a tristate symbol can often lead to problems like this.

Fixes 8061d1c31f1a ("vfio-pci/zdev: add open/close device hooks")
Cc Matthew Rosato <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Pierre Morel <[email protected]>
Cc Eric Farman <[email protected]>
---
  drivers/vfio/pci/Kconfig |    2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/vfio/pci/Kconfig +1 -1
@@ -46,7 +46,7 @@ endif
 
  config VFIO_PCI_ZDEV_KVM
  	bool "VFIO PCI extensions for s390x KVM passthrough"
-	depends on S390 && KVM
+	depends on S390 && KVM=y
  	default y
  	help
  	  Support s390x-specific extensions to enable support for enhancements
Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#2 lore raw

Hi--

Quoted 12 lines · Pierre Morel

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +0 -0
index 6287a843e8bc..1733339cc4eb 100644

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 5 lines
arch/s390/include/asm/kvm_host.h +1 -1
@@ -1038,7 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
  #define __KVM_HAVE_ARCH_VM_FREE
  void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)

This all looks good except for the line above.
It should be:

#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)

Thanks.

Quoted 26 lines
arch/s390/include/asm/kvm_host.h +0 -0
  int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
  void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
  #else
drivers/vfio/pci/Kconfig +0 -0
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD

 endif

 config VFIO_PCI_ZDEV_KVM
-       bool "VFIO PCI extensions for s390x KVM passthrough"
+       def_tristate y
+       prompt "VFIO PCI extensions for s390x KVM passthrough" depends on S390 && KVM
-       default y
        help
          Support s390x-specific extensions to enable support for enhancements to KVM passthrough capabilities, such as interpretive execution of

----

What do you think? It seems to me it solves the problem, what do you think?

Regards,
Pierre

Signature · 1 line
~Randy
#3 lore raw
Quoted 24 lines · Randy Dunlap

Hi--

Quoted 12 lines · Pierre Morel

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +0 -0
index 6287a843e8bc..1733339cc4eb 100644

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 5 lines
arch/s390/include/asm/kvm_host.h +1 -1
@@ -1038,7 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
  #define __KVM_HAVE_ARCH_VM_FREE
  void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)

This all looks good except for the line above.
It should be:

Quoted 3 lines

#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)

Thanks.

Yes, better, thanks.
How do we do? Should I repost it with reported-by you or do you want to post it?

Pierre

Quoted 30 lines
Quoted 28 lines
arch/s390/include/asm/kvm_host.h +0 -0
  int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
  void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
  #else
drivers/vfio/pci/Kconfig +2 -3
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD
  endif
 
  config VFIO_PCI_ZDEV_KVM
-       bool "VFIO PCI extensions for s390x KVM passthrough"
+       def_tristate y
+       prompt "VFIO PCI extensions for s390x KVM passthrough"
         depends on S390 && KVM
-       default y
         help
           Support s390x-specific extensions to enable support for enhancements
           to KVM passthrough capabilities, such as interpretive execution of
 
----

What do you think? It seems to me it solves the problem, what do you think?

Regards,
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#4 lore raw

Randy,

I need to provide the correction patch rapidly.
Without answer I will propose the patch.

Regards,
Pierre

Quoted 70 lines · Pierre Morel
Quoted 26 lines · Randy Dunlap

Hi--

Quoted 12 lines · Pierre Morel

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +0 -0
index 6287a843e8bc..1733339cc4eb 100644

arch/s390/include/asm/kvm_host.h static inline void · line 1043

Quoted 7 lines
arch/s390/include/asm/kvm_host.h +1 -1
@@ -1038,7 +1038,7 @@ static inline void 
 kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
   #define __KVM_HAVE_ARCH_VM_FREE
   void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || 
 defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)

This all looks good except for the line above.
It should be:

Quoted 3 lines

#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)

Thanks.

Yes, better, thanks.
How do we do? Should I repost it with reported-by you or do you want to post it?

Pierre

Quoted 30 lines
Quoted 28 lines
arch/s390/include/asm/kvm_host.h +0 -0
   int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);

  void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev); #else

drivers/vfio/pci/Kconfig +0 -0
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD

  endif

  config VFIO_PCI_ZDEV_KVM
-       bool "VFIO PCI extensions for s390x KVM passthrough"
+       def_tristate y
+       prompt "VFIO PCI extensions for s390x KVM passthrough" depends on S390 && KVM
-       default y
         help
           Support s390x-specific extensions to enable support for enhancements to KVM passthrough capabilities, such as interpretive execution of

----

What do you think? It seems to me it solves the problem, what do you think?

Regards,
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#5 lore raw
Quoted 7 lines · Pierre Morel

Randy,

I need to provide the correction patch rapidly.
Without answer I will propose the patch.

Regards,
Pierre

Please go ahead with it.
Thanks.

Quoted 71 lines · Pierre Morel
Quoted 68 lines · Randy Dunlap
Quoted 24 lines

Hi--

Quoted 12 lines · Pierre Morel

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +0 -0
index 6287a843e8bc..1733339cc4eb 100644

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 5 lines
arch/s390/include/asm/kvm_host.h +1 -1
@@ -1038,7 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
   #define __KVM_HAVE_ARCH_VM_FREE
   void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)

This all looks good except for the line above.
It should be:

Quoted 3 lines

#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)

Thanks.

Yes, better, thanks.
How do we do? Should I repost it with reported-by you or do you want to post it?

Pierre

Quoted 30 lines
Quoted 28 lines
arch/s390/include/asm/kvm_host.h +0 -0
   int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
   void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
   #else
drivers/vfio/pci/Kconfig +0 -0
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD

  endif

  config VFIO_PCI_ZDEV_KVM
-       bool "VFIO PCI extensions for s390x KVM passthrough"
+       def_tristate y
+       prompt "VFIO PCI extensions for s390x KVM passthrough" depends on S390 && KVM
-       default y
         help
           Support s390x-specific extensions to enable support for enhancements to KVM passthrough capabilities, such as interpretive execution of

----

What do you think? It seems to me it solves the problem, what do you think?

Regards,
Pierre

Signature · 1 line
~Randy
maintainer
#6 lore raw
In reply to Pierre Morel
Quoted 27 lines · Pierre Morel
Quoted 24 lines · Randy Dunlap

Hi--

Quoted 12 lines · Pierre Morel

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +0 -0
index 6287a843e8bc..1733339cc4eb 100644

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 5 lines
arch/s390/include/asm/kvm_host.h +1 -1
@@ -1038,7 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
   #define __KVM_HAVE_ARCH_VM_FREE
   void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)

This all looks good except for the line above.
It should be:

Quoted 8 lines
Quoted 3 lines

#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)

Thanks.

Yes, better, thanks.
How do we do? Should I repost it with reported-by you or do you want to post it?

Pierre

Thanks for looking into this while I was away.

I think the issue is not just CONFIG_KVM=m && CONFIG_VFIO_PCI_ZDEV_KVM=y -- it also requires CONFIG_VFIO_PCI=y && CONFIG_VFIO_PCI_CORE=y. This combination results in building in vfio_pci (which tries to link the calls to kvm_s390_pci_register_kvm and kvm_s390_pci_unregister_kvm which is not built in).

However... this tristate + IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) check in kvm_host.h will not solve the issue. Rather, due to the #ifdef CONFIG_VFIO_PCI_ZDEV_KVM in include/linux/vfio_pci_core.h, this change will just cause us to never call kvm_s390_pci_register_kvm or kvm_s390_pci_unregister_kvm when CONFIG_VFIO_PCI_ZDEV_KVM=m, effectively treating CONFIG_VFIO_PCI_ZDEV_KVM=m as a 'n' and we don't get the zdev kvm extensions, which I don't think was the intent.

I'm still thinking & am open to other ideas, but one solution to avoiding building in KVM would be to go back to using symbol_get for these 2 interfaces (kvm_s390_pci_register_kvm and kvm_s390_pci_unregister_kvm) as was done in a prior version of this series like virt/kvm/vfio.c does and otherwise leave CONFIG_VFIO_PCI_ZDEV_KVM as-is.

drivers/vfio/pci/vfio_pci_zdev.c +19 -2
index e163aa9f6144..09c2758134c7 100644
@@ -144,6 +144,8 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
 {
        struct zpci_dev *zdev = to_zpci(vdev->pdev);
+       int (*fn)(struct zpci_dev *zdev, struct kvm *kvm);
+       int rc;
 
        if (!zdev)
                return -ENODEV;
@@ -151,15 +153,30 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
        if (!vdev->vdev.kvm)
                return 0;
 
-       return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+       fn = symbol_get(kvm_s390_pci_register_kvm);
+       if (!fn)
+               return -EPERM;
+
+       rc = fn(zdev, vdev->vdev.kvm);
+
+       symbol_put(kvm_s390_pci_register_kvm);
+
+       return rc;
 }
 
 void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
 {
        struct zpci_dev *zdev = to_zpci(vdev->pdev);
+       void (*fn)(struct zpci_dev *zdev);
 
        if (!zdev || !vdev->vdev.kvm)
                return;
 
-       kvm_s390_pci_unregister_kvm(zdev);
+       fn = symbol_get(kvm_s390_pci_unregister_kvm);
+       if (!fn)
+               return;
+
+       fn(zdev);
+
+       symbol_put(kvm_s390_pci_unregister_kvm);
 }
#7 lore raw
Quoted 83 lines · Matthew Rosato
Quoted 27 lines · Pierre Morel
Quoted 24 lines · Randy Dunlap

Hi--

Quoted 12 lines · Pierre Morel

Thank you Randy for this good catch.
However forcing KVM to be include statically in the kernel when using VFIO_PCI extensions is not a good solution for us I think.

I suggest we better do something like:

----

arch/s390/include/asm/kvm_host.h +0 -0
index 6287a843e8bc..1733339cc4eb 100644

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 5 lines
arch/s390/include/asm/kvm_host.h +1 -1
@@ -1038,7 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
   #define __KVM_HAVE_ARCH_VM_FREE
   void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if defined(CONFIG_VFIO_PCI_ZDEV_KVM) || defined(CONFIG_VFIO_PCI_ZDEV_KVM_MODULE)

This all looks good except for the line above.
It should be:

Quoted 8 lines
Quoted 3 lines

#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)

Thanks.

Yes, better, thanks.
How do we do? Should I repost it with reported-by you or do you want to post it?

Pierre

Thanks for looking into this while I was away.

I think the issue is not just CONFIG_KVM=m && CONFIG_VFIO_PCI_ZDEV_KVM=y -- it also requires CONFIG_VFIO_PCI=y && CONFIG_VFIO_PCI_CORE=y. This combination results in building in vfio_pci (which tries to link the calls to kvm_s390_pci_register_kvm and kvm_s390_pci_unregister_kvm which is not built in).

However... this tristate + IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) check in kvm_host.h will not solve the issue. Rather, due to the #ifdef CONFIG_VFIO_PCI_ZDEV_KVM in include/linux/vfio_pci_core.h, this change will just cause us to never call kvm_s390_pci_register_kvm or kvm_s390_pci_unregister_kvm when CONFIG_VFIO_PCI_ZDEV_KVM=m, effectively treating CONFIG_VFIO_PCI_ZDEV_KVM=m as a 'n' and we don't get the zdev kvm extensions, which I don't think was the intent.

I'm still thinking & am open to other ideas, but one solution to avoiding building in KVM would be to go back to using symbol_get for these 2 interfaces (kvm_s390_pci_register_kvm and kvm_s390_pci_unregister_kvm) as was done in a prior version of this series like virt/kvm/vfio.c does and otherwise leave CONFIG_VFIO_PCI_ZDEV_KVM as-is.

drivers/vfio/pci/vfio_pci_zdev.c +12 -1
index e163aa9f6144..09c2758134c7 100644
@@ -144,6 +144,8 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
  int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
  {
         struct zpci_dev *zdev = to_zpci(vdev->pdev);
+       int (*fn)(struct zpci_dev *zdev, struct kvm *kvm);
+       int rc;
 
         if (!zdev)
                 return -ENODEV;
@@ -151,15 +153,30 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
         if (!vdev->vdev.kvm)
                 return 0;
 
-       return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+       fn = symbol_get(kvm_s390_pci_register_kvm);
+       if (!fn)
+               return -EPERM;
+
+       rc = fn(zdev, vdev->vdev.kvm);
+
+       symbol_put(kvm_s390_pci_register_kvm);
+
+       return rc;
  }
 
  void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
  {
         struct zpci_dev *zdev = to_zpci(vdev->pdev);
+       void (*fn)(struct zpci_dev *zdev);
 
         if (!zdev || !vdev->vdev.kvm)
                 return;

drivers/vfio/pci/vfio_pci_zdev.c int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev) · line 182

Quoted 10 lines
drivers/vfio/pci/vfio_pci_zdev.c +7 -1
 
-       kvm_s390_pci_unregister_kvm(zdev);
+       fn = symbol_get(kvm_s390_pci_unregister_kvm);
+       if (!fn)
+               return;
+
+       fn(zdev);
+
+       symbol_put(kvm_s390_pci_unregister_kvm);
  }

Hello Matt,

In between I came to another solution that seems to satisfy the dependencies. Still need to check that the functionality is still intact :)

I send you the proposition as a reply.

Regards,
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#8 lore raw

PATCH 1/1 KVM: s390: pci: VFIO_PCI ZDEV configuration fix

Fixing configuration for VFIO PCI interpretation.

Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
Fixes c435c54639aa5 ("vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM..")
---
 arch/s390/include/asm/kvm_host.h | 9 ---------
 arch/s390/kvm/Makefile           | 2 +-
 arch/s390/kvm/pci.c              | 4 ++--
 drivers/vfio/pci/Kconfig         | 4 ++--
 include/linux/vfio_pci_core.h    | 2 +-
 5 files changed, 6 insertions(+), 15 deletions(-)
arch/s390/include/asm/kvm_host.h +0 -9
index f39092e0ceaa..f6cf961731af 100644
@@ -1038,16 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
 int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
 void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
 
 #endif
arch/s390/kvm/Makefile +1 -1
index 02217fb4ae10..be36afcfd6ff 100644
@@ -9,6 +9,6 @@ ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
 kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o
 kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o
+kvm-y += pci.o
 
-kvm-$(CONFIG_VFIO_PCI_ZDEV_KVM) += pci.o
 obj-$(CONFIG_KVM) += kvm.o
arch/s390/kvm/pci.c +2 -2
index 4946fb7757d6..cf8ab72a2109 100644
@@ -435,7 +435,7 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
 {
 	int rc;
 
-	if (!zdev)
+	if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) || !zdev)
 		return -EINVAL;
 
 	mutex_lock(&zdev->kzdev_lock);
@@ -516,7 +516,7 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 {
 	struct kvm *kvm;
 
-	if (!zdev)
+	if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) || !zdev)
 		return;
 
 	mutex_lock(&zdev->kzdev_lock);
drivers/vfio/pci/Kconfig +2 -2
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD
 endif
 
 config VFIO_PCI_ZDEV_KVM
-	bool "VFIO PCI extensions for s390x KVM passthrough"
+	def_tristate y
+	prompt "VFIO PCI extensions for s390x KVM passthrough"
 	depends on S390 && KVM
-	default y
 	help
 	  Support s390x-specific extensions to enable support for enhancements
 	  to KVM passthrough capabilities, such as interpretive execution of
include/linux/vfio_pci_core.h +1 -1
index 5579ece4347b..7db3bb8129b1 100644
@@ -205,7 +205,7 @@ static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev)
 }
 #endif
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)
 int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 				struct vfio_info_cap *caps);
 int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev);
Signature · 1 line
2.31.1
maintainer
#9 lore raw

Re: [PATCH] KVM: s390: pci: VFIO_PCI ZDEV configuration fix

Quoted 42 lines · Pierre Morel

Fixing configuration for VFIO PCI interpretation.

Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
Fixes c435c54639aa5 ("vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM..")
---
 arch/s390/include/asm/kvm_host.h | 9 ---------
 arch/s390/kvm/Makefile           | 2 +-
 arch/s390/kvm/pci.c              | 4 ++--
 drivers/vfio/pci/Kconfig         | 4 ++--
 include/linux/vfio_pci_core.h    | 2 +-
 5 files changed, 6 insertions(+), 15 deletions(-)
arch/s390/include/asm/kvm_host.h +0 -9
index f39092e0ceaa..f6cf961731af 100644
@@ -1038,16 +1038,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
 int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
 void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
 
 #endif
arch/s390/kvm/Makefile +0 -0
index 02217fb4ae10..be36afcfd6ff 100644

arch/s390/kvm/Makefile ccflags-y := -Ivirt/kvm -Iarch/s390/kvm · line 13

Quoted 6 lines
arch/s390/kvm/Makefile +1 -1
@@ -9,6 +9,6 @@ ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 
 kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o
 kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o
+kvm-y += pci.o
 
-kvm-$(CONFIG_VFIO_PCI_ZDEV_KVM) += pci.o

You would need to switch this to CONFIG_PCI at least, else we get build errors with CONFIG_PCI=n.

Quoted 44 lines
arch/s390/kvm/Makefile +0 -0
 obj-$(CONFIG_KVM) += kvm.o
arch/s390/kvm/pci.c +2 -2
index 4946fb7757d6..cf8ab72a2109 100644
@@ -435,7 +435,7 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
 {
 	int rc;
 
-	if (!zdev)
+	if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) || !zdev)
 		return -EINVAL;
 
 	mutex_lock(&zdev->kzdev_lock);
@@ -516,7 +516,7 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 {
 	struct kvm *kvm;
 
-	if (!zdev)
+	if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) || !zdev)
 		return;
 
 	mutex_lock(&zdev->kzdev_lock);
drivers/vfio/pci/Kconfig +2 -2
index f9d0c908e738..bbc375b028ef 100644
@@ -45,9 +45,9 @@ config VFIO_PCI_IGD
 endif
 
 config VFIO_PCI_ZDEV_KVM
-	bool "VFIO PCI extensions for s390x KVM passthrough"
+	def_tristate y
+	prompt "VFIO PCI extensions for s390x KVM passthrough"
 	depends on S390 && KVM
-	default y
 	help
 	  Support s390x-specific extensions to enable support for enhancements
 	  to KVM passthrough capabilities, such as interpretive execution of
include/linux/vfio_pci_core.h +0 -0
index 5579ece4347b..7db3bb8129b1 100644

include/linux/vfio_pci_core.h static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev) · line 211

Quoted 8 lines
include/linux/vfio_pci_core.h +1 -1
@@ -205,7 +205,7 @@ static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev)
 }
 #endif
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+#if IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)
 int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 				struct vfio_info_cap *caps);
 int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev);

This still doesn't seem quite right... I tried some variations:

1)
CONFIG_KVM=m
CONFIG_VFIO_PCI_CORE=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_ZDEV_KVM=m

compiles, works with a small change:

include/linux/sched/user.h +1 -1
index f054d0360a75..99734e135420 100644
@@ -25,7 +25,7 @@ struct user_struct {
 
 #if defined(CONFIG_PERF_EVENTS) || defined(CONFIG_BPF_SYSCALL) || \
        defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \
-       defined(CONFIG_VFIO_PCI_ZDEV_KVM)
+       IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)
 
 

2)
CONFIG_KVM=y
CONFIG_VFIO_PCI_CORE=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_ZDEV_KVM=m

compiles, works with above user.h change

3)
CONFIG_KVM=y
CONFIG_VFIO_PCI_CORE=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_ZDEV_KVM=y

compiles, works with above user.h change

4)
CONFIG_KVM=m
CONFIG_VFIO_PCI_CORE=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_ZDEV_KVM=m

fails with:

ld: drivers/vfio/pci/vfio_pci_core.o: in function `vfio_pci_core_enable':
/usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:320: undefined reference to `vfio_pci_zdev_open_device'
ld: /usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:349: undefined reference to `vfio_pci_zdev_close_device'
ld: drivers/vfio/pci/vfio_pci_core.o: in function `vfio_pci_core_disable':
/usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:428: undefined reference to `vfio_pci_zdev_close_device'
ld: drivers/vfio/pci/vfio_pci_core.o: in function `vfio_pci_core_ioctl':
/usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:712: undefined reference to `vfio_pci_info_zdev_add_caps'

5)
CONFIG_KVM=m
CONFIG_VFIO_PCI_CORE=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_ZDEV_KVM=y

This forces CONFIG_VFIO_PCI_ZDEV_KVM to 'm' and fails as above.

#10 lore raw

Re: [PATCH] KVM: s390: pci: VFIO_PCI ZDEV configuration fix

Quoted 21 lines · Matthew Rosato
Quoted 2 lines · Pierre Morel

Fixing configuration for VFIO PCI interpretation.

4)
CONFIG_KVM=m
CONFIG_VFIO_PCI_CORE=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_ZDEV_KVM=m

fails with:

ld: drivers/vfio/pci/vfio_pci_core.o: in function `vfio_pci_core_enable':
/usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:320: undefined reference to `vfio_pci_zdev_open_device'
ld: /usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:349: undefined reference to `vfio_pci_zdev_close_device'
ld: drivers/vfio/pci/vfio_pci_core.o: in function `vfio_pci_core_disable':
/usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:428: undefined reference to `vfio_pci_zdev_close_device'
ld: drivers/vfio/pci/vfio_pci_core.o: in function `vfio_pci_core_ioctl':
/usr/src/linux/drivers/vfio/pci/vfio_pci_core.c:712: undefined reference to `vfio_pci_info_zdev_add_caps'

Quoted 10 lines

5)
CONFIG_KVM=m
CONFIG_VFIO_PCI_CORE=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_ZDEV_KVM=y

This forces CONFIG_VFIO_PCI_ZDEV_KVM to 'm' and fails as above.

yes right
I was really too stupid yesterday
I think you are right and you should go as you proposed or use some pointer hook inside S390 to register with KVM.

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#11 lore raw

PATCH 1/1 KVM: s390: pci: Hook to access KVM lowlevel from VFIO

In reply to Pierre Morel

We have a cross dependency between KVM and VFIO.
To be able to keep both subsystem modular we add a registering hook inside the S390 core code.

This fixes a build problem when VFIO is built-in and KVM is built as a module or excluded.

Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
---
 arch/s390/include/asm/kvm_host.h | 17 ++++++-----------
 arch/s390/kvm/pci.c              | 10 ++++++----
 arch/s390/pci/Makefile           |  2 ++
 arch/s390/pci/pci_kvm_hook.c     | 11 +++++++++++
 drivers/vfio/pci/vfio_pci_zdev.c |  8 ++++++--
 5 files changed, 31 insertions(+), 17 deletions(-)
 create mode 100644 arch/s390/pci/pci_kvm_hook.c
arch/s390/include/asm/kvm_host.h +6 -11
index f39092e0ceaa..8312ed9d1937 100644
@@ -1038,16 +1038,11 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
+struct kvm_register_hook {
+	int (*kvm_register)(void *opaque, struct kvm *kvm);
+	void (*kvm_unregister)(void *opaque);
+};
+
+extern struct kvm_register_hook kvm_pci_hook;
 
 #endif
arch/s390/kvm/pci.c +6 -4
index 4946fb7757d6..e173fce64c4f 100644
@@ -431,8 +431,9 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
  * available, enable them and let userspace indicate whether or not they will
  * be used (specify SHM bit to disable).
  */
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
 {
+	struct zpci_dev *zdev = opaque;
 	int rc;
 
 	if (!zdev)
@@ -510,10 +511,10 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
 	kvm_put_kvm(kvm);
 	return rc;
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
 
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+static void kvm_s390_pci_unregister_kvm(void *opaque)
 {
+	struct zpci_dev *zdev = opaque;
 	struct kvm *kvm;
 
 	if (!zdev)
@@ -566,7 +567,6 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 
 	kvm_put_kvm(kvm);
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_unregister_kvm);
 
 void kvm_s390_pci_init_list(struct kvm *kvm)
 {
@@ -678,6 +678,8 @@ int kvm_s390_pci_init(void)
 
 	spin_lock_init(&aift->gait_lock);
 	mutex_init(&aift->aift_lock);
+	kvm_pci_hook.kvm_register = kvm_s390_pci_register_kvm;
+	kvm_pci_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
 
 	return 0;
 }
arch/s390/pci/Makefile +2 -0
index bf557a1b789c..c02dbfb415d9 100644
@@ -7,3 +7,5 @@ obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
 			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
 			   pci_bus.o
 obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
+
+obj-y += pci_kvm_hook.o
arch/s390/pci/pci_kvm_hook.c new file +11 -0
new file mode 100644
index 000000000000..9d8799b72dbf
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2022.  All rights reserved.
+ *	Author(s): Pierre Morel <[email protected]>
+ */
+#include <linux/kvm_host.h>
+
+struct kvm_register_hook kvm_pci_hook;
+EXPORT_SYMBOL_GPL(kvm_pci_hook);
drivers/vfio/pci/vfio_pci_zdev.c +6 -2
index e163aa9f6144..3b7a707e2fe5 100644
@@ -151,7 +151,10 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
 	if (!vdev->vdev.kvm)
 		return 0;
 
-	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+	if (kvm_pci_hook.kvm_register)
+		return kvm_pci_hook.kvm_register(zdev, vdev->vdev.kvm);
+
+	return -ENOENT;
 }
 
 void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
@@ -161,5 +164,6 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
 	if (!zdev || !vdev->vdev.kvm)
 		return;
 
-	kvm_s390_pci_unregister_kvm(zdev);
+	if (kvm_pci_hook.kvm_unregister)
+		return kvm_pci_hook.kvm_unregister(zdev);
 }
Signature · 1 line
2.31.1
maintainer
#12 lore raw

Re: [PATCH] KVM: s390: pci: Hook to access KVM lowlevel from VFIO

Quoted 1 line · Pierre Morel

We have a cross dependency between KVM and VFIO.

maybe add something like 'when using s390 vfio_pci_zdev extensions for PCI passthrough'

Quoted 5 lines

To be able to keep both subsystem modular we add a registering hook inside the S390 core code.

This fixes a build problem when VFIO is built-in and KVM is built as a module or excluded.

s/or excluded//

There's no problem when KVM is excluded, that forces CONFIG_VFIO_PCI_ZDEV_KVM=n because of the 'depends on S390 && KVM'.

Quoted 25 lines
Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
---
 arch/s390/include/asm/kvm_host.h | 17 ++++++-----------
 arch/s390/kvm/pci.c              | 10 ++++++----
 arch/s390/pci/Makefile           |  2 ++
 arch/s390/pci/pci_kvm_hook.c     | 11 +++++++++++
 drivers/vfio/pci/vfio_pci_zdev.c |  8 ++++++--
 5 files changed, 31 insertions(+), 17 deletions(-)
 create mode 100644 arch/s390/pci/pci_kvm_hook.c
arch/s390/include/asm/kvm_host.h +0 -2
index f39092e0ceaa..8312ed9d1937 100644
@@ -1038,16 +1038,11 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 10 lines
arch/s390/include/asm/kvm_host.h +1 -9
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
+struct kvm_register_hook {

Nit: zpci_kvm_register_hook ? Just to make it clear it's for zpci.

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1046

Quoted 5 lines
arch/s390/include/asm/kvm_host.h +5 -0
+	int (*kvm_register)(void *opaque, struct kvm *kvm);
+	void (*kvm_unregister)(void *opaque);
+};
+
+extern struct kvm_register_hook kvm_pci_hook;

Nit: kvm_zpci_hook ?

Quoted 52 lines
arch/s390/include/asm/kvm_host.h +0 -0
 #endif
arch/s390/kvm/pci.c +6 -4
index 4946fb7757d6..e173fce64c4f 100644
@@ -431,8 +431,9 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
  * available, enable them and let userspace indicate whether or not they will
  * be used (specify SHM bit to disable).
  */
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
 {
+	struct zpci_dev *zdev = opaque;
 	int rc;
 
 	if (!zdev)
@@ -510,10 +511,10 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
 	kvm_put_kvm(kvm);
 	return rc;
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
 
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+static void kvm_s390_pci_unregister_kvm(void *opaque)
 {
+	struct zpci_dev *zdev = opaque;
 	struct kvm *kvm;
 
 	if (!zdev)
@@ -566,7 +567,6 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 
 	kvm_put_kvm(kvm);
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_unregister_kvm);
 
 void kvm_s390_pci_init_list(struct kvm *kvm)
 {
@@ -678,6 +678,8 @@ int kvm_s390_pci_init(void)
 
 	spin_lock_init(&aift->gait_lock);
 	mutex_init(&aift->aift_lock);
+	kvm_pci_hook.kvm_register = kvm_s390_pci_register_kvm;
+	kvm_pci_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
 
 	return 0;
 }
arch/s390/pci/Makefile +0 -0
index bf557a1b789c..c02dbfb415d9 100644

arch/s390/pci/Makefile obj-$(CONFIG_PCI) += pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \ · line 11

Quoted 5 lines
arch/s390/pci/Makefile +2 -0
@@ -7,3 +7,5 @@ obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
 			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
 			   pci_bus.o
 obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
+
+obj-y += pci_kvm_hook.o

I guess it doesn't harm anything to add this unconditionally, but I think it would also be OK to just include this in the CONFIG_PCI list - vfio_pci_zdev and arch/s390/kvm/pci all rely on CONFIG_PCI via CONFIG_VFIO_PCI_ZDEV_KVM which implies PCI via VFIO_PCI.

Quoted 7 lines
arch/s390/pci/pci_kvm_hook.c new file +1 -0
new file mode 100644
index 000000000000..9d8799b72dbf
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-only

arch/s390/pci/pci_kvm_hook.c line 11

Quoted 10 lines
arch/s390/pci/pci_kvm_hook.c new file +10 -0
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2022.  All rights reserved.
+ *	Author(s): Pierre Morel <[email protected]>
+ */
+#include <linux/kvm_host.h>
+
+struct kvm_register_hook kvm_pci_hook;
+EXPORT_SYMBOL_GPL(kvm_pci_hook);

Following the comments above, zpci_kvm_register_hook, kvm_zpci_hook ?

I'm not sure if this really needs to be in a separate file or if it could just go into arch/s390/pci.c with the zpci_aipb -- If going the route of a separate file, up to Niklas whether he wants this under the S390 PCI maintainership or added to the list for s390 vfio-pci like arch/kvm/pci* and vfio_pci_zdev.

Quoted 17 lines
drivers/vfio/pci/vfio_pci_zdev.c +4 -1
index e163aa9f6144..3b7a707e2fe5 100644
@@ -151,7 +151,10 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
 	if (!vdev->vdev.kvm)
 		return 0;
 
-	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+	if (kvm_pci_hook.kvm_register)
+		return kvm_pci_hook.kvm_register(zdev, vdev->vdev.kvm);
+
+	return -ENOENT;
 }
 
 void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)

drivers/vfio/pci/vfio_pci_zdev.c void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev) · line 168

Quoted 6 lines
drivers/vfio/pci/vfio_pci_zdev.c +2 -1
@@ -161,5 +164,6 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
 	if (!zdev || !vdev->vdev.kvm)
 		return;
 
-	kvm_s390_pci_unregister_kvm(zdev);
+	if (kvm_pci_hook.kvm_unregister)
+		return kvm_pci_hook.kvm_unregister(zdev);

No need for the return here, this is a void function calling a void function.

Overall, this looks good to me and survives a series of compile and device passthrough tests on my end, just a matter of a few of these minor comments above. Thanks for tackling this Pierre!

#13 lore raw

Re: [PATCH] KVM: s390: pci: Hook to access KVM lowlevel from VFIO

Quoted 8 lines · Matthew Rosato
Quoted 1 line · Pierre Morel

We have a cross dependency between KVM and VFIO.

maybe add something like 'when using s390 vfio_pci_zdev extensions for PCI passthrough'

Quoted 3 lines

To be able to keep both subsystem modular we add a registering hook inside the S390 core code.

Quoted 6 lines
Quoted 2 lines

This fixes a build problem when VFIO is built-in and KVM is built as a module or excluded.

s/or excluded//

There's no problem when KVM is excluded, that forces CONFIG_VFIO_PCI_ZDEV_KVM=n because of the 'depends on S390 && KVM'.

OK

Quoted 24 lines
Quoted 24 lines
Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
---
  arch/s390/include/asm/kvm_host.h | 17 ++++++-----------
  arch/s390/kvm/pci.c              | 10 ++++++----
  arch/s390/pci/Makefile           |  2 ++
  arch/s390/pci/pci_kvm_hook.c     | 11 +++++++++++
  drivers/vfio/pci/vfio_pci_zdev.c |  8 ++++++--
  5 files changed, 31 insertions(+), 17 deletions(-)

create mode 100644 arch/s390/pci/pci_kvm_hook.c

arch/s390/include/asm/kvm_host.h +0 -0
index f39092e0ceaa..8312ed9d1937 100644
@@ -1038,16 +1038,11 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
  #define __KVM_HAVE_ARCH_VM_FREE
  void kvm_arch_free_vm(struct kvm *kvm);
 
Quoted 14 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 12 lines
arch/s390/include/asm/kvm_host.h +1 -11
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
+struct kvm_register_hook {

Nit: zpci_kvm_register_hook ? Just to make it clear it's for zpci.

OK

Quoted 8 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1046

Quoted 6 lines
arch/s390/include/asm/kvm_host.h +5 -0
+	int (*kvm_register)(void *opaque, struct kvm *kvm);
+	void (*kvm_unregister)(void *opaque);
+};
+
+extern struct kvm_register_hook kvm_pci_hook;

Nit: kvm_zpci_hook ?

OK too,

Quoted 46 lines
Quoted 46 lines
arch/s390/include/asm/kvm_host.h +0 -0
  #endif
arch/s390/kvm/pci.c +6 -4
index 4946fb7757d6..e173fce64c4f 100644
@@ -431,8 +431,9 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
   * available, enable them and let userspace indicate whether or not they will
   * be used (specify SHM bit to disable).
   */
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
  {
+	struct zpci_dev *zdev = opaque;
  	int rc;
 
  	if (!zdev)
@@ -510,10 +511,10 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
  	kvm_put_kvm(kvm);
  	return rc;
  }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
 
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+static void kvm_s390_pci_unregister_kvm(void *opaque)
  {
+	struct zpci_dev *zdev = opaque;
  	struct kvm *kvm;
 
  	if (!zdev)
@@ -566,7 +567,6 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 
  	kvm_put_kvm(kvm);
  }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_unregister_kvm);
 
  void kvm_s390_pci_init_list(struct kvm *kvm)
  {
@@ -678,6 +678,8 @@ int kvm_s390_pci_init(void)
 
  	spin_lock_init(&aift->gait_lock);
  	mutex_init(&aift->aift_lock);
+	kvm_pci_hook.kvm_register = kvm_s390_pci_register_kvm;
+	kvm_pci_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
 
Quoted 14 lines

arch/s390/pci/Makefile obj-$(CONFIG_PCI) += pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \ · line 11

Quoted 12 lines
arch/s390/kvm/pci.c +0 -0
  	return 0;
  }
arch/s390/pci/Makefile +2 -0
index bf557a1b789c..c02dbfb415d9 100644
@@ -7,3 +7,5 @@ obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
  			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
  			   pci_bus.o
  obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
+
+obj-y += pci_kvm_hook.o

I guess it doesn't harm anything to add this unconditionally, but I think it would also be OK to just include this in the CONFIG_PCI list - vfio_pci_zdev and arch/s390/kvm/pci all rely on CONFIG_PCI via CONFIG_VFIO_PCI_ZDEV_KVM which implies PCI via VFIO_PCI.

Right,CONFIG_PCI is a bool so we can put the hook in arch/s390/pci/pci.c and use a defined(CONFIG_PCI) to protect the initialization inside KVM.

Quoted 12 lines
Quoted 12 lines
arch/s390/pci/pci_kvm_hook.c new file +5 -0
new file mode 100644
index 000000000000..9d8799b72dbf
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2022.  All rights reserved.
Quoted 8 lines

arch/s390/pci/pci_kvm_hook.c line 11

Quoted 6 lines
arch/s390/pci/pci_kvm_hook.c new file +6 -0
+ *	Author(s): Pierre Morel <[email protected]>
+ */
+#include <linux/kvm_host.h>
+
+struct kvm_register_hook kvm_pci_hook;
+EXPORT_SYMBOL_GPL(kvm_pci_hook);

Following the comments above, zpci_kvm_register_hook, kvm_zpci_hook ?

OK

Quoted 2 lines

I'm not sure if this really needs to be in a separate file or if it could just go into arch/s390/pci.c with the zpci_aipb -- If going the route of a separate file, up to Niklas whether he wants this under the S390 PCI maintainership or added to the list for s390 vfio-pci like arch/kvm/pci* and vfio_pci_zdev.

agreed no need for a separate file, it is much better.

Quoted 21 lines
Quoted 21 lines
drivers/vfio/pci/vfio_pci_zdev.c +4 -1
index e163aa9f6144..3b7a707e2fe5 100644
@@ -151,7 +151,10 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
  	if (!vdev->vdev.kvm)
  		return 0;
 
-	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+	if (kvm_pci_hook.kvm_register)
+		return kvm_pci_hook.kvm_register(zdev, vdev->vdev.kvm);
+
+	return -ENOENT;
  }
 
  void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
@@ -161,5 +164,6 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
  	if (!zdev || !vdev->vdev.kvm)
  		return;
 
Quoted 5 lines

drivers/vfio/pci/vfio_pci_zdev.c void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev) · line 168

Quoted 3 lines
drivers/vfio/pci/vfio_pci_zdev.c +2 -1
-	kvm_s390_pci_unregister_kvm(zdev);
+	if (kvm_pci_hook.kvm_unregister)
+		return kvm_pci_hook.kvm_unregister(zdev);

No need for the return here, this is a void function calling a void function.

right.

Quoted 4 lines

Overall, this looks good to me and survives a series of compile and device passthrough tests on my end, just a matter of a few of these minor comments above. Thanks for tackling this Pierre!

Thanks,
Pierre

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
maintainer
#14 lore raw

Re: [PATCH] KVM: s390: pci: Hook to access KVM lowlevel from VFIO

In reply to Matthew Rosato
Quoted 29 lines · Matthew Rosato
Quoted 1 line · Pierre Morel

We have a cross dependency between KVM and VFIO.

maybe add something like 'when using s390 vfio_pci_zdev extensions for PCI passthrough'

Quoted 5 lines

To be able to keep both subsystem modular we add a registering hook inside the S390 core code.

This fixes a build problem when VFIO is built-in and KVM is built as a module or excluded.

s/or excluded//

There's no problem when KVM is excluded, that forces CONFIG_VFIO_PCI_ZDEV_KVM=n because of the 'depends on S390 && KVM'.

Quoted 14 lines
Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
---
 arch/s390/include/asm/kvm_host.h | 17 ++++++-----------
 arch/s390/kvm/pci.c              | 10 ++++++----
 arch/s390/pci/Makefile           |  2 ++
 arch/s390/pci/pci_kvm_hook.c     | 11 +++++++++++
 drivers/vfio/pci/vfio_pci_zdev.c |  8 ++++++--
 5 files changed, 31 insertions(+), 17 deletions(-)
 create mode 100644 arch/s390/pci/pci_kvm_hook.c
Quoted 4 lines

arch/s390/include/asm/kvm_host.h

Quoted 4 lines
arch/s390/include/asm/kvm_host.h +0 -0
index f39092e0ceaa..8312ed9d1937 100644

I added Janosch as second S390 KVM maintainer in case he wants to chime in.

Quoted 4 lines
Quoted 4 lines
arch/s390/include/asm/kvm_host.h +0 -0
@@ -1038,16 +1038,11 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
Quoted 14 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 12 lines
arch/s390/include/asm/kvm_host.h +1 -11
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
+struct kvm_register_hook {

Nit: zpci_kvm_register_hook ? Just to make it clear it's for zpci.

Hmm, I guess one could re-use the same struct for another such KVM dependency but I lean towards the same thinking as Matt, for now this is for zpci so stay specific we can always generalize later.

Nit: For me hook and register together sound a bit redudant, maybe "zpci_kvm_register"? Also question for Matt as a native speaker, should it rather be "registration" when used as a noun here?

Quoted 3 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1043

Quoted 3 lines
arch/s390/include/asm/kvm_host.h +2 -0
+	int (*kvm_register)(void *opaque, struct kvm *kvm);
+	void (*kvm_unregister)(void *opaque);

I do wonder if this needs to be opague "struct zpci_dev" should be defined even if CONFIG_PCI is unset.

Quoted 5 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1046

Quoted 3 lines
arch/s390/include/asm/kvm_host.h +3 -0
+};
+
+extern struct kvm_register_hook kvm_pci_hook;

Nit: kvm_zpci_hook ?

Analogous to zpci_kvm_regist(er|ration) I would call the variable simply zpci_kvm i.e. the type is a registration and the variable is the instance of it that links zpci and kvm.

Quoted 81 lines
Quoted 53 lines
arch/s390/include/asm/kvm_host.h +0 -0
 #endif
arch/s390/kvm/pci.c +6 -4
index 4946fb7757d6..e173fce64c4f 100644
@@ -431,8 +431,9 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
  * available, enable them and let userspace indicate whether or not they will
  * be used (specify SHM bit to disable).
  */
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
 {
+	struct zpci_dev *zdev = opaque;
 	int rc;
 
 	if (!zdev)
@@ -510,10 +511,10 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
 	kvm_put_kvm(kvm);
 	return rc;
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
 
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+static void kvm_s390_pci_unregister_kvm(void *opaque)
 {
+	struct zpci_dev *zdev = opaque;
 	struct kvm *kvm;
 
 	if (!zdev)
@@ -566,7 +567,6 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 
 	kvm_put_kvm(kvm);
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_unregister_kvm);
 
 void kvm_s390_pci_init_list(struct kvm *kvm)
 {
@@ -678,6 +678,8 @@ int kvm_s390_pci_init(void)
 
 	spin_lock_init(&aift->gait_lock);
 	mutex_init(&aift->aift_lock);
+	kvm_pci_hook.kvm_register = kvm_s390_pci_register_kvm;
+	kvm_pci_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
 
 	return 0;
 }
arch/s390/pci/Makefile +0 -0
index bf557a1b789c..c02dbfb415d9 100644

arch/s390/pci/Makefile obj-$(CONFIG_PCI) += pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \ · line 11

Quoted 5 lines
arch/s390/pci/Makefile +2 -0
@@ -7,3 +7,5 @@ obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
 			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
 			   pci_bus.o
 obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
+
+obj-y += pci_kvm_hook.o

I guess it doesn't harm anything to add this unconditionally, but I think it would also be OK to just include this in the CONFIG_PCI list - vfio_pci_zdev and arch/s390/kvm/pci all rely on CONFIG_PCI via CONFIG_VFIO_PCI_ZDEV_KVM which implies PCI via VFIO_PCI.

Quoted 7 lines
arch/s390/pci/pci_kvm_hook.c new file +1 -0
new file mode 100644
index 000000000000..9d8799b72dbf
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-only

arch/s390/pci/pci_kvm_hook.c line 11

Quoted 10 lines
arch/s390/pci/pci_kvm_hook.c new file +10 -0
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2022.  All rights reserved.
+ *	Author(s): Pierre Morel <[email protected]>
+ */
+#include <linux/kvm_host.h>
+
+struct kvm_register_hook kvm_pci_hook;
+EXPORT_SYMBOL_GPL(kvm_pci_hook);

Following the comments above, zpci_kvm_register_hook, kvm_zpci_hook ?

Quoted 1 line

I'm not sure if this really needs to be in a separate file or if it could just go into arch/s390/pci.c with the zpci_aipb -- If going the route of a separate file, up to Niklas whether he wants this under the S390 PCI maintainership or added to the list for s390 vfio-pci like arch/kvm/pci* and vfio_pci_zdev.

I'm fine with a separate file, pci.c is long enough as it is. I also don't have a problem with having it maintained as part of S390 PCI but logically I think it does fall more under arch/kvm/pci* so one could argue it should be added in the MAINTAINERS file in that section. If you change the struct name as I proposed above I would probably go with "pci_kvm_register.c"

Quoted 21 lines
Quoted 21 lines
drivers/vfio/pci/vfio_pci_zdev.c +4 -1
index e163aa9f6144..3b7a707e2fe5 100644
@@ -151,7 +151,10 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
 	if (!vdev->vdev.kvm)
 		return 0;
 
-	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+	if (kvm_pci_hook.kvm_register)
+		return kvm_pci_hook.kvm_register(zdev, vdev->vdev.kvm);
+
+	return -ENOENT;
 }
 
 void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
@@ -161,5 +164,6 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
 	if (!zdev || !vdev->vdev.kvm)
 		return;
 
Quoted 8 lines

drivers/vfio/pci/vfio_pci_zdev.c void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev) · line 168

Quoted 3 lines
drivers/vfio/pci/vfio_pci_zdev.c +2 -1
-	kvm_s390_pci_unregister_kvm(zdev);
+	if (kvm_pci_hook.kvm_unregister)
+		return kvm_pci_hook.kvm_unregister(zdev);

No need for the return here, this is a void function calling a void function.

Overall, this looks good to me and survives a series of compile and device passthrough tests on my end, just a matter of a few of these minor comments above. Thanks for tackling this Pierre!

Yes I agree, overall this looks good to me though I'm admittedly not very knowledgable about how to best handle module dependencies like this. It does look cleaner than the symbol_get() alternative we discussed.

maintainer
#15 lore raw

Re: [PATCH] KVM: s390: pci: Hook to access KVM lowlevel from VFIO

Quoted 57 lines · Niklas Schnelle
Quoted 29 lines · Matthew Rosato
Quoted 1 line · Pierre Morel

We have a cross dependency between KVM and VFIO.

maybe add something like 'when using s390 vfio_pci_zdev extensions for PCI passthrough'

Quoted 5 lines

To be able to keep both subsystem modular we add a registering hook inside the S390 core code.

This fixes a build problem when VFIO is built-in and KVM is built as a module or excluded.

s/or excluded//

There's no problem when KVM is excluded, that forces CONFIG_VFIO_PCI_ZDEV_KVM=n because of the 'depends on S390 && KVM'.

Quoted 14 lines
Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
---
 arch/s390/include/asm/kvm_host.h | 17 ++++++-----------
 arch/s390/kvm/pci.c              | 10 ++++++----
 arch/s390/pci/Makefile           |  2 ++
 arch/s390/pci/pci_kvm_hook.c     | 11 +++++++++++
 drivers/vfio/pci/vfio_pci_zdev.c |  8 ++++++--
 5 files changed, 31 insertions(+), 17 deletions(-)
 create mode 100644 arch/s390/pci/pci_kvm_hook.c
Quoted 4 lines

arch/s390/include/asm/kvm_host.h

Quoted 4 lines
arch/s390/include/asm/kvm_host.h +0 -0
index f39092e0ceaa..8312ed9d1937 100644

I added Janosch as second S390 KVM maintainer in case he wants to chime in.

Quoted 4 lines
Quoted 4 lines
arch/s390/include/asm/kvm_host.h +0 -0
@@ -1038,16 +1038,11 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
Quoted 14 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 12 lines
arch/s390/include/asm/kvm_host.h +1 -11
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
+struct kvm_register_hook {

Nit: zpci_kvm_register_hook ? Just to make it clear it's for zpci.

Quoted 3 lines

Hmm, I guess one could re-use the same struct for another such KVM dependency but I lean towards the same thinking as Matt, for now this is for zpci so stay specific we can always generalize later.

Yes, let's keep this zpci-specific.

Quoted 5 lines

Nit: For me hook and register together sound a bit redudant, maybe "zpci_kvm_register"? Also question for Matt as a native speaker, should it rather be "registration" when used as a noun here?

Maybe just drop the 'register'. If there is a need for a 3rd function later, for example, it might not be related to registration.

e.g. struct kvm_zpci_hook {
...
};

extern struct kvm_zpci_hook zpci_kvm;

Quoted 15 lines
Quoted 4 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1043

Quoted 4 lines
arch/s390/include/asm/kvm_host.h +2 -0
+	int (*kvm_register)(void *opaque, struct kvm *kvm);
+	void (*kvm_unregister)(void *opaque);

I do wonder if this needs to be opague "struct zpci_dev" should be defined even if CONFIG_PCI is unset.

Quoted 5 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1046

Quoted 3 lines
arch/s390/include/asm/kvm_host.h +3 -0
+};
+
+extern struct kvm_register_hook kvm_pci_hook;

Nit: kvm_zpci_hook ?

Quoted 4 lines

Analogous to zpci_kvm_regist(er|ration) I would call the variable simply zpci_kvm i.e. the type is a registration and the variable is the instance of it that links zpci and kvm.

Yeah, see above.

Quoted 81 lines
Quoted 81 lines
Quoted 53 lines
arch/s390/include/asm/kvm_host.h +0 -0
 #endif
arch/s390/kvm/pci.c +6 -4
index 4946fb7757d6..e173fce64c4f 100644
@@ -431,8 +431,9 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
  * available, enable them and let userspace indicate whether or not they will
  * be used (specify SHM bit to disable).
  */
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+static int kvm_s390_pci_register_kvm(void *opaque, struct kvm *kvm)
 {
+	struct zpci_dev *zdev = opaque;
 	int rc;
 
 	if (!zdev)
@@ -510,10 +511,10 @@ int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
 	kvm_put_kvm(kvm);
 	return rc;
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
 
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+static void kvm_s390_pci_unregister_kvm(void *opaque)
 {
+	struct zpci_dev *zdev = opaque;
 	struct kvm *kvm;
 
 	if (!zdev)
@@ -566,7 +567,6 @@ void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
 
 	kvm_put_kvm(kvm);
 }
-EXPORT_SYMBOL_GPL(kvm_s390_pci_unregister_kvm);
 
 void kvm_s390_pci_init_list(struct kvm *kvm)
 {
@@ -678,6 +678,8 @@ int kvm_s390_pci_init(void)
 
 	spin_lock_init(&aift->gait_lock);
 	mutex_init(&aift->aift_lock);
+	kvm_pci_hook.kvm_register = kvm_s390_pci_register_kvm;
+	kvm_pci_hook.kvm_unregister = kvm_s390_pci_unregister_kvm;
 
 	return 0;
 }
arch/s390/pci/Makefile +0 -0
index bf557a1b789c..c02dbfb415d9 100644

arch/s390/pci/Makefile obj-$(CONFIG_PCI) += pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \ · line 11

Quoted 5 lines
arch/s390/pci/Makefile +2 -0
@@ -7,3 +7,5 @@ obj-$(CONFIG_PCI)	+= pci.o pci_irq.o pci_dma.o pci_clp.o pci_sysfs.o \
 			   pci_event.o pci_debug.o pci_insn.o pci_mmio.o \
 			   pci_bus.o
 obj-$(CONFIG_PCI_IOV)	+= pci_iov.o
+
+obj-y += pci_kvm_hook.o

I guess it doesn't harm anything to add this unconditionally, but I think it would also be OK to just include this in the CONFIG_PCI list - vfio_pci_zdev and arch/s390/kvm/pci all rely on CONFIG_PCI via CONFIG_VFIO_PCI_ZDEV_KVM which implies PCI via VFIO_PCI.

Quoted 7 lines
arch/s390/pci/pci_kvm_hook.c new file +1 -0
new file mode 100644
index 000000000000..9d8799b72dbf
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-only

arch/s390/pci/pci_kvm_hook.c line 11

Quoted 10 lines
arch/s390/pci/pci_kvm_hook.c new file +10 -0
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2022.  All rights reserved.
+ *	Author(s): Pierre Morel <[email protected]>
+ */
+#include <linux/kvm_host.h>
+
+struct kvm_register_hook kvm_pci_hook;
+EXPORT_SYMBOL_GPL(kvm_pci_hook);

Following the comments above, zpci_kvm_register_hook, kvm_zpci_hook ?

Quoted 8 lines
Quoted 1 line

I'm not sure if this really needs to be in a separate file or if it could just go into arch/s390/pci.c with the zpci_aipb -- If going the route of a separate file, up to Niklas whether he wants this under the S390 PCI maintainership or added to the list for s390 vfio-pci like arch/kvm/pci* and vfio_pci_zdev.

I'm fine with a separate file, pci.c is long enough as it is. I also don't have a problem with having it maintained as part of S390 PCI but logically I think it does fall more under arch/kvm/pci* so one could argue it should be added in the MAINTAINERS file in that section. If you change the struct name as I proposed above I would probably go with "pci_kvm_register.c"

OK, no problem with me for a separate file then, or maintaining said file. But I guess not pci_kvm_register.c per my comments above

Quoted 37 lines
Quoted 22 lines
Quoted 22 lines
drivers/vfio/pci/vfio_pci_zdev.c +4 -1
index e163aa9f6144..3b7a707e2fe5 100644
@@ -151,7 +151,10 @@ int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
 	if (!vdev->vdev.kvm)
 		return 0;
 
-	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+	if (kvm_pci_hook.kvm_register)
+		return kvm_pci_hook.kvm_register(zdev, vdev->vdev.kvm);
+
+	return -ENOENT;
 }
 
 void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
@@ -161,5 +164,6 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
 	if (!zdev || !vdev->vdev.kvm)
 		return;
 
Quoted 8 lines

drivers/vfio/pci/vfio_pci_zdev.c void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev) · line 168

Quoted 3 lines
drivers/vfio/pci/vfio_pci_zdev.c +2 -1
-	kvm_s390_pci_unregister_kvm(zdev);
+	if (kvm_pci_hook.kvm_unregister)
+		return kvm_pci_hook.kvm_unregister(zdev);

No need for the return here, this is a void function calling a void function.

Overall, this looks good to me and survives a series of compile and device passthrough tests on my end, just a matter of a few of these minor comments above. Thanks for tackling this Pierre!

Yes I agree, overall this looks good to me though I'm admittedly not very knowledgable about how to best handle module dependencies like this. It does look cleaner than the symbol_get() alternative we discussed.

maintainer
#16 lore raw

Re: [PATCH] KVM: s390: pci: Hook to access KVM lowlevel from VFIO

Quoted 62 lines · Matthew Rosato
Quoted 57 lines · Niklas Schnelle
Quoted 29 lines · Matthew Rosato
Quoted 1 line · Pierre Morel

We have a cross dependency between KVM and VFIO.

maybe add something like 'when using s390 vfio_pci_zdev extensions for PCI passthrough'

Quoted 5 lines

To be able to keep both subsystem modular we add a registering hook inside the S390 core code.

This fixes a build problem when VFIO is built-in and KVM is built as a module or excluded.

s/or excluded//

There's no problem when KVM is excluded, that forces CONFIG_VFIO_PCI_ZDEV_KVM=n because of the 'depends on S390 && KVM'.

Quoted 14 lines
Fixes 09340b2fca007 ("KVM: s390: pci: add routines to start/stop inter..")
---
 arch/s390/include/asm/kvm_host.h | 17 ++++++-----------
 arch/s390/kvm/pci.c              | 10 ++++++----
 arch/s390/pci/Makefile           |  2 ++
 arch/s390/pci/pci_kvm_hook.c     | 11 +++++++++++
 drivers/vfio/pci/vfio_pci_zdev.c |  8 ++++++--
 5 files changed, 31 insertions(+), 17 deletions(-)
 create mode 100644 arch/s390/pci/pci_kvm_hook.c
Quoted 4 lines

arch/s390/include/asm/kvm_host.h

Quoted 4 lines
arch/s390/include/asm/kvm_host.h +0 -0
index f39092e0ceaa..8312ed9d1937 100644

I added Janosch as second S390 KVM maintainer in case he wants to chime in.

Quoted 4 lines
Quoted 4 lines
arch/s390/include/asm/kvm_host.h +0 -0
@@ -1038,16 +1038,11 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 #define __KVM_HAVE_ARCH_VM_FREE
 void kvm_arch_free_vm(struct kvm *kvm);
 
Quoted 14 lines

arch/s390/include/asm/kvm_host.h static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {} · line 1041

Quoted 12 lines
arch/s390/include/asm/kvm_host.h +1 -11
-#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
-int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
-void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
-#else
-static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
-					    struct kvm *kvm)
-{
-	return -EPERM;
-}
-static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
-#endif
+struct kvm_register_hook {

Nit: zpci_kvm_register_hook ? Just to make it clear it's for zpci.

Quoted 3 lines

Hmm, I guess one could re-use the same struct for another such KVM dependency but I lean towards the same thinking as Matt, for now this is for zpci so stay specific we can always generalize later.

Quoted 8 lines

Yes, let's keep this zpci-specific.

Quoted 4 lines

Nit: For me hook and register together sound a bit redudant, maybe "zpci_kvm_register"? Also question for Matt as a native speaker, should it rather be "registration" when used as a noun here?

Maybe just drop the 'register'. If there is a need for a 3rd function later, for example, it might not be related to registration.

Yes, that sounds good and makes sense so "zpci_kvm_hook".

Quoted 7 lines
e.g. struct kvm_zpci_hook {
   ...
};

extern struct kvm_zpci_hook zpci_kvm;

---8<---

Quoted 23 lines
Quoted 23 lines
Quoted 22 lines
Quoted 8 lines
arch/s390/pci/pci_kvm_hook.c new file +1 -0
new file mode 100644
index 000000000000..9d8799b72dbf
@@ -0,0 +1,11 @@
+// SPDX-License-Identifier: GPL-2.0-only

arch/s390/pci/pci_kvm_hook.c line 11

Quoted 10 lines
arch/s390/pci/pci_kvm_hook.c new file +10 -0
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2022.  All rights reserved.
+ *	Author(s): Pierre Morel <[email protected]>
+ */
+#include <linux/kvm_host.h>
+
+struct kvm_register_hook kvm_pci_hook;
+EXPORT_SYMBOL_GPL(kvm_pci_hook);

Following the comments above, zpci_kvm_register_hook, kvm_zpci_hook ?

I'm not sure if this really needs to be in a separate file or if it could just go into arch/s390/pci.c with the zpci_aipb -- If going the route of a separate file, up to Niklas whether he wants this under the S390 PCI maintainership or added to the list for s390 vfio-pci like arch/kvm/pci* and vfio_pci_zdev.

Quoted 8 lines
Quoted 6 lines

I'm fine with a separate file, pci.c is long enough as it is. I also don't have a problem with having it maintained as part of S390 PCI but logically I think it does fall more under arch/kvm/pci* so one could argue it should be added in the MAINTAINERS file in that section. If you change the struct name as I proposed above I would probably go with "pci_kvm_register.c"

OK, no problem with me for a separate file then, or maintaining said file. But I guess not pci_kvm_register.c per my comments above

Yes, let's go with pci_kvm_hook.c then

Quoted 1 line