Skip to contentSkip to conversation

Retrieving zPCI specific info with VFIO

PATCH In review Pierre Morel 3 participants 9 messages May 23, 2019 – May 28, 2019

Subsystems (5)
KVM s390 VFIO driver s390 PCI +1 more subsystems
lore
Share & tools
lore
#0 lore raw

v3 cover Retrieving zPCI specific info with VFIO

We define a new configuration entry for VFIO/PCI, VFIO_PCI_ZDEV to configure access to a zPCI region dedicated for retrieving zPCI features.

When the VFIO_PCI_ZDEV feature is configured we initialize a new device region, VFIO_REGION_SUBTYPE_ZDEV_CLP, to hold the information from the ZPCI device the userland needs to give to a guest driving the zPCI function.

Note that in the current state we do not use the CLP instructions to access the firmware but get the information directly from the zdev device.

-This means that the patch 1, "s390: pci: Exporting access to CLP PCI function and PCI group" is not used and can be let out of this series without denying the good working of the other patches.
- But we will need this later, eventually in the next iteration to retrieve values not being saved inside the zdev structure. like maxstbl and the PCI supported version

To share the code with arch/s390/pci/pci_clp.c the original functions in pci_clp.c to query PCI functions and PCI functions group are modified so that they can be exported.

A new function clp_query_pci() replaces clp_query_pci_fn() and the previous calls to clp_query_pci_fn() and clp_query_pci_fngrp() are replaced with calls to zdev_query_pci_fn() and zdev_query_pci_fngrp() using a zdev pointer as argument.

Pierre Morel (3):
s390: pci: Exporting access to CLP PCI function and PCI group
vfio: zpci: defining the VFIO headers
vfio: pci: Using a device region to retrieve zPCI information

 arch/s390/include/asm/pci.h         |  3 ++
 arch/s390/pci/pci_clp.c             | 70 ++++++++++++++++---------------
 drivers/vfio/pci/Kconfig            |  7 ++++
 drivers/vfio/pci/Makefile           |  1 +
 drivers/vfio/pci/vfio_pci.c         |  9 ++++
 drivers/vfio/pci/vfio_pci_private.h | 10 +++++
 drivers/vfio/pci/vfio_pci_zdev.c    | 83 +++++++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h           |  4 ++
 include/uapi/linux/vfio_zdev.h      | 34 +++++++++++++++
 9 files changed, 187 insertions(+), 34 deletions(-)
 create mode 100644 drivers/vfio/pci/vfio_pci_zdev.c
 create mode 100644 include/uapi/linux/vfio_zdev.h
Signature · 1 line
2.7.4
#1 lore raw

PATCH v3 1/3 s390: pci: Exporting access to CLP PCI function and PCI group

For the generic implementation of VFIO PCI we need to retrieve the hardware configuration for the PCI functions and the PCI function groups.

We modify the internal function using CLP Query PCI function and CLP query PCI function group so that they can be called from outside the S390 architecture PCI code and prefix the two functions with "zdev" to make clear that they can be called knowing only the associated zdevice.

---
 arch/s390/include/asm/pci.h |  3 ++
 arch/s390/pci/pci_clp.c     | 70 +++++++++++++++++++++++----------------------
 2 files changed, 39 insertions(+), 34 deletions(-)
arch/s390/include/asm/pci.h +3 -0
index 305befd..e66b246 100644
@@ -261,4 +261,7 @@ cpumask_of_pcibus(const struct pci_bus *bus)
 
 #endif /* CONFIG_NUMA */
 
+int zdev_query_pci_fngrp(struct zpci_dev *zdev,
+			 struct clp_req_rsp_query_pci_grp *rrb);
+int zdev_query_pci_fn(struct zpci_dev *zdev, struct clp_req_rsp_query_pci *rrb);
 #endif
arch/s390/pci/pci_clp.c +36 -34

106 lines not on this page · raw message

Signature · 1 line
2.7.4
#2 lore raw

PATCH v3 2/3 vfio: zpci: defining the VFIO headers

In reply to Pierre Morel

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userland.

We create a new file, vfio_zdev.h to define the structure of the new region we defined in vfio.h

---
 include/uapi/linux/vfio.h      |  4 ++++
 include/uapi/linux/vfio_zdev.h | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +4 -0
index 8f10748..56595b8 100644
@@ -306,6 +306,10 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_TYPE_GFX                    (1)
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
+/* IBM Subtypes */
+#define VFIO_REGION_TYPE_IBM_ZDEV		(1)
+#define VFIO_REGION_SUBTYPE_ZDEV_CLP		(1)
+
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
  *
include/uapi/linux/vfio_zdev.h new file +34 -0
new file mode 100644
index 0000000..84b1a82
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2019
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u64 dasm;
+	__u64 start_dma;
+	__u64 end_dma;
+	__u64 msi_addr;
+	__u64 flags;
+	__u16 pchid;
+	__u16 mui;
+	__u16 noi;
+	__u8 gid;
+	__u8 version;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1
+	__u8 util_str[CLP_UTIL_STR_LEN];
+} __packed;
+
+#endif
Signature · 1 line
2.7.4
#3 lore raw

Re: [PATCH v3 2/3] vfio: zpci: defining the VFIO headers

Quoted 19 lines · On Thu, 23 May 2019 14:25:25 +0200 Pierre Morel <[email protected]> wrote:

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userland.

We create a new file, vfio_zdev.h to define the structure of the new region we defined in vfio.h

---
 include/uapi/linux/vfio.h      |  4 ++++
 include/uapi/linux/vfio_zdev.h | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +0 -0
index 8f10748..56595b8 100644

include/uapi/linux/vfio.h struct vfio_region_info_cap_type { · line 311

Quoted 6 lines
include/uapi/linux/vfio.h +3 -0
@@ -306,6 +306,10 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_TYPE_GFX                    (1)
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
+/* IBM Subtypes */
+#define VFIO_REGION_TYPE_IBM_ZDEV		(1)
+#define VFIO_REGION_SUBTYPE_ZDEV_CLP		(1)

I'm afraid that confuses me a bit. You want to add the region to every vfio-pci device when we're running under s390, right? So this does not depend on the device type of the actual device (which may or may not be from IBM), but only on the architecture?

(Generally speaking, I think using regions for this makes sense, though.)

Quoted 44 lines
include/uapi/linux/vfio.h +1 -0
+
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
  *
include/uapi/linux/vfio_zdev.h new file +34 -0
new file mode 100644
index 0000000..84b1a82
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2019
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u64 dasm;
+	__u64 start_dma;
+	__u64 end_dma;
+	__u64 msi_addr;
+	__u64 flags;
+	__u16 pchid;
+	__u16 mui;
+	__u16 noi;
+	__u8 gid;
+	__u8 version;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1
+	__u8 util_str[CLP_UTIL_STR_LEN];
+} __packed;
+
+#endif
#4 lore raw

Re: [PATCH v3 2/3] vfio: zpci: defining the VFIO headers

Quoted 25 lines · On Thu, 23 May 2019 18:24:33 +0200 Cornelia Huck <[email protected]> wrote:
Quoted 22 lines · On Thu, 23 May 2019 14:25:25 +0200 Pierre Morel <[email protected]> wrote:

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userland.

We create a new file, vfio_zdev.h to define the structure of the new region we defined in vfio.h

---
 include/uapi/linux/vfio.h      |  4 ++++
 include/uapi/linux/vfio_zdev.h | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +0 -0
index 8f10748..56595b8 100644
@@ -306,6 +306,10 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_TYPE_GFX                    (1)
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
Quoted 8 lines

include/uapi/linux/vfio.h struct vfio_region_info_cap_type { · line 311

Quoted 3 lines
include/uapi/linux/vfio.h +3 -0
+/* IBM Subtypes */
+#define VFIO_REGION_TYPE_IBM_ZDEV		(1)
+#define VFIO_REGION_SUBTYPE_ZDEV_CLP		(1)  

I'm afraid that confuses me a bit. You want to add the region to every vfio-pci device when we're running under s390, right? So this does not depend on the device type of the actual device (which may or may not be from IBM), but only on the architecture?

FWIW, I don't really have a strong opinion here but I welcome the discussion. It seems fair to me that a PCI vendor type could be used for either a device with that vendor ID or by the vendor of the platform. We've got a lot of address space if want to use VFIO_REGION_TYPE_IBM_ZDEV rather than a PCI vendor type (so long as it's updated not to conflict with the GFX type). Thanks,

Alex

Quoted 48 lines

(Generally speaking, I think using regions for this makes sense, though.)

Quoted 44 lines
include/uapi/linux/vfio.h +1 -0
+
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
  *
(quoted diff) +0 -0

b/include/uapi/linux/vfio_zdev.h new file mode 100644 index 0000000..84b1a82

include/uapi/linux/vfio_zdev.h new file +34 -0
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2019
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u64 dasm;
+	__u64 start_dma;
+	__u64 end_dma;
+	__u64 msi_addr;
+	__u64 flags;
+	__u16 pchid;
+	__u16 mui;
+	__u16 noi;
+	__u8 gid;
+	__u8 version;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1
+	__u8 util_str[CLP_UTIL_STR_LEN];
+} __packed;
+
+#endif  
#5 lore raw

Re: [PATCH v3 2/3] vfio: zpci: defining the VFIO headers

In reply to Pierre Morel
Quoted 19 lines · On Thu, 23 May 2019 14:25:25 +0200 Pierre Morel <[email protected]> wrote:

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userland.

We create a new file, vfio_zdev.h to define the structure of the new region we defined in vfio.h

---
 include/uapi/linux/vfio.h      |  4 ++++
 include/uapi/linux/vfio_zdev.h | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +0 -0
index 8f10748..56595b8 100644

include/uapi/linux/vfio.h struct vfio_region_info_cap_type { · line 310

Quoted 5 lines
include/uapi/linux/vfio.h +2 -0
@@ -306,6 +306,10 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_TYPE_GFX                    (1)
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
+/* IBM Subtypes */
+#define VFIO_REGION_TYPE_IBM_ZDEV		(1)

This one defines (but never uses) a conflicting region type to GFX above.

include/uapi/linux/vfio.h struct vfio_region_info_cap_type { · line 311

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+#define VFIO_REGION_SUBTYPE_ZDEV_CLP		(1)

If we're using a PCI vendor type, which the next patch indicates we are, this is the one you need. But please also specify it as a PCI vendor sub-type with the hex PCI vendor ID, and perhaps group it with the Intel vendor sub-types.

Quoted 31 lines
include/uapi/linux/vfio.h +1 -0
+
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
  *
include/uapi/linux/vfio_zdev.h new file +21 -0
new file mode 100644
index 0000000..84b1a82
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2019
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u64 dasm;
+	__u64 start_dma;

include/uapi/linux/vfio_zdev.h line 31

Quoted 10 lines
include/uapi/linux/vfio_zdev.h new file +10 -0
+	__u64 end_dma;
+	__u64 msi_addr;
+	__u64 flags;
+	__u16 pchid;
+	__u16 mui;
+	__u16 noi;
+	__u8 gid;
+	__u8 version;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1
+	__u8 util_str[CLP_UTIL_STR_LEN];

I don't see where CLP_UTIL_STR_LEN is defined in a uapi consumable header. Should this simply be [] where the string length is implied by the remaining region size? QEMU hard codes it, that doesn't validate the vfio uapi.

Quoted 3 lines
include/uapi/linux/vfio_zdev.h new file +3 -0
+} __packed;
+
+#endif
#6 lore raw

Re: [PATCH v3 2/3] vfio: zpci: defining the VFIO headers

Quoted 31 lines · On Tue, 28 May 2019 15:43:42 -0600 Alex Williamson…
Quoted 19 lines · On Thu, 23 May 2019 14:25:25 +0200 Pierre Morel <[email protected]> wrote:

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userland.

We create a new file, vfio_zdev.h to define the structure of the new region we defined in vfio.h

---
 include/uapi/linux/vfio.h      |  4 ++++
 include/uapi/linux/vfio_zdev.h | 34 ++++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +0 -0
index 8f10748..56595b8 100644

include/uapi/linux/vfio.h struct vfio_region_info_cap_type { · line 310

Quoted 5 lines
include/uapi/linux/vfio.h +2 -0
@@ -306,6 +306,10 @@ struct vfio_region_info_cap_type {
 #define VFIO_REGION_TYPE_GFX                    (1)
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
 
+/* IBM Subtypes */
+#define VFIO_REGION_TYPE_IBM_ZDEV		(1)  

This one defines (but never uses) a conflicting region type to GFX above.

Quoted 6 lines

include/uapi/linux/vfio.h struct vfio_region_info_cap_type { · line 311

Quoted 1 line
include/uapi/linux/vfio.h +1 -0
+#define VFIO_REGION_SUBTYPE_ZDEV_CLP		(1)  

If we're using a PCI vendor type, which the next patch indicates we are, this is the one you need. But please also specify it as a PCI vendor sub-type with the hex PCI vendor ID, and perhaps group it with the Intel vendor sub-types.

BTW, we've already started a set of IBM sub-types:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/include/uapi/linux/vfio.h?id=7f92891778dff62303c070ac81de7b7d80de331a

Quoted 51 lines
Quoted 31 lines
include/uapi/linux/vfio.h +1 -0
+
 /**
  * struct vfio_region_gfx_edid - EDID region layout.
  *
include/uapi/linux/vfio_zdev.h new file +21 -0
new file mode 100644
index 0000000..84b1a82
@@ -0,0 +1,34 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2019
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u64 dasm;
+	__u64 start_dma;

include/uapi/linux/vfio_zdev.h line 31

Quoted 10 lines
include/uapi/linux/vfio_zdev.h new file +10 -0
+	__u64 end_dma;
+	__u64 msi_addr;
+	__u64 flags;
+	__u16 pchid;
+	__u16 mui;
+	__u16 noi;
+	__u8 gid;
+	__u8 version;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1
+	__u8 util_str[CLP_UTIL_STR_LEN];  

I don't see where CLP_UTIL_STR_LEN is defined in a uapi consumable header. Should this simply be [] where the string length is implied by the remaining region size? QEMU hard codes it, that doesn't validate the vfio uapi.

Quoted 3 lines
include/uapi/linux/vfio_zdev.h new file +3 -0
+} __packed;
+
+#endif  
#7 lore raw

PATCH v3 3/3 vfio: pci: Using a device region to retrieve zPCI information

In reply to Pierre Morel

We define a new configuration entry for VFIO/PCI, VFIO_PCI_ZDEV

When the VFIO_PCI_ZDEV feature is configured we initialize a new device region, VFIO_REGION_SUBTYPE_ZDEV_CLP, to hold the information from the ZPCI device the userland needs to give to a guest driving the zPCI function.

---
 drivers/vfio/pci/Kconfig            |  7 ++++
 drivers/vfio/pci/Makefile           |  1 +
 drivers/vfio/pci/vfio_pci.c         |  9 ++++
 drivers/vfio/pci/vfio_pci_private.h | 10 +++++
 drivers/vfio/pci/vfio_pci_zdev.c    | 83 +++++++++++++++++++++++++++++++++++++
 5 files changed, 110 insertions(+)
 create mode 100644 drivers/vfio/pci/vfio_pci_zdev.c
drivers/vfio/pci/Kconfig +7 -0
index d0f8e4f..9c1181c 100644
@@ -44,3 +44,10 @@ config VFIO_PCI_NVLINK2
 	depends on VFIO_PCI && PPC_POWERNV
 	help
 	  VFIO PCI support for P9 Witherspoon machine with NVIDIA V100 GPUs
+
+config VFIO_PCI_ZDEV
+	tristate "VFIO PCI Generic for ZPCI devices"
+	depends on VFIO_PCI && S390
+	default y
+	help
+	  VFIO PCI support for S390 Z-PCI devices
drivers/vfio/pci/Makefile +1 -0
index 9662c06..fd53819 100644
@@ -2,5 +2,6 @@
 vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
 vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
 vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o
+vfio-pci-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o
 
 obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
drivers/vfio/pci/vfio_pci.c +9 -0
index 3fa20e9..b6087d6 100644
@@ -362,6 +362,15 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev)
 		}
 	}
 
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV)) {
+		ret = vfio_pci_zdev_init(vdev);
+		if (ret) {
+			dev_warn(&vdev->pdev->dev,
+				 "Failed to setup ZDEV regions\n");
+			goto disable_exit;
+		}
+	}
+
 	vfio_pci_probe_mmaps(vdev);
 
 	return 0;
drivers/vfio/pci/vfio_pci_private.h +10 -0
index 1812cf2..db73cdf 100644
@@ -189,4 +189,14 @@ static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
 	return -ENODEV;
 }
 #endif
+
+#ifdef(IS_ENABLED_VFIO_PCI_ZDEV)
+extern int vfio_pci_zdev_init(struct vfio_pci_device *vdev);
+#else
+static inline int vfio_pci_zdev_init(struct vfio_pci_device *vdev)
+{
+	return -ENODEV;
+}
+#endif
+
 #endif /* VFIO_PCI_PRIVATE_H */
drivers/vfio/pci/vfio_pci_zdev.c new file +83 -0

83 lines not on this page · raw message

Signature · 1 line
2.7.4
#8 lore raw

Re: [PATCH v3 3/3] vfio: pci: Using a device region to retrieve zPCI information

Quoted 22 lines · On Thu, 23 May 2019 14:25:26 +0200 Pierre Morel <[email protected]> wrote:

We define a new configuration entry for VFIO/PCI, VFIO_PCI_ZDEV

When the VFIO_PCI_ZDEV feature is configured we initialize a new device region, VFIO_REGION_SUBTYPE_ZDEV_CLP, to hold the information from the ZPCI device the userland needs to give to a guest driving the zPCI function.

---
 drivers/vfio/pci/Kconfig            |  7 ++++
 drivers/vfio/pci/Makefile           |  1 +
 drivers/vfio/pci/vfio_pci.c         |  9 ++++
 drivers/vfio/pci/vfio_pci_private.h | 10 +++++
 drivers/vfio/pci/vfio_pci_zdev.c    | 83 +++++++++++++++++++++++++++++++++++++
 5 files changed, 110 insertions(+)
 create mode 100644 drivers/vfio/pci/vfio_pci_zdev.c
drivers/vfio/pci/Kconfig +0 -0
index d0f8e4f..9c1181c 100644

drivers/vfio/pci/Kconfig config VFIO_PCI_NVLINK2 · line 49

Quoted 6 lines
drivers/vfio/pci/Kconfig +3 -0
@@ -44,3 +44,10 @@ config VFIO_PCI_NVLINK2
 	depends on VFIO_PCI && PPC_POWERNV
 	help
 	  VFIO PCI support for P9 Witherspoon machine with NVIDIA V100 GPUs
+
+config VFIO_PCI_ZDEV
+	tristate "VFIO PCI Generic for ZPCI devices"

Shouldn't this be 'bool'?

Quoted 40 lines
drivers/vfio/pci/Kconfig +4 -0
+	depends on VFIO_PCI && S390
+	default y
+	help
+	  VFIO PCI support for S390 Z-PCI devices
drivers/vfio/pci/Makefile +1 -0
index 9662c06..fd53819 100644
@@ -2,5 +2,6 @@
 vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
 vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
 vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o
+vfio-pci-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o
 
 obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
drivers/vfio/pci/vfio_pci.c +9 -0
index 3fa20e9..b6087d6 100644
@@ -362,6 +362,15 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev)
 		}
 	}
 
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV)) {
+		ret = vfio_pci_zdev_init(vdev);
+		if (ret) {
+			dev_warn(&vdev->pdev->dev,
+				 "Failed to setup ZDEV regions\n");
+			goto disable_exit;
+		}
+	}
+
 	vfio_pci_probe_mmaps(vdev);
 
 	return 0;
drivers/vfio/pci/vfio_pci_private.h +0 -0
index 1812cf2..db73cdf 100644

drivers/vfio/pci/vfio_pci_private.h static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev) · line 193

Quoted 5 lines
drivers/vfio/pci/vfio_pci_private.h +2 -0
@@ -189,4 +189,14 @@ static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
 	return -ENODEV;
 }
 #endif
+
+#ifdef(IS_ENABLED_VFIO_PCI_ZDEV)

I thought this might be some clever new macro, but is it just a typo? Seems it should just be

#ifdef CONFIG_VFIO_PCI_ZDEV

Quoted 42 lines
drivers/vfio/pci/vfio_pci_private.h +8 -0
+extern int vfio_pci_zdev_init(struct vfio_pci_device *vdev);
+#else
+static inline int vfio_pci_zdev_init(struct vfio_pci_device *vdev)
+{
+	return -ENODEV;
+}
+#endif
+
 #endif /* VFIO_PCI_PRIVATE_H */
drivers/vfio/pci/vfio_pci_zdev.c new file +27 -0
new file mode 100644
index 0000000..230a4e4
@@ -0,0 +1,83 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * VFIO ZPCI devices support
+ *
+ * Copyright (C) IBM Corp. 2019.  All rights reserved.
+ *	Author: Pierre Morel <[email protected]>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+#include <linux/io.h>
+#include <linux/pci.h>
+#include <linux/uaccess.h>
+#include <linux/vfio.h>
+#include <linux/vfio_zdev.h>
+
+#include "vfio_pci_private.h"
+
+static size_t vfio_pci_zdev_rw(struct vfio_pci_device *vdev,
+			       char __user *buf, size_t count, loff_t *ppos,
+			       bool iswrite)
+{
+	struct vfio_region_zpci_info *region;
+	struct zpci_dev *zdev;
+	unsigned int index = VFIO_PCI_OFFSET_TO_INDEX(*ppos);

drivers/vfio/pci/vfio_pci_zdev.c line 37

Quoted 10 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +10 -0
+
+	if (!vdev->pdev->bus)
+		return -ENODEV;
+
+	zdev = vdev->pdev->bus->sysdata;
+	if (!zdev)
+		return -ENODEV;
+
+	if ((*ppos & VFIO_PCI_OFFSET_MASK) || (count != sizeof(*region)))
+		return -EINVAL;

Why? This sort of restriction would need to be documented in the ABI.

drivers/vfio/pci/vfio_pci_zdev.c line 48

Quoted 11 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +11 -0
+
+	region = vdev->region[index - VFIO_PCI_NUM_REGIONS].data;
+	region->dasm = zdev->dma_mask;
+	region->start_dma = zdev->start_dma;
+	region->end_dma = zdev->end_dma;
+	region->msi_addr = zdev->msi_addr;
+	region->flags = VFIO_PCI_ZDEV_FLAGS_REFRESH;
+	region->gid = zdev->pfgid;
+	region->mui = zdev->fmb_update;
+	region->noi = zdev->max_msi;
+	memcpy(region->util_str, zdev->util_str, CLP_UTIL_STR_LEN);

Does anything here change? Why not do this in the init function?

drivers/vfio/pci/vfio_pci_zdev.c line 51

Quoted 2 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +2 -0
+	if (copy_to_user(buf, region, count))
+		return -EFAULT;

It's really not that difficult to make this support arbitrary reads.

Quoted 16 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +16 -0
+
+	return count;
+}
+
+static void vfio_pci_zdev_release(struct vfio_pci_device *vdev,
+				  struct vfio_pci_region *region)
+{
+	kfree(region->data);
+}
+
+static const struct vfio_pci_regops vfio_pci_zdev_regops = {
+	.rw		= vfio_pci_zdev_rw,
+	.release	= vfio_pci_zdev_release,
+};
+
+int vfio_pci_zdev_init(struct vfio_pci_device *vdev)

drivers/vfio/pci/vfio_pci_zdev.c line 77

Quoted 10 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +10 -0
+{
+	struct vfio_region_zpci_info *region;
+	int ret;
+
+	region = kmalloc(sizeof(*region), GFP_KERNEL);
+	if (!region)
+		return -ENOMEM;
+
+	ret = vfio_pci_register_dev_region(vdev,
+		PCI_VENDOR_ID_IBM | VFIO_REGION_TYPE_PCI_VENDOR_TYPE,

The uapi should specify 0x1014 as the vendor ID to eliminate any confusion.

drivers/vfio/pci/vfio_pci_zdev.c line 79

Quoted 2 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +2 -0
+		VFIO_REGION_SUBTYPE_ZDEV_CLP,
+		&vfio_pci_zdev_regops, sizeof(*region),

'sizeof(*region) + CLP_UTIL_STR_LEN' if suggestion in previous patch is used.

drivers/vfio/pci/vfio_pci_zdev.c line 80

Quoted 1 line
drivers/vfio/pci/vfio_pci_zdev.c new file +1 -0
+		VFIO_REGION_INFO_FLAG_READ, region);

This FLAG_READ only tells the user what is supported, it's up to your .rw callback to reject iswrite.

Quoted 3 lines
drivers/vfio/pci/vfio_pci_zdev.c new file +3 -0
+
+	return ret;
+}