Skip to contentSkip to conversation

Pass zPCI hardware information via VFIO

PATCH Superseded Matthew Rosato 3 participants 14 messages Oct 2, 2020 – Oct 6, 2020

Subsystems (5)
KVM s390 VFIO driver s390 PCI +1 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Matthew Rosato

author maintainer
#0 lore raw

v2 cover Pass zPCI hardware information via VFIO

This patchset provides a means by which hardware information about the underlying PCI device can be passed up to userspace (ie, QEMU) so that this hardware information can be used rather than previously hard-coded assumptions. A new VFIO region type is defined which holds this information.

A form of these patches saw some rounds last year but has been back- tabled for a while. The original work for this feature was done by Pierre Morel. I'd like to refresh the discussion on this and get this finished up so that we can move forward with better-supporting additional types of PCI-attached devices. The proposal here presents a completely different region mapping vs the prior approach, taking inspiration from vfio info capability chains to provide device CLP information in a way that allows for future expansion (new CLP features).

This feature is toggled via the CONFIG_VFIO_PCI_ZDEV configuration entry.

Changes from v1:
- Added ACKs (thanks!)
- Patch 2: Minor change:s/util_avail/util_str_avail/ per Niklas
- Patch 3: removed __packed
- Patch 3: rework various descriptions / comment blocks
- New patch: MAINTAINERS hit to cover new files.

Matthew Rosato (5):
s390/pci: stash version in the zpci_dev
s390/pci: track whether util_str is valid in the zpci_dev vfio-pci/zdev: define the vfio_zdev header vfio-pci/zdev: use a device region to retrieve zPCI information
MAINTAINERS: Add entry for s390 vfio-pci

 MAINTAINERS                         |   8 ++
 arch/s390/include/asm/pci.h         |   4 +-
 arch/s390/pci/pci_clp.c             |   2 +
 drivers/vfio/pci/Kconfig            |  13 ++
 drivers/vfio/pci/Makefile           |   1 +
 drivers/vfio/pci/vfio_pci.c         |   8 ++
 drivers/vfio/pci/vfio_pci_private.h |  10 ++
 drivers/vfio/pci/vfio_pci_zdev.c    | 242 ++++++++++++++++++++++++++++++++++++
 include/uapi/linux/vfio.h           |   5 +
 include/uapi/linux/vfio_zdev.h      | 118 ++++++++++++++++++
 10 files changed, 410 insertions(+), 1 deletion(-)
 create mode 100644 drivers/vfio/pci/vfio_pci_zdev.c
 create mode 100644 include/uapi/linux/vfio_zdev.h
Signature · 1 line
1.8.3.1
author maintainer
#1 lore raw

PATCH v2 1/5 s390/pci: stash version in the zpci_dev

In preparation for passing the info on to vfio-pci devices, stash the supported PCI version for the target device in the zpci_dev.

---
 arch/s390/include/asm/pci.h | 1 +
 arch/s390/pci/pci_clp.c     | 1 +
 2 files changed, 2 insertions(+)
arch/s390/include/asm/pci.h +1 -0
index 99b92c3..882e233 100644
@@ -179,6 +179,7 @@ struct zpci_dev {
 	atomic64_t mapped_pages;
 	atomic64_t unmapped_pages;
 
+	u8		version;
 	enum pci_bus_speed max_bus_speed;
 
 	struct dentry	*debugfs_dev;
arch/s390/pci/pci_clp.c +1 -0
index 7e735f4..48bf316 100644
@@ -102,6 +102,7 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev,
 	zdev->msi_addr = response->msia;
 	zdev->max_msi = response->noi;
 	zdev->fmb_update = response->mui;
+	zdev->version = response->version;
 
 	switch (response->version) {
 	case 1:
Signature · 1 line
1.8.3.1
author maintainer
#2 lore raw

PATCH v2 2/5 s390/pci: track whether util_str is valid in the zpci_dev

In reply to Matthew Rosato

We'll need to keep track of whether or not the byte string in util_str is valid and thus needs to be passed to a vfio-pci passthrough device.

---
 arch/s390/include/asm/pci.h | 3 ++-
 arch/s390/pci/pci_clp.c     | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)
arch/s390/include/asm/pci.h +2 -1
index 882e233..fa1fed4 100644
@@ -132,7 +132,8 @@ struct zpci_dev {
 	u8		rid_available	: 1;
 	u8		has_hp_slot	: 1;
 	u8		is_physfn	: 1;
-	u8		reserved	: 5;
+	u8		util_str_avail	: 1;
+	u8		reserved	: 4;
 	unsigned int	devfn;		/* DEVFN part of the RID*/
 
 	struct mutex lock;
arch/s390/pci/pci_clp.c +1 -0
index 48bf316..322689b 100644
@@ -168,6 +168,7 @@ static int clp_store_query_pci_fn(struct zpci_dev *zdev,
 	if (response->util_str_avail) {
 		memcpy(zdev->util_str, response->util_str,
 		       sizeof(zdev->util_str));
+		zdev->util_str_avail = 1;
 	}
 	zdev->mio_capable = response->mio_addr_avail;
 	for (i = 0; i < PCI_STD_NUM_BARS; i++) {
Signature · 1 line
1.8.3.1
Acked-by on v2 2/5
#3 lore raw

Re: [PATCH v2 2/5] s390/pci: track whether util_str is valid in the zpci_dev

Quoted 10 lines · On Fri, 2 Oct 2020 16:00:41 -0400 Matthew Rosato <[email protected]>…

We'll need to keep track of whether or not the byte string in util_str is valid and thus needs to be passed to a vfio-pci passthrough device.

---
 arch/s390/include/asm/pci.h | 3 ++-
 arch/s390/pci/pci_clp.c     | 1 +
 2 files changed, 3 insertions(+), 1 deletion(-)

FWIW:

author maintainer
#4 lore raw

PATCH v2 3/5 vfio-pci/zdev: define the vfio_zdev header

In reply to Matthew Rosato

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userspace.

We create a new file, vfio_zdev.h to define the structure of the new region defined in vfio.h

---
 include/uapi/linux/vfio.h      |   5 ++
 include/uapi/linux/vfio_zdev.h | 118 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +5 -0
index 9204705..65eb367 100644
@@ -326,6 +326,11 @@ struct vfio_region_info_cap_type {
  * to do TLB invalidation on a GPU.
  */
 #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
+/*
+ * IBM zPCI specific hardware feature information for a devcie.  The contents
+ * of this region are mapped by struct vfio_region_zpci_info.
+ */
+#define VFIO_REGION_SUBTYPE_IBM_ZPCI_CLP	(2)
 
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
include/uapi/linux/vfio_zdev.h new file +118 -0

118 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#5 lore raw

Re: [PATCH v2 3/5] vfio-pci/zdev: define the vfio_zdev header

Quoted 143 lines · On Fri, 2 Oct 2020 16:00:42 -0400 Matthew Rosato <[email protected]>…

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userspace.

We create a new file, vfio_zdev.h to define the structure of the new region defined in vfio.h

---
 include/uapi/linux/vfio.h      |   5 ++
 include/uapi/linux/vfio_zdev.h | 118 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
 create mode 100644 include/uapi/linux/vfio_zdev.h
include/uapi/linux/vfio.h +5 -0
index 9204705..65eb367 100644
@@ -326,6 +326,11 @@ struct vfio_region_info_cap_type {
  * to do TLB invalidation on a GPU.
  */
 #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
+/*
+ * IBM zPCI specific hardware feature information for a devcie.  The contents
+ * of this region are mapped by struct vfio_region_zpci_info.
+ */
+#define VFIO_REGION_SUBTYPE_IBM_ZPCI_CLP	(2)
 
 /* sub-types for VFIO_REGION_TYPE_GFX */
 #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
include/uapi/linux/vfio_zdev.h new file +108 -0
new file mode 100644
index 0000000..1c8fb62
@@ -0,0 +1,118 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2020
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ *            Matthew Rosato <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information
+ *
+ * This region provides zPCI specific hardware feature information for a
+ * device.
+ *
+ * The ZPCI information structure is presented as a chain of CLP features
+ * defined below. argsz provides the size of the entire region, and offset
+ * provides the location of the first CLP feature in the chain.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u32 argsz;		/* Size of entire payload */
+	__u32 offset;		/* Location of first entry */
+};
+
+/**
+ * struct vfio_region_zpci_info_hdr - ZPCI header information
+ *
+ * This structure is included at the top of each CLP feature to define what
+ * type of CLP feature is presented / the structure version. The next value
+ * defines the offset of the next CLP feature, and is an offset from the very
+ * beginning of the region (vfio_region_zpci_info).
+ *
+ * Each CLP feature must have it's own unique 'id'.
+ */
+struct vfio_region_zpci_info_hdr {
+	__u16 id;		/* Identifies the CLP type */
+	__u16	version;	/* version of the CLP data */
+	__u32 next;		/* Offset of next entry */
+};
+
+/**
+ * struct vfio_region_zpci_info_pci - Base PCI Function information
+ *
+ * This region provides a set of descriptive information about the associated
+ * PCI function.
+ */
+#define VFIO_REGION_ZPCI_INFO_BASE	1
+
+struct vfio_region_zpci_info_base {
+	struct vfio_region_zpci_info_hdr hdr;
+	__u64 start_dma;	/* Start of available DMA addresses */
+	__u64 end_dma;		/* End of available DMA addresses */
+	__u16 pchid;		/* Physical Channel ID */
+	__u16 vfn;		/* Virtual function number */
+	__u16 fmb_length;	/* Measurement Block Length (in bytes) */
+	__u8 pft;		/* PCI Function Type */
+	__u8 gid;		/* PCI function group ID */
+};
+
+
+/**
+ * struct vfio_region_zpci_info_group - Base PCI Function Group information
+ *
+ * This region provides a set of descriptive information about the group of PCI
+ * functions that the associated device belongs to.
+ */
+#define VFIO_REGION_ZPCI_INFO_GROUP	2
+
+struct vfio_region_zpci_info_group {
+	struct vfio_region_zpci_info_hdr hdr;
+	__u64 dasm;		/* DMA Address space mask */
+	__u64 msi_addr;		/* MSI address */
+	__u64 flags;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1 /* Use program-specified TLB refresh */
+	__u16 mui;		/* Measurement Block Update Interval */
+	__u16 noi;		/* Maximum number of MSIs */
+	__u16 maxstbl;		/* Maximum Store Block Length */
+	__u8 version;		/* Supported PCI Version */
+};
+
+/**
+ * struct vfio_region_zpci_info_util - Utility String
+ *
+ * This region provides the utility string for the associated device, which is
+ * a device identifier string made up of EBCDID characters.  'size' specifies
+ * the length of 'util_str'.
+ */
+#define VFIO_REGION_ZPCI_INFO_UTIL	3
+
+struct vfio_region_zpci_info_util {
+	struct vfio_region_zpci_info_hdr hdr;
+	__u32 size;
+	__u8 util_str[];
+};
+
+/**
+ * struct vfio_region_zpci_info_pfip - PCI Function Path
+ *
+ * This region provides the PCI function path string, which is an identifier
+ * that describes the internal hardware path of the device. 'size' specifies
+ * the length of 'pfip'.

include/uapi/linux/vfio_zdev.h line 118

Quoted 10 lines
include/uapi/linux/vfio_zdev.h new file +10 -0
+ */
+#define VFIO_REGION_ZPCI_INFO_PFIP	4
+
+struct vfio_region_zpci_info_pfip {
+struct vfio_region_zpci_info_hdr hdr;
+	__u32 size;
+	__u8 pfip[];
+};
+
+#endif

Can you discuss why a region with embedded capability chain is a better solution than extending the VFIO_DEVICE_GET_INFO ioctl to support a capability chain and providing this info there? This all appears to be read-only info, so what's the benefit of duplicating yet another capability chain in a region? It would also be possible to define four separate device specific regions, one for each of these capabilities rather than creating this chain. It just seems like a strange approach TBH. Thanks,

Alex

author maintainer
#6 lore raw

Re: [PATCH v2 3/5] vfio-pci/zdev: define the vfio_zdev header

Quoted 153 lines · Alex Williamson
Quoted 150 lines · On Fri, 2 Oct 2020 16:00:42 -0400 Matthew Rosato <[email protected]>…

We define a new device region in vfio.h to be able to get the ZPCI CLP information by reading this region from userspace.

We create a new file, vfio_zdev.h to define the structure of the new region defined in vfio.h

---
  include/uapi/linux/vfio.h      |   5 ++
  include/uapi/linux/vfio_zdev.h | 118 +++++++++++++++++++++++++++++++++++++++++
  2 files changed, 123 insertions(+)

create mode 100644 include/uapi/linux/vfio_zdev.h

include/uapi/linux/vfio.h +5 -0
index 9204705..65eb367 100644
@@ -326,6 +326,11 @@ struct vfio_region_info_cap_type {
   * to do TLB invalidation on a GPU.
   */
  #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD	(1)
+/*
+ * IBM zPCI specific hardware feature information for a devcie.  The contents
+ * of this region are mapped by struct vfio_region_zpci_info.
+ */
+#define VFIO_REGION_SUBTYPE_IBM_ZPCI_CLP	(2)
 
  /* sub-types for VFIO_REGION_TYPE_GFX */
  #define VFIO_REGION_SUBTYPE_GFX_EDID            (1)
include/uapi/linux/vfio_zdev.h new file +115 -0
new file mode 100644
index 0000000..1c8fb62
@@ -0,0 +1,118 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Region definition for ZPCI devices
+ *
+ * Copyright IBM Corp. 2020
+ *
+ * Author(s): Pierre Morel <[email protected]>
+ *            Matthew Rosato <[email protected]>
+ */
+
+#ifndef _VFIO_ZDEV_H_
+#define _VFIO_ZDEV_H_
+
+#include <linux/types.h>
+
+/**
+ * struct vfio_region_zpci_info - ZPCI information
+ *
+ * This region provides zPCI specific hardware feature information for a
+ * device.
+ *
+ * The ZPCI information structure is presented as a chain of CLP features
+ * defined below. argsz provides the size of the entire region, and offset
+ * provides the location of the first CLP feature in the chain.
+ *
+ */
+struct vfio_region_zpci_info {
+	__u32 argsz;		/* Size of entire payload */
+	__u32 offset;		/* Location of first entry */
+};
+
+/**
+ * struct vfio_region_zpci_info_hdr - ZPCI header information
+ *
+ * This structure is included at the top of each CLP feature to define what
+ * type of CLP feature is presented / the structure version. The next value
+ * defines the offset of the next CLP feature, and is an offset from the very
+ * beginning of the region (vfio_region_zpci_info).
+ *
+ * Each CLP feature must have it's own unique 'id'.
+ */
+struct vfio_region_zpci_info_hdr {
+	__u16 id;		/* Identifies the CLP type */
+	__u16	version;	/* version of the CLP data */
+	__u32 next;		/* Offset of next entry */
+};
+
+/**
+ * struct vfio_region_zpci_info_pci - Base PCI Function information
+ *
+ * This region provides a set of descriptive information about the associated
+ * PCI function.
+ */
+#define VFIO_REGION_ZPCI_INFO_BASE	1
+
+struct vfio_region_zpci_info_base {
+	struct vfio_region_zpci_info_hdr hdr;
+	__u64 start_dma;	/* Start of available DMA addresses */
+	__u64 end_dma;		/* End of available DMA addresses */
+	__u16 pchid;		/* Physical Channel ID */
+	__u16 vfn;		/* Virtual function number */
+	__u16 fmb_length;	/* Measurement Block Length (in bytes) */
+	__u8 pft;		/* PCI Function Type */
+	__u8 gid;		/* PCI function group ID */
+};
+
+
+/**
+ * struct vfio_region_zpci_info_group - Base PCI Function Group information
+ *
+ * This region provides a set of descriptive information about the group of PCI
+ * functions that the associated device belongs to.
+ */
+#define VFIO_REGION_ZPCI_INFO_GROUP	2
+
+struct vfio_region_zpci_info_group {
+	struct vfio_region_zpci_info_hdr hdr;
+	__u64 dasm;		/* DMA Address space mask */
+	__u64 msi_addr;		/* MSI address */
+	__u64 flags;
+#define VFIO_PCI_ZDEV_FLAGS_REFRESH 1 /* Use program-specified TLB refresh */
+	__u16 mui;		/* Measurement Block Update Interval */
+	__u16 noi;		/* Maximum number of MSIs */
+	__u16 maxstbl;		/* Maximum Store Block Length */
+	__u8 version;		/* Supported PCI Version */
+};
+
+/**
+ * struct vfio_region_zpci_info_util - Utility String
+ *
+ * This region provides the utility string for the associated device, which is
+ * a device identifier string made up of EBCDID characters.  'size' specifies
+ * the length of 'util_str'.
+ */
+#define VFIO_REGION_ZPCI_INFO_UTIL	3
+
+struct vfio_region_zpci_info_util {
+	struct vfio_region_zpci_info_hdr hdr;
+	__u32 size;
+	__u8 util_str[];
+};
+
+/**
+ * struct vfio_region_zpci_info_pfip - PCI Function Path
+ *
+ * This region provides the PCI function path string, which is an identifier
+ * that describes the internal hardware path of the device. 'size' specifies
+ * the length of 'pfip'.
+ */
+#define VFIO_REGION_ZPCI_INFO_PFIP	4
+
+struct vfio_region_zpci_info_pfip {
+struct vfio_region_zpci_info_hdr hdr;
+	__u32 size;
+	__u8 pfip[];
Quoted 8 lines

include/uapi/linux/vfio_zdev.h line 118

Quoted 3 lines
include/uapi/linux/vfio_zdev.h new file +3 -0
+};
+
+#endif

Can you discuss why a region with embedded capability chain is a better solution than extending the VFIO_DEVICE_GET_INFO ioctl to support a capability chain and providing this info there? This all appears to be read-only info, so what's the benefit of duplicating yet another

It is indeed read-only info, and the device region was defined as such.

I would not necessarily be opposed to extending VFIO_DEVICE_GET_INFO with these defined as capabilities; I'd say a primary motivating factor to putting these in their own region was to avoid stuffing a bunch of s390-specific capabilities into a general-purpose ioctl response.

But if you're OK with that notion, I can give that a crack in v3.

Quoted 3 lines

capability chain in a region? It would also be possible to define four separate device specific regions, one for each of these capabilities rather than creating this chain. It just seems like a strange approach

I'm not sure if creating separate regions would be the right approach though; these are just the first 4. There will definitely be additional capabilities in support of new zPCI features moving forward, I'm not sure how many regions we really want to end up with. Some might be as small as a single field, which seems more in-line with capabilities vs an entire region.

#7 lore raw

Re: [PATCH v2 3/5] vfio-pci/zdev: define the vfio_zdev header

Quoted 7 lines · On Mon, 5 Oct 2020 09:52:25 -0400 Matthew Rosato <[email protected]> wrote:
Quoted 4 lines · Alex Williamson

Can you discuss why a region with embedded capability chain is a better solution than extending the VFIO_DEVICE_GET_INFO ioctl to support a capability chain and providing this info there? This all appears to be read-only info, so what's the benefit of duplicating yet another

Quoted 6 lines

It is indeed read-only info, and the device region was defined as such.

I would not necessarily be opposed to extending VFIO_DEVICE_GET_INFO with these defined as capabilities; I'd say a primary motivating factor to putting these in their own region was to avoid stuffing a bunch of s390-specific capabilities into a general-purpose ioctl response.

Can't you make the zdev code register the capabilities? That would put them nicely into their own configurable part.

Quoted 7 lines

But if you're OK with that notion, I can give that a crack in v3.

Quoted 3 lines

capability chain in a region? It would also be possible to define four separate device specific regions, one for each of these capabilities rather than creating this chain. It just seems like a strange approach

Quoted 6 lines

I'm not sure if creating separate regions would be the right approach though; these are just the first 4. There will definitely be additional capabilities in support of new zPCI features moving forward, I'm not sure how many regions we really want to end up with. Some might be as small as a single field, which seems more in-line with capabilities vs an entire region.

If we are expecting more of these in the future, going with GET_INFO capabilities when adding new ones seems like the best approach.

author maintainer
#8 lore raw

Re: [PATCH v2 3/5] vfio-pci/zdev: define the vfio_zdev header

Quoted 12 lines · Cornelia Huck
Quoted 9 lines · On Mon, 5 Oct 2020 09:52:25 -0400 Matthew Rosato <[email protected]> wrote:
Quoted 4 lines · Alex Williamson

Can you discuss why a region with embedded capability chain is a better solution than extending the VFIO_DEVICE_GET_INFO ioctl to support a capability chain and providing this info there? This all appears to be read-only info, so what's the benefit of duplicating yet another

It is indeed read-only info, and the device region was defined as such.

Quoted 8 lines
Quoted 4 lines

I would not necessarily be opposed to extending VFIO_DEVICE_GET_INFO with these defined as capabilities; I'd say a primary motivating factor to putting these in their own region was to avoid stuffing a bunch of s390-specific capabilities into a general-purpose ioctl response.

Can't you make the zdev code register the capabilities? That would put them nicely into their own configurable part.

I can still keep the code that adds these capabilities in the zdev .c file, thus meaning they will only be added for s390 zpci devices -- but the actual definition of them should probably instead be in vfio.h, no? (maybe that's what you mean, but let's lay it out just in case)

The capability IDs would be shared with any other potential user of VFIO_DEVICE_GET_INFO (I guess there is precedent for this already, nvlink2 does this for vfio_region_info, see VFIO_REGION_INFO_CAP_NVLINK2_SSATGT as an example).

Today, ZPCI would be the only users of VFIO_DEVICE_GET_INFO capability chains. Tomorrow, some other type might use them too. Unless we want to put a stake in the ground that says there will never be a case for a capability that all devices share on VFIO_DEVICE_GET_INFO, I think we should keep the IDs unique and define the capabilities in vfio.h but do the corresponding add_capability() calls from a zdev-specific file.

Quoted 17 lines
Quoted 7 lines

But if you're OK with that notion, I can give that a crack in v3.

Quoted 3 lines

capability chain in a region? It would also be possible to define four separate device specific regions, one for each of these capabilities rather than creating this chain. It just seems like a strange approach

Quoted 6 lines

I'm not sure if creating separate regions would be the right approach though; these are just the first 4. There will definitely be additional capabilities in support of new zPCI features moving forward, I'm not sure how many regions we really want to end up with. Some might be as small as a single field, which seems more in-line with capabilities vs an entire region.

If we are expecting more of these in the future, going with GET_INFO capabilities when adding new ones seems like the best approach.

#9 lore raw

Re: [PATCH v2 3/5] vfio-pci/zdev: define the vfio_zdev header

Quoted 32 lines · On Mon, 5 Oct 2020 12:16:10 -0400 Matthew Rosato <[email protected]> wrote:
Quoted 12 lines · Cornelia Huck
Quoted 9 lines · On Mon, 5 Oct 2020 09:52:25 -0400 Matthew Rosato <[email protected]> wrote:
Quoted 4 lines · Alex Williamson

Can you discuss why a region with embedded capability chain is a better solution than extending the VFIO_DEVICE_GET_INFO ioctl to support a capability chain and providing this info there? This all appears to be read-only info, so what's the benefit of duplicating yet another

It is indeed read-only info, and the device region was defined as such.

Quoted 8 lines
Quoted 4 lines

I would not necessarily be opposed to extending VFIO_DEVICE_GET_INFO with these defined as capabilities; I'd say a primary motivating factor to putting these in their own region was to avoid stuffing a bunch of s390-specific capabilities into a general-purpose ioctl response.

Can't you make the zdev code register the capabilities? That would put them nicely into their own configurable part.

I can still keep the code that adds these capabilities in the zdev .c file, thus meaning they will only be added for s390 zpci devices -- but the actual definition of them should probably instead be in vfio.h, no? (maybe that's what you mean, but let's lay it out just in case)

The capability IDs would be shared with any other potential user of VFIO_DEVICE_GET_INFO (I guess there is precedent for this already, nvlink2 does this for vfio_region_info, see VFIO_REGION_INFO_CAP_NVLINK2_SSATGT as an example).

Quoted 6 lines

Today, ZPCI would be the only users of VFIO_DEVICE_GET_INFO capability chains. Tomorrow, some other type might use them too. Unless we want to put a stake in the ground that says there will never be a case for a capability that all devices share on VFIO_DEVICE_GET_INFO, I think we should keep the IDs unique and define the capabilities in vfio.h but do the corresponding add_capability() calls from a zdev-specific file.

Agreed. We should have enough space for multiple users, and I do not consider reserving the IDs cluttering.

author maintainer
#10 lore raw

PATCH v2 4/5 vfio-pci/zdev: use a device region to retrieve zPCI information

In reply to Matthew Rosato

Define a new configuration entry VFIO_PCI_ZDEV for VFIO/PCI.

When this s390-only feature is configured we initialize a new device region, VFIO_REGION_SUBTYPE_IBM_ZPCI_CLP, to hold information provided by the underlying hardware.

This patch is based on work previously done by Pierre Morel.

---
 drivers/vfio/pci/Kconfig            |  13 ++
 drivers/vfio/pci/Makefile           |   1 +
 drivers/vfio/pci/vfio_pci.c         |   8 ++
 drivers/vfio/pci/vfio_pci_private.h |  10 ++
 drivers/vfio/pci/vfio_pci_zdev.c    | 242 ++++++++++++++++++++++++++++++++++++
 5 files changed, 274 insertions(+)
 create mode 100644 drivers/vfio/pci/vfio_pci_zdev.c
drivers/vfio/pci/Kconfig +13 -0
index ac3c1dd..07b4a35 100644
@@ -45,3 +45,16 @@ config VFIO_PCI_NVLINK2
 	depends on VFIO_PCI && PPC_POWERNV
 	help
 	  VFIO PCI support for P9 Witherspoon machine with NVIDIA V100 GPUs
+
+config VFIO_PCI_ZDEV
+	bool "VFIO PCI ZPCI device CLP support"
+	depends on VFIO_PCI && S390
+	default y
+	help
+	  Enabling this options exposes a region containing hardware
+	  configuration for zPCI devices. This enables userspace (e.g. QEMU)
+	  to supply proper configuration values instead of hard-coded defaults
+	  for zPCI devices passed through via VFIO on s390.
+
+	  Say Y here.
+
drivers/vfio/pci/Makefile +1 -0
index f027f8a..781e080 100644
@@ -3,5 +3,6 @@
 vfio-pci-y := vfio_pci.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
 vfio-pci-$(CONFIG_VFIO_PCI_IGD) += vfio_pci_igd.o
 vfio-pci-$(CONFIG_VFIO_PCI_NVLINK2) += vfio_pci_nvlink2.o
+vfio-pci-$(CONFIG_VFIO_PCI_ZDEV) += vfio_pci_zdev.o
 
 obj-$(CONFIG_VFIO_PCI) += vfio-pci.o
drivers/vfio/pci/vfio_pci.c +8 -0
index 1ab1f5c..cfb04d9 100644
@@ -409,6 +409,14 @@ static int vfio_pci_enable(struct vfio_pci_device *vdev)
 		}
 	}
 
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV)) {
+		ret = vfio_pci_zdev_init(vdev);
+		if (ret && ret != -ENODEV) {
+			pci_warn(pdev, "Failed to setup zPCI CLP region\n");
+			goto disable_exit;
+		}
+	}
+
 	vfio_pci_probe_mmaps(vdev);
 
 	return 0;
drivers/vfio/pci/vfio_pci_private.h +10 -0
index 61ca8ab..729af20 100644
@@ -213,4 +213,14 @@ static inline int vfio_pci_ibm_npu2_init(struct vfio_pci_device *vdev)
 	return -ENODEV;
 }
 #endif
+
+#ifdef CONFIG_VFIO_PCI_ZDEV
+extern int vfio_pci_zdev_init(struct vfio_pci_device *vdev);
+#else
+static inline int vfio_pci_zdev_init(struct vfio_pci_device *vdev)
+{
+	return -ENODEV;
+}
+#endif
+
 #endif /* VFIO_PCI_PRIVATE_H */
drivers/vfio/pci/vfio_pci_zdev.c new file +242 -0

242 lines not on this page · raw message

Signature · 1 line
1.8.3.1
author maintainer
#11 lore raw

PATCH v2 5/5 MAINTAINERS: Add entry for s390 vfio-pci

In reply to Matthew Rosato

Add myself to cover s390-specific items related to vfio-pci.

---
 MAINTAINERS | 8 ++++++++
 1 file changed, 8 insertions(+)
MAINTAINERS +8 -0
index 190c7fa..389c4ad 100644
@@ -15162,6 +15162,14 @@ F:	Documentation/s390/vfio-ccw.rst
 F:	drivers/s390/cio/vfio_ccw*
 F:	include/uapi/linux/vfio_ccw.h
 
+S390 VFIO-PCI DRIVER
+M:	Matthew Rosato <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/vfio_pci_zdev.c
+F:	include/uapi/linux/vfio_zdev.h
+
 S390 ZCRYPT DRIVER
 M:	Harald Freudenberger <[email protected]>
 L:	[email protected]
Signature · 1 line
1.8.3.1
Acked-by on v2 5/5
#12 lore raw

Re: [PATCH v2 5/5] MAINTAINERS: Add entry for s390 vfio-pci

Quoted 16 lines · On Fri, 2 Oct 2020 16:00:44 -0400 Matthew Rosato <[email protected]>…

Add myself to cover s390-specific items related to vfio-pci.

---
 MAINTAINERS | 8 ++++++++
 1 file changed, 8 insertions(+)
MAINTAINERS +1 -0
index 190c7fa..389c4ad 100644
@@ -15162,6 +15162,14 @@ F:	Documentation/s390/vfio-ccw.rst
 F:	drivers/s390/cio/vfio_ccw*
 F:	include/uapi/linux/vfio_ccw.h
 
+S390 VFIO-PCI DRIVER

MAINTAINERS F: Documentation/s390/vfio-ccw.rst · line 15175

Quoted 10 lines
MAINTAINERS +7 -0
+M:	Matthew Rosato <[email protected]>
+L:	[email protected]
+L:	[email protected]
+S:	Supported
+F:	drivers/vfio/pci/vfio_pci_zdev.c
+F:	include/uapi/linux/vfio_zdev.h
+
 S390 ZCRYPT DRIVER
 M:	Harald Freudenberger <[email protected]>
 L:	[email protected]
author maintainer
#13 lore raw
In reply to Matthew Rosato
Quoted 15 lines · Matthew Rosato

This patchset provides a means by which hardware information about the underlying PCI device can be passed up to userspace (ie, QEMU) so that this hardware information can be used rather than previously hard-coded assumptions. A new VFIO region type is defined which holds this information.

A form of these patches saw some rounds last year but has been back- tabled for a while. The original work for this feature was done by Pierre Morel. I'd like to refresh the discussion on this and get this finished up so that we can move forward with better-supporting additional types of PCI-attached devices. The proposal here presents a completely different region mapping vs the prior approach, taking inspiration from vfio info capability chains to provide device CLP information in a way that allows for future expansion (new CLP features).

Quoted 9 lines

This feature is toggled via the CONFIG_VFIO_PCI_ZDEV configuration entry.

Changes from v1:
- Added ACKs (thanks!)
- Patch 2: Minor change:s/util_avail/util_str_avail/ per Niklas
- Patch 3: removed __packed
- Patch 3: rework various descriptions / comment blocks
- New patch: MAINTAINERS hit to cover new files.

Link to latest QEMU patchset:
https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00673.html

Quoted 21 lines

Matthew Rosato (5):
s390/pci: stash version in the zpci_dev
s390/pci: track whether util_str is valid in the zpci_dev vfio-pci/zdev: define the vfio_zdev header vfio-pci/zdev: use a device region to retrieve zPCI information
MAINTAINERS: Add entry for s390 vfio-pci

  MAINTAINERS                         |   8 ++
  arch/s390/include/asm/pci.h         |   4 +-
  arch/s390/pci/pci_clp.c             |   2 +
  drivers/vfio/pci/Kconfig            |  13 ++
  drivers/vfio/pci/Makefile           |   1 +
  drivers/vfio/pci/vfio_pci.c         |   8 ++
  drivers/vfio/pci/vfio_pci_private.h |  10 ++
  drivers/vfio/pci/vfio_pci_zdev.c    | 242 ++++++++++++++++++++++++++++++++++++
  include/uapi/linux/vfio.h           |   5 +
  include/uapi/linux/vfio_zdev.h      | 118 ++++++++++++++++++
  10 files changed, 410 insertions(+), 1 deletion(-)

create mode 100644 drivers/vfio/pci/vfio_pci_zdev.c create mode 100644 include/uapi/linux/vfio_zdev.h