Skip to contentSkip to conversation

Retrieve zPCI hardware information from VFIO

PATCH In review Matthew Rosato 2 participants 18 messages Oct 7, 2020 – Oct 9, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v3 cover Retrieve zPCI hardware information from VFIO

This patchset adds code to s390 pci to examine the VFIO_DEVICE_GET_INFO capability chain, looking for capabilities that describe the underlying hardware and features of the passed-through device.

The retrieval of this information is done once per function (and for a subset of data, once per function group) and is performed at time of device plug. Some elements provided via the capability chain must still be forced to default values for now to reflect what QEMU actually provides support for.

The original work for this feature was done by Pierre Morel.

Associated kernel patchset:
https://lkml.org/lkml/2020/10/7/813

Tested using a host kernel with/without the VFIO_DEVICE_GET_INFO changes and a QEMU with/without the VFIO_DEVICE_GET_INFO changes to validate VFIO_DEVICE_GET_INFO remains backwards-compatible.

Changes since v2:
- Added ACKs/RBs (thanks!)
- Squashed the first 2 patches, and tried again using by formatting the set with --find-renames / diff.renames. (Thanks Richard and Paolo!) As before, this first patch can be applied separately but is included here for the sake of simplicity.
- Copied patch 'vfio: Create shared routine for scanning info capabilities' from the 'Accomodate vfio DMA limiting' set, as it's now needed here too. We can drop it from this set later once the former is applied.
- Refresh the linux-header placeholder. Manually omit the qatomic->atomic pvrdma revert for now.
- New patch to add a helper function for finding VFIO_DEVICE_GET_INFO capabilities. Depends on 'vfio: Create shared routine for scanning info capabilities'
- re-write the last patch to use VFIO_DEVICE_GET_INFO capabilities instead of a vfio device region

Matthew Rosato (7):
s390x/pci: Move header files to include/hw/s390x
vfio: Create shared routine for scanning info capabilities
update-linux-headers: Add vfio_zdev.h
linux-headers: update against 5.9-rc8
s390x/pci: clean up s390 PCI groups
vfio: Add routine for finding VFIO_DEVICE_GET_INFO capabilities s390x/pci: get zPCI function info from host

Pierre Morel (3):
s390x/pci: create a header dedicated to PCI CLP
s390x/pci: use a PCI Group structure
s390x/pci: use a PCI Function structure

 MAINTAINERS                                        |   1 +
 hw/s390x/meson.build                               |   1 +
 hw/s390x/s390-pci-bus.c                            |  86 ++++++++-
 hw/s390x/s390-pci-inst.c                           |  33 ++--
 hw/s390x/s390-pci-vfio.c                           | 197 +++++++++++++++++++++
 hw/s390x/s390-virtio-ccw.c                         |   2 +-
 hw/vfio/common.c                                   |  31 +++-
 {hw => include/hw}/s390x/s390-pci-bus.h            |  13 ++
 .../hw/s390x/s390-pci-clp.h                        | 123 ++-----------
 include/hw/s390x/s390-pci-inst.h                   | 116 ++++++++++++
 include/hw/s390x/s390-pci-vfio.h                   |  19 ++
 include/hw/vfio/vfio-common.h                      |   2 +
 linux-headers/linux/kvm.h                          |   6 +-
 linux-headers/linux/vfio.h                         |  11 ++
 linux-headers/linux/vfio_zdev.h                    |  78 ++++++++
 scripts/update-linux-headers.sh                    |   2 +-
 16 files changed, 575 insertions(+), 146 deletions(-)
 create mode 100644 hw/s390x/s390-pci-vfio.c
 rename {hw => include/hw}/s390x/s390-pci-bus.h (96%)
 rename hw/s390x/s390-pci-inst.h => include/hw/s390x/s390-pci-clp.h (59%)
 create mode 100644 include/hw/s390x/s390-pci-inst.h
 create mode 100644 include/hw/s390x/s390-pci-vfio.h
 create mode 100644 linux-headers/linux/vfio_zdev.h
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v3 1/10 s390x/pci: Move header files to include/hw/s390x

Seems a more appropriate location for them.

---
 MAINTAINERS                              | 1 +
 hw/s390x/s390-pci-bus.c                  | 4 ++--
 hw/s390x/s390-pci-inst.c                 | 4 ++--
 hw/s390x/s390-virtio-ccw.c               | 2 +-
 {hw => include/hw}/s390x/s390-pci-bus.h  | 0
 {hw => include/hw}/s390x/s390-pci-inst.h | 0
 6 files changed, 6 insertions(+), 5 deletions(-)
 rename {hw => include/hw}/s390x/s390-pci-bus.h (100%)
 rename {hw => include/hw}/s390x/s390-pci-inst.h (100%)
MAINTAINERS +1 -0
index e9d85cc..0eb8a7f 100644
@@ -1443,6 +1443,7 @@ S390 PCI
 M: Matthew Rosato <[email protected]>
 S: Supported
 F: hw/s390x/s390-pci*
+F: include/hw/s390x/s390-pci*
 L: [email protected]
 
 UniCore32 Machines
hw/s390x/s390-pci-bus.c +2 -2
index fb4cee8..a929340 100644
@@ -15,8 +15,8 @@
 #include "qapi/error.h"
 #include "qapi/visitor.h"
 #include "cpu.h"
-#include "s390-pci-bus.h"
-#include "s390-pci-inst.h"
+#include "hw/s390x/s390-pci-bus.h"
+#include "hw/s390x/s390-pci-inst.h"
 #include "hw/pci/pci_bus.h"
 #include "hw/qdev-properties.h"
 #include "hw/pci/pci_bridge.h"
hw/s390x/s390-pci-inst.c +2 -2
index 2f7a7d7..639b13c 100644
@@ -13,12 +13,12 @@
 
 #include "qemu/osdep.h"
 #include "cpu.h"
-#include "s390-pci-inst.h"
-#include "s390-pci-bus.h"
 #include "exec/memop.h"
 #include "exec/memory-internal.h"
 #include "qemu/error-report.h"
 #include "sysemu/hw_accel.h"
+#include "hw/s390x/s390-pci-inst.h"
+#include "hw/s390x/s390-pci-bus.h"
 #include "hw/s390x/tod.h"
 
 #ifndef DEBUG_S390PCI_INST
hw/s390x/s390-virtio-ccw.c +1 -1
index 28266a3..f31ae2e 100644
@@ -28,7 +28,7 @@
 #include "qemu/error-report.h"
 #include "qemu/option.h"
 #include "qemu/qemu-print.h"
-#include "s390-pci-bus.h"
+#include "hw/s390x/s390-pci-bus.h"
 #include "sysemu/reset.h"
 #include "hw/s390x/storage-keys.h"
 #include "hw/s390x/storage-attributes.h"
include/hw/s390x/s390-pci-bus.h renamed from hw/s390x/s390-pci-bus.h +0 -0
similarity index 100%
rename from hw/s390x/s390-pci-bus.h
rename to include/hw/s390x/s390-pci-bus.h
include/hw/s390x/s390-pci-inst.h renamed from hw/s390x/s390-pci-inst.h +0 -0
similarity index 100%
rename from hw/s390x/s390-pci-inst.h
rename to include/hw/s390x/s390-pci-inst.h
Signature · 1 line
1.8.3.1
Reviewed-by on v3 1/10
#2 lore raw

Re: [PATCH v3 01/10] s390x/pci: Move header files to include/hw/s390x

Quoted 13 lines · On Wed, 7 Oct 2020 15:04:06 -0400 Matthew Rosato <[email protected]>…

Seems a more appropriate location for them.

---
 MAINTAINERS                              | 1 +
 hw/s390x/s390-pci-bus.c                  | 4 ++--
 hw/s390x/s390-pci-inst.c                 | 4 ++--
 hw/s390x/s390-virtio-ccw.c               | 2 +-
 {hw => include/hw}/s390x/s390-pci-bus.h  | 0
 {hw => include/hw}/s390x/s390-pci-inst.h | 0
 6 files changed, 6 insertions(+), 5 deletions(-)
 rename {hw => include/hw}/s390x/s390-pci-bus.h (100%)
 rename {hw => include/hw}/s390x/s390-pci-inst.h (100%)

Much better :)

(I don't have anything on s390-next atm, or I would have queued it straight away. Better wait until we can do a proper headers sync and queue the whole thing.)

#3 lore raw

PATCH v3 2/10 vfio: Create shared routine for scanning info capabilities

In reply to Matthew Rosato

Rather than duplicating the same loop in multiple locations, create a static function to do the work.

---
 hw/vfio/common.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)
hw/vfio/common.c +13 -8
index 13471ae..e47a4d7 100644
@@ -825,17 +825,12 @@ static void vfio_listener_release(VFIOContainer *container)
     }
 }
 
-struct vfio_info_cap_header *
-vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)
+static struct vfio_info_cap_header *
+vfio_get_cap(void *ptr, uint32_t cap_offset, uint16_t id)
 {
     struct vfio_info_cap_header *hdr;
-    void *ptr = info;
-
-    if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) {
-        return NULL;
-    }
 
-    for (hdr = ptr + info->cap_offset; hdr != ptr; hdr = ptr + hdr->next) {
+    for (hdr = ptr + cap_offset; hdr != ptr; hdr = ptr + hdr->next) {
         if (hdr->id == id) {
             return hdr;
         }
@@ -844,6 +839,16 @@ vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)
     return NULL;
 }
 
+struct vfio_info_cap_header *
+vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)
+{
+    if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) {
+        return NULL;
+    }
+
+    return vfio_get_cap((void *)info, info->cap_offset, id);
+}
+
 static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
                                           struct vfio_region_info *info)
 {
Signature · 1 line
1.8.3.1
#4 lore raw

PATCH v3 3/10 update-linux-headers: Add vfio_zdev.h

In reply to Matthew Rosato

vfio_zdev.h is used by s390x zPCI support to pass device-specific CLP information between host and userspace.

---
 scripts/update-linux-headers.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/update-linux-headers.sh +1 -1
index 29c27f4..9efbaf2 100755
@@ -141,7 +141,7 @@ done
 
 rm -rf "$output/linux-headers/linux"
 mkdir -p "$output/linux-headers/linux"
-for header in kvm.h vfio.h vfio_ccw.h vhost.h \
+for header in kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \
               psci.h psp-sev.h userfaultfd.h mman.h; do
     cp "$tmpdir/include/linux/$header" "$output/linux-headers/linux"
 done
Signature · 1 line
1.8.3.1
#5 lore raw

PATCH v3 4/10 linux-headers: update against 5.9-rc8

In reply to Matthew Rosato

PLACEHOLDER as the kernel patch driving the need for this ("vfio: Introduce capability definitions for VFIO_DEVICE_GET_INFO") isn't merged yet.

---
 linux-headers/linux/kvm.h       |  6 ++--
 linux-headers/linux/vfio.h      | 11 ++++++
 linux-headers/linux/vfio_zdev.h | 78 +++++++++++++++++++++++++++++++++++++++++
 3 files changed, 93 insertions(+), 2 deletions(-)
 create mode 100644 linux-headers/linux/vfio_zdev.h
linux-headers/linux/kvm.h +4 -2
index 6683e2e..43580c7 100644
@@ -790,9 +790,10 @@ struct kvm_ppc_resize_hpt {
 #define KVM_VM_PPC_HV 1
 #define KVM_VM_PPC_PR 2
 
-/* on MIPS, 0 forces trap & emulate, 1 forces VZ ASE */
-#define KVM_VM_MIPS_TE		0
+/* on MIPS, 0 indicates auto, 1 forces VZ ASE, 2 forces trap & emulate */
+#define KVM_VM_MIPS_AUTO	0
 #define KVM_VM_MIPS_VZ		1
+#define KVM_VM_MIPS_TE		2
 
 #define KVM_S390_SIE_PAGE_OFFSET 1
 
@@ -1035,6 +1036,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_LAST_CPU 184
 #define KVM_CAP_SMALLER_MAXPHYADDR 185
 #define KVM_CAP_S390_DIAG318 186
+#define KVM_CAP_STEAL_TIME 187
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
linux-headers/linux/vfio.h +11 -0
index a906724..467eeab 100644
@@ -201,8 +201,10 @@ struct vfio_device_info {
 #define VFIO_DEVICE_FLAGS_AMBA  (1 << 3)	/* vfio-amba device */
 #define VFIO_DEVICE_FLAGS_CCW	(1 << 4)	/* vfio-ccw device */
 #define VFIO_DEVICE_FLAGS_AP	(1 << 5)	/* vfio-ap device */
+#define VFIO_DEVICE_FLAGS_CAPS	(1 << 6)	/* Info supports caps */
 	__u32	num_regions;	/* Max region index + 1 */
 	__u32	num_irqs;	/* Max IRQ index + 1 */
+	__u32   cap_offset;	/* Offset within info struct of first cap */
 };
 #define VFIO_DEVICE_GET_INFO		_IO(VFIO_TYPE, VFIO_BASE + 7)
 
@@ -218,6 +220,15 @@ struct vfio_device_info {
 #define VFIO_DEVICE_API_CCW_STRING		"vfio-ccw"
 #define VFIO_DEVICE_API_AP_STRING		"vfio-ap"
 
+/*
+ * The following capabilities are unique to s390 zPCI devices.  Their contents
+ * are further-defined in vfio_zdev.h
+ */
+#define VFIO_DEVICE_INFO_CAP_ZPCI_BASE		1
+#define VFIO_DEVICE_INFO_CAP_ZPCI_GROUP		2
+#define VFIO_DEVICE_INFO_CAP_ZPCI_UTIL		3
+#define VFIO_DEVICE_INFO_CAP_ZPCI_PFIP		4
+
 /**
  * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8,
  *				       struct vfio_region_info)
linux-headers/linux/vfio_zdev.h new file +78 -0

78 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#6 lore raw

PATCH v3 5/10 s390x/pci: create a header dedicated to PCI CLP

In reply to Matthew Rosato

From: Pierre Morel <[email protected]>

To have a clean separation between s390-pci-bus.h and s390-pci-inst.h headers we export the PCI CLP instructions in a dedicated header.

---
 include/hw/s390x/s390-pci-bus.h  |   1 +
 include/hw/s390x/s390-pci-clp.h  | 211 +++++++++++++++++++++++++++++++++++++++
 include/hw/s390x/s390-pci-inst.h | 196 ------------------------------------
 3 files changed, 212 insertions(+), 196 deletions(-)
 create mode 100644 include/hw/s390x/s390-pci-clp.h
include/hw/s390x/s390-pci-bus.h +1 -0
index 97464d0..04e7cb6 100644
@@ -19,6 +19,7 @@
 #include "hw/s390x/sclp.h"
 #include "hw/s390x/s390_flic.h"
 #include "hw/s390x/css.h"
+#include "hw/s390x/s390-pci-clp.h"
 #include "qom/object.h"
 
 #define TYPE_S390_PCI_HOST_BRIDGE "s390-pcihost"
include/hw/s390x/s390-pci-clp.h new file +211 -0

211 lines not on this page · raw message

include/hw/s390x/s390-pci-inst.h +0 -196

202 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#7 lore raw

PATCH v3 6/10 s390x/pci: use a PCI Group structure

In reply to Matthew Rosato

From: Pierre Morel <[email protected]>

We use a S390PCIGroup structure to hold the information related to a zPCI Function group.

This allows us to be ready to support multiple groups and to retrieve the group information from the host.

---
 hw/s390x/s390-pci-bus.c         | 42 +++++++++++++++++++++++++++++++++++++++++
 hw/s390x/s390-pci-inst.c        | 23 +++++++++++++---------
 include/hw/s390x/s390-pci-bus.h | 10 ++++++++++
 3 files changed, 66 insertions(+), 9 deletions(-)
hw/s390x/s390-pci-bus.c +42 -0
index a929340..c99f2f0 100644
@@ -737,6 +737,46 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn)
     object_unref(OBJECT(iommu));
 }
 
+static S390PCIGroup *s390_group_create(int id)
+{
+    S390PCIGroup *group;
+    S390pciState *s = s390_get_phb();
+
+    group = g_new0(S390PCIGroup, 1);
+    group->id = id;
+    QTAILQ_INSERT_TAIL(&s->zpci_groups, group, link);
+    return group;
+}
+
+S390PCIGroup *s390_group_find(int id)
+{
+    S390PCIGroup *group;
+    S390pciState *s = s390_get_phb();
+
+    QTAILQ_FOREACH(group, &s->zpci_groups, link) {
+        if (group->id == id) {
+            return group;
+        }
+    }
+    return NULL;
+}
+
+static void s390_pci_init_default_group(void)
+{
+    S390PCIGroup *group;
+    ClpRspQueryPciGrp *resgrp;
+
+    group = s390_group_create(ZPCI_DEFAULT_FN_GRP);
+    resgrp = &group->zpci_group;
+    resgrp->fr = 1;
+    stq_p(&resgrp->dasm, 0);
+    stq_p(&resgrp->msia, ZPCI_MSI_ADDR);
+    stw_p(&resgrp->mui, DEFAULT_MUI);
+    stw_p(&resgrp->i, 128);
+    stw_p(&resgrp->maxstbl, 128);
+    resgrp->version = 0;
+}
+
 static void s390_pcihost_realize(DeviceState *dev, Error **errp)
 {
     PCIBus *b;
@@ -764,7 +804,9 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp)
     s->bus_no = 0;
     QTAILQ_INIT(&s->pending_sei);
     QTAILQ_INIT(&s->zpci_devs);
+    QTAILQ_INIT(&s->zpci_groups);
 
+    s390_pci_init_default_group();
     css_register_io_adapters(CSS_IO_ADAPTER_PCI, true, false,
                              S390_ADAPTER_SUPPRESSIBLE, errp);
 }
hw/s390x/s390-pci-inst.c +14 -9
index 639b13c..aeb8b5f 100644
@@ -284,21 +284,25 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
         stq_p(&resquery->edma, ZPCI_EDMA_ADDR);
         stl_p(&resquery->fid, pbdev->fid);
         stw_p(&resquery->pchid, 0);
-        stw_p(&resquery->ug, 1);
+        stw_p(&resquery->ug, ZPCI_DEFAULT_FN_GRP);
         stl_p(&resquery->uid, pbdev->uid);
         stw_p(&resquery->hdr.rsp, CLP_RC_OK);
         break;
     }
     case CLP_QUERY_PCI_FNGRP: {
         ClpRspQueryPciGrp *resgrp = (ClpRspQueryPciGrp *)resh;
-        resgrp->fr = 1;
-        stq_p(&resgrp->dasm, 0);
-        stq_p(&resgrp->msia, ZPCI_MSI_ADDR);
-        stw_p(&resgrp->mui, DEFAULT_MUI);
-        stw_p(&resgrp->i, 128);
-        stw_p(&resgrp->maxstbl, 128);
-        resgrp->version = 0;
 
+        ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh;
+        S390PCIGroup *group;
+
+        group = s390_group_find(reqgrp->g);
+        if (!group) {
+            /* We do not allow access to unknown groups */
+            /* The group must have been obtained with a vfio device */
+            stw_p(&resgrp->hdr.rsp, CLP_RC_QUERYPCIFG_PFGID);
+            goto out;
+        }
+        memcpy(resgrp, &group->zpci_group, sizeof(ClpRspQueryPciGrp));
         stw_p(&resgrp->hdr.rsp, CLP_RC_OK);
         break;
     }
@@ -754,7 +758,8 @@ int pcistb_service_call(S390CPU *cpu, uint8_t r1, uint8_t r3, uint64_t gaddr,
     }
     /* Length must be greater than 8, a multiple of 8 */
     /* and not greater than maxstbl */
-    if ((len <= 8) || (len % 8) || (len > pbdev->maxstbl)) {
+    if ((len <= 8) || (len % 8) ||
+        (len > pbdev->pci_group->zpci_group.maxstbl)) {
         goto specification_error;
     }
     /* Do not cross a 4K-byte boundary */
include/hw/s390x/s390-pci-bus.h +10 -0
index 04e7cb6..52f6e7b 100644
@@ -308,6 +308,14 @@ typedef struct ZpciFmb {
 } ZpciFmb;
 QEMU_BUILD_BUG_MSG(offsetof(ZpciFmb, fmt0) != 48, "padding in ZpciFmb");
 
+#define ZPCI_DEFAULT_FN_GRP 0x20
+typedef struct S390PCIGroup {
+    ClpRspQueryPciGrp zpci_group;
+    int id;
+    QTAILQ_ENTRY(S390PCIGroup) link;
+} S390PCIGroup;
+S390PCIGroup *s390_group_find(int id);
+
 struct S390PCIBusDevice {
     DeviceState qdev;
     PCIDevice *pdev;
@@ -325,6 +333,7 @@ struct S390PCIBusDevice {
     uint16_t noi;
     uint16_t maxstbl;
     uint8_t sum;
+    S390PCIGroup *pci_group;
     S390MsixInfo msix;
     AdapterRoutes routes;
     S390PCIIOMMU *iommu;
@@ -349,6 +358,7 @@ struct S390pciState {
     GHashTable *zpci_table;
     QTAILQ_HEAD(, SeiContainer) pending_sei;
     QTAILQ_HEAD(, S390PCIBusDevice) zpci_devs;
+    QTAILQ_HEAD(, S390PCIGroup) zpci_groups;
 };
 
 S390pciState *s390_get_phb(void);
Signature · 1 line
1.8.3.1
Reviewed-by on v3 6/10
#8 lore raw

Re: [PATCH v3 06/10] s390x/pci: use a PCI Group structure

Quoted 8 lines · On Wed, 7 Oct 2020 15:04:11 -0400 Matthew Rosato <[email protected]>…

From: Pierre Morel <[email protected]>

We use a S390PCIGroup structure to hold the information related to a zPCI Function group.

This allows us to be ready to support multiple groups and to retrieve the group information from the host.

Quoted 7 lines
---
 hw/s390x/s390-pci-bus.c         | 42 +++++++++++++++++++++++++++++++++++++++++
 hw/s390x/s390-pci-inst.c        | 23 +++++++++++++---------
 include/hw/s390x/s390-pci-bus.h | 10 ++++++++++
 3 files changed, 66 insertions(+), 9 deletions(-)
#9 lore raw

PATCH v3 7/10 s390x/pci: clean up s390 PCI groups

In reply to Matthew Rosato

Add a step to remove all stashed PCI groups to avoid stale data between machine resets.

---
 hw/s390x/s390-pci-bus.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)
hw/s390x/s390-pci-bus.c +12 -0
index c99f2f0..c34f14a 100644
@@ -811,6 +811,17 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp)
                              S390_ADAPTER_SUPPRESSIBLE, errp);
 }
 
+static void s390_pcihost_unrealize(DeviceState *dev)
+{
+    S390PCIGroup *group;
+    S390pciState *s = S390_PCI_HOST_BRIDGE(dev);
+
+    while (!QTAILQ_EMPTY(&s->zpci_groups)) {
+        group = QTAILQ_FIRST(&s->zpci_groups);
+        QTAILQ_REMOVE(&s->zpci_groups, group, link);
+    }
+}
+
 static int s390_pci_msix_init(S390PCIBusDevice *pbdev)
 {
     char *name;
@@ -1165,6 +1176,7 @@ static void s390_pcihost_class_init(ObjectClass *klass, void *data)
 
     dc->reset = s390_pcihost_reset;
     dc->realize = s390_pcihost_realize;
+    dc->unrealize = s390_pcihost_unrealize;
     hc->pre_plug = s390_pcihost_pre_plug;
     hc->plug = s390_pcihost_plug;
     hc->unplug_request = s390_pcihost_unplug_request;
Signature · 1 line
1.8.3.1
#11 lore raw

PATCH v3 8/10 s390x/pci: use a PCI Function structure

In reply to Matthew Rosato

From: Pierre Morel <[email protected]>

We use a ClpRspQueryPci structure to hold the information related to a zPCI Function.

This allows us to be ready to support different zPCI functions and to retrieve the zPCI function information from the host.

---
 hw/s390x/s390-pci-bus.c         | 22 +++++++++++++++++-----
 hw/s390x/s390-pci-inst.c        |  8 ++------
 include/hw/s390x/s390-pci-bus.h |  1 +
 3 files changed, 20 insertions(+), 11 deletions(-)
hw/s390x/s390-pci-bus.c +17 -5
index c34f14a..3dc8a10 100644
@@ -777,6 +777,17 @@ static void s390_pci_init_default_group(void)
     resgrp->version = 0;
 }
 
+static void set_pbdev_info(S390PCIBusDevice *pbdev)
+{
+    pbdev->zpci_fn.sdma = ZPCI_SDMA_ADDR;
+    pbdev->zpci_fn.edma = ZPCI_EDMA_ADDR;
+    pbdev->zpci_fn.pchid = 0;
+    pbdev->zpci_fn.ug = ZPCI_DEFAULT_FN_GRP;
+    pbdev->zpci_fn.fid = pbdev->fid;
+    pbdev->zpci_fn.uid = pbdev->uid;
+    pbdev->pci_group = s390_group_find(ZPCI_DEFAULT_FN_GRP);
+}
+
 static void s390_pcihost_realize(DeviceState *dev, Error **errp)
 {
     PCIBus *b;
@@ -994,17 +1005,18 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
             }
         }
 
+        pbdev->pdev = pdev;
+        pbdev->iommu = s390_pci_get_iommu(s, pci_get_bus(pdev), pdev->devfn);
+        pbdev->iommu->pbdev = pbdev;
+        pbdev->state = ZPCI_FS_DISABLED;
+        set_pbdev_info(pbdev);
+
         if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) {
             pbdev->fh |= FH_SHM_VFIO;
         } else {
             pbdev->fh |= FH_SHM_EMUL;
         }
 
-        pbdev->pdev = pdev;
-        pbdev->iommu = s390_pci_get_iommu(s, pci_get_bus(pdev), pdev->devfn);
-        pbdev->iommu->pbdev = pbdev;
-        pbdev->state = ZPCI_FS_DISABLED;
-
         if (s390_pci_msix_init(pbdev)) {
             error_setg(errp, "MSI-X support is mandatory "
                        "in the S390 architecture");
hw/s390x/s390-pci-inst.c +2 -6
index aeb8b5f..cc3d274 100644
@@ -267,6 +267,8 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
             goto out;
         }
 
+        memcpy(resquery, &pbdev->zpci_fn, sizeof(*resquery));
+
         for (i = 0; i < PCI_BAR_COUNT; i++) {
             uint32_t data = pci_get_long(pbdev->pdev->config +
                 PCI_BASE_ADDRESS_0 + (i * 4));
@@ -280,12 +282,6 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)
                     resquery->bar_size[i]);
         }
 
-        stq_p(&resquery->sdma, ZPCI_SDMA_ADDR);
-        stq_p(&resquery->edma, ZPCI_EDMA_ADDR);
-        stl_p(&resquery->fid, pbdev->fid);
-        stw_p(&resquery->pchid, 0);
-        stw_p(&resquery->ug, ZPCI_DEFAULT_FN_GRP);
-        stl_p(&resquery->uid, pbdev->uid);
         stw_p(&resquery->hdr.rsp, CLP_RC_OK);
         break;
     }
include/hw/s390x/s390-pci-bus.h +1 -0
index 52f6e7b..79206be 100644
@@ -334,6 +334,7 @@ struct S390PCIBusDevice {
     uint16_t maxstbl;
     uint8_t sum;
     S390PCIGroup *pci_group;
+    ClpRspQueryPci zpci_fn;
     S390MsixInfo msix;
     AdapterRoutes routes;
     S390PCIIOMMU *iommu;
Signature · 1 line
1.8.3.1
Reviewed-by on v3 8/10
#12 lore raw

Re: [PATCH v3 08/10] s390x/pci: use a PCI Function structure

Quoted 8 lines · On Wed, 7 Oct 2020 15:04:13 -0400 Matthew Rosato <[email protected]>…

From: Pierre Morel <[email protected]>

We use a ClpRspQueryPci structure to hold the information related to a zPCI Function.

This allows us to be ready to support different zPCI functions and to retrieve the zPCI function information from the host.

Quoted 7 lines
---
 hw/s390x/s390-pci-bus.c         | 22 +++++++++++++++++-----
 hw/s390x/s390-pci-inst.c        |  8 ++------
 include/hw/s390x/s390-pci-bus.h |  1 +
 3 files changed, 20 insertions(+), 11 deletions(-)
#13 lore raw

PATCH v3 9/10 vfio: Add routine for finding VFIO_DEVICE_GET_INFO capabilities

In reply to Matthew Rosato

Now that VFIO_DEVICE_GET_INFO supports capability chains, add a helper function to find specific capabilities in the chain.

---
 hw/vfio/common.c              | 10 ++++++++++
 include/hw/vfio/vfio-common.h |  2 ++
 2 files changed, 12 insertions(+)
hw/vfio/common.c +10 -0
index e47a4d7..7cda9d2 100644
@@ -849,6 +849,16 @@ vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)
     return vfio_get_cap((void *)info, info->cap_offset, id);
 }
 
+struct vfio_info_cap_header *
+vfio_get_device_info_cap(struct vfio_device_info *info, uint16_t id)
+{
+    if (!(info->flags & VFIO_DEVICE_FLAGS_CAPS)) {
+        return NULL;
+    }
+
+    return vfio_get_cap((void *)info, info->cap_offset, id);
+}
+
 static int vfio_setup_region_sparse_mmaps(VFIORegion *region,
                                           struct vfio_region_info *info)
 {
include/hw/vfio/vfio-common.h +2 -0
index c78f3ff..3c236ed 100644
@@ -191,6 +191,8 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,
 bool vfio_has_region_cap(VFIODevice *vbasedev, int region, uint16_t cap_type);
 struct vfio_info_cap_header *
 vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id);
+struct vfio_info_cap_header *
+vfio_get_device_info_cap(struct vfio_device_info *info, uint16_t id);
 #endif
 extern const MemoryListener vfio_prereg_listener;
 
Signature · 1 line
1.8.3.1
Reviewed-by on v3 9/10
#14 lore raw

Re: [PATCH v3 09/10] vfio: Add routine for finding VFIO_DEVICE_GET_INFO capabilities

Quoted 8 lines · On Wed, 7 Oct 2020 15:04:14 -0400 Matthew Rosato <[email protected]>…

Now that VFIO_DEVICE_GET_INFO supports capability chains, add a helper function to find specific capabilities in the chain.

---
 hw/vfio/common.c              | 10 ++++++++++
 include/hw/vfio/vfio-common.h |  2 ++
 2 files changed, 12 insertions(+)
#15 lore raw

PATCH v3 10/10 s390x/pci: get zPCI function info from host

In reply to Matthew Rosato

We use the capability chains of the VFIO_DEVICE_GET_INFO ioctl to retrieve the CLP information that the kernel exports.

To be compatible with previous kernel versions we fall back on previous predefined values, same as the emulation values, when the ioctl is found to not support capability chains. If individual CLP capabilities are not found, we fall back on default values for only those capabilities missing from the chain.

This patch is based on work previously done by Pierre Morel.

---
 hw/s390x/meson.build             |   1 +
 hw/s390x/s390-pci-bus.c          |  10 +-
 hw/s390x/s390-pci-vfio.c         | 197 +++++++++++++++++++++++++++++++++++++++
 include/hw/s390x/s390-pci-bus.h  |   1 +
 include/hw/s390x/s390-pci-clp.h  |  12 ++-
 include/hw/s390x/s390-pci-vfio.h |  19 ++++
 6 files changed, 233 insertions(+), 7 deletions(-)
 create mode 100644 hw/s390x/s390-pci-vfio.c
 create mode 100644 include/hw/s390x/s390-pci-vfio.h
hw/s390x/meson.build +1 -0
index 948ceae..3ee4594 100644
@@ -10,6 +10,7 @@ s390x_ss.add(files(
   's390-ccw.c',
   's390-pci-bus.c',
   's390-pci-inst.c',
+  's390-pci-vfio.c',
   's390-skeys.c',
   's390-stattrib.c',
   's390-virtio-hcall.c',
hw/s390x/s390-pci-bus.c +7 -3
index 3dc8a10..182e76e 100644
@@ -17,6 +17,7 @@
 #include "cpu.h"
 #include "hw/s390x/s390-pci-bus.h"
 #include "hw/s390x/s390-pci-inst.h"
+#include "hw/s390x/s390-pci-vfio.h"
 #include "hw/pci/pci_bus.h"
 #include "hw/qdev-properties.h"
 #include "hw/pci/pci_bridge.h"
@@ -737,7 +738,7 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn)
     object_unref(OBJECT(iommu));
 }
 
-static S390PCIGroup *s390_group_create(int id)
+S390PCIGroup *s390_group_create(int id)
 {
     S390PCIGroup *group;
     S390pciState *s = s390_get_phb();
@@ -782,7 +783,7 @@ static void set_pbdev_info(S390PCIBusDevice *pbdev)
     pbdev->zpci_fn.sdma = ZPCI_SDMA_ADDR;
     pbdev->zpci_fn.edma = ZPCI_EDMA_ADDR;
     pbdev->zpci_fn.pchid = 0;
-    pbdev->zpci_fn.ug = ZPCI_DEFAULT_FN_GRP;
+    pbdev->zpci_fn.pfgid = ZPCI_DEFAULT_FN_GRP;
     pbdev->zpci_fn.fid = pbdev->fid;
     pbdev->zpci_fn.uid = pbdev->uid;
     pbdev->pci_group = s390_group_find(ZPCI_DEFAULT_FN_GRP);
@@ -861,7 +862,8 @@ static int s390_pci_msix_init(S390PCIBusDevice *pbdev)
     name = g_strdup_printf("msix-s390-%04x", pbdev->uid);
     memory_region_init_io(&pbdev->msix_notify_mr, OBJECT(pbdev),
                           &s390_msi_ctrl_ops, pbdev, name, PAGE_SIZE);
-    memory_region_add_subregion(&pbdev->iommu->mr, ZPCI_MSI_ADDR,
+    memory_region_add_subregion(&pbdev->iommu->mr,
+                                pbdev->pci_group->zpci_group.msia,
                                 &pbdev->msix_notify_mr);
     g_free(name);
 
@@ -1013,6 +1015,8 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,
 
         if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) {
             pbdev->fh |= FH_SHM_VFIO;
+            /* Fill in CLP information passed via the vfio region */
+            s390_pci_get_clp_info(pbdev);
         } else {
             pbdev->fh |= FH_SHM_EMUL;
         }
hw/s390x/s390-pci-vfio.c new file +197 -0

197 lines not on this page · raw message

include/hw/s390x/s390-pci-bus.h +1 -0
index 79206be..945d5c1 100644
@@ -314,6 +314,7 @@ typedef struct S390PCIGroup {
     int id;
     QTAILQ_ENTRY(S390PCIGroup) link;
 } S390PCIGroup;
+S390PCIGroup *s390_group_create(int id);
 S390PCIGroup *s390_group_find(int id);
 
 struct S390PCIBusDevice {
include/hw/s390x/s390-pci-clp.h +8 -4
index 3708acd..ea2b137 100644
@@ -79,6 +79,7 @@ typedef struct ClpFhListEntry {
 #define CLP_SET_DISABLE_PCI_FN 1 /* Yes, 1 disables it */
 
 #define CLP_UTIL_STR_LEN 64
+#define CLP_PFIP_NR_SEGMENTS 4
 
 #define CLP_MASK_FMT 0xf0000000
 
@@ -120,14 +121,17 @@ typedef struct ClpRspQueryPci {
     uint32_t fmt;
     uint64_t reserved1;
     uint16_t vfn; /* virtual fn number */
-#define CLP_RSP_QPCI_MASK_UTIL  0x100
-#define CLP_RSP_QPCI_MASK_PFGID 0xff
-    uint16_t ug;
+#define CLP_RSP_QPCI_MASK_UTIL  0x01
+    uint8_t flags;
+    uint8_t pfgid;
     uint32_t fid; /* pci function id */
     uint8_t bar_size[PCI_BAR_COUNT];
     uint16_t pchid;
     uint32_t bar[PCI_BAR_COUNT];
-    uint64_t reserved2;
+    uint8_t pfip[CLP_PFIP_NR_SEGMENTS];
+    uint16_t reserved2;
+    uint8_t fmbl;
+    uint8_t pft;
     uint64_t sdma; /* start dma as */
     uint64_t edma; /* end dma as */
     uint32_t reserved3[11];
include/hw/s390x/s390-pci-vfio.h new file +19 -0
new file mode 100644
index 0000000..f690dae
@@ -0,0 +1,19 @@
+/*
+ * s390 vfio-pci interfaces
+ *
+ * Copyright 2020 IBM Corp.
+ * Author(s): Matthew Rosato <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#ifndef HW_S390_PCI_VFIO_H
+#define HW_S390_PCI_VFIO_H
+
+#include "hw/s390x/s390-pci-bus.h"
+
+void s390_pci_get_clp_info(S390PCIBusDevice *pbdev);
+
+#endif
Signature · 1 line
1.8.3.1
#16 lore raw

Re: [PATCH v3 10/10] s390x/pci: get zPCI function info from host

Quoted 22 lines · On Wed, 7 Oct 2020 15:04:15 -0400 Matthew Rosato <[email protected]>…

We use the capability chains of the VFIO_DEVICE_GET_INFO ioctl to retrieve the CLP information that the kernel exports.

To be compatible with previous kernel versions we fall back on previous predefined values, same as the emulation values, when the ioctl is found to not support capability chains. If individual CLP capabilities are not found, we fall back on default values for only those capabilities missing from the chain.

This patch is based on work previously done by Pierre Morel.

---
 hw/s390x/meson.build             |   1 +
 hw/s390x/s390-pci-bus.c          |  10 +-
 hw/s390x/s390-pci-vfio.c         | 197 +++++++++++++++++++++++++++++++++++++++
 include/hw/s390x/s390-pci-bus.h  |   1 +
 include/hw/s390x/s390-pci-clp.h  |  12 ++-
 include/hw/s390x/s390-pci-vfio.h |  19 ++++
 6 files changed, 233 insertions(+), 7 deletions(-)
 create mode 100644 hw/s390x/s390-pci-vfio.c
 create mode 100644 include/hw/s390x/s390-pci-vfio.h

(...)

Quoted 27 lines
hw/s390x/s390-pci-vfio.c new file +21 -0
new file mode 100644
index 0000000..43684c6
@@ -0,0 +1,197 @@
+/*
+ * s390 vfio-pci interfaces
+ *
+ * Copyright 2020 IBM Corp.
+ * Author(s): Matthew Rosato <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include <sys/ioctl.h>
+#include <linux/vfio.h>
+#include <linux/vfio_zdev.h>
+
+#include "qemu/osdep.h"
+#include "hw/s390x/s390-pci-bus.h"
+#include "hw/s390x/s390-pci-clp.h"
+#include "hw/s390x/s390-pci-vfio.h"
+#include "hw/vfio/pci.h"
+

hw/s390x/s390-pci-vfio.c line 31

Quoted 10 lines
hw/s390x/s390-pci-vfio.c new file +10 -0
+#ifndef DEBUG_S390PCI_VFIO
+#define DEBUG_S390PCI_VFIO  0
+#endif
+
+#define DPRINTF(fmt, ...)                                          \
+    do {                                                           \
+        if (DEBUG_S390PCI_VFIO) {                                  \
+            fprintf(stderr, "S390pci-vfio: " fmt, ## __VA_ARGS__); \
+        }                                                          \
+    } while (0)

Not really a fan of DPRINTF. Can you maybe use trace events instead?

Other than that, looks good to me.

#17 lore raw

Re: [PATCH v3 10/10] s390x/pci: get zPCI function info from host

Quoted 59 lines · Cornelia Huck
Quoted 22 lines · On Wed, 7 Oct 2020 15:04:15 -0400 Matthew Rosato <[email protected]>…

We use the capability chains of the VFIO_DEVICE_GET_INFO ioctl to retrieve the CLP information that the kernel exports.

To be compatible with previous kernel versions we fall back on previous predefined values, same as the emulation values, when the ioctl is found to not support capability chains. If individual CLP capabilities are not found, we fall back on default values for only those capabilities missing from the chain.

This patch is based on work previously done by Pierre Morel.

---
  hw/s390x/meson.build             |   1 +
  hw/s390x/s390-pci-bus.c          |  10 +-
  hw/s390x/s390-pci-vfio.c         | 197 +++++++++++++++++++++++++++++++++++++++
  include/hw/s390x/s390-pci-bus.h  |   1 +
  include/hw/s390x/s390-pci-clp.h  |  12 ++-
  include/hw/s390x/s390-pci-vfio.h |  19 ++++
  6 files changed, 233 insertions(+), 7 deletions(-)

create mode 100644 hw/s390x/s390-pci-vfio.c create mode 100644 include/hw/s390x/s390-pci-vfio.h

(...)

Quoted 31 lines
hw/s390x/s390-pci-vfio.c new file +25 -0
new file mode 100644
index 0000000..43684c6
@@ -0,0 +1,197 @@
+/*
+ * s390 vfio-pci interfaces
+ *
+ * Copyright 2020 IBM Corp.
+ * Author(s): Matthew Rosato <[email protected]>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or (at
+ * your option) any later version. See the COPYING file in the top-level
+ * directory.
+ */
+
+#include <sys/ioctl.h>
+#include <linux/vfio.h>
+#include <linux/vfio_zdev.h>
+
+#include "qemu/osdep.h"
+#include "hw/s390x/s390-pci-bus.h"
+#include "hw/s390x/s390-pci-clp.h"
+#include "hw/s390x/s390-pci-vfio.h"
+#include "hw/vfio/pci.h"
+
+#ifndef DEBUG_S390PCI_VFIO
+#define DEBUG_S390PCI_VFIO  0
+#endif
+
Quoted 9 lines

hw/s390x/s390-pci-vfio.c line 31

Quoted 6 lines
hw/s390x/s390-pci-vfio.c new file +6 -0
+#define DPRINTF(fmt, ...)                                          \
+    do {                                                           \
+        if (DEBUG_S390PCI_VFIO) {                                  \
+            fprintf(stderr, "S390pci-vfio: " fmt, ## __VA_ARGS__); \
+        }                                                          \
+    } while (0)

Not really a fan of DPRINTF. Can you maybe use trace events instead?

Sure, I was just continuing what -inst.c and -bus.c do today. I'll remove DPRINTF here and look at what trace-events make sense, with a note to convert the rest of s390-pci* to use trace events at some later time.

Quoted 2 lines

Other than that, looks good to me.

Thanks! Assuming nobody has further comments, I'll plan to send both this set (with the change above) and the 's390x/pci: Accomodate vfio DMA limiting' set merged together with a single linux header sync once 5.10-rc1 is available, sound OK?