Retrieve zPCI hardware information from VFIO
Subsystems (2)
v3 cover Retrieve zPCI hardware information from VFIO
This patchset adds code to s390 pci to examine the VFIO_DEVICE_GET_INFO capability chain, looking for capabilities that describe the underlying hardware and features of the passed-through device.
The retrieval of this information is done once per function (and for a subset of data, once per function group) and is performed at time of device plug. Some elements provided via the capability chain must still be forced to default values for now to reflect what QEMU actually provides support for.
The original work for this feature was done by Pierre Morel.
Associated kernel patchset:
https://lkml.org/lkml/2020/10/7/813
Tested using a host kernel with/without the VFIO_DEVICE_GET_INFO changes and a QEMU with/without the VFIO_DEVICE_GET_INFO changes to validate VFIO_DEVICE_GET_INFO remains backwards-compatible.
Changes since v2:
- Added ACKs/RBs (thanks!)
- Squashed the first 2 patches, and tried again using by formatting the set with --find-renames / diff.renames. (Thanks Richard and Paolo!) As before, this first patch can be applied separately but is included here for the sake of simplicity.
- Copied patch 'vfio: Create shared routine for scanning info capabilities' from the 'Accomodate vfio DMA limiting' set, as it's now needed here too. We can drop it from this set later once the former is applied.
- Refresh the linux-header placeholder. Manually omit the qatomic->atomic pvrdma revert for now.
- New patch to add a helper function for finding VFIO_DEVICE_GET_INFO capabilities. Depends on 'vfio: Create shared routine for scanning info capabilities'
- re-write the last patch to use VFIO_DEVICE_GET_INFO capabilities instead of a vfio device region
Matthew Rosato (7):
s390x/pci: Move header files to include/hw/s390x
vfio: Create shared routine for scanning info capabilities
update-linux-headers: Add vfio_zdev.h
linux-headers: update against 5.9-rc8
s390x/pci: clean up s390 PCI groups
vfio: Add routine for finding VFIO_DEVICE_GET_INFO capabilities s390x/pci: get zPCI function info from host
Pierre Morel (3):
s390x/pci: create a header dedicated to PCI CLP
s390x/pci: use a PCI Group structure
s390x/pci: use a PCI Function structure
MAINTAINERS | 1 +
hw/s390x/meson.build | 1 +
hw/s390x/s390-pci-bus.c | 86 ++++++++-
hw/s390x/s390-pci-inst.c | 33 ++--
hw/s390x/s390-pci-vfio.c | 197 +++++++++++++++++++++
hw/s390x/s390-virtio-ccw.c | 2 +-
hw/vfio/common.c | 31 +++-
{hw => include/hw}/s390x/s390-pci-bus.h | 13 ++
.../hw/s390x/s390-pci-clp.h | 123 ++-----------
include/hw/s390x/s390-pci-inst.h | 116 ++++++++++++
include/hw/s390x/s390-pci-vfio.h | 19 ++
include/hw/vfio/vfio-common.h | 2 +
linux-headers/linux/kvm.h | 6 +-
linux-headers/linux/vfio.h | 11 ++
linux-headers/linux/vfio_zdev.h | 78 ++++++++
scripts/update-linux-headers.sh | 2 +-
16 files changed, 575 insertions(+), 146 deletions(-)
create mode 100644 hw/s390x/s390-pci-vfio.c
rename {hw => include/hw}/s390x/s390-pci-bus.h (96%)
rename hw/s390x/s390-pci-inst.h => include/hw/s390x/s390-pci-clp.h (59%)
create mode 100644 include/hw/s390x/s390-pci-inst.h
create mode 100644 include/hw/s390x/s390-pci-vfio.h
create mode 100644 linux-headers/linux/vfio_zdev.hSignature · 1 line
1.8.3.1
PATCH v3 1/10 s390x/pci: Move header files to include/hw/s390x
Seems a more appropriate location for them.
---
MAINTAINERS | 1 +
hw/s390x/s390-pci-bus.c | 4 ++--
hw/s390x/s390-pci-inst.c | 4 ++--
hw/s390x/s390-virtio-ccw.c | 2 +-
{hw => include/hw}/s390x/s390-pci-bus.h | 0
{hw => include/hw}/s390x/s390-pci-inst.h | 0
6 files changed, 6 insertions(+), 5 deletions(-)
rename {hw => include/hw}/s390x/s390-pci-bus.h (100%)
rename {hw => include/hw}/s390x/s390-pci-inst.h (100%)MAINTAINERS +1 -0
@@ -1443,6 +1443,7 @@ S390 PCI M: Matthew Rosato <[email protected]> S: Supported F: hw/s390x/s390-pci* +F: include/hw/s390x/s390-pci* L: [email protected] UniCore32 Machines
hw/s390x/s390-pci-bus.c +2 -2
@@ -15,8 +15,8 @@#include "qapi/error.h" #include "qapi/visitor.h" #include "cpu.h" -#include "s390-pci-bus.h" -#include "s390-pci-inst.h" +#include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-inst.h" #include "hw/pci/pci_bus.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h"
hw/s390x/s390-pci-inst.c +2 -2
@@ -13,12 +13,12 @@#include "qemu/osdep.h" #include "cpu.h" -#include "s390-pci-inst.h" -#include "s390-pci-bus.h" #include "exec/memop.h" #include "exec/memory-internal.h" #include "qemu/error-report.h" #include "sysemu/hw_accel.h" +#include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST
hw/s390x/s390-virtio-ccw.c +1 -1
@@ -28,7 +28,7 @@#include "qemu/error-report.h" #include "qemu/option.h" #include "qemu/qemu-print.h" -#include "s390-pci-bus.h" +#include "hw/s390x/s390-pci-bus.h" #include "sysemu/reset.h" #include "hw/s390x/storage-keys.h" #include "hw/s390x/storage-attributes.h"
include/hw/s390x/s390-pci-bus.h renamed from hw/s390x/s390-pci-bus.h +0 -0
include/hw/s390x/s390-pci-inst.h renamed from hw/s390x/s390-pci-inst.h +0 -0
Signature · 1 line
1.8.3.1
Re: [PATCH v3 01/10] s390x/pci: Move header files to include/hw/s390x
Quoted 13 lines · On Wed, 7 Oct 2020 15:04:06 -0400 Matthew Rosato <[email protected]>…
Seems a more appropriate location for them.
Signed-off-by Matthew Rosato <[email protected]>--- MAINTAINERS | 1 + hw/s390x/s390-pci-bus.c | 4 ++-- hw/s390x/s390-pci-inst.c | 4 ++-- hw/s390x/s390-virtio-ccw.c | 2 +- {hw => include/hw}/s390x/s390-pci-bus.h | 0 {hw => include/hw}/s390x/s390-pci-inst.h | 0 6 files changed, 6 insertions(+), 5 deletions(-) rename {hw => include/hw}/s390x/s390-pci-bus.h (100%) rename {hw => include/hw}/s390x/s390-pci-inst.h (100%)
Much better :)
(I don't have anything on s390-next atm, or I would have queued it straight away. Better wait until we can do a proper headers sync and queue the whole thing.)
PATCH v3 2/10 vfio: Create shared routine for scanning info capabilities
In reply to Matthew RosatoRather than duplicating the same loop in multiple locations, create a static function to do the work.
--- hw/vfio/common.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
hw/vfio/common.c +13 -8
@@ -825,17 +825,12 @@ static void vfio_listener_release(VFIOContainer *container) } } -struct vfio_info_cap_header * -vfio_get_region_info_cap (struct vfio_region_info *info,uint16_t id) +static struct vfio_info_cap_header * +vfio_get_cap (void *ptr,uint32_t cap_offset,uint16_t id) {struct vfio_info_cap_header *hdr; -void *ptr = info; - -if (!(info->flags &VFIO_REGION_INFO_FLAG_CAPS )) { -return NULL ; - } -for (hdr = ptr + info->cap_offset; hdr != ptr; hdr = ptr + hdr->next) { +for (hdr = ptr + cap_offset; hdr != ptr; hdr = ptr + hdr->next) {if (hdr->id == id) {return hdr; }
@@ -844,6 +839,16 @@ vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)return NULL ; } +struct vfio_info_cap_header * +vfio_get_region_info_cap (struct vfio_region_info *info,uint16_t id) +{ +if (!(info->flags &VFIO_REGION_INFO_FLAG_CAPS )) { +return NULL ; + } + +return vfio_get_cap ((void *)info, info->cap_offset, id); +} +static int vfio_setup_region_sparse_mmaps (VFIORegion *region,struct vfio_region_info *info) {
Signature · 1 line
1.8.3.1
PATCH v3 3/10 update-linux-headers: Add vfio_zdev.h
In reply to Matthew Rosatovfio_zdev.h is used by s390x zPCI support to pass device-specific CLP information between host and userspace.
--- scripts/update-linux-headers.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/update-linux-headers.sh +1 -1
@@ -141,7 +141,7 @@ done rm -rf "$output/linux-headers/linux" mkdir -p "$output/linux-headers/linux" -for header in kvm.h vfio.h vfio_ccw.h vhost.h \ +for header in kvm.h vfio.h vfio_ccw.h vfio_zdev.h vhost.h \ psci.h psp-sev.h userfaultfd.h mman.h; do cp "$tmpdir/include/linux/$header" "$output/linux-headers/linux" done
Signature · 1 line
1.8.3.1
PATCH v3 4/10 linux-headers: update against 5.9-rc8
In reply to Matthew RosatoPLACEHOLDER as the kernel patch driving the need for this ("vfio: Introduce capability definitions for VFIO_DEVICE_GET_INFO") isn't merged yet.
--- linux-headers/linux/kvm.h | 6 ++-- linux-headers/linux/vfio.h | 11 ++++++ linux-headers/linux/vfio_zdev.h | 78 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 93 insertions(+), 2 deletions(-) create mode 100644 linux-headers/linux/vfio_zdev.h
linux-headers/linux/kvm.h +4 -2
@@ -790,9 +790,10 @@ struct kvm_ppc_resize_hpt {#define KVM_VM_PPC_HV 1#define KVM_VM_PPC_PR 2 -/* on MIPS, 0 forces trap & emulate, 1 forces VZ ASE */ -#define KVM_VM_MIPS_TE 0 +/* on MIPS, 0 indicates auto, 1 forces VZ ASE, 2 forces trap & emulate */ +#define KVM_VM_MIPS_AUTO 0#define KVM_VM_MIPS_VZ 1 +#define KVM_VM_MIPS_TE 2#define KVM_S390_SIE_PAGE_OFFSET 1
@@ -1035,6 +1036,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_LAST_CPU 184#define KVM_CAP_SMALLER_MAXPHYADDR 185#define KVM_CAP_S390_DIAG318 186 +#define KVM_CAP_STEAL_TIME 187#ifdef KVM_CAP_IRQ_ROUTING
linux-headers/linux/vfio.h +11 -0
@@ -201,8 +201,10 @@ struct vfio_device_info {#define VFIO_DEVICE_FLAGS_AMBA (1 << 3)/* vfio-amba device */ #define VFIO_DEVICE_FLAGS_CCW (1 << 4)/* vfio-ccw device */ #define VFIO_DEVICE_FLAGS_AP (1 << 5)/* vfio-ap device */ +#define VFIO_DEVICE_FLAGS_CAPS (1 << 6)/* Info supports caps */ __u32 num_regions;/* Max region index + 1 */ __u32 num_irqs;/* Max IRQ index + 1 */ +__u32 cap_offset;/* Offset within info struct of first cap */ };#define VFIO_DEVICE_GET_INFO _IO(VFIO_TYPE, VFIO_BASE + 7)
@@ -218,6 +220,15 @@ struct vfio_device_info {#define VFIO_DEVICE_API_CCW_STRING "vfio-ccw"#define VFIO_DEVICE_API_AP_STRING "vfio-ap" +/* +* The following capabilities are unique to s390 zPCI devices. Their contents +* are further-defined in vfio_zdev.h +*/ +#define VFIO_DEVICE_INFO_CAP_ZPCI_BASE 1 +#define VFIO_DEVICE_INFO_CAP_ZPCI_GROUP 2 +#define VFIO_DEVICE_INFO_CAP_ZPCI_UTIL 3 +#define VFIO_DEVICE_INFO_CAP_ZPCI_PFIP 4 +/** * VFIO_DEVICE_GET_REGION_INFO - _IOWR(VFIO_TYPE, VFIO_BASE + 8, * struct vfio_region_info)
linux-headers/linux/vfio_zdev.h new file +78 -0
78 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v3 5/10 s390x/pci: create a header dedicated to PCI CLP
In reply to Matthew RosatoFrom: Pierre Morel <[email protected]>
To have a clean separation between s390-pci-bus.h and s390-pci-inst.h headers we export the PCI CLP instructions in a dedicated header.
--- include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-clp.h | 211 +++++++++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-inst.h | 196 ------------------------------------ 3 files changed, 212 insertions(+), 196 deletions(-) create mode 100644 include/hw/s390x/s390-pci-clp.h
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -19,6 +19,7 @@#include "hw/s390x/sclp.h" #include "hw/s390x/s390_flic.h" #include "hw/s390x/css.h" +#include "hw/s390x/s390-pci-clp.h" #include "qom/object.h" #define TYPE_S390_PCI_HOST_BRIDGE "s390-pcihost"
include/hw/s390x/s390-pci-clp.h new file +211 -0
211 lines not on this page · raw message
include/hw/s390x/s390-pci-inst.h +0 -196
202 lines not on this page · raw message
Signature · 1 line
1.8.3.1
PATCH v3 6/10 s390x/pci: use a PCI Group structure
In reply to Matthew RosatoFrom: Pierre Morel <[email protected]>
We use a S390PCIGroup structure to hold the information related to a zPCI Function group.
This allows us to be ready to support multiple groups and to retrieve the group information from the host.
--- hw/s390x/s390-pci-bus.c | 42 +++++++++++++++++++++++++++++++++++++++++ hw/s390x/s390-pci-inst.c | 23 +++++++++++++--------- include/hw/s390x/s390-pci-bus.h | 10 ++++++++++ 3 files changed, 66 insertions(+), 9 deletions(-)
hw/s390x/s390-pci-bus.c +42 -0
@@ -737,6 +737,46 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn)object_unref (OBJECT (iommu)); } +static S390PCIGroup *s390_group_create (int id) +{ +S390PCIGroup *group; +S390pciState *s =s390_get_phb (); + + group =g_new0 (S390PCIGroup,1 ); + group->id = id; +QTAILQ_INSERT_TAIL (&s->zpci_groups, group, link); +return group; +} + +S390PCIGroup *s390_group_find (int id) +{ +S390PCIGroup *group; +S390pciState *s =s390_get_phb (); + + QTAILQ_FOREACH(group, &s->zpci_groups, link) { +if (group->id == id) { +return group; + } + } +return NULL ; +} + +static void s390_pci_init_default_group (void ) +{ +S390PCIGroup *group; +ClpRspQueryPciGrp *resgrp; + + group =s390_group_create (ZPCI_DEFAULT_FN_GRP ); + resgrp = &group->zpci_group; + resgrp->fr =1 ; +stq_p (&resgrp->dasm,0 ); +stq_p (&resgrp->msia,ZPCI_MSI_ADDR ); +stw_p (&resgrp->mui,DEFAULT_MUI ); +stw_p (&resgrp->i,128 ); +stw_p (&resgrp->maxstbl,128 ); + resgrp->version =0 ; +} +static void s390_pcihost_realize (DeviceState *dev,Error **errp) {PCIBus *b;
@@ -764,7 +804,9 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp) s->bus_no =0 ;QTAILQ_INIT (&s->pending_sei);QTAILQ_INIT (&s->zpci_devs); +QTAILQ_INIT (&s->zpci_groups); +s390_pci_init_default_group ();css_register_io_adapters (CSS_IO_ADAPTER_PCI , true, false,S390_ADAPTER_SUPPRESSIBLE , errp); }
hw/s390x/s390-pci-inst.c +14 -9
@@ -284,21 +284,25 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra)stq_p (&resquery->edma,ZPCI_EDMA_ADDR );stl_p (&resquery->fid, pbdev->fid);stw_p (&resquery->pchid,0 ); -stw_p (&resquery->ug,1 ); +stw_p (&resquery->ug,ZPCI_DEFAULT_FN_GRP );stl_p (&resquery->uid, pbdev->uid);stw_p (&resquery->hdr.rsp,CLP_RC_OK );break ; }case CLP_QUERY_PCI_FNGRP : {ClpRspQueryPciGrp *resgrp = (ClpRspQueryPciGrp *)resh; - resgrp->fr =1 ; -stq_p (&resgrp->dasm,0 ); -stq_p (&resgrp->msia,ZPCI_MSI_ADDR ); -stw_p (&resgrp->mui,DEFAULT_MUI ); -stw_p (&resgrp->i,128 ); -stw_p (&resgrp->maxstbl,128 ); - resgrp->version =0 ; +ClpReqQueryPciGrp *reqgrp = (ClpReqQueryPciGrp *)reqh; +S390PCIGroup *group; + + group =s390_group_find (reqgrp->g); +if (!group) { +/* We do not allow access to unknown groups */ +/* The group must have been obtained with a vfio device */ +stw_p (&resgrp->hdr.rsp,CLP_RC_QUERYPCIFG_PFGID ); + goto out; + } +memcpy (resgrp, &group->zpci_group,sizeof (ClpRspQueryPciGrp));stw_p (&resgrp->hdr.rsp,CLP_RC_OK );break ; }
@@ -754,7 +758,8 @@ int pcistb_service_call(S390CPU *cpu, uint8_t r1, uint8_t r3, uint64_t gaddr, }/* Length must be greater than 8, a multiple of 8 */ /* and not greater than maxstbl */ -if ((len <=8 ) || (len %8 ) || (len > pbdev->maxstbl)) { +if ((len <=8 ) || (len %8 ) || + (len > pbdev->pci_group->zpci_group.maxstbl)) { goto specification_error; }/* Do not cross a 4K-byte boundary */
include/hw/s390x/s390-pci-bus.h +10 -0
@@ -308,6 +308,14 @@ typedef struct ZpciFmb { } ZpciFmb;QEMU_BUILD_BUG_MSG (offsetof(ZpciFmb , fmt0) !=48 ,"padding in ZpciFmb" ); +#define ZPCI_DEFAULT_FN_GRP 0x20 +typedef struct S390PCIGroup { +ClpRspQueryPciGrp zpci_group; +int id; +QTAILQ_ENTRY (S390PCIGroup) link; +}S390PCIGroup ; +S390PCIGroup *s390_group_find (int id); +struct S390PCIBusDevice {DeviceState qdev;PCIDevice *pdev;
@@ -325,6 +333,7 @@ struct S390PCIBusDevice {uint16_t noi;uint16_t maxstbl;uint8_t sum; +S390PCIGroup *pci_group;S390MsixInfo msix;AdapterRoutes routes;S390PCIIOMMU *iommu;
@@ -349,6 +358,7 @@ struct S390pciState {GHashTable *zpci_table; QTAILQ_HEAD(, SeiContainer) pending_sei; QTAILQ_HEAD(, S390PCIBusDevice) zpci_devs; + QTAILQ_HEAD(, S390PCIGroup) zpci_groups; };S390pciState *s390_get_phb (void );
Signature · 1 line
1.8.3.1
Re: [PATCH v3 06/10] s390x/pci: use a PCI Group structure
Quoted 8 lines · On Wed, 7 Oct 2020 15:04:11 -0400 Matthew Rosato <[email protected]>…
From: Pierre Morel <[email protected]>
We use a S390PCIGroup structure to hold the information related to a zPCI Function group.
This allows us to be ready to support multiple groups and to retrieve the group information from the host.
Quoted 7 lines
Signed-off-by Pierre Morel <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 42 +++++++++++++++++++++++++++++++++++++++++ hw/s390x/s390-pci-inst.c | 23 +++++++++++++--------- include/hw/s390x/s390-pci-bus.h | 10 ++++++++++ 3 files changed, 66 insertions(+), 9 deletions(-)
PATCH v3 7/10 s390x/pci: clean up s390 PCI groups
In reply to Matthew RosatoAdd a step to remove all stashed PCI groups to avoid stale data between machine resets.
--- hw/s390x/s390-pci-bus.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
hw/s390x/s390-pci-bus.c +12 -0
@@ -811,6 +811,17 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp) S390_ADAPTER_SUPPRESSIBLE, errp); } +static void s390_pcihost_unrealize (DeviceState *dev) +{ +S390PCIGroup *group; +S390pciState *s =S390_PCI_HOST_BRIDGE (dev); + +while (!QTAILQ_EMPTY (&s->zpci_groups)) { + group =QTAILQ_FIRST (&s->zpci_groups); +QTAILQ_REMOVE (&s->zpci_groups, group, link); + } +} +static int s390_pci_msix_init (S390PCIBusDevice *pbdev) {char *name;
@@ -1165,6 +1176,7 @@ static void s390_pcihost_class_init(ObjectClass *klass, void *data) dc->reset = s390_pcihost_reset; dc->realize = s390_pcihost_realize; + dc->unrealize = s390_pcihost_unrealize; hc->pre_plug = s390_pcihost_pre_plug; hc->plug = s390_pcihost_plug; hc->unplug_request = s390_pcihost_unplug_request;
Signature · 1 line
1.8.3.1
Re: [PATCH v3 07/10] s390x/pci: clean up s390 PCI groups
Quoted 7 lines · On Wed, 7 Oct 2020 15:04:12 -0400 Matthew Rosato <[email protected]>…
Add a step to remove all stashed PCI groups to avoid stale data between machine resets.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
PATCH v3 8/10 s390x/pci: use a PCI Function structure
In reply to Matthew RosatoFrom: Pierre Morel <[email protected]>
We use a ClpRspQueryPci structure to hold the information related to a zPCI Function.
This allows us to be ready to support different zPCI functions and to retrieve the zPCI function information from the host.
--- hw/s390x/s390-pci-bus.c | 22 +++++++++++++++++----- hw/s390x/s390-pci-inst.c | 8 ++------ include/hw/s390x/s390-pci-bus.h | 1 + 3 files changed, 20 insertions(+), 11 deletions(-)
hw/s390x/s390-pci-bus.c +17 -5
@@ -777,6 +777,17 @@ static void s390_pci_init_default_group(void) resgrp->version =0 ; } +static void set_pbdev_info (S390PCIBusDevice *pbdev) +{ + pbdev->zpci_fn.sdma =ZPCI_SDMA_ADDR ; + pbdev->zpci_fn.edma =ZPCI_EDMA_ADDR ; + pbdev->zpci_fn.pchid =0 ; + pbdev->zpci_fn.ug =ZPCI_DEFAULT_FN_GRP ; + pbdev->zpci_fn.fid = pbdev->fid; + pbdev->zpci_fn.uid = pbdev->uid; + pbdev->pci_group =s390_group_find (ZPCI_DEFAULT_FN_GRP ); +} +static void s390_pcihost_realize (DeviceState *dev,Error **errp) {PCIBus *b;
@@ -994,17 +1005,18 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } + pbdev->pdev = pdev; + pbdev->iommu =s390_pci_get_iommu (s,pci_get_bus (pdev), pdev->devfn); + pbdev->iommu->pbdev = pbdev; + pbdev->state =ZPCI_FS_DISABLED ; +set_pbdev_info (pbdev); +if (object_dynamic_cast (OBJECT (dev),"vfio-pci" )) { pbdev->fh |=FH_SHM_VFIO ; }else { pbdev->fh |=FH_SHM_EMUL ; } - pbdev->pdev = pdev; - pbdev->iommu =s390_pci_get_iommu (s,pci_get_bus (pdev), pdev->devfn); - pbdev->iommu->pbdev = pbdev; - pbdev->state =ZPCI_FS_DISABLED ; -if (s390_pci_msix_init (pbdev)) {error_setg (errp,"MSI-X support is mandatory " "in the S390 architecture" );
hw/s390x/s390-pci-inst.c +2 -6
@@ -267,6 +267,8 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } +memcpy (resquery, &pbdev->zpci_fn,sizeof (*resquery)); +for (i =0 ; i <PCI_BAR_COUNT ; i++) {uint32_t data =pci_get_long (pbdev->pdev->config +PCI_BASE_ADDRESS_0 + (i *4 ));
@@ -280,12 +282,6 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) resquery->bar_size[i]); } -stq_p (&resquery->sdma,ZPCI_SDMA_ADDR ); -stq_p (&resquery->edma,ZPCI_EDMA_ADDR ); -stl_p (&resquery->fid, pbdev->fid); -stw_p (&resquery->pchid,0 ); -stw_p (&resquery->ug,ZPCI_DEFAULT_FN_GRP ); -stl_p (&resquery->uid, pbdev->uid);stw_p (&resquery->hdr.rsp,CLP_RC_OK );break ; }
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -334,6 +334,7 @@ struct S390PCIBusDevice {uint16_t maxstbl;uint8_t sum;S390PCIGroup *pci_group; +ClpRspQueryPci zpci_fn;S390MsixInfo msix;AdapterRoutes routes;S390PCIIOMMU *iommu;
Signature · 1 line
1.8.3.1
Re: [PATCH v3 08/10] s390x/pci: use a PCI Function structure
Quoted 8 lines · On Wed, 7 Oct 2020 15:04:13 -0400 Matthew Rosato <[email protected]>…
From: Pierre Morel <[email protected]>
We use a ClpRspQueryPci structure to hold the information related to a zPCI Function.
This allows us to be ready to support different zPCI functions and to retrieve the zPCI function information from the host.
Quoted 7 lines
Signed-off-by Pierre Morel <[email protected]>Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 22 +++++++++++++++++----- hw/s390x/s390-pci-inst.c | 8 ++------ include/hw/s390x/s390-pci-bus.h | 1 + 3 files changed, 20 insertions(+), 11 deletions(-)
PATCH v3 9/10 vfio: Add routine for finding VFIO_DEVICE_GET_INFO capabilities
In reply to Matthew RosatoNow that VFIO_DEVICE_GET_INFO supports capability chains, add a helper function to find specific capabilities in the chain.
--- hw/vfio/common.c | 10 ++++++++++ include/hw/vfio/vfio-common.h | 2 ++ 2 files changed, 12 insertions(+)
hw/vfio/common.c +10 -0
@@ -849,6 +849,16 @@ vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id)return vfio_get_cap ((void *)info, info->cap_offset, id); } +struct vfio_info_cap_header * +vfio_get_device_info_cap (struct vfio_device_info *info,uint16_t id) +{ +if (!(info->flags &VFIO_DEVICE_FLAGS_CAPS )) { +return NULL ; + } + +return vfio_get_cap ((void *)info, info->cap_offset, id); +} +static int vfio_setup_region_sparse_mmaps (VFIORegion *region,struct vfio_region_info *info) {
include/hw/vfio/vfio-common.h +2 -0
@@ -191,6 +191,8 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type,bool vfio_has_region_cap (VFIODevice *vbasedev,int region,uint16_t cap_type);struct vfio_info_cap_header *vfio_get_region_info_cap (struct vfio_region_info *info,uint16_t id); +struct vfio_info_cap_header * +vfio_get_device_info_cap (struct vfio_device_info *info,uint16_t id);#endif extern const MemoryListener vfio_prereg_listener;
Signature · 1 line
1.8.3.1
Re: [PATCH v3 09/10] vfio: Add routine for finding VFIO_DEVICE_GET_INFO capabilities
Quoted 8 lines · On Wed, 7 Oct 2020 15:04:14 -0400 Matthew Rosato <[email protected]>…
Now that
VFIO_DEVICE_GET_INFOsupports capability chains, add a helper function to find specific capabilities in the chain.Signed-off-by Matthew Rosato <[email protected]>--- hw/vfio/common.c | 10 ++++++++++ include/hw/vfio/vfio-common.h | 2 ++ 2 files changed, 12 insertions(+)
PATCH v3 10/10 s390x/pci: get zPCI function info from host
In reply to Matthew RosatoWe use the capability chains of the VFIO_DEVICE_GET_INFO ioctl to retrieve the CLP information that the kernel exports.
To be compatible with previous kernel versions we fall back on previous predefined values, same as the emulation values, when the ioctl is found to not support capability chains. If individual CLP capabilities are not found, we fall back on default values for only those capabilities missing from the chain.
This patch is based on work previously done by Pierre Morel.
--- hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 10 +- hw/s390x/s390-pci-vfio.c | 197 +++++++++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-clp.h | 12 ++- include/hw/s390x/s390-pci-vfio.h | 19 ++++ 6 files changed, 233 insertions(+), 7 deletions(-) create mode 100644 hw/s390x/s390-pci-vfio.c create mode 100644 include/hw/s390x/s390-pci-vfio.h
hw/s390x/meson.build +1 -0
@@ -10,6 +10,7 @@ s390x_ss.add(files( 's390-ccw.c', 's390-pci-bus.c', 's390-pci-inst.c', + 's390-pci-vfio.c', 's390-skeys.c', 's390-stattrib.c', 's390-virtio-hcall.c',
hw/s390x/s390-pci-bus.c +7 -3
@@ -17,6 +17,7 @@#include "cpu.h" #include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-inst.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/pci/pci_bus.h" #include "hw/qdev-properties.h" #include "hw/pci/pci_bridge.h"
@@ -737,7 +738,7 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn)object_unref (OBJECT (iommu)); } -static S390PCIGroup *s390_group_create (int id) +S390PCIGroup *s390_group_create (int id) {S390PCIGroup *group;S390pciState *s =s390_get_phb ();
@@ -782,7 +783,7 @@ static void set_pbdev_info(S390PCIBusDevice *pbdev) pbdev->zpci_fn.sdma =ZPCI_SDMA_ADDR ; pbdev->zpci_fn.edma =ZPCI_EDMA_ADDR ; pbdev->zpci_fn.pchid =0 ; - pbdev->zpci_fn.ug =ZPCI_DEFAULT_FN_GRP ; + pbdev->zpci_fn.pfgid =ZPCI_DEFAULT_FN_GRP ; pbdev->zpci_fn.fid = pbdev->fid; pbdev->zpci_fn.uid = pbdev->uid; pbdev->pci_group =s390_group_find (ZPCI_DEFAULT_FN_GRP );
@@ -861,7 +862,8 @@ static int s390_pci_msix_init(S390PCIBusDevice *pbdev) name =g_strdup_printf ("msix-s390-%04x" , pbdev->uid);memory_region_init_io (&pbdev->msix_notify_mr,OBJECT (pbdev), &s390_msi_ctrl_ops, pbdev, name,PAGE_SIZE ); -memory_region_add_subregion (&pbdev->iommu->mr,ZPCI_MSI_ADDR , +memory_region_add_subregion (&pbdev->iommu->mr, + pbdev->pci_group->zpci_group.msia, &pbdev->msix_notify_mr);g_free (name);
@@ -1013,6 +1015,8 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (object_dynamic_cast (OBJECT (dev),"vfio-pci" )) { pbdev->fh |=FH_SHM_VFIO ; +/* Fill in CLP information passed via the vfio region */ +s390_pci_get_clp_info (pbdev); }else { pbdev->fh |=FH_SHM_EMUL ; }
hw/s390x/s390-pci-vfio.c new file +197 -0
197 lines not on this page · raw message
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -314,6 +314,7 @@ typedef struct S390PCIGroup {int id;QTAILQ_ENTRY (S390PCIGroup) link; } S390PCIGroup; +S390PCIGroup *s390_group_create (int id);S390PCIGroup *s390_group_find (int id);struct S390PCIBusDevice {
include/hw/s390x/s390-pci-clp.h +8 -4
@@ -79,6 +79,7 @@ typedef struct ClpFhListEntry {#define CLP_SET_DISABLE_PCI_FN 1/* Yes, 1 disables it */ #define CLP_UTIL_STR_LEN 64 +#define CLP_PFIP_NR_SEGMENTS 4#define CLP_MASK_FMT 0xf0000000
@@ -120,14 +121,17 @@ typedef struct ClpRspQueryPci {uint32_t fmt;uint64_t reserved1;uint16_t vfn;/* virtual fn number */ -#define CLP_RSP_QPCI_MASK_UTIL 0x100 -#define CLP_RSP_QPCI_MASK_PFGID 0xff -uint16_t ug; +#define CLP_RSP_QPCI_MASK_UTIL 0x01 +uint8_t flags; +uint8_t pfgid;uint32_t fid;/* pci function id */ uint8_t bar_size[PCI_BAR_COUNT ];uint16_t pchid;uint32_t bar[PCI_BAR_COUNT ]; -uint64_t reserved2; +uint8_t pfip[CLP_PFIP_NR_SEGMENTS ]; +uint16_t reserved2; +uint8_t fmbl; +uint8_t pft;uint64_t sdma;/* start dma as */ uint64_t edma;/* end dma as */ uint32_t reserved3[11 ];
include/hw/s390x/s390-pci-vfio.h new file +19 -0
@@ -0,0 +1,19 @@ +/* +* s390 vfio-pci interfaces +* +* Copyright 2020 IBM Corp. +* Author(s): Matthew Rosato <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#ifndef HW_S390_PCI_VFIO_H +#define HW_S390_PCI_VFIO_H + +#include "hw/s390x/s390-pci-bus.h" + +void s390_pci_get_clp_info (S390PCIBusDevice *pbdev); + +#endif
Signature · 1 line
1.8.3.1
Re: [PATCH v3 10/10] s390x/pci: get zPCI function info from host
Quoted 22 lines · On Wed, 7 Oct 2020 15:04:15 -0400 Matthew Rosato <[email protected]>…
We use the capability chains of the
VFIO_DEVICE_GET_INFOioctl to retrieve the CLP information that the kernel exports.To be compatible with previous kernel versions we fall back on previous predefined values, same as the emulation values, when the ioctl is found to not support capability chains. If individual CLP capabilities are not found, we fall back on default values for only those capabilities missing from the chain.
This patch is based on work previously done by Pierre Morel.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 10 +- hw/s390x/s390-pci-vfio.c | 197 +++++++++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-clp.h | 12 ++- include/hw/s390x/s390-pci-vfio.h | 19 ++++ 6 files changed, 233 insertions(+), 7 deletions(-) create mode 100644 hw/s390x/s390-pci-vfio.c create mode 100644 include/hw/s390x/s390-pci-vfio.h
(...)
Quoted 27 lines
hw/s390x/s390-pci-vfio.c new file +21
-0@@ -0,0 +1,197 @@ +/* +* s390 vfio-pci interfaces +* +* Copyright 2020 IBM Corp. +* Author(s): Matthew Rosato <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#include <sys/ioctl.h> +#include <linux/vfio.h> +#include <linux/vfio_zdev.h> + +#include "qemu/osdep.h" +#include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-clp.h" +#include "hw/s390x/s390-pci-vfio.h" +#include "hw/vfio/pci.h" +
hw/s390x/s390-pci-vfio.c line 31
Quoted 10 lines
hw/s390x/s390-pci-vfio.c new file +10
-0+#ifndef DEBUG_S390PCI_VFIO +#define DEBUG_S390PCI_VFIO 0 +#endif + +#define DPRINTF (fmt, ...) \ + do { \ + if (DEBUG_S390PCI_VFIO) { \ + fprintf(stderr, "S390pci-vfio: " fmt, ## __VA_ARGS__); \ + } \ + } while (0)
Not really a fan of DPRINTF. Can you maybe use trace events instead?
Other than that, looks good to me.
Re: [PATCH v3 10/10] s390x/pci: get zPCI function info from host
Quoted 59 lines · Cornelia Huck
Quoted 22 lines · On Wed, 7 Oct 2020 15:04:15 -0400 Matthew Rosato <[email protected]>…
We use the capability chains of the
VFIO_DEVICE_GET_INFOioctl to retrieve the CLP information that the kernel exports.To be compatible with previous kernel versions we fall back on previous predefined values, same as the emulation values, when the ioctl is found to not support capability chains. If individual CLP capabilities are not found, we fall back on default values for only those capabilities missing from the chain.
This patch is based on work previously done by Pierre Morel.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 10 +- hw/s390x/s390-pci-vfio.c | 197 +++++++++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-clp.h | 12 ++- include/hw/s390x/s390-pci-vfio.h | 19 ++++ 6 files changed, 233 insertions(+), 7 deletions(-)create mode 100644
hw/s390x/s390-pci-vfio.ccreate mode 100644include/hw/s390x/s390-pci-vfio.h(...)
Quoted 31 lines
hw/s390x/s390-pci-vfio.c new file +25
-0@@ -0,0 +1,197 @@ +/* +* s390 vfio-pci interfaces +* +* Copyright 2020 IBM Corp. +* Author(s): Matthew Rosato <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#include <sys/ioctl.h> +#include <linux/vfio.h> +#include <linux/vfio_zdev.h> + +#include "qemu/osdep.h" +#include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-clp.h" +#include "hw/s390x/s390-pci-vfio.h" +#include "hw/vfio/pci.h" + +#ifndef DEBUG_S390PCI_VFIO +#define DEBUG_S390PCI_VFIO 0 +#endif +
Quoted 9 lines
hw/s390x/s390-pci-vfio.c line 31
Quoted 6 lines
hw/s390x/s390-pci-vfio.c new file +6
-0+#define DPRINTF (fmt, ...) \ + do { \ + if (DEBUG_S390PCI_VFIO) { \ + fprintf(stderr, "S390pci-vfio: " fmt, ## __VA_ARGS__); \ + } \ + } while (0)Not really a fan of DPRINTF. Can you maybe use trace events instead?
Sure, I was just continuing what -inst.c and -bus.c do today. I'll remove DPRINTF here and look at what trace-events make sense, with a note to convert the rest of s390-pci* to use trace events at some later time.
Quoted 2 lines
Other than that, looks good to me.
Thanks! Assuming nobody has further comments, I'll plan to send both this set (with the change above) and the 's390x/pci: Accomodate vfio DMA limiting' set merged together with a single linux header sync once 5.10-rc1 is available, sound OK?