s390x/pci: zPCI interpretation support
v2 cover s390x/pci: zPCI interpretation support
For QEMU, the majority of the work in enabling instruction interpretation is handled via new VFIO ioctls to SET the appropriate interpretation and interrupt forwarding modes, and to GET the function handle to use for interpretive execution.
This series implements these new ioctls, as well as adding a new, optional 'intercept' parameter to zpci to request interpretation support not be used as well as an 'intassist' parameter to determine whether or not the firmware assist will be used for interrupt delivery or whether the host will be responsible for delivering all interrupts.
The ZPCI_INTERP CPU feature is added beginning with the z14 model to enable this support.
As a consequence of implementing zPCI interpretation, ISM devices now become eligible for passthrough (but only when zPCI interpretation is available).
From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu.
Associated kernel series:
https://lore.kernel.org/kvm/[email protected]/
Changes v1->v2:
- Update kernel headers sync
- Drop some pre-req patches that are now merged
- Add some R-bs (Thanks!)
- fence FEAT_ZPCI_INTERP for QEMU 6.2 and older (Christian)
- switch from container_of to VFIO_PCI and drop asserts (Thomas)
- re-arrange g_autofree so we malloc at time of declaration (Thomas)
Matthew Rosato (9):
Update linux headers
target/s390x: add zpci-interp to cpu models
fixup: force interp off for QEMU machine 6.2 and older s390x/pci: enable for load/store intepretation
s390x/pci: don't fence interpreted devices without MSI-X s390x/pci: enable adapter event notification for interpreted devices s390x/pci: use I/O Address Translation assist when interpreting s390x/pci: use dtsm provided from vfio capabilities for interpreted
devices
s390x/pci: let intercept devices have separate PCI groups
hw/s390x/s390-pci-bus.c | 121 ++++++++++- hw/s390x/s390-pci-inst.c | 168 ++++++++++++++- hw/s390x/s390-pci-vfio.c | 204 +++++++++++++++++- hw/s390x/s390-virtio-ccw.c | 1 + include/hw/s390x/s390-pci-bus.h | 8 +- include/hw/s390x/s390-pci-inst.h | 2 +- include/hw/s390x/s390-pci-vfio.h | 45 ++++ include/standard-headers/asm-x86/kvm_para.h | 1 + include/standard-headers/drm/drm_fourcc.h | 11 + include/standard-headers/linux/ethtool.h | 1 + include/standard-headers/linux/fuse.h | 60 +++++- include/standard-headers/linux/pci_regs.h | 4 + include/standard-headers/linux/virtio_iommu.h | 8 +- linux-headers/asm-mips/unistd_n32.h | 1 + linux-headers/asm-mips/unistd_n64.h | 1 + linux-headers/asm-mips/unistd_o32.h | 1 + linux-headers/asm-powerpc/unistd_32.h | 1 + linux-headers/asm-powerpc/unistd_64.h | 1 + linux-headers/asm-s390/kvm.h | 1 + linux-headers/asm-s390/unistd_32.h | 1 + linux-headers/asm-s390/unistd_64.h | 1 + linux-headers/linux/kvm.h | 1 + linux-headers/linux/vfio.h | 22 ++ linux-headers/linux/vfio_zdev.h | 51 +++++ target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 + target/s390x/kvm/kvm.c | 1 + 27 files changed, 693 insertions(+), 27 deletions(-)
Signature · 1 line
2.27.0
PATCH v2 1/9 Update linux headers
This is a placeholder that pulls in 5.17 + unmerged kernel changes required by this item. A proper header sync can be done once the associated kernel code merges.
--- include/standard-headers/asm-x86/kvm_para.h | 1 + include/standard-headers/drm/drm_fourcc.h | 11 ++++ include/standard-headers/linux/ethtool.h | 1 + include/standard-headers/linux/fuse.h | 60 +++++++++++++++++-- include/standard-headers/linux/pci_regs.h | 4 ++ include/standard-headers/linux/virtio_iommu.h | 8 ++- linux-headers/asm-mips/unistd_n32.h | 1 + linux-headers/asm-mips/unistd_n64.h | 1 + linux-headers/asm-mips/unistd_o32.h | 1 + linux-headers/asm-powerpc/unistd_32.h | 1 + linux-headers/asm-powerpc/unistd_64.h | 1 + linux-headers/asm-s390/kvm.h | 1 + linux-headers/asm-s390/unistd_32.h | 1 + linux-headers/asm-s390/unistd_64.h | 1 + linux-headers/linux/kvm.h | 1 + linux-headers/linux/vfio.h | 22 +++++++ linux-headers/linux/vfio_zdev.h | 51 ++++++++++++++++ 17 files changed, 162 insertions(+), 5 deletions(-)
include/standard-headers/asm-x86/kvm_para.h +1 -0
@@ -8,6 +8,7 @@* should be used to determine that a VM is running under KVM. */ #define KVM_CPUID_SIGNATURE 0x40000000 +#define KVM_SIGNATURE "KVMKVMKVM\0\0\0"/* This CPUID returns two feature bitmaps in eax, edx. Before enabling * a particular paravirtualization, the appropriate feature bit should
include/standard-headers/drm/drm_fourcc.h +11 -0
@@ -313,6 +313,13 @@ extern "C" {*/ #define DRM_FORMAT_P016 fourcc_code('P', '0', '1', '6')/* 2x2 subsampled Cr:Cb plane 16 bits per channel */ +/* 2 plane YCbCr420. +* 3 10 bit components and 2 padding bits packed into 4 bytes. +* index 0 = Y plane, [31:0] x:Y2:Y1:Y0 2:10:10:10 little endian +* index 1 = Cr:Cb plane, [63:0] x:Cr2:Cb2:Cr1:x:Cb1:Cr0:Cb0 [2:10:10:10:2:10:10:10] little endian +*/ +#define DRM_FORMAT_P030 fourcc_code('P', '0', '3', '0')/* 2x2 subsampled Cr:Cb plane 10 bits per channel packed */ +/* 3 plane non-subsampled (444) YCbCr * 16 bits per component, but only 10 bits are used and 6 bits are padded * index 0: Y plane, [15:0] Y:x [10:6] little endian
@@ -853,6 +860,10 @@ drm_fourcc_canonicalize_nvidia_format_mod(uint64_t modifier)* and UV. Some SAND-using hardware stores UV in a separate tiled * image from Y to reduce the column height, which is not supported * with these modifiers. +* +* The DRM_FORMAT_MOD_BROADCOM_SAND128_COL_HEIGHT modifier is also +* supported for DRM_FORMAT_P030 where the columns remain as 128 bytes +* wide, but as this is a 10 bpp format that translates to 96 pixels. */ #define DRM_FORMAT_MOD_BROADCOM_SAND32_COL_HEIGHT (v) \
include/standard-headers/linux/ethtool.h +1 -0
@@ -231,6 +231,7 @@ enum tunable_id {ETHTOOL_RX_COPYBREAK ,ETHTOOL_TX_COPYBREAK ,ETHTOOL_PFC_PREVENTION_TOUT ,/* timeout in msecs */ +ETHTOOL_TX_COPYBREAK_BUF_SIZE ,/* * Add your fresh new tunable attribute above and remember to update * tunable_strings[] in net/ethtool/common.c
include/standard-headers/linux/fuse.h +56 -4
127 lines not on this page · raw message
include/standard-headers/linux/pci_regs.h +4 -0
@@ -1086,7 +1086,11 @@/* Designated Vendor-Specific (DVSEC, PCI_EXT_CAP_ID_DVSEC) */ #define PCI_DVSEC_HEADER1 0x4/* Designated Vendor-Specific Header1 */ +#define PCI_DVSEC_HEADER1_VID (x) ((x) & 0xffff) +#define PCI_DVSEC_HEADER1_REV (x) (((x) >> 16) & 0xf) +#define PCI_DVSEC_HEADER1_LEN (x) (((x) >> 20) & 0xfff)#define PCI_DVSEC_HEADER2 0x8/* Designated Vendor-Specific Header2 */ +#define PCI_DVSEC_HEADER2_ID (x) ((x) & 0xffff)/* Data Link Feature */ #define PCI_DLF_CAP 0x04/* Capabilities Register */
include/standard-headers/linux/virtio_iommu.h +7 -1
@@ -16,6 +16,7 @@#define VIRTIO_IOMMU_F_BYPASS 3#define VIRTIO_IOMMU_F_PROBE 4#define VIRTIO_IOMMU_F_MMIO 5 +#define VIRTIO_IOMMU_F_BYPASS_CONFIG 6struct virtio_iommu_range_64 {uint64_t start;
@@ -36,6 +37,8 @@ struct virtio_iommu_config {struct virtio_iommu_range_32 domain_range;/* Probe buffer size */ uint32_t probe_size; +uint8_t bypass; +uint8_t reserved[3 ]; };/* Request types */
@@ -66,11 +69,14 @@ struct virtio_iommu_req_tail {uint8_t reserved[3 ]; }; +#define VIRTIO_IOMMU_ATTACH_F_BYPASS (1 << 0) +struct virtio_iommu_req_attach {struct virtio_iommu_req_head head;uint32_t domain;uint32_t endpoint; -uint8_t reserved[8 ]; +uint32_t flags; +uint8_t reserved[4 ];struct virtio_iommu_req_tail tail; };
linux-headers/asm-mips/unistd_n32.h +1 -0
@@ -377,5 +377,6 @@#define __NR_landlock_add_rule (__NR_Linux + 445)#define __NR_landlock_restrict_self (__NR_Linux + 446)#define __NR_process_mrelease (__NR_Linux + 448) +#define __NR_futex_waitv (__NR_Linux + 449)#endif /* _ASM_UNISTD_N32_H */
linux-headers/asm-mips/unistd_n64.h +1 -0
@@ -353,5 +353,6 @@#define __NR_landlock_add_rule (__NR_Linux + 445)#define __NR_landlock_restrict_self (__NR_Linux + 446)#define __NR_process_mrelease (__NR_Linux + 448) +#define __NR_futex_waitv (__NR_Linux + 449)#endif /* _ASM_UNISTD_N64_H */
linux-headers/asm-mips/unistd_o32.h +1 -0
@@ -423,5 +423,6 @@#define __NR_landlock_add_rule (__NR_Linux + 445)#define __NR_landlock_restrict_self (__NR_Linux + 446)#define __NR_process_mrelease (__NR_Linux + 448) +#define __NR_futex_waitv (__NR_Linux + 449)#endif /* _ASM_UNISTD_O32_H */
linux-headers/asm-powerpc/unistd_32.h +1 -0
@@ -430,6 +430,7 @@#define __NR_landlock_add_rule 445#define __NR_landlock_restrict_self 446#define __NR_process_mrelease 448 +#define __NR_futex_waitv 449#endif /* _ASM_UNISTD_32_H */
linux-headers/asm-powerpc/unistd_64.h +1 -0
@@ -402,6 +402,7 @@#define __NR_landlock_add_rule 445#define __NR_landlock_restrict_self 446#define __NR_process_mrelease 448 +#define __NR_futex_waitv 449#endif /* _ASM_UNISTD_64_H */
linux-headers/asm-s390/kvm.h +1 -0
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {#define KVM_S390_VM_CPU_FEAT_PFMFI 11#define KVM_S390_VM_CPU_FEAT_SIGPIF 12#define KVM_S390_VM_CPU_FEAT_KSS 13 +#define KVM_S390_VM_CPU_FEAT_ZPCI_INTERP 14struct kvm_s390_vm_cpu_feat {__u64 feat[16 ]; };
linux-headers/asm-s390/unistd_32.h +1 -0
@@ -420,5 +420,6 @@#define __NR_landlock_add_rule 445#define __NR_landlock_restrict_self 446#define __NR_process_mrelease 448 +#define __NR_futex_waitv 449#endif /* _ASM_S390_UNISTD_32_H */
linux-headers/asm-s390/unistd_64.h +1 -0
@@ -368,5 +368,6 @@#define __NR_landlock_add_rule 445#define __NR_landlock_restrict_self 446#define __NR_process_mrelease 448 +#define __NR_futex_waitv 449#endif /* _ASM_S390_UNISTD_64_H */
linux-headers/linux/kvm.h +1 -0
@@ -1130,6 +1130,7 @@ struct kvm_ppc_resize_hpt {#define KVM_CAP_BINARY_STATS_FD 203#define KVM_CAP_EXIT_ON_EMULATION_FAILURE 204#define KVM_CAP_ARM_MTE 205 +#define KVM_CAP_VM_MOVE_ENC_CONTEXT_FROM 206#ifdef KVM_CAP_IRQ_ROUTING
linux-headers/linux/vfio.h +22 -0
@@ -1002,6 +1002,28 @@ struct vfio_device_feature {*/ #define VFIO_DEVICE_FEATURE_PCI_VF_TOKEN (0) +/* +* Provide support for enabling interpretation of zPCI instructions. This +* feature is only valid for s390x PCI devices. Data provided when setting +* and getting this feature is futher described in vfio_zdev.h +*/ +#define VFIO_DEVICE_FEATURE_ZPCI_INTERP (1) + +/* +* Provide support for enbaling adapter interruption forwarding for zPCI +* devices. This feature is only valid for s390x PCI devices. Data provided +* when setting and getting this feature is further described in vfio_zdev.h +*/ +#define VFIO_DEVICE_FEATURE_ZPCI_AIF (2) + +/* +* Provide support for enabling guest I/O address translation assistance for +* zPCI devices. This feature is only valid for s390x PCI devices. Data +* provided when setting and getting this feature is further described in +* vfio_zdev.h +*/ +#define VFIO_DEVICE_FEATURE_ZPCI_IOAT (3) +/* -------- API for Type1 VFIO IOMMU -------- */ /**
linux-headers/linux/vfio_zdev.h +51 -0
61 lines not on this page · raw message
Signature · 1 line
2.27.0
PATCH v2 2/9 target/s390x: add zpci-interp to cpu models
In reply to Matthew RosatoThe zpci-interp feature is used to specify whether zPCI interpretation is to be used for this guest.
--- target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 ++ target/s390x/kvm/kvm.c | 1 + 3 files changed, 4 insertions(+)
target/s390x/cpu_features_def.h.inc +1 -0
@@ -146,6 +146,7 @@ DEF_FEAT(SIE_CEI, "cei", SCLP_CPU, 43, "SIE: Conditional-external-interception f DEF_FEAT(DAT_ENH_2, "dateh2", MISC, 0, "DAT-enhancement facility 2") DEF_FEAT(CMM, "cmm", MISC, 0, "Collaborative-memory-management facility") DEF_FEAT(AP, "ap", MISC, 0, "AP instructions installed") +DEF_FEAT(ZPCI_INTERP, "zpci-interp", MISC, 0, "zPCI interpretation") /* Features exposed via the PLO instruction. */ DEF_FEAT(PLO_CL, "plo-cl", PLO, 0, "PLO Compare and load (32 bit in general registers)")
target/s390x/gen-features.c +2 -0
@@ -554,6 +554,7 @@ static uint16_t full_GEN14_GA1[] = {S390_FEAT_HPMA2 ,S390_FEAT_SIE_KSS ,S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF , + S390_FEAT_ZPCI_INTERP, };#define full_GEN14_GA2 EmptyFeat
@@ -650,6 +651,7 @@ static uint16_t default_GEN14_GA1[] = {S390_FEAT_GROUP_MSA_EXT_8 ,S390_FEAT_MULTIPLE_EPOCH ,S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF , + S390_FEAT_ZPCI_INTERP, };#define default_GEN14_GA2 EmptyFeat
target/s390x/kvm/kvm.c +1 -0
@@ -2294,6 +2294,7 @@ static int kvm_to_feat[][2] = { { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI}, { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF}, { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS}, + { KVM_S390_VM_CPU_FEAT_ZPCI_INTERP, S390_FEAT_ZPCI_INTERP }, };static int query_cpu_feat (S390FeatBitmap features)
Signature · 1 line
2.27.0
PATCH v2 3/9 fixup: force interp off for QEMU machine 6.2 and older
In reply to Matthew RosatoDouble-check I'm doing this right + test.
--- hw/s390x/s390-virtio-ccw.c | 1 + 1 file changed, 1 insertion(+)
hw/s390x/s390-virtio-ccw.c +1 -0
@@ -803,6 +803,7 @@ DEFINE_CCW_MACHINE(7_0, "7.0", true);static void ccw_machine_6_2_instance_options (MachineState *machine) {ccw_machine_7_0_instance_options (machine); +s390_cpudef_featoff_greater (14 ,1 ,S390_FEAT_ZPCI_INTERP ); }static void ccw_machine_6_2_class_options (MachineClass *mc)
Signature · 1 line
2.27.0
Re: [PATCH v2 3/9] fixup: force interp off for QEMU machine 6.2 and older
Quoted 2 lines · Matthew Rosato
Double-check I'm doing this right + test.
Argh... This should have been squashed into the preceding patch 'target/s390x: add zpci-interp to cpu models'
Quoted 17 lines
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-virtio-ccw.c | 1 + 1 file changed, 1 insertion(+)hw/s390x/s390-virtio-ccw.c +1
-0@@ -803,6 +803,7 @@ DEFINE_CCW_MACHINE(7_0, "7.0", true);static void ccw_machine_6_2_instance_options (MachineState *machine) {ccw_machine_7_0_instance_options (machine); +s390_cpudef_featoff_greater (14 ,1 ,S390_FEAT_ZPCI_INTERP ); }static void ccw_machine_6_2_class_options (MachineClass *mc)
PATCH v2 4/9 s390x/pci: enable for load/store intepretation
In reply to Matthew RosatoUse the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
--- hw/s390x/s390-pci-bus.c | 70 +++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 63 +++++++++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 52 ++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 15 +++++++ 5 files changed, 199 insertions(+), 2 deletions(-)
hw/s390x/s390-pci-bus.c +69 -1
@@ -971,12 +971,58 @@ static void s390_pci_update_subordinate(PCIDevice *dev, uint32_t nr) } } +static int s390_pci_interp_plug (S390pciState *s,S390PCIBusDevice *pbdev) +{ +uint32_t idx; +int rc; + + rc =s390_pci_probe_interp (pbdev); +if (rc) { +return rc; + } + + rc =s390_pci_update_passthrough_fh (pbdev); +if (rc) { +return rc; + } + +/* +* The host device is already in an enabled state, but we always present +* the initial device state to the guest as disabled (ZPCI_FS_DISABLED). +* Therefore, mask off the enable bit from the passthrough handle until +* the guest issues a CLP SET PCI FN later to enable the device. +*/ + pbdev->fh &= ~FH_MASK_ENABLE ; + +/* Next, see if the idx is already in-use */ + idx = pbdev->fh &FH_MASK_INDEX ; +if (pbdev->idx != idx) { +if (s390_pci_find_dev_by_idx (s, idx)) { +return -EINVAL ; + } +/* +* Update the idx entry with the passed through idx +* If the relinquished idx is lower than next_idx, use it +* to replace next_idx +*/ +g_hash_table_remove (s->zpci_table, &pbdev->idx); +if (idx < s->next_idx) { + s->next_idx = idx; + } + pbdev->idx = idx; +g_hash_table_insert (s->zpci_table, &pbdev->idx, pbdev); + } + +return 0 ; +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {S390pciState *s =S390_PCI_HOST_BRIDGE (hotplug_dev);PCIDevice *pdev =NULL ;S390PCIBusDevice *pbdev =NULL ; +int rc;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) {PCIBridge *pb =PCI_BRIDGE (dev);
@@ -1022,12 +1068,33 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,set_pbdev_info (pbdev);if (object_dynamic_cast (OBJECT (dev),"vfio-pci" )) { - pbdev->fh |=FH_SHM_VFIO ; +/* +* By default, interpretation is always requested; if the available +* facilities indicate it is not available, fallback to the +* intercept model. +*/ +if (pbdev->interp && !s390_has_feat (S390_FEAT_ZPCI_INTERP )) { +DPRINTF ("zPCI interpretation facilities missing.\n" ); + pbdev->interp = false; + } +if (pbdev->interp) { + rc =s390_pci_interp_plug (s, pbdev); +if (rc) { +error_setg (errp,"zpci interp plug failed: %d" , rc); +return ; + } + } pbdev->iommu->dma_limit =s390_pci_start_dma_count (s, pbdev);/* Fill in CLP information passed via the vfio region */ s390_pci_get_clp_info (pbdev); +if (!pbdev->interp) { +/* Do vfio passthrough but intercept for I/O */ + pbdev->fh |=FH_SHM_VFIO ; + } }else { pbdev->fh |=FH_SHM_EMUL ; +/* Always intercept emulated devices */ + pbdev->interp = false; }if (s390_pci_msix_init (pbdev)) {
@@ -1360,6 +1427,7 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_UINT16 ("uid" , S390PCIBusDevice, uid,UID_UNDEFINED ),DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target), +DEFINE_PROP_BOOL ("interp" , S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };
hw/s390x/s390-pci-inst.c +62 -1
87 lines not on this page · raw message
hw/s390x/s390-pci-vfio.c +52 -0
58 lines not on this page · raw message
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -350,6 +350,7 @@ struct S390PCIBusDevice {IndAddr *indicator;bool pci_unplug_request_processed;bool unplug_requested; +bool interp;QTAILQ_ENTRY (S390PCIBusDevice) link; };
include/hw/s390x/s390-pci-vfio.h +15 -0
@@ -20,6 +20,9 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail);S390PCIDMACount *s390_pci_start_dma_count (S390pciState *s,S390PCIBusDevice *pbdev);void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt); +int s390_pci_probe_interp (S390PCIBusDevice *pbdev); +int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable); +int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)
@@ -33,6 +36,18 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, }static inline void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt) { } +int s390_pci_probe_interp (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +} +static inline int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable) +{ +return -EINVAL ; +} +static inline int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
Signature · 1 line
2.27.0
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
Quoted 81 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 70 +++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 63 +++++++++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 52 ++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 15 +++++++ 5 files changed, 199 insertions(+), 2 deletions(-)hw/s390x/s390-pci-bus.c +46
-0@@ -971,12 +971,58 @@ static void s390_pci_update_subordinate(PCIDevice *dev, uint32_t nr) } } +static int s390_pci_interp_plug (S390pciState *s,S390PCIBusDevice *pbdev) +{ +uint32_t idx; +int rc; + + rc =s390_pci_probe_interp (pbdev); +if (rc) { +return rc; + } + + rc =s390_pci_update_passthrough_fh (pbdev); +if (rc) { +return rc; + } + +/* +* The host device is already in an enabled state, but we always present +* the initial device state to the guest as disabled (ZPCI_FS_DISABLED). +* Therefore, mask off the enable bit from the passthrough handle until +* the guest issues a CLP SET PCI FN later to enable the device. +*/ + pbdev->fh &= ~FH_MASK_ENABLE ; + +/* Next, see if the idx is already in-use */ + idx = pbdev->fh &FH_MASK_INDEX ; +if (pbdev->idx != idx) { +if (s390_pci_find_dev_by_idx (s, idx)) { +return -EINVAL ; + } +/* +* Update the idx entry with the passed through idx +* If the relinquished idx is lower than next_idx, use it +* to replace next_idx +*/ +g_hash_table_remove (s->zpci_table, &pbdev->idx); +if (idx < s->next_idx) { + s->next_idx = idx; + } + pbdev->idx = idx; +g_hash_table_insert (s->zpci_table, &pbdev->idx, pbdev); + } + +return 0 ; +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {S390pciState *s =S390_PCI_HOST_BRIDGE (hotplug_dev);PCIDevice *pdev =NULL ;S390PCIBusDevice *pbdev =NULL ; +int rc;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) {PCIBridge *pb =PCI_BRIDGE (dev);@@ -1022,12 +1068,33 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,set_pbdev_info (pbdev);if (object_dynamic_cast (OBJECT (dev),"vfio-pci" )) {
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 1079
Quoted 10 lines
hw/s390x/s390-pci-bus.c +9
-1- pbdev->fh |=FH_SHM_VFIO ; +/* +* By default, interpretation is always requested; if the available +* facilities indicate it is not available, fallback to the +* intercept model. +*/ +if (pbdev->interp && !s390_has_feat (S390_FEAT_ZPCI_INTERP )) { +DPRINTF ("zPCI interpretation facilities missing.\n" ); + pbdev->interp = false; + }
Wrong indentation in the above three lines.
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 1083
Quoted 4 lines
hw/s390x/s390-pci-bus.c +4
-0+if (pbdev->interp) { + rc =s390_pci_interp_plug (s, pbdev); +if (rc) { +error_setg (errp,"zpci interp plug failed: %d" , rc);
The error message is a little bit scarce for something that might be presented to the user - maybe write at least "interpretation" instead of "interp" ?
Quoted 5 lines
hw/s390x/s390-pci-bus.c +3
-0+return ; + } + } pbdev->iommu->dma_limit =s390_pci_start_dma_count (s, pbdev);/* Fill in CLP information passed via the vfio region */
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 1098
Quoted 10 lines
hw/s390x/s390-pci-bus.c +6
-0s390_pci_get_clp_info (pbdev); +if (!pbdev->interp) { +/* Do vfio passthrough but intercept for I/O */ + pbdev->fh |=FH_SHM_VFIO ; + } }else { pbdev->fh |=FH_SHM_EMUL ; +/* Always intercept emulated devices */ + pbdev->interp = false; }
Thomas
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
Quoted 1 line · Thomas Huth
On 14/01/2022 21.38, Matthew Rosato wrote:
...
Quoted 20 lines
Quoted 20 lines
static void
s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp)
{
S390pciState *s =S390_PCI_HOST_BRIDGE(hotplug_dev); PCIDevice *pdev = NULL;
S390PCIBusDevice *pbdev = NULL;
+ int rc;
if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) { PCIBridge *pb =PCI_BRIDGE(dev);hw/s390x/s390-pci-bus.c +2
-1@@ -1022,12 +1068,33 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, set_pbdev_info(pbdev); if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) { - pbdev->fh |= FH_SHM_VFIO; + /* +* By default, interpretation is always requested; if the available
+ * facilities indicate it is not available, fallback to the + * intercept model.
+ */
Quoted 8 lines
Quoted 6 lines
+ if (
pbdev->interp&& !s390_has_feat(S390_FEAT_ZPCI_INTERP)) {
+DPRINTF("zPCI interpretation facilities missing.\n");
+pbdev->interp= false;
+ }Wrong indentation in the above three lines.
Thanks
Quoted 8 lines
Quoted 5 lines
+ if (
pbdev->interp) {
+ rc =s390_pci_interp_plug(s, pbdev);
+ if (rc) {
+error_setg(errp, "zpci interp plug failed: %d", rc);The error message is a little bit scarce for something that might be presented to the user - maybe write at least "interpretation" instead of "interp" ?
Good point, I'll re-word to something like "Plug failed for zPCI device in interpretation mode: %d"
Quoted 18 lines
Quoted 7 lines
+ return;
+ }
+ }
pbdev->iommu->dma_limit=s390_pci_start_dma_count(s, pbdev); /* Fill in CLP information passed via the vfio region */s390_pci_get_clp_info(pbdev);
+ if (!pbdev->interp) {Quoted 8 lines
+ /* Do vfio passthrough but intercept for I/O */ +
pbdev->fh|=FH_SHM_VFIO;
+ }
} else {
pbdev->fh|=FH_SHM_EMUL;
+ /* Always intercept emulated devices */
+pbdev->interp= false;
}Thomas
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
In reply to Matthew RosatoQuoted 18 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 70 +++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 63 +++++++++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 52 ++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 15 +++++++ 5 files changed, 199 insertions(+), 2 deletions(-)hw/s390x/s390-pci-bus.c +0
-0
...
hw/s390x/s390-pci-bus.c static Property s390_pci_device_properties[] = { · line 1432
Quoted 7 lines
hw/s390x/s390-pci-bus.c +1
-0@@ -1360,6 +1427,7 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_UINT16 ("uid" , S390PCIBusDevice, uid,UID_UNDEFINED ),DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target), +DEFINE_PROP_BOOL ("interp" , S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };
Since this is something that the user can see, would it maybe make sense to provide a full word instead of an abbreviation here? I.e. "interpret" or "interpretation" instead of "interp" ?
Thomas
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
Quoted 20 lines · Thomas Huth
Quoted 18 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 70 +++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 63 +++++++++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 52 ++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 +include/hw/s390x/s390-pci-vfio.h| 15 +++++++5 files changed, 199 insertions(+), 2 deletions(-)hw/s390x/s390-pci-bus.c +0
-0...
Quoted 9 lines
hw/s390x/s390-pci-bus.c static Property s390_pci_device_properties[] = { · line 1432
Quoted 7 lines
hw/s390x/s390-pci-bus.c +1
-0@@ -1360,6 +1427,7 @@ static Property s390_pci_device_properties[] = { DEFINE_PROP_UINT16("uid", S390PCIBusDevice, uid, UID_UNDEFINED), DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid), DEFINE_PROP_STRING("target", S390PCIBusDevice, target), + DEFINE_PROP_BOOL("interp", S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };Since this is something that the user can see, would it maybe make sense to provide a full word instead of an abbreviation here? I.e. "interpret" or "interpretation" instead of "interp" ?
I'll go with "interpret" unless someone else has a strong opinion on it.
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
In reply to Matthew RosatoQuoted 22 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 70 +++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 63 +++++++++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 52 ++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 15 +++++++ 5 files changed, 199 insertions(+), 2 deletions(-)hw/s390x/s390-pci-bus.c +0
-0@@ -971,12 +971,58 @@ static void s390_pci_update_subordinate(PCIDevice *dev, uint32_t nr) } }
...snip...
Quoted 15 lines
include/hw/s390x/s390-pci-vfio.h +3
-0@@ -20,6 +20,9 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail);S390PCIDMACount *s390_pci_start_dma_count (S390pciState *s,S390PCIBusDevice *pbdev);void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt); +int s390_pci_probe_interp (S390PCIBusDevice *pbdev); +int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable); +int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)
include/hw/s390x/s390-pci-vfio.h static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, · line 39
Quoted 4 lines
include/hw/s390x/s390-pci-vfio.h +1
-0@@ -33,6 +36,18 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, }static inline void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt) { } +int s390_pci_probe_interp (S390PCIBusDevice *pbdev)
static inline ?
Quoted 15 lines
include/hw/s390x/s390-pci-vfio.h +11
-0+{ +return -EINVAL ; +} +static inline int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable) +{ +return -EINVAL ; +} +static inline int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
In reply to Matthew RosatoQuoted 4 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
I wonder if we should not explain here that having interpretation as a default and silently fall back to interception allows backward compatibility while allowing performence be chosing by default. (You can say it better as I do :) )
Quoted 74 lines
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 70 +++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 63 +++++++++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 52 ++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 15 +++++++ 5 files changed, 199 insertions(+), 2 deletions(-)hw/s390x/s390-pci-bus.c +46
-0@@ -971,12 +971,58 @@ static void s390_pci_update_subordinate(PCIDevice *dev, uint32_t nr) } } +static int s390_pci_interp_plug (S390pciState *s,S390PCIBusDevice *pbdev) +{ +uint32_t idx; +int rc; + + rc =s390_pci_probe_interp (pbdev); +if (rc) { +return rc; + } + + rc =s390_pci_update_passthrough_fh (pbdev); +if (rc) { +return rc; + } + +/* +* The host device is already in an enabled state, but we always present +* the initial device state to the guest as disabled (ZPCI_FS_DISABLED). +* Therefore, mask off the enable bit from the passthrough handle until +* the guest issues a CLP SET PCI FN later to enable the device. +*/ + pbdev->fh &= ~FH_MASK_ENABLE ; + +/* Next, see if the idx is already in-use */ + idx = pbdev->fh &FH_MASK_INDEX ; +if (pbdev->idx != idx) { +if (s390_pci_find_dev_by_idx (s, idx)) { +return -EINVAL ; + } +/* +* Update the idx entry with the passed through idx +* If the relinquished idx is lower than next_idx, use it +* to replace next_idx +*/ +g_hash_table_remove (s->zpci_table, &pbdev->idx); +if (idx < s->next_idx) { + s->next_idx = idx; + } + pbdev->idx = idx; +g_hash_table_insert (s->zpci_table, &pbdev->idx, pbdev); + } + +return 0 ; +} +static void s390_pcihost_plug (HotplugHandler *hotplug_dev,DeviceState *dev,Error **errp) {S390pciState *s =S390_PCI_HOST_BRIDGE (hotplug_dev);PCIDevice *pdev =NULL ;S390PCIBusDevice *pbdev =NULL ; +int rc;if (object_dynamic_cast (OBJECT (dev),TYPE_PCI_BRIDGE )) {PCIBridge *pb =PCI_BRIDGE (dev);
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 1074
Quoted 8 lines
hw/s390x/s390-pci-bus.c +4
-1@@ -1022,12 +1068,33 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,set_pbdev_info (pbdev);if (object_dynamic_cast (OBJECT (dev),"vfio-pci" )) { - pbdev->fh |=FH_SHM_VFIO ; +/* +* By default, interpretation is always requested; if the available +* facilities indicate it is not available, fallback to the +* intercept model.
s/intercept/interception/ ?
hw/s390x/s390-pci-bus.c static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, · line 1086
Quoted 12 lines
hw/s390x/s390-pci-bus.c +12
-0+*/ +if (pbdev->interp && !s390_has_feat (S390_FEAT_ZPCI_INTERP )) { +DPRINTF ("zPCI interpretation facilities missing.\n" ); + pbdev->interp = false; + } +if (pbdev->interp) { + rc =s390_pci_interp_plug (s, pbdev); +if (rc) { +error_setg (errp,"zpci interp plug failed: %d" , rc); +return ; + } + }
Can't we rearrange that as
if (pbdev->interp) {
if (s390_has_feat) {
} else {
}
}Quoted 216 lines
hw/s390x/s390-pci-bus.c +7
-0pbdev->iommu->dma_limit =s390_pci_start_dma_count (s, pbdev);/* Fill in CLP information passed via the vfio region */ s390_pci_get_clp_info (pbdev); +if (!pbdev->interp) { +/* Do vfio passthrough but intercept for I/O */ + pbdev->fh |=FH_SHM_VFIO ; + } }else { pbdev->fh |=FH_SHM_EMUL ; +/* Always intercept emulated devices */ + pbdev->interp = false; }if (s390_pci_msix_init (pbdev)) {@@ -1360,6 +1427,7 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_UINT16 ("uid" , S390PCIBusDevice, uid,UID_UNDEFINED ),DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target), +DEFINE_PROP_BOOL ("interp" , S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };hw/s390x/s390-pci-inst.c +62
-1@@ -18,6 +18,7 @@#include "sysemu/hw_accel.h" #include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST @@ -156,6 +157,47 @@ out:return rc; } +static int clp_enable_interp (S390PCIBusDevice *pbdev) +{ +int rc; + + rc =s390_pci_set_interp (pbdev, true); +if (rc) { +DPRINTF ("Failed to enable interpretation\n" ); +return rc; + } + rc =s390_pci_update_passthrough_fh (pbdev); +if (rc) { +DPRINTF ("Failed to update passthrough fh\n" ); +return rc; + } +if (!(pbdev->fh &FH_MASK_ENABLE )) { +DPRINTF ("Passthrough handle is not enabled\n" ); +return -EINVAL ; + } + +return 0 ; +} + +static int clp_disable_interp (S390PCIBusDevice *pbdev) +{ +int rc; + + rc =s390_pci_set_interp (pbdev, false); +if (rc) { +DPRINTF ("Failed to disable interpretation\n" ); +return rc; + } + + rc =s390_pci_update_passthrough_fh (pbdev); +if (rc) { +DPRINTF ("Failed to update passthrough fh\n" ); +return rc; + } + +return 0 ; +} +int clp_service_call (S390CPU *cpu,uint8_t r2,uintptr_t ra) {ClpReqHdr *reqh;@@ -246,7 +288,19 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } - pbdev->fh |=FH_MASK_ENABLE ; +/* +* If interpretation is specified, attempt to enable this now and +* update with the host fh +*/ +if (pbdev->interp) { +if (clp_enable_interp (pbdev)) { +stw_p (&ressetpci->hdr.rsp,CLP_RC_SETPCIFN_ERR ); + goto out; + } + }else { + pbdev->fh |=FH_MASK_ENABLE ; + } + pbdev->state =ZPCI_FS_ENABLED ;stl_p (&ressetpci->fh, pbdev->fh);stw_p (&ressetpci->hdr.rsp,CLP_RC_OK );@@ -257,6 +311,13 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; }device_legacy_reset (DEVICE (pbdev)); +if (pbdev->interp) { +if (clp_disable_interp (pbdev)) { +stw_p (&ressetpci->hdr.rsp,CLP_RC_SETPCIFN_ERR ); + goto out; + } + } +/* Mask off the enabled bit for interpreted devices too */ pbdev->fh &= ~FH_MASK_ENABLE ; pbdev->state =ZPCI_FS_DISABLED ;stl_p (&ressetpci->fh, pbdev->fh);hw/s390x/s390-pci-vfio.c +52
-0@@ -97,6 +97,58 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt) } } +int s390_pci_probe_interp (S390PCIBusDevice *pbdev) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_feature feat = { + .argsz =sizeof (struct vfio_device_feature ), + .flags =VFIO_DEVICE_FEATURE_PROBE |VFIO_DEVICE_FEATURE_ZPCI_INTERP + }; + +return ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , &feat); +} + +int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_zpci_interp *data; +int size =sizeof (struct vfio_device_feature ) +sizeof (*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); + + feat->argsz = size; + feat->flags =VFIO_DEVICE_FEATURE_SET +VFIO_DEVICE_FEATURE_ZPCI_INTERP ; + + data = (struct vfio_device_zpci_interp *)&feat->data; +if (enable) { + data->flags =VFIO_DEVICE_ZPCI_FLAG_INTERP ; + }else { + data->flags =0 ; + } + +return ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , feat); +} + +int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_zpci_interp *data; +int size =sizeof (struct vfio_device_feature ) +sizeof (*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); +int rc; + + feat->argsz = size; + feat->flags =VFIO_DEVICE_FEATURE_GET +VFIO_DEVICE_FEATURE_ZPCI_INTERP ; + + rc =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , feat); +if (rc) { +return rc; + } + + data = (struct vfio_device_zpci_interp *)&feat->data; + pbdev->fh = data->fh; +return 0 ; +} +static void s390_pci_read_base (S390PCIBusDevice *pbdev,struct vfio_device_info *info) {include/hw/s390x/s390-pci-bus.h +1
-0@@ -350,6 +350,7 @@ struct S390PCIBusDevice {IndAddr *indicator;bool pci_unplug_request_processed;bool unplug_requested; +bool interp;QTAILQ_ENTRY (S390PCIBusDevice) link; };include/hw/s390x/s390-pci-vfio.h +8
-0@@ -20,6 +20,9 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail);S390PCIDMACount *s390_pci_start_dma_count (S390pciState *s,S390PCIBusDevice *pbdev);void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt); +int s390_pci_probe_interp (S390PCIBusDevice *pbdev); +int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable); +int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)@@ -33,6 +36,18 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, }static inline void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt) { } +int s390_pci_probe_interp (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +} +static inline int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable)
include/hw/s390x/s390-pci-vfio.h static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, · line 53
Quoted 10 lines
include/hw/s390x/s390-pci-vfio.h +7
-0+{ +return -EINVAL ; +} +static inline int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
LGTM
With the corrections proposed by Thomas.
Mine... you see what you prefer.
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v2 4/9] s390x/pci: enable for load/store intepretation
Quoted 10 lines · Pierre Morel
Quoted 4 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable interpretation for devices when requested. As part of this process, we must use the host function handle rather than a QEMU-generated one -- this is provided as part of the ioctl payload.
I wonder if we should not explain here that having interpretation as a default and silently fall back to interception allows backward compatibility while allowing performence be chosing by default. (You can say it better as I do :) )
Good suggestion, I'll think of something to add to the commit message.
Quoted 4 lines
Quoted 4 lines
hw/s390x/s390-pci-bus.c +0
-0@@ -1022,12 +1068,33 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, set_pbdev_info(pbdev); if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) {
Quoted 9 lines
Quoted 6 lines
hw/s390x/s390-pci-bus.c +2
-1- pbdev->fh |= FH_SHM_VFIO; + /* +* By default, interpretation is always requested; if the available
+ * facilities indicate it is not available, fallback to the + * intercept model.s/intercept/interception/ ?
OK
Quoted 16 lines
Quoted 6 lines
+ */
+ if (pbdev->interp&& !s390_has_feat(S390_FEAT_ZPCI_INTERP)) {
+DPRINTF("zPCI interpretation facilities missing.\n");
+pbdev->interp= false;Quoted 8 lines
+ }
+ if (pbdev->interp) {
+ rc =s390_pci_interp_plug(s, pbdev);
+ if (rc) {
+error_setg(errp, "zpci interp plug failed: %d", rc); + return;
+ }
+ }
Quoted 6 lines
Can't we rearrange that as
if (pbdev->interp) {
if (s390_has_feat) {
} else {
}
}
Yep, sure
...
Quoted 6 lines
LGTM
With the corrections proposed by Thomas.
Mine... you see what you prefer.Reviewed-by Pierre Morel <[email protected]>
Thanks!
PATCH v2 5/9 s390x/pci: don't fence interpreted devices without MSI-X
In reply to Matthew RosatoLack of MSI-X support is not an issue for interpreted passthrough devices, so let's let these in. This will allow, for example, ISM devices to be passed through -- but only when interpretation is available and being used.
--- hw/s390x/s390-pci-bus.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-bus.c +1 -1
@@ -1097,7 +1097,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->interp = false; } -if (s390_pci_msix_init (pbdev)) { +if (s390_pci_msix_init (pbdev) && !pbdev->interp) {error_setg (errp,"MSI-X support is mandatory " "in the S390 architecture" );return ;
Signature · 1 line
2.27.0
PATCH v2 6/9 s390x/pci: enable adapter event notification for interpreted devices
In reply to Matthew RosatoUse the associated vfio feature ioctl to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'intassist' setting.
--- hw/s390x/s390-pci-bus.c | 24 ++++++++-- hw/s390x/s390-pci-inst.c | 54 +++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 79 ++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 20 ++++++++ 5 files changed, 173 insertions(+), 5 deletions(-)
hw/s390x/s390-pci-bus.c +21 -3
@@ -189,7 +189,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc =SCLP_RC_NO_ACTION_REQUIRED ;break ;default : -if (pbdev->summary_ind) { +if (pbdev->interp) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif (pbdev,NULL , false, pbdev->intassist); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
@@ -981,6 +984,11 @@ static int s390_pci_interp_plug(S390pciState *s, S390PCIBusDevice *pbdev)return rc; } + rc =s390_pci_probe_aif (pbdev); +if (rc) { +return rc; + } + rc =s390_pci_update_passthrough_fh (pbdev);if (rc) {return rc;
@@ -1076,6 +1084,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (pbdev->interp && !s390_has_feat (S390_FEAT_ZPCI_INTERP )) {DPRINTF ("zPCI interpretation facilities missing.\n" ); pbdev->interp = false; + pbdev->intassist = false; }if (pbdev->interp) { rc =s390_pci_interp_plug (s, pbdev);
@@ -1090,11 +1099,13 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (!pbdev->interp) {/* Do vfio passthrough but intercept for I/O */ pbdev->fh |=FH_SHM_VFIO ; + pbdev->intassist = false; } }else { pbdev->fh |=FH_SHM_EMUL ;/* Always intercept emulated devices */ pbdev->interp = false; + pbdev->intassist = false; }if (s390_pci_msix_init (pbdev) && !pbdev->interp) {
@@ -1244,7 +1255,10 @@ static void s390_pcihost_reset(DeviceState *dev)/* Process all pending unplug requests */ QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) {if (pbdev->unplug_requested) { -if (pbdev->summary_ind) { +if (pbdev->interp) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif (pbdev,NULL , false, pbdev->intassist); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
@@ -1382,7 +1396,10 @@ static void s390_pci_device_reset(DeviceState *dev)break ; } -if (pbdev->summary_ind) { +if (pbdev->interp) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif (pbdev,NULL , false, pbdev->intassist); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
@@ -1428,6 +1445,7 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target),DEFINE_PROP_BOOL ("interp" , S390PCIBusDevice, interp, true), +DEFINE_PROP_BOOL ("intassist" , S390PCIBusDevice, intassist, true), DEFINE_PROP_END_OF_LIST(), };
hw/s390x/s390-pci-inst.c +52 -2
72 lines not on this page · raw message
hw/s390x/s390-pci-vfio.c +79 -0
85 lines not on this page · raw message
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -351,6 +351,7 @@ struct S390PCIBusDevice {bool pci_unplug_request_processed;bool unplug_requested;bool interp; +bool intassist;QTAILQ_ENTRY (S390PCIBusDevice) link; };
include/hw/s390x/s390-pci-vfio.h +20 -0
@@ -13,6 +13,7 @@#define HW_S390_PCI_VFIO_H #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-inst.h" #include CONFIG_DEVICES #ifdef CONFIG_VFIO
@@ -23,6 +24,11 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt);int s390_pci_probe_interp (S390PCIBusDevice *pbdev);int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable);int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev); +int s390_pci_probe_aif (S390PCIBusDevice *pbdev); +int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib,bool enable, +bool assist); +int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable,bool assist); +void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)
@@ -48,6 +54,20 @@ static inline int s390_pci_update_passthrough_fh(S390PCIBusDevice *pbdev) {return -EINVAL ; } +static inline int s390_pci_probe_aif (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +} +static inline int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib, +bool enable,bool assist) +{ +return -EINVAL ; +} +static inline int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable, +bool assist) +{ +return -EINVAL ; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
Signature · 1 line
2.27.0
Re: [PATCH v2 6/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 3 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'intassist' setting.
It is a personal opinion but I do not like the term 'intassist'.
Why not using 'gisa' ?
I would find it more relevant.
If something for the profane admin then interrupt_assist or irq_assist or even int_assist.
Or is it forwarding_assist ?
Quoted 85 lines
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 24 ++++++++-- hw/s390x/s390-pci-inst.c | 54 +++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 79 ++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-vfio.h | 20 ++++++++ 5 files changed, 173 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +20
-3@@ -189,7 +189,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc =SCLP_RC_NO_ACTION_REQUIRED ;break ;default : -if (pbdev->summary_ind) { +if (pbdev->interp) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif (pbdev,NULL , false, pbdev->intassist); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {@@ -981,6 +984,11 @@ static int s390_pci_interp_plug(S390pciState *s, S390PCIBusDevice *pbdev)return rc; } + rc =s390_pci_probe_aif (pbdev); +if (rc) { +return rc; + } + rc =s390_pci_update_passthrough_fh (pbdev);if (rc) {return rc;@@ -1076,6 +1084,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (pbdev->interp && !s390_has_feat (S390_FEAT_ZPCI_INTERP )) {DPRINTF ("zPCI interpretation facilities missing.\n" ); pbdev->interp = false; + pbdev->intassist = false; }if (pbdev->interp) { rc =s390_pci_interp_plug (s, pbdev);@@ -1090,11 +1099,13 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (!pbdev->interp) {/* Do vfio passthrough but intercept for I/O */ pbdev->fh |=FH_SHM_VFIO ; + pbdev->intassist = false; } }else { pbdev->fh |=FH_SHM_EMUL ;/* Always intercept emulated devices */ pbdev->interp = false; + pbdev->intassist = false; }if (s390_pci_msix_init (pbdev) && !pbdev->interp) {@@ -1244,7 +1255,10 @@ static void s390_pcihost_reset(DeviceState *dev)/* Process all pending unplug requests */ QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) {if (pbdev->unplug_requested) { -if (pbdev->summary_ind) { +if (pbdev->interp) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif (pbdev,NULL , false, pbdev->intassist); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {@@ -1382,7 +1396,10 @@ static void s390_pci_device_reset(DeviceState *dev)break ; } -if (pbdev->summary_ind) { +if (pbdev->interp) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif (pbdev,NULL , false, pbdev->intassist); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
hw/s390x/s390-pci-bus.c static Property s390_pci_device_properties[] = { · line 1448
Quoted 4 lines
hw/s390x/s390-pci-bus.c +1
-0@@ -1428,6 +1445,7 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target),DEFINE_PROP_BOOL ("interp" , S390PCIBusDevice, interp, true), + DEFINE_PROP_BOOL("intassist", S390PCIBusDevice, intassist, true),
We allow to disable IRQ forwarding only for test or debug purpose don't we? Then shouldn't we make it explicit ?
Quoted 149 lines
hw/s390x/s390-pci-bus.c +0
-0DEFINE_PROP_END_OF_LIST(), };hw/s390x/s390-pci-inst.c +52
-2@@ -1111,6 +1111,46 @@ static void fmb_update(void *opaque)timer_mod (pbdev->fmb_timer, t + pbdev->pci_group->zpci_group.mui); } +static int mpcifc_reg_int_interp (S390PCIBusDevice *pbdev,ZpciFib *fib) +{ +int rc; + +/* Interpreted devices must also use interrupt forwarding */ + rc =s390_pci_get_aif (pbdev, false, pbdev->intassist); +if (rc) { +DPRINTF ("Bad interrupt forwarding state\n" ); +return rc; + } + + rc =s390_pci_set_aif (pbdev, fib, true, pbdev->intassist); +if (rc) { +DPRINTF ("Failed to enable interrupt forwarding\n" ); +return rc; + } + +return 0 ; +} + +static int mpcifc_dereg_int_interp (S390PCIBusDevice *pbdev,ZpciFib *fib) +{ +int rc; + +/* Interpreted devices were using interrupt forwarding */ + rc =s390_pci_get_aif (pbdev, true, pbdev->intassist); +if (rc) { +DPRINTF ("Bad interrupt forwarding state\n" ); +return rc; + } + + rc =s390_pci_set_aif (pbdev, fib, false, pbdev->intassist); +if (rc) { +DPRINTF ("Failed to disable interrupt forwarding\n" ); +return rc; + } + +return 0 ; +} +int mpcifc_service_call (S390CPU *cpu,uint8_t r1,uint64_t fiba,uint8_t ar,uintptr_t ra) {@@ -1165,7 +1205,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar,switch (oc) {case ZPCI_MOD_FC_REG_INT : -if (pbdev->summary_ind) { +if (pbdev->interp) { +if (mpcifc_reg_int_interp (pbdev, &fib)) { + cc =ZPCI_PCI_LS_ERR ; +s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); + } + }else if (pbdev->summary_ind) { cc =ZPCI_PCI_LS_ERR ;s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); }else if (reg_irqs (env, pbdev, fib)) {@@ -1174,7 +1219,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, }break ;case ZPCI_MOD_FC_DEREG_INT : -if (!pbdev->summary_ind) { +if (pbdev->interp) { +if (mpcifc_dereg_int_interp (pbdev, &fib)) { + cc =ZPCI_PCI_LS_ERR ; +s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); + } + }else if (!pbdev->summary_ind) { cc =ZPCI_PCI_LS_ERR ;s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); }else {hw/s390x/s390-pci-vfio.c +59
-0@@ -149,6 +149,85 @@ int s390_pci_update_passthrough_fh(S390PCIBusDevice *pbdev)return 0 ; } +int s390_pci_probe_aif (S390PCIBusDevice *pbdev) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_feature feat = { + .argsz =sizeof (struct vfio_device_feature ), + .flags =VFIO_DEVICE_FEATURE_PROBE +VFIO_DEVICE_FEATURE_ZPCI_AIF + }; + +return ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , &feat); +} + +int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib,bool enable, +bool assist) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_zpci_aif *data; +int size =sizeof (struct vfio_device_feature ) +sizeof (*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); + + feat->argsz = size; + feat->flags =VFIO_DEVICE_FEATURE_SET +VFIO_DEVICE_FEATURE_ZPCI_AIF ; + + data = (struct vfio_device_zpci_aif *)&feat->data; +if (enable) { + data->flags =VFIO_DEVICE_ZPCI_FLAG_AIF_FLOAT ; +if (!pbdev->intassist) { + data->flags |=VFIO_DEVICE_ZPCI_FLAG_AIF_HOST ; + } +/* Fill in the guest fib info */ + data->ibv = fib->aibv; + data->sb = fib->aisb; + data->noi =FIB_DATA_NOI (fib->data); + data->isc =FIB_DATA_ISC (fib->data); + data->sbo =FIB_DATA_AISBO (fib->data); + }else { + data->flags =0 ; + } + +return ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , feat); +} + +int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable,bool assist) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_zpci_aif *data; +int size =sizeof (struct vfio_device_feature ) +sizeof (*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); +int rc; + + feat->argsz = size; + feat->flags =VFIO_DEVICE_FEATURE_GET +VFIO_DEVICE_FEATURE_ZPCI_AIF ; + + rc =ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , feat); +if (rc) { +return rc; + } + +/* Determine if current interrupt settings match the host */ + data = (struct vfio_device_zpci_aif *)&feat->data;
hw/s390x/s390-pci-vfio.c int s390_pci_update_passthrough_fh(S390PCIBusDevice *pbdev) · line 220
Quoted 10 lines
hw/s390x/s390-pci-vfio.c +10
-0+if (enable && (!(data->flags &VFIO_DEVICE_ZPCI_FLAG_AIF_FLOAT ))) { + rc = -EINVAL ; + }else if (!enable && (data->flags &VFIO_DEVICE_ZPCI_FLAG_AIF_FLOAT )) { + rc = -EINVAL ; + } + +/* +* When enabled for interrupts, the assist and forced host-delivery are +* mututally exclusive +*/
assist is unclear for me int_assist , forwarding_assist ?
Quoted 61 lines
hw/s390x/s390-pci-vfio.c +10
-0+if (enable && assist && (data->flags &VFIO_DEVICE_ZPCI_FLAG_AIF_HOST )) { + rc = -EINVAL ; + }else if (enable && (!assist) && (!(data->flags & +VFIO_DEVICE_ZPCI_FLAG_AIF_HOST ))) { + rc = -EINVAL ; + } + +return rc; +} +static void s390_pci_read_base (S390PCIBusDevice *pbdev,struct vfio_device_info *info) {include/hw/s390x/s390-pci-bus.h +1
-0@@ -351,6 +351,7 @@ struct S390PCIBusDevice {bool pci_unplug_request_processed;bool unplug_requested;bool interp; +bool intassist;QTAILQ_ENTRY (S390PCIBusDevice) link; };include/hw/s390x/s390-pci-vfio.h +13
-0@@ -13,6 +13,7 @@#define HW_S390_PCI_VFIO_H #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-inst.h" #include CONFIG_DEVICES #ifdef CONFIG_VFIO @@ -23,6 +24,11 @@ void s390_pci_end_dma_count(S390pciState *s, S390PCIDMACount *cnt);int s390_pci_probe_interp (S390PCIBusDevice *pbdev);int s390_pci_set_interp (S390PCIBusDevice *pbdev,bool enable);int s390_pci_update_passthrough_fh (S390PCIBusDevice *pbdev); +int s390_pci_probe_aif (S390PCIBusDevice *pbdev); +int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib,bool enable, +bool assist); +int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable,bool assist); +void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)@@ -48,6 +54,20 @@ static inline int s390_pci_update_passthrough_fh(S390PCIBusDevice *pbdev) {return -EINVAL ; } +static inline int s390_pci_probe_aif (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +} +static inline int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib, +bool enable,bool assist) +{
include/hw/s390x/s390-pci-vfio.h static inline int s390_pci_update_passthrough_fh(S390PCIBusDevice *pbdev) · line 73
Quoted 10 lines
include/hw/s390x/s390-pci-vfio.h +7
-0+ return -EINVAL; +} +static inline int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable, +bool assist) +{ +return -EINVAL ; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
apart of the naming (which may be a personal taste) LGTM
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v2 6/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 9 lines · Pierre Morel
Quoted 3 lines · Matthew Rosato
Use the associated vfio feature ioctl to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'intassist' setting.
It is a personal opinion but I do not like the term 'intassist'. Why not using 'gisa' ?
Well, I don't think 'gisa' is quite accurate, we're not toggling the delivery mechanism here, we're toggling who is responsible for doing the delivery (forwarding). The default (intassist=on) says firmware will do it if it can and if not, KVM will be tasked with delivery. intassist=off means KVM will always be told to do the delivery.
Quoted 4 lines
I would find it more relevant.
If something for the profane admin theninterrupt_assistorirq_assistor evenint_assist.Or is it
forwarding_assist?
Yeah, perahps 'forwarding_assist' is the right phrase, as we are disabling firmware's ability to assist in the forwarding of guest interrupts.
Quoted 87 lines
Quoted 87 lines
Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 24 ++++++++-- hw/s390x/s390-pci-inst.c | 54 +++++++++++++++++++++- hw/s390x/s390-pci-vfio.c | 79 ++++++++++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 +include/hw/s390x/s390-pci-vfio.h| 20 ++++++++5 files changed, 173 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-0@@ -189,7 +189,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break;default:
- if (pbdev->summary_ind) {
+ if (pbdev->interp) {
+ /* Interpreted devices were using interrupt forwarding */ +s390_pci_set_aif(pbdev, NULL, false, pbdev->intassist); + } else if (pbdev->summary_ind) {pci_dereg_irqs(pbdev);
}
if (pbdev->iommu->enabled) {hw/s390x/s390-pci-bus.c +6
-0@@ -981,6 +984,11 @@ static int s390_pci_interp_plug(S390pciState *s, S390PCIBusDevice *pbdev) return rc; } + rc = s390_pci_probe_aif(pbdev); + if (rc) { + return rc; + } + rc = s390_pci_update_passthrough_fh(pbdev); if (rc) { return rc;@@ -1076,6 +1084,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (pbdev->interp && !s390_has_feat(S390_FEAT_ZPCI_INTERP)) { DPRINTF("zPCI interpretation facilities missing.\n"); pbdev->interp = false; + pbdev->intassist = false; } if (pbdev->interp) {rc =
s390_pci_interp_plug(s, pbdev);hw/s390x/s390-pci-bus.c +4
-1@@ -1090,11 +1099,13 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, if (!pbdev->interp) { /* Do vfio passthrough but intercept for I/O */ pbdev->fh |= FH_SHM_VFIO; + pbdev->intassist = false; } } else { pbdev->fh |= FH_SHM_EMUL; /* Always intercept emulated devices */ pbdev->interp = false; + pbdev->intassist = false; } if (s390_pci_msix_init(pbdev) && !pbdev->interp) {@@ -1244,7 +1255,10 @@ static void s390_pcihost_reset(DeviceState *dev) /* Process all pending unplug requests */ QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) { if (pbdev->unplug_requested) { - if (pbdev->summary_ind) { + if (pbdev->interp) { + /* Interpreted devices were using interruptforwarding */
+s390_pci_set_aif(pbdev, NULL, false, pbdev->intassist); + } else if (pbdev->summary_ind) {pci_dereg_irqs(pbdev);
}
if (pbdev->iommu->enabled) {hw/s390x/s390-pci-bus.c +4
-1@@ -1382,7 +1396,10 @@ static void s390_pci_device_reset(DeviceState *dev) break; } - if (pbdev->summary_ind) { + if (pbdev->interp) { + /* Interpreted devices were using interrupt forwarding */ + s390_pci_set_aif(pbdev, NULL, false, pbdev->intassist); + } else if (pbdev->summary_ind) { pci_dereg_irqs(pbdev); } if (pbdev->iommu->enabled) {
Quoted 8 lines
hw/s390x/s390-pci-bus.c static Property s390_pci_device_properties[] = { · line 1448
Quoted 5 lines
hw/s390x/s390-pci-bus.c +1
-0@@ -1428,6 +1445,7 @@ static Property s390_pci_device_properties[] = { DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid), DEFINE_PROP_STRING("target", S390PCIBusDevice, target), DEFINE_PROP_BOOL("interp", S390PCIBusDevice, interp, true), + DEFINE_PROP_BOOL("intassist", S390PCIBusDevice, intassist, true),We allow to disable IRQ forwarding only for test or debug purpose don't we? Then shouldn't we make it explicit ?
Not sure I follow.
This setting makes forwarding assist (intassist=on) an implicit default which can be subsequently overruled by an explicit 'intassist=off'.
So, the idea is that by default if interpretation is available and enabled we will also use the firmware assist to do interrupt forwarding -- This is the preferred setup (performance) and nothing needs to be explicitly specified. Explicitly specifying 'intassist=on' does nothing additional.
Conversely, if you want to disable the firmware forwarding (for as you say a test/debug scenario) only then must you explicitly specify 'intassist=off'.
Or were you instead suggesting to make the bool a default of false that must be explicity set to true? In that case, it would probably need a different name (e.g. implicit default is host_forwarding=off, override with host_forwarding=on and only when host_forwarding=on will we force host delivery of interrupts)
FWIW, I chose a default of 'true' because I preferred the idea of having to specify something negative like 'intassist=off' when forcing host delivery as this implies you are turning a feature off (because you are).
Quoted 159 lines
Quoted 159 lines
hw/s390x/s390-pci-bus.c +0
-0DEFINE_PROP_END_OF_LIST(), };hw/s390x/s390-pci-inst.c +40
-0@@ -1111,6 +1111,46 @@ static void fmb_update(void *opaque) timer_mod(pbdev->fmb_timer, t + pbdev->pci_group->zpci_group.mui); } +static int mpcifc_reg_int_interp (S390PCIBusDevice *pbdev,ZpciFib *fib) +{ + int rc; + + /* Interpreted devices must also use interrupt forwarding */ + rc = s390_pci_get_aif(pbdev, false, pbdev->intassist); + if (rc) { + DPRINTF("Bad interrupt forwarding state\n"); + return rc; + } + + rc = s390_pci_set_aif(pbdev, fib, true, pbdev->intassist); + if (rc) { + DPRINTF("Failed to enable interrupt forwarding\n"); + return rc; + } + + return 0; +} + +static int mpcifc_dereg_int_interp (S390PCIBusDevice *pbdev,ZpciFib *fib) +{ + int rc; + + /* Interpreted devices were using interrupt forwarding */ + rc = s390_pci_get_aif(pbdev, true, pbdev->intassist); + if (rc) { + DPRINTF("Bad interrupt forwarding state\n"); + return rc; + } + + rc = s390_pci_set_aif(pbdev, fib, false, pbdev->intassist); + if (rc) { + DPRINTF("Failed to disable interrupt forwarding\n"); + return rc; + } + + return 0; +} + int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, uintptr_t ra) {@@ -1165,7 +1205,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, switch (oc) {case
ZPCI_MOD_FC_REG_INT:
- if (pbdev->summary_ind) {
+ if (pbdev->interp) {
+ if (mpcifc_reg_int_interp(pbdev, &fib)) {
+ cc =ZPCI_PCI_LS_ERR;
+s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); + }
+ } else if (pbdev->summary_ind) {
cc =ZPCI_PCI_LS_ERR;s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); } else if (reg_irqs(env, pbdev, fib)) {hw/s390x/s390-pci-inst.c +0
-0@@ -1174,7 +1219,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, } break;case
ZPCI_MOD_FC_DEREG_INT:
- if (!pbdev->summary_ind) {
+ if (pbdev->interp) {
+ if (mpcifc_dereg_int_interp(pbdev, &fib)) {
+ cc =ZPCI_PCI_LS_ERR;
+s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); + }
+ } else if (!pbdev->summary_ind) {
cc =ZPCI_PCI_LS_ERR;s390_set_status_code(env, r1, ZPCI_MOD_ST_SEQUENCE); } else {hw/s390x/s390-pci-vfio.c +6
-0@@ -149,6 +149,85 @@ int s390_pci_update_passthrough_fh(S390PCIBusDevice *pbdev) return 0; } +int s390_pci_probe_aif(S390PCIBusDevice *pbdev) +{ + VFIOPCIDevice *vdev = VFIO_PCI(pbdev->pdev); + struct vfio_device_feature feat = { + .argsz = sizeof(struct vfio_device_feature), + .flags = VFIO_DEVICE_FEATURE_PROBE +
VFIO_DEVICE_FEATURE_ZPCI_AIF
+ };hw/s390x/s390-pci-vfio.c +14
-0+ + return ioctl(vdev->vbasedev.fd, VFIO_DEVICE_FEATURE, &feat); +} + +int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib,bool enable, + bool assist) +{ + VFIOPCIDevice *vdev = VFIO_PCI(pbdev->pdev); + struct vfio_device_zpci_aif *data; + int size = sizeof(struct vfio_device_feature) + sizeof(*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); + + feat->argsz = size; + feat->flags = VFIO_DEVICE_FEATURE_SET +
VFIO_DEVICE_FEATURE_ZPCI_AIF;hw/s390x/s390-pci-vfio.c +30
-0+ + data = (struct vfio_device_zpci_aif *)&feat->data; + if (enable) { + data->flags = VFIO_DEVICE_ZPCI_FLAG_AIF_FLOAT; + if (!pbdev->intassist) { + data->flags |= VFIO_DEVICE_ZPCI_FLAG_AIF_HOST; + } + /* Fill in the guest fib info */ + data->ibv = fib->aibv; + data->sb = fib->aisb; + data->noi = FIB_DATA_NOI(fib->data); + data->isc = FIB_DATA_ISC(fib->data); + data->sbo = FIB_DATA_AISBO(fib->data); + } else { + data->flags = 0; + } + + return ioctl(vdev->vbasedev.fd, VFIO_DEVICE_FEATURE, feat); +} + +int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable,bool assist) +{ + VFIOPCIDevice *vdev = VFIO_PCI(pbdev->pdev); + struct vfio_device_zpci_aif *data; + int size = sizeof(struct vfio_device_feature) + sizeof(*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); + int rc; + + feat->argsz = size; + feat->flags = VFIO_DEVICE_FEATURE_GET +
VFIO_DEVICE_FEATURE_ZPCI_AIF;hw/s390x/s390-pci-vfio.c +11
-0+ + rc = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_FEATURE, feat); + if (rc) { + return rc; + } + + /* Determine if current interrupt settings match the host */ + data = (struct vfio_device_zpci_aif *)&feat->data; + if (enable && (!(data->flags & VFIO_DEVICE_ZPCI_FLAG_AIF_FLOAT))) { + rc = -EINVAL; + } else if (!enable && (data->flags &
VFIO_DEVICE_ZPCI_FLAG_AIF_FLOAT)) {
+ rc =-EINVAL;
+ }
Quoted 9 lines
Quoted 6 lines
+
+ /*
+ * When enabled for interrupts, the assist and forced host-delivery are
+ * mututally exclusive
+ */assist is unclear for me
int_assist,forwarding_assist?
Yes, my wording here is confusing. Here, 'enable' is meant to represent whether forwarding (AIF) is currently enabled. And then 'assist' is the intassist (forwarding_assist) setting. And my use of 'mutually exclusive' is just plain confusing and should be removed.
I think that otherwise the code is logically correct -- I'll re-visit this and change the variable names and/or improve the comments. To be clear, what we are trying to verify here is:
1) if qemu believes AIF is currently disabled, then kernel should also show the same (AIF_FLOAT is off)
2) if qemu believes AIF is already enabled, kernel should also show the same (AIF_FLOAT is on). Additionally, if qemu believes host delivery is forced (instassist=off) then kernel should also show this (AIF_HOST is on).
This is where my really confusing use of 'mutally exclusive' comes in to play -- the AIF_HOST kernel state is an inversion of the QEMU state, so the values should never match -- if intassist=off then AIF_HOST should be on. If intassist=on then AIF_HOST should be off. But really the comment should just read that we are ensuring that kernel space indicates what we expect for the current state.
PATCH v2 7/9 s390x/pci: use I/O Address Translation assist when interpreting
In reply to Matthew RosatoAllow the underlying kvm host to handle the Refresh PCI Translation instruction intercepts.
--- hw/s390x/s390-pci-bus.c | 6 ++-- hw/s390x/s390-pci-inst.c | 51 ++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-vfio.c | 27 +++++++++++++++++ include/hw/s390x/s390-pci-inst.h | 2 +- include/hw/s390x/s390-pci-vfio.h | 10 +++++++ 5 files changed, 89 insertions(+), 7 deletions(-)
hw/s390x/s390-pci-bus.c +3 -3
@@ -196,7 +196,7 @@ void s390_pci_sclp_deconfigure(SCCB *sccb)pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) { -pci_dereg_ioat (pbdev->iommu); +pci_dereg_ioat (pbdev); } pbdev->state =ZPCI_FS_STANDBY ; rc =SCLP_RC_NORMAL_COMPLETION ;
@@ -1262,7 +1262,7 @@ static void s390_pcihost_reset(DeviceState *dev)pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) { -pci_dereg_ioat (pbdev->iommu); +pci_dereg_ioat (pbdev); } pbdev->state =ZPCI_FS_STANDBY ;s390_pci_perform_unplug (pbdev);
@@ -1403,7 +1403,7 @@ static void s390_pci_device_reset(DeviceState *dev)pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) { -pci_dereg_ioat (pbdev->iommu); +pci_dereg_ioat (pbdev); }fmb_timer_free (pbdev);
hw/s390x/s390-pci-inst.c +48 -3
82 lines not on this page · raw message
hw/s390x/s390-pci-vfio.c +27 -0
@@ -228,6 +228,33 @@ int s390_pci_get_aif(S390PCIBusDevice *pbdev, bool enable, bool assist)return rc; } +int s390_pci_probe_ioat (S390PCIBusDevice *pbdev) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_feature feat = { + .argsz =sizeof (struct vfio_device_feature ), + .flags =VFIO_DEVICE_FEATURE_PROBE +VFIO_DEVICE_FEATURE_ZPCI_IOAT + }; + +return ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , &feat); +} + +int s390_pci_set_ioat (S390PCIBusDevice *pbdev,uint64_t iota) +{ +VFIOPCIDevice *vdev =VFIO_PCI (pbdev->pdev); +struct vfio_device_zpci_ioat *data; +int size =sizeof (struct vfio_device_feature ) +sizeof (*data); + g_autofree struct vfio_device_feature *feat = g_malloc0(size); + + feat->argsz = size; + feat->flags =VFIO_DEVICE_FEATURE_SET +VFIO_DEVICE_FEATURE_ZPCI_IOAT ; + + data = (struct vfio_device_zpci_ioat *)&feat->data; + data->iota = iota; + +return ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , feat); +} +static void s390_pci_read_base (S390PCIBusDevice *pbdev,struct vfio_device_info *info) {
include/hw/s390x/s390-pci-inst.h +1 -1
@@ -99,7 +99,7 @@ typedef struct ZpciFib { }QEMU_PACKED ZpciFib;int pci_dereg_irqs (S390PCIBusDevice *pbdev); -void pci_dereg_ioat (S390PCIIOMMU *iommu); +void pci_dereg_ioat (S390PCIBusDevice *pbdev);int clp_service_call (S390CPU *cpu,uint8_t r2,uintptr_t ra);int pcilg_service_call (S390CPU *cpu,uint8_t r1,uint8_t r2,uintptr_t ra);int pcistg_service_call (S390CPU *cpu,uint8_t r1,uint8_t r2,uintptr_t ra);
include/hw/s390x/s390-pci-vfio.h +10 -0
@@ -28,6 +28,8 @@ int s390_pci_probe_aif(S390PCIBusDevice *pbdev);int s390_pci_set_aif (S390PCIBusDevice *pbdev,ZpciFib *fib,bool enable,bool assist);int s390_pci_get_aif (S390PCIBusDevice *pbdev,bool enable,bool assist); +int s390_pci_probe_ioat (S390PCIBusDevice *pbdev); +int s390_pci_set_ioat (S390PCIBusDevice *pbdev,uint64_t iota);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else
@@ -68,6 +70,14 @@ static inline int s390_pci_get_aif(S390PCIBusDevice *pbdev, bool enable, {return -EINVAL ; } +static inline int s390_pci_probe_ioat (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +} +static inline int s390_pci_set_ioat (S390PCIBusDevice *pbdev,uint64_t iota) +{ +return -EINVAL ; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
Signature · 1 line
2.27.0
PATCH v2 8/9 s390x/pci: use dtsm provided from vfio capabilities for interpreted devices
In reply to Matthew RosatoWhen using the IOAT assist via interpretation, we should advertise what the host driver supports, not QEMU.
--- hw/s390x/s390-pci-vfio.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
hw/s390x/s390-pci-vfio.c +5 -1
@@ -318,7 +318,11 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, resgrp->i = cap->noi; resgrp->maxstbl = cap->maxstbl; resgrp->version = cap->version; - resgrp->dtsm =ZPCI_DTSM ; +if (hdr->version >=2 && pbdev->interp) { + resgrp->dtsm = cap->dtsm; + }else { + resgrp->dtsm =ZPCI_DTSM ; + } } }
Signature · 1 line
2.27.0
PATCH v2 9/9 s390x/pci: let intercept devices have separate PCI groups
In reply to Matthew RosatoLet's use the reserved pool of simulated PCI groups to allow intercept devices to have separate groups from interpreted devices as some group values may be different. If we run out of simulated PCI groups, subsequent intercept devices just get the default group.
Furthermore, if we encounter any PCI groups from hostdevs that are marked as simulated, let's just assign them to the default group to avoid conflicts between host simulated groups and our own simulated groups.
--- hw/s390x/s390-pci-bus.c | 19 ++++++++++++++-- hw/s390x/s390-pci-vfio.c | 40 ++++++++++++++++++++++++++++++--- include/hw/s390x/s390-pci-bus.h | 6 ++++- 3 files changed, 59 insertions(+), 6 deletions(-)
hw/s390x/s390-pci-bus.c +17 -2
@@ -747,13 +747,14 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn)object_unref (OBJECT (iommu)); } -S390PCIGroup *s390_group_create (int id) +S390PCIGroup *s390_group_create (int id,int host_id) {S390PCIGroup *group;S390pciState *s =s390_get_phb (); group =g_new0 (S390PCIGroup,1 ); group->id = id; + group->host_id = host_id;QTAILQ_INSERT_TAIL (&s->zpci_groups, group, link);return group; }
@@ -771,12 +772,25 @@ S390PCIGroup *s390_group_find(int id)return NULL ; } +S390PCIGroup *s390_group_find_host_sim (int host_id) +{ +S390PCIGroup *group; +S390pciState *s =s390_get_phb (); + + QTAILQ_FOREACH(group, &s->zpci_groups, link) { +if (group->id >=ZPCI_SIM_GRP_START && group->host_id == host_id) { +return group; + } + } +return NULL ; +} +static void s390_pci_init_default_group (void ) {S390PCIGroup *group;ClpRspQueryPciGrp *resgrp; - group =s390_group_create (ZPCI_DEFAULT_FN_GRP ); + group =s390_group_create (ZPCI_DEFAULT_FN_GRP ,ZPCI_DEFAULT_FN_GRP ); resgrp = &group->zpci_group; resgrp->fr =1 ; resgrp->dasm =0 ;
@@ -824,6 +838,7 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp) NULL, g_free); s->zpci_table =g_hash_table_new_full (g_int_hash, g_int_equal,NULL ,NULL ); s->bus_no =0 ; + s->next_sim_grp =ZPCI_SIM_GRP_START ;QTAILQ_INIT (&s->pending_sei);QTAILQ_INIT (&s->zpci_devs);QTAILQ_INIT (&s->zpci_dma_limit);
hw/s390x/s390-pci-vfio.c +37 -3
@@ -287,13 +287,17 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, {struct vfio_info_cap_header *hdr;struct vfio_device_info_cap_zpci_group *cap; +S390pciState *s =s390_get_phb ();ClpRspQueryPciGrp *resgrp;VFIOPCIDevice *vpci =container_of (pbdev->pdev, VFIOPCIDevice, pdev); hdr =vfio_get_device_info_cap (info,VFIO_DEVICE_INFO_CAP_ZPCI_GROUP ); -/* If capability not provided, just use the default group */ -if (hdr ==NULL ) { +/* +* If capability not provided or the underlying hostdev is simulated, just +* use the default group. +*/ +if (hdr ==NULL || pbdev->zpci_fn.pfgid >=ZPCI_SIM_GRP_START ) {trace_s390_pci_clp_cap (vpci->vbasedev.name,VFIO_DEVICE_INFO_CAP_ZPCI_GROUP ); pbdev->zpci_fn.pfgid =ZPCI_DEFAULT_FN_GRP ;
@@ -302,11 +306,41 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, } cap = (void *) hdr; +/* +* For an intercept device, let's use an existing simulated group if one +* one was already created for other intercept devices in this group. +* If not, create a new simulated group if any are still available. +* If all else fails, just fall back on the default group. +*/ +if (!pbdev->interp) { + pbdev->pci_group =s390_group_find_host_sim (pbdev->zpci_fn.pfgid); +if (pbdev->pci_group) { +/* Use existing simulated group */ + pbdev->zpci_fn.pfgid = pbdev->pci_group->id; +return ; + }else { +if (s->next_sim_grp ==ZPCI_DEFAULT_FN_GRP ) { +/* All out of simulated groups, use default */ +trace_s390_pci_clp_cap (vpci->vbasedev.name, +VFIO_DEVICE_INFO_CAP_ZPCI_GROUP ); + pbdev->zpci_fn.pfgid =ZPCI_DEFAULT_FN_GRP ; + pbdev->pci_group =s390_group_find (ZPCI_DEFAULT_FN_GRP ); +return ; + }else { +/* We can assign a new simulated group */ + pbdev->zpci_fn.pfgid = s->next_sim_grp; + s->next_sim_grp++; +/* Fall through to create the new sim group using CLP info */ + } + } + } +/* See if the PCI group is already defined, create if not */ pbdev->pci_group =s390_group_find (pbdev->zpci_fn.pfgid);if (!pbdev->pci_group) { - pbdev->pci_group =s390_group_create (pbdev->zpci_fn.pfgid); + pbdev->pci_group =s390_group_create (pbdev->zpci_fn.pfgid, + pbdev->zpci_fn.pfgid); resgrp = &pbdev->pci_group->zpci_group;if (cap->flags &VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH ) {
include/hw/s390x/s390-pci-bus.h +5 -1
@@ -315,13 +315,16 @@ typedef struct ZpciFmb {QEMU_BUILD_BUG_MSG (offsetof(ZpciFmb , fmt0) !=48 ,"padding in ZpciFmb" );#define ZPCI_DEFAULT_FN_GRP 0xFF +#define ZPCI_SIM_GRP_START 0xF0typedef struct S390PCIGroup {ClpRspQueryPciGrp zpci_group;int id; +int host_id;QTAILQ_ENTRY (S390PCIGroup) link; }S390PCIGroup ; -S390PCIGroup *s390_group_create (int id); +S390PCIGroup *s390_group_create (int id,int host_id);S390PCIGroup *s390_group_find (int id); +S390PCIGroup *s390_group_find_host_sim (int host_id);struct S390PCIBusDevice {DeviceState qdev;
@@ -370,6 +373,7 @@ struct S390pciState { QTAILQ_HEAD(, S390PCIBusDevice) zpci_devs; QTAILQ_HEAD(, S390PCIDMACount) zpci_dma_limit; QTAILQ_HEAD(, S390PCIGroup) zpci_groups; +uint8_t next_sim_grp; };S390pciState *s390_get_phb (void );
Signature · 1 line
2.27.0
Quoted 10 lines · Matthew Rosato
For QEMU, the majority of the work in enabling instruction interpretation is handled via new VFIO ioctls to SET the appropriate interpretation and interrupt forwarding modes, and to GET the function handle to use for interpretive execution.
This series implements these new ioctls, as well as adding a new, optional 'intercept' parameter to zpci to request interpretation support not be used as well as an 'intassist' parameter to determine whether or not the firmware assist will be used for interrupt delivery or whether the host will be responsible for delivering all interrupts.
Hi Matthew,
would it make sense to create a docs/system/s390x/zpci.rst doc file, too, where you could describe such new parameters like 'intassist' and 'intercept' (or is it 'interp') ? ... otherwise hardly anybody except you will know how to use these parameters later...
Thomas
Quoted 12 lines · Thomas Huth
Quoted 10 lines · Matthew Rosato
For QEMU, the majority of the work in enabling instruction interpretation is handled via new VFIO ioctls to SET the appropriate interpretation and interrupt forwarding modes, and to GET the function handle to use for interpretive execution.
This series implements these new ioctls, as well as adding a new, optional 'intercept' parameter to zpci to request interpretation support not be used as well as an 'intassist' parameter to determine whether or not the firmware assist will be used for interrupt delivery or whether the host will be responsible for delivering all interrupts.
Quoted 3 lines
Hi Matthew,
would it make sense to create a
docs/system/s390x/zpci.rstdoc file,
This is a good idea and probably something that was due for zpci before this series even.
Quoted 1 line
too, where you could describe such new parameters like 'intassist' and 'intercept' (or is it 'interp') ? ... otherwise hardly anybody except
Oops, 'intercept' was a holdover from a previous version and effectively had an inverted meaning. It is indeed 'interp' with this current series (and subject to change again per other thread)
Quoted 18 lines · Matthew Rosato
For QEMU, the majority of the work in enabling instruction interpretation is handled via new VFIO ioctls to SET the appropriate interpretation and interrupt forwarding modes, and to GET the function handle to use for interpretive execution.
This series implements these new ioctls, as well as adding a new, optional 'intercept' parameter to zpci to request interpretation support not be used as well as an 'intassist' parameter to determine whether or not the firmware assist will be used for interrupt delivery or whether the host will be responsible for delivering all interrupts.
The
ZPCI_INTERPCPU feature is added beginning with the z14 model to enable this support.As a consequence of implementing zPCI interpretation, ISM devices now become eligible for passthrough (but only when zPCI interpretation is available).
Quoted 6 lines
Quoted 1 line
From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by default if available in kernel+qemu.
Associated kernel series:
https://lore.kernel.org/kvm/[email protected]/
I took a quick look and don't see much to object to, but I don't know much about the topic either.
Pls use the s390 tree for this.
Quoted 52 lines
Changes
v1->v2:- Update kernel headers sync
- Drop some pre-req patches that are now merged
- Add some R-bs (Thanks!)
- fenceFEAT_ZPCI_INTERPfor QEMU 6.2 and older (Christian)
- switch fromcontainer_oftoVFIO_PCIand drop asserts (Thomas)
- re-arrangeg_autofreeso we malloc at time of declaration (Thomas)Matthew Rosato (9):
Update linux headers
target/s390x: add zpci-interp to cpu models
fixup: force interp off for QEMU machine 6.2 and older s390x/pci: enable for load/store intepretation
s390x/pci: don't fence interpreted devices without MSI-X s390x/pci: enable adapter event notification for interpreted devices s390x/pci: use I/O Address Translation assist when interpreting s390x/pci: use dtsm provided from vfio capabilities for interpreted
devices
s390x/pci: let intercept devices have separate PCI groupshw/s390x/s390-pci-bus.c | 121 ++++++++++- hw/s390x/s390-pci-inst.c | 168 ++++++++++++++- hw/s390x/s390-pci-vfio.c | 204 +++++++++++++++++- hw/s390x/s390-virtio-ccw.c | 1 + include/hw/s390x/s390-pci-bus.h | 8 +- include/hw/s390x/s390-pci-inst.h | 2 +- include/hw/s390x/s390-pci-vfio.h | 45 ++++ include/standard-headers/asm-x86/kvm_para.h | 1 + include/standard-headers/drm/drm_fourcc.h | 11 + include/standard-headers/linux/ethtool.h | 1 + include/standard-headers/linux/fuse.h | 60 +++++- include/standard-headers/linux/pci_regs.h | 4 + include/standard-headers/linux/virtio_iommu.h | 8 +- linux-headers/asm-mips/unistd_n32.h | 1 + linux-headers/asm-mips/unistd_n64.h | 1 + linux-headers/asm-mips/unistd_o32.h | 1 + linux-headers/asm-powerpc/unistd_32.h | 1 + linux-headers/asm-powerpc/unistd_64.h | 1 + linux-headers/asm-s390/kvm.h | 1 + linux-headers/asm-s390/unistd_32.h | 1 + linux-headers/asm-s390/unistd_64.h | 1 + linux-headers/linux/kvm.h | 1 + linux-headers/linux/vfio.h | 22 ++ linux-headers/linux/vfio_zdev.h | 51 +++++ target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 + target/s390x/kvm/kvm.c | 1 + 27 files changed, 693 insertions(+), 27 deletions(-)Signature · 1 line
2.27.0