s390x/pci: zPCI interpretation support
Subsystems (2)
v5 cover s390x/pci: zPCI interpretation support
For QEMU, the majority of the work in enabling instruction interpretation is handled via SHM bit settings (to indicate to firmware whether or not interpretive execution facilities are to be used) + a new KVM ioctl is used to setup firmware-interpreted forwarding of Adapter Event Notifications.
This series also adds a new, optional 'interpret' parameter to zpci which can be used to disable interpretation support (interpret=off) as well as an 'forwarding_assist' parameter to determine whether or not the firmware assist will be used for adapter event delivery (default when interpretation is in use) or whether the host will be responsible for delivering all adapter event notifications (forwarding_assist=off).
The ZPCI_INTERP CPU feature is added beginning with the z14 model to enable this support.
As a consequence of implementing zPCI interpretation, ISM devices now become eligible for passthrough (but only when zPCI interpretation is available).
From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu.
Will reply with a link to the associated kernel series.
Changelog v4->v5:
- Update to match latest interface from kernel code. Major changes are:
1) we no longer issue any ioctls to set a device to interpreted mode; rather, this will be done automatically if supported by the host kernel at the time the vfio group is associated with the KVM. Then, the SHM bit setting will indicate whether or not interpretation is actually used.
2) the RPCIT enhancments (IOMMU changes) are removed from this series, so the code associated with indicating a desired IOMMU are also removed. With this series s390x-pci will continue to use only type1 IOMMU for now.
- Refresh the linux headers sync. Added a patch to tolerate some vfio uapi renames that will happen in 5.18 (this can be discarded if there is something else underway to address this)
Matthew Rosato (9):
Update linux headers
vfio: tolerate migration protocol v1 uapi renames target/s390x: add zpci-interp to cpu models
s390x/pci: add routine to get host function handle from CLP info s390x/pci: enable for load/store intepretation
s390x/pci: don't fence interpreted devices without MSI-X s390x/pci: enable adapter event notification for interpreted devices s390x/pci: let intercept devices have separate PCI groups s390x/pci: reflect proper maxstbl for groups of interpreted devices
hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 107 ++++- hw/s390x/s390-pci-inst.c | 52 ++- hw/s390x/s390-pci-kvm.c | 51 +++ hw/s390x/s390-pci-vfio.c | 129 +++++- hw/s390x/s390-virtio-ccw.c | 1 + hw/vfio/common.c | 2 +- hw/vfio/migration.c | 19 +- include/hw/s390x/s390-pci-bus.h | 8 +- include/hw/s390x/s390-pci-kvm.h | 38 ++ include/hw/s390x/s390-pci-vfio.h | 6 + .../linux/input-event-codes.h | 4 +- .../standard-headers/linux/virtio_config.h | 6 + .../standard-headers/linux/virtio_crypto.h | 82 +++- linux-headers/asm-arm64/kvm.h | 16 + linux-headers/asm-generic/mman-common.h | 2 + linux-headers/asm-mips/mman.h | 2 + linux-headers/asm-s390/kvm.h | 1 + linux-headers/linux/kvm.h | 50 ++- linux-headers/linux/psci.h | 4 + linux-headers/linux/userfaultfd.h | 8 +- linux-headers/linux/vfio.h | 406 +++++++++--------- linux-headers/linux/vfio_zdev.h | 7 + linux-headers/linux/vhost.h | 7 + target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 + target/s390x/kvm/kvm.c | 8 + target/s390x/kvm/kvm_s390x.h | 1 + 28 files changed, 763 insertions(+), 258 deletions(-) create mode 100644 hw/s390x/s390-pci-kvm.c create mode 100644 include/hw/s390x/s390-pci-kvm.h
Signature · 1 line
2.27.0
PATCH v5 1/9 Update linux headers
This is a placeholder that pulls in 5.18-rc1 + unmerged kernel changes required by this item. A proper header sync can be done once the associated kernel code merges.
--- .../linux/input-event-codes.h | 4 +- .../standard-headers/linux/virtio_config.h | 6 + .../standard-headers/linux/virtio_crypto.h | 82 +++- linux-headers/asm-arm64/kvm.h | 16 + linux-headers/asm-generic/mman-common.h | 2 + linux-headers/asm-mips/mman.h | 2 + linux-headers/asm-s390/kvm.h | 1 + linux-headers/linux/kvm.h | 50 ++- linux-headers/linux/psci.h | 4 + linux-headers/linux/userfaultfd.h | 8 +- linux-headers/linux/vfio.h | 406 +++++++++--------- linux-headers/linux/vfio_zdev.h | 7 + linux-headers/linux/vhost.h | 7 + 13 files changed, 376 insertions(+), 219 deletions(-)
include/standard-headers/linux/input-event-codes.h +3 -1
@@ -278,7 +278,8 @@#define KEY_PAUSECD 201#define KEY_PROG3 202#define KEY_PROG4 203 -#define KEY_DASHBOARD 204/* AL Dashboard */ +#define KEY_ALL_APPLICATIONS 204/* AC Desktop Show All Applications */ +#define KEY_DASHBOARD KEY_ALL_APPLICATIONS#define KEY_SUSPEND 205#define KEY_CLOSE 206/* AC Close */ #define KEY_PLAY 207
@@ -612,6 +613,7 @@#define KEY_ASSISTANT 0x247/* AL Context-aware desktop assistant */ #define KEY_KBD_LAYOUT_NEXT 0x248/* AC Next Keyboard Layout Select */ #define KEY_EMOJI_PICKER 0x249/* Show/hide emoji picker (HUTRR101) */ +#define KEY_DICTATE 0x24a/* Start or Stop Voice Dictation Session (HUTRR99) */ #define KEY_BRIGHTNESS_MIN 0x250/* Set Brightness to Minimum */ #define KEY_BRIGHTNESS_MAX 0x251/* Set Brightness to Maximum */
include/standard-headers/linux/virtio_config.h +6 -0
@@ -80,6 +80,12 @@/* This feature indicates support for the packed virtqueue layout. */ #define VIRTIO_F_RING_PACKED 34 +/* +* Inorder feature indicates that all buffers are used by the device +* in the same order in which they have been made available. +*/ +#define VIRTIO_F_IN_ORDER 35 +/* * This feature indicates that memory accesses by the driver and the * device are ordered in a way described by the platform.
include/standard-headers/linux/virtio_crypto.h +81 -1
136 lines not on this page · raw message
linux-headers/asm-arm64/kvm.h +16 -0
@@ -281,6 +281,11 @@ struct kvm_arm_copy_mte_tags {#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED 3#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED (1U << 4) +#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3 KVM_REG_ARM_FW_REG(3) +#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3_NOT_AVAIL 0 +#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3_AVAIL 1 +#define KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_3_NOT_REQUIRED 2 +/* SVE registers */ #define KVM_REG_ARM64_SVE (0x15 << KVM_REG_ARM_COPROC_SHIFT)
@@ -362,6 +367,7 @@ struct kvm_arm_copy_mte_tags {#define KVM_ARM_VCPU_PMU_V3_IRQ 0#define KVM_ARM_VCPU_PMU_V3_INIT 1#define KVM_ARM_VCPU_PMU_V3_FILTER 2 +#define KVM_ARM_VCPU_PMU_V3_SET_PMU 3#define KVM_ARM_VCPU_TIMER_CTRL 1#define KVM_ARM_VCPU_TIMER_IRQ_VTIMER 0#define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1
@@ -411,6 +417,16 @@ struct kvm_arm_copy_mte_tags {#define KVM_PSCI_RET_INVAL PSCI_RET_INVALID_PARAMS#define KVM_PSCI_RET_DENIED PSCI_RET_DENIED +/* arm64-specific kvm_run::system_event flags */ +/* +* Reset caused by a PSCI v1.1 SYSTEM_RESET2 call. +* Valid only when the system event has a type of KVM_SYSTEM_EVENT_RESET. +*/ +#define KVM_SYSTEM_EVENT_RESET_FLAG_PSCI_RESET2 (1ULL << 0) + +/* run->fail_entry.hardware_entry_failure_reason codes. */ +#define KVM_EXIT_FAIL_ENTRY_CPU_UNSUPPORTED (1ULL << 0) +#endif #endif /* __ARM_KVM_H__ */
linux-headers/asm-generic/mman-common.h +2 -0
@@ -75,6 +75,8 @@#define MADV_POPULATE_READ 22/* populate (prefault) page tables readable */ #define MADV_POPULATE_WRITE 23/* populate (prefault) page tables writable */ +#define MADV_DONTNEED_LOCKED 24/* like DONTNEED, but drop locked pages too */ +/* compatibility flags */ #define MAP_FILE 0
linux-headers/asm-mips/mman.h +2 -0
@@ -101,6 +101,8 @@#define MADV_POPULATE_READ 22/* populate (prefault) page tables readable */ #define MADV_POPULATE_WRITE 23/* populate (prefault) page tables writable */ +#define MADV_DONTNEED_LOCKED 24/* like DONTNEED, but drop locked pages too */ +/* compatibility flags */ #define MAP_FILE 0
linux-headers/asm-s390/kvm.h +1 -0
@@ -130,6 +130,7 @@ struct kvm_s390_vm_cpu_machine {#define KVM_S390_VM_CPU_FEAT_PFMFI 11#define KVM_S390_VM_CPU_FEAT_SIGPIF 12#define KVM_S390_VM_CPU_FEAT_KSS 13 +#define KVM_S390_VM_CPU_FEAT_ZPCI_INTERP 14struct kvm_s390_vm_cpu_feat {__u64 feat[16 ]; };
linux-headers/linux/kvm.h +45 -5
88 lines not on this page · raw message
linux-headers/linux/psci.h +4 -0
@@ -82,6 +82,10 @@#define PSCI_0_2_TOS_UP_NO_MIGRATE 1#define PSCI_0_2_TOS_MP 2 +/* PSCI v1.1 reset type encoding for SYSTEM_RESET2 */ +#define PSCI_1_1_RESET_TYPE_SYSTEM_WARM_RESET 0 +#define PSCI_1_1_RESET_TYPE_VENDOR_START 0x80000000U +/* PSCI version decoding (independent of PSCI version) */ #define PSCI_VERSION_MAJOR_SHIFT 16#define PSCI_VERSION_MINOR_MASK \
linux-headers/linux/userfaultfd.h +7 -1
@@ -32,7 +32,8 @@ UFFD_FEATURE_SIGBUS | \ UFFD_FEATURE_THREAD_ID | \ UFFD_FEATURE_MINOR_HUGETLBFS | \ - UFFD_FEATURE_MINOR_SHMEM) + UFFD_FEATURE_MINOR_SHMEM | \ + UFFD_FEATURE_EXACT_ADDRESS)#define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \
@@ -189,6 +190,10 @@ struct uffdio_api {* * UFFD_FEATURE_MINOR_SHMEM indicates the same support as * UFFD_FEATURE_MINOR_HUGETLBFS, but for shmem-backed pages instead. +* +* UFFD_FEATURE_EXACT_ADDRESS indicates that the exact address of page +* faults would be provided and the offset within the page would not be +* masked. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0)#define UFFD_FEATURE_EVENT_FORK (1<<1)
@@ -201,6 +206,7 @@ struct uffdio_api {#define UFFD_FEATURE_THREAD_ID (1<<8)#define UFFD_FEATURE_MINOR_HUGETLBFS (1<<9)#define UFFD_FEATURE_MINOR_SHMEM (1<<10) +#define UFFD_FEATURE_EXACT_ADDRESS (1<<11)__u64 features;__u64 ioctls;
linux-headers/linux/vfio.h +195 -211
433 lines not on this page · raw message
linux-headers/linux/vfio_zdev.h +7 -0
@@ -29,6 +29,9 @@ struct vfio_device_info_cap_zpci_base {__u16 fmb_length;/* Measurement Block Length (in bytes) */ __u8 pft;/* PCI Function Type */ __u8 gid;/* PCI function group ID */ +/* End of version 1 */ +__u32 fh;/* PCI function handle */ +/* End of version 2 */ };/**
@@ -47,6 +50,10 @@ struct vfio_device_info_cap_zpci_group {__u16 noi;/* Maximum number of MSIs */ __u16 maxstbl;/* Maximum Store Block Length */ __u8 version;/* Supported PCI Version */ +/* End of version 1 */ +__u8 reserved; +__u16 imaxstbl;/* Maximum Interpreted Store Block Length */ +/* End of version 2 */ };/**
linux-headers/linux/vhost.h +7 -0
@@ -150,4 +150,11 @@/* Get the valid iova range */ #define VHOST_VDPA_GET_IOVA_RANGE _IOR(VHOST_VIRTIO, 0x78, \ struct vhost_vdpa_iova_range) + +/* Get the config size */ +#define VHOST_VDPA_GET_CONFIG_SIZE _IOR(VHOST_VIRTIO, 0x79, __u32) + +/* Get the count of all virtqueues */ +#define VHOST_VDPA_GET_VQS_COUNT _IOR(VHOST_VIRTIO, 0x80, __u32) +#endif
Signature · 1 line
2.27.0
PATCH v5 2/9 vfio: tolerate migration protocol v1 uapi renames
In reply to Matthew RosatoThe v1 uapi is deprecated and will be replaced by v2 at some point; this patch just tolerates the renaming of uapi fields to reflect v1 / deprecated status.
--- hw/vfio/common.c | 2 +- hw/vfio/migration.c | 19 +++++++++++-------- 2 files changed, 12 insertions(+), 9 deletions(-)
hw/vfio/common.c +1 -1
@@ -380,7 +380,7 @@ static bool vfio_devices_all_running_and_saving(VFIOContainer *container)return false; } -if ((migration->device_state &VFIO_DEVICE_STATE_SAVING ) && +if ((migration->device_state &VFIO_DEVICE_STATE_V1_SAVING ) && (migration->device_state &VFIO_DEVICE_STATE_RUNNING )) {continue ; }else {
hw/vfio/migration.c +11 -8
@@ -432,7 +432,7 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) } ret =vfio_migration_set_state (vbasedev,VFIO_DEVICE_STATE_MASK , -VFIO_DEVICE_STATE_SAVING ); +VFIO_DEVICE_STATE_V1_SAVING );if (ret) {error_report ("%s: Failed to set state SAVING" , vbasedev->name);return ret;
@@ -532,7 +532,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)int ret; ret =vfio_migration_set_state (vbasedev, ~VFIO_DEVICE_STATE_RUNNING , -VFIO_DEVICE_STATE_SAVING ); +VFIO_DEVICE_STATE_V1_SAVING );if (ret) {error_report ("%s: Failed to set state STOP and SAVING" , vbasedev->name);
@@ -569,7 +569,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)return ret; } - ret =vfio_migration_set_state (vbasedev, ~VFIO_DEVICE_STATE_SAVING ,0 ); + ret =vfio_migration_set_state (vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING ,0 );if (ret) {error_report ("%s: Failed to set state STOPPED" , vbasedev->name);return ret;
@@ -730,7 +730,7 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state)* start saving data. */ if (state ==RUN_STATE_SAVE_VM ) { - value =VFIO_DEVICE_STATE_SAVING ; + value =VFIO_DEVICE_STATE_V1_SAVING ; }else { value =0 ; }
@@ -768,8 +768,9 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case MIGRATION_STATUS_FAILED : bytes_transferred =0 ; ret =vfio_migration_set_state (vbasedev, - ~(VFIO_DEVICE_STATE_SAVING |VFIO_DEVICE_STATE_RESUMING ), -VFIO_DEVICE_STATE_RUNNING ); + ~(VFIO_DEVICE_STATE_V1_SAVING | +VFIO_DEVICE_STATE_RESUMING ), +VFIO_DEVICE_STATE_RUNNING );if (ret) {error_report ("%s: Failed to set state RUNNING" , vbasedev->name); }
@@ -864,8 +865,10 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) goto add_blocker; } - ret =vfio_get_dev_region_info (vbasedev,VFIO_REGION_TYPE_MIGRATION , -VFIO_REGION_SUBTYPE_MIGRATION , &info); + ret =vfio_get_dev_region_info (vbasedev, +VFIO_REGION_TYPE_MIGRATION_DEPRECATED , +VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED , + &info);if (ret) { goto add_blocker; }
Signature · 1 line
2.27.0
Re: [PATCH v5 2/9] vfio: tolerate migration protocol v1 uapi renames
Quoted 9 lines · Matthew Rosato
The v1 uapi is deprecated and will be replaced by v2 at some point; this patch just tolerates the renaming of uapi fields to reflect v1 / deprecated status.
Signed-off-by Matthew Rosato <[email protected]>--- hw/vfio/common.c | 2 +- hw/vfio/migration.c | 19 +++++++++++-------- 2 files changed, 12 insertions(+), 9 deletions(-)
I do not understand why you need this patch in this series. Shouldn't it be separate?
Quoted 80 lines
hw/vfio/common.c +1
-1@@ -380,7 +380,7 @@ static bool vfio_devices_all_running_and_saving(VFIOContainer *container)return false; } -if ((migration->device_state &VFIO_DEVICE_STATE_SAVING ) && +if ((migration->device_state &VFIO_DEVICE_STATE_V1_SAVING ) && (migration->device_state &VFIO_DEVICE_STATE_RUNNING )) {continue ; }else {hw/vfio/migration.c +11
-8@@ -432,7 +432,7 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) } ret =vfio_migration_set_state (vbasedev,VFIO_DEVICE_STATE_MASK , -VFIO_DEVICE_STATE_SAVING ); +VFIO_DEVICE_STATE_V1_SAVING );if (ret) {error_report ("%s: Failed to set state SAVING" , vbasedev->name);return ret;@@ -532,7 +532,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)int ret; ret =vfio_migration_set_state (vbasedev, ~VFIO_DEVICE_STATE_RUNNING , -VFIO_DEVICE_STATE_SAVING ); +VFIO_DEVICE_STATE_V1_SAVING );if (ret) {error_report ("%s: Failed to set state STOP and SAVING" , vbasedev->name);@@ -569,7 +569,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque)return ret; } - ret =vfio_migration_set_state (vbasedev, ~VFIO_DEVICE_STATE_SAVING ,0 ); + ret =vfio_migration_set_state (vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING ,0 );if (ret) {error_report ("%s: Failed to set state STOPPED" , vbasedev->name);return ret;@@ -730,7 +730,7 @@ static void vfio_vmstate_change(void *opaque, bool running, RunState state)* start saving data. */ if (state ==RUN_STATE_SAVE_VM ) { - value =VFIO_DEVICE_STATE_SAVING ; + value =VFIO_DEVICE_STATE_V1_SAVING ; }else { value =0 ; }@@ -768,8 +768,9 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case MIGRATION_STATUS_FAILED : bytes_transferred =0 ; ret =vfio_migration_set_state (vbasedev, - ~(VFIO_DEVICE_STATE_SAVING |VFIO_DEVICE_STATE_RESUMING ), -VFIO_DEVICE_STATE_RUNNING ); + ~(VFIO_DEVICE_STATE_V1_SAVING | +VFIO_DEVICE_STATE_RESUMING ), +VFIO_DEVICE_STATE_RUNNING );if (ret) {error_report ("%s: Failed to set state RUNNING" , vbasedev->name); }@@ -864,8 +865,10 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) goto add_blocker; } - ret =vfio_get_dev_region_info (vbasedev,VFIO_REGION_TYPE_MIGRATION , -VFIO_REGION_SUBTYPE_MIGRATION , &info); + ret =vfio_get_dev_region_info (vbasedev, +VFIO_REGION_TYPE_MIGRATION_DEPRECATED , +VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED , + &info);if (ret) { goto add_blocker; }
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 2/9] vfio: tolerate migration protocol v1 uapi renames
Quoted 14 lines · Pierre Morel
Quoted 9 lines · Matthew Rosato
The v1 uapi is deprecated and will be replaced by v2 at some point; this patch just tolerates the renaming of uapi fields to reflect v1 / deprecated status.
Signed-off-by Matthew Rosato <[email protected]>---
hw/vfio/common.c | 2 +-
hw/vfio/migration.c| 19 +++++++++++--------2 files changed, 12 insertions(+), 9 deletions(-)
Quoted 2 lines
I do not understand why you need this patch in this series. Shouldn't it be separate?
This patch is included because of the patch 1 kernel header sync, which pulls in uapi headers from kernel version 5.18-rc1 + my unmerged kernel uapi changes.
This patch is unnecessary without a header sync (and in fact would break QEMU compile), and is unrelated to the rest of the series -- but QEMU will not compile without it once you update linux uapi headers to 5.18-rc1 (or greater) due to the v1 uapi for vfio migration being deprecated [1]. This means that ANY series that does a linux header sync starting from here on will need something like this patch to go along with the header sync (or a series that replaces v1 usage with v2?).
If this patch looks good it could be included whenever a header sync is next needed, doesn't necessarily have to be with this series.
[1] https://www.spinics.net/lists/kernel/msg4288200.html
Quoted 90 lines
Quoted 89 lines
hw/vfio/common.c +1
-1@@ -380,7 +380,7 @@ static bool vfio_devices_all_running_and_saving(VFIOContainer *container) return false; } - if ((migration->device_state & VFIO_DEVICE_STATE_SAVING) && + if ((migration->device_state &
VFIO_DEVICE_STATE_V1_SAVING) &&
(migration->device_state&VFIO_DEVICE_STATE_RUNNING)) { continue;
} else {hw/vfio/migration.c +3
-3@@ -432,7 +432,7 @@ static int vfio_save_setup(QEMUFile *f, void *opaque) } ret = vfio_migration_set_state(vbasedev, VFIO_DEVICE_STATE_MASK, - VFIO_DEVICE_STATE_SAVING); + VFIO_DEVICE_STATE_V1_SAVING); if (ret) { error_report("%s: Failed to set state SAVING", vbasedev->name); return ret;@@ -532,7 +532,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) int ret; ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_RUNNING, - VFIO_DEVICE_STATE_SAVING); + VFIO_DEVICE_STATE_V1_SAVING); if (ret) { error_report("%s: Failed to set state STOP and SAVING", vbasedev->name);@@ -569,7 +569,7 @@ static int vfio_save_complete_precopy(QEMUFile *f, void *opaque) return ret; } - ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_SAVING, 0); + ret = vfio_migration_set_state(vbasedev, ~VFIO_DEVICE_STATE_V1_SAVING ,0 ); if (ret) {
error_report("%s: Failed to set state STOPPED", vbasedev->name); return ret;hw/vfio/migration.c +1
-1@@ -730,7 +730,7 @@ static void vfio_vmstate_change(void *opaque, boolrunning, RunState state) * start saving data. */ if (state == RUN_STATE_SAVE_VM) { - value = VFIO_DEVICE_STATE_SAVING; + value = VFIO_DEVICE_STATE_V1_SAVING; } else { value = 0;}
hw/vfio/migration.c +0
-0@@ -768,8 +768,9 @@ static void vfio_migration_state_notifier(Notifier *notifier, void *data)case
MIGRATION_STATUS_FAILED:
bytes_transferred= 0;
ret = vfio_migration_set_state(vbasedev,
- ~(VFIO_DEVICE_STATE_SAVING |VFIO_DEVICE_STATE_RESUMING),
-VFIO_DEVICE_STATE_RUNNING);
+ ~(VFIO_DEVICE_STATE_V1_SAVING | +VFIO_DEVICE_STATE_RESUMING), +VFIO_DEVICE_STATE_RUNNING); if (ret) {
error_report("%s: Failed to set state RUNNING", vbasedev->name); }hw/vfio/migration.c +4
-2@@ -864,8 +865,10 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) goto add_blocker; } - ret = vfio_get_dev_region_info(vbasedev, VFIO_REGION_TYPE_MIGRATION, - VFIO_REGION_SUBTYPE_MIGRATION, &info); + ret = vfio_get_dev_region_info(vbasedev, +VFIO_REGION_TYPE_MIGRATION_DEPRECATED , +VFIO_REGION_SUBTYPE_MIGRATION_DEPRECATED , + &info);if (ret) {
gotoadd_blocker;
}
Re: [PATCH v5 2/9] vfio: tolerate migration protocol v1 uapi renames
Quoted 18 lines · Matthew Rosato
Quoted 14 lines · Pierre Morel
Quoted 9 lines · Matthew Rosato
The v1 uapi is deprecated and will be replaced by v2 at some point; this patch just tolerates the renaming of uapi fields to reflect v1 / deprecated status.
Signed-off-by Matthew Rosato <[email protected]>---
hw/vfio/common.c | 2 +-
hw/vfio/migration.c| 19 +++++++++++--------2 files changed, 12 insertions(+), 9 deletions(-)Quoted 2 lines
I do not understand why you need this patch in this series. Shouldn't it be separate?
Quoted 8 lines
This patch is included because of the patch 1 kernel header sync, which pulls in uapi headers from kernel version 5.18-rc1 + my unmerged kernel uapi changes.
This patch is unnecessary without a header sync (and in fact would break QEMU compile), and is unrelated to the rest of the series -- but QEMU will not compile without it once you update linux uapi headers to 5.18-rc1 (or greater) due to the v1 uapi for vfio migration being deprecated [1]. This means that ANY series that does a linux header sync starting from here on will need something like this patch to go along with the header sync (or a series that replaces v1 usage with v2?).
If this patch looks good it could be included whenever a header sync is next needed, doesn't necessarily have to be with this series.
arrg, seems I will need it too then.
Thanks,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
PATCH v5 3/9 target/s390x: add zpci-interp to cpu models
In reply to Matthew RosatoThe zpci-interp feature is used to specify whether zPCI interpretation is to be used for this guest.
--- hw/s390x/s390-virtio-ccw.c | 1 + target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 ++ target/s390x/kvm/kvm.c | 1 + 4 files changed, 5 insertions(+)
hw/s390x/s390-virtio-ccw.c +1 -0
@@ -805,6 +805,7 @@ static void ccw_machine_6_2_instance_options(MachineState *machine)static const S390FeatInit qemu_cpu_feat = {S390_FEAT_LIST_QEMU_V6_2 };ccw_machine_7_0_instance_options (machine); +s390_cpudef_featoff_greater (14 ,1 ,S390_FEAT_ZPCI_INTERP );s390_set_qemu_cpu_model (0x3906 ,14 ,2 , qemu_cpu_feat); }
target/s390x/cpu_features_def.h.inc +1 -0
@@ -146,6 +146,7 @@ DEF_FEAT(SIE_CEI, "cei", SCLP_CPU, 43, "SIE: Conditional-external-interception f DEF_FEAT(DAT_ENH_2, "dateh2", MISC, 0, "DAT-enhancement facility 2") DEF_FEAT(CMM, "cmm", MISC, 0, "Collaborative-memory-management facility") DEF_FEAT(AP, "ap", MISC, 0, "AP instructions installed") +DEF_FEAT(ZPCI_INTERP, "zpci-interp", MISC, 0, "zPCI interpretation") /* Features exposed via the PLO instruction. */ DEF_FEAT(PLO_CL, "plo-cl", PLO, 0, "PLO Compare and load (32 bit in general registers)")
target/s390x/gen-features.c +2 -0
@@ -554,6 +554,7 @@ static uint16_t full_GEN14_GA1[] = {S390_FEAT_HPMA2 ,S390_FEAT_SIE_KSS ,S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF , + S390_FEAT_ZPCI_INTERP, };#define full_GEN14_GA2 EmptyFeat
@@ -650,6 +651,7 @@ static uint16_t default_GEN14_GA1[] = {S390_FEAT_GROUP_MSA_EXT_8 ,S390_FEAT_MULTIPLE_EPOCH ,S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF , + S390_FEAT_ZPCI_INTERP, };#define default_GEN14_GA2 EmptyFeat
target/s390x/kvm/kvm.c +1 -0
@@ -2294,6 +2294,7 @@ static int kvm_to_feat[][2] = { { KVM_S390_VM_CPU_FEAT_PFMFI, S390_FEAT_SIE_PFMFI}, { KVM_S390_VM_CPU_FEAT_SIGPIF, S390_FEAT_SIE_SIGPIF}, { KVM_S390_VM_CPU_FEAT_KSS, S390_FEAT_SIE_KSS}, + { KVM_S390_VM_CPU_FEAT_ZPCI_INTERP, S390_FEAT_ZPCI_INTERP }, };static int query_cpu_feat (S390FeatBitmap features)
Signature · 1 line
2.27.0
Re: [PATCH v5 3/9] target/s390x: add zpci-interp to cpu models
Quoted 48 lines · Matthew Rosato
The zpci-interp feature is used to specify whether zPCI interpretation is to be used for this guest.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-virtio-ccw.c | 1 + target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 ++ target/s390x/kvm/kvm.c | 1 + 4 files changed, 5 insertions(+)hw/s390x/s390-virtio-ccw.c +1
-0@@ -805,6 +805,7 @@ static void ccw_machine_6_2_instance_options(MachineState *machine)static const S390FeatInit qemu_cpu_feat = {S390_FEAT_LIST_QEMU_V6_2 };ccw_machine_7_0_instance_options (machine); +s390_cpudef_featoff_greater (14 ,1 ,S390_FEAT_ZPCI_INTERP );s390_set_qemu_cpu_model (0x3906 ,14 ,2 , qemu_cpu_feat); }target/s390x/cpu_features_def.h.inc +1
-0@@ -146,6 +146,7 @@ DEF_FEAT(SIE_CEI, "cei", SCLP_CPU, 43, "SIE: Conditional-external-interception f DEF_FEAT(DAT_ENH_2, "dateh2", MISC, 0, "DAT-enhancement facility 2") DEF_FEAT(CMM, "cmm", MISC, 0, "Collaborative-memory-management facility") DEF_FEAT(AP, "ap", MISC, 0, "AP instructions installed") +DEF_FEAT(ZPCI_INTERP, "zpci-interp", MISC, 0, "zPCI interpretation") /* Features exposed via the PLO instruction. */ DEF_FEAT(PLO_CL, "plo-cl", PLO, 0, "PLO Compare and load (32 bit in general registers)")target/s390x/gen-features.c +1
-0@@ -554,6 +554,7 @@ static uint16_t full_GEN14_GA1[] = {S390_FEAT_HPMA2 ,S390_FEAT_SIE_KSS ,S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF , + S390_FEAT_ZPCI_INTERP, };#define full_GEN14_GA2 EmptyFeat
target/s390x/gen-features.c static uint16_t default_GEN14_GA1[] = { · line 655
Quoted 5 lines
target/s390x/gen-features.c +1
-0@@ -650,6 +651,7 @@ static uint16_t default_GEN14_GA1[] = {S390_FEAT_GROUP_MSA_EXT_8 ,S390_FEAT_MULTIPLE_EPOCH ,S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF , + S390_FEAT_ZPCI_INTERP, };
If you add something to the default model, I think you also need to add some compatibility handling to the machine types. See e.g. commit 84176c7906f as an example.
Thomas
Re: [PATCH v5 3/9] target/s390x: add zpci-interp to cpu models
Quoted 46 lines · Thomas Huth
Quoted 45 lines · Matthew Rosato
The zpci-interp feature is used to specify whether zPCI interpretation is to be used for this guest.
Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-virtio-ccw.c | 1 +
target/s390x/cpu_features_def.h.inc | 1 +
target/s390x/gen-features.c | 2 ++
target/s390x/kvm/kvm.c | 1 +4 files changed, 5 insertions(+)hw/s390x/s390-virtio-ccw.c +1
-0@@ -805,6 +805,7 @@ static void ccw_machine_6_2_instance_options(MachineState *machine) static const S390FeatInit qemu_cpu_feat = { S390_FEAT_LIST_QEMU_V6_2 }; ccw_machine_7_0_instance_options(machine); + s390_cpudef_featoff_greater(14, 1, S390_FEAT_ZPCI_INTERP); s390_set_qemu_cpu_model(0x3906, 14, 2, qemu_cpu_feat); }target/s390x/cpu_features_def.h.inc +0
-0@@ -146,6 +146,7 @@ DEF_FEAT(SIE_CEI, "cei", SCLP_CPU, 43, "SIE:Conditional-external-interception f
DEF_FEAT(DAT_ENH_2, "dateh2", MISC, 0, "DAT-enhancement facility 2")DEF_FEAT(CMM, "cmm", MISC, 0, "Collaborative-memory-management facility")DEF_FEAT(AP, "ap", MISC, 0, "AP instructions installed")+DEF_FEAT(ZPCI_INTERP, "zpci-interp", MISC, 0, "zPCI interpretation")
/* Features exposed via the PLO instruction. */
DEF_FEAT(PLO_CL, "plo-cl", PLO, 0, "PLO Compare and load (32 bit in general registers)")target/s390x/gen-features.c +1
-0@@ -554,6 +554,7 @@ static uint16_t full_GEN14_GA1[] = { S390_FEAT_HPMA2, S390_FEAT_SIE_KSS, S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF, + S390_FEAT_ZPCI_INTERP, }; #define full_GEN14_GA2 EmptyFeat
Quoted 8 lines
target/s390x/gen-features.c static uint16_t default_GEN14_GA1[] = { · line 655
Quoted 6 lines
target/s390x/gen-features.c +1
-0@@ -650,6 +651,7 @@ static uint16_t default_GEN14_GA1[] = { S390_FEAT_GROUP_MSA_EXT_8, S390_FEAT_MULTIPLE_EPOCH, S390_FEAT_GROUP_MULTIPLE_EPOCH_PTFF, + S390_FEAT_ZPCI_INTERP, };If you add something to the default model, I think you also need to add some compatibility handling to the machine types. See e.g. commit
84176c7906fas an example.
Ah, never mind, it's there some lines earlier in the patch ... I guess I did not have not enough coffee today yet...
Thomas
Re: [PATCH v5 3/9] target/s390x: add zpci-interp to cpu models
In reply to Matthew RosatoQuoted 16 lines · Matthew Rosato
The zpci-interp feature is used to specify whether zPCI interpretation is to be used for this guest.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-virtio-ccw.c | 1 + target/s390x/cpu_features_def.h.inc | 1 + target/s390x/gen-features.c | 2 ++ target/s390x/kvm/kvm.c | 1 + 4 files changed, 5 insertions(+)hw/s390x/s390-virtio-ccw.c +0
-0
hw/s390x/s390-virtio-ccw.c static void ccw_machine_6_2_instance_options(MachineState *machine) · line 810
Quoted 6 lines
hw/s390x/s390-virtio-ccw.c +1
-0@@ -805,6 +805,7 @@ static void ccw_machine_6_2_instance_options(MachineState *machine)static const S390FeatInit qemu_cpu_feat = {S390_FEAT_LIST_QEMU_V6_2 };ccw_machine_7_0_instance_options (machine); +s390_cpudef_featoff_greater (14 ,1 ,S390_FEAT_ZPCI_INTERP );s390_set_qemu_cpu_model (0x3906 ,14 ,2 , qemu_cpu_feat); }
This needs to be moved into ccw_machine_7_0_instance_options() now that 7.0 has been released without this feature.
Thomas
PATCH v5 4/9 s390x/pci: add routine to get host function handle from CLP info
In reply to Matthew RosatoIn order to interface with the underlying host zPCI device, we need to know it's function handle. Add a routine to grab this from the vfio CLP capabilities chain.
--- hw/s390x/s390-pci-vfio.c | 83 ++++++++++++++++++++++++++------ include/hw/s390x/s390-pci-vfio.h | 6 +++ 2 files changed, 73 insertions(+), 16 deletions(-)
hw/s390x/s390-pci-vfio.c +67 -16
@@ -124,6 +124,27 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, pbdev->zpci_fn.pft =0 ; } +static bool get_host_fh (S390PCIBusDevice *pbdev,struct vfio_device_info *info, +uint32_t *fh) +{ +struct vfio_info_cap_header *hdr; +struct vfio_device_info_cap_zpci_base *cap; +VFIOPCIDevice *vpci =container_of (pbdev->pdev, VFIOPCIDevice, pdev); + + hdr =vfio_get_device_info_cap (info,VFIO_DEVICE_INFO_CAP_ZPCI_BASE ); + +/* Can only get the host fh with version 2 or greater */ +if (hdr ==NULL || hdr->version <2 ) { +trace_s390_pci_clp_cap (vpci->vbasedev.name, +VFIO_DEVICE_INFO_CAP_ZPCI_BASE ); +return false; + } + cap = (void *) hdr; + + *fh = cap->fh; +return true; +} +static void s390_pci_read_group (S390PCIBusDevice *pbdev,struct vfio_device_info *info) {
@@ -217,25 +238,13 @@ static void s390_pci_read_pfip(S390PCIBusDevice *pbdev,memcpy (pbdev->zpci_fn.pfip, cap->pfip,CLP_PFIP_NR_SEGMENTS ); } -/* -* This function will issue the VFIO_DEVICE_GET_INFO ioctl and look for -* capabilities that contain information about CLP features provided by the -* underlying host. -* On entry, defaults have already been placed into the guest CLP response -* buffers. On exit, defaults will have been overwritten for any CLP features -* found in the capability chain; defaults will remain for any CLP features not -* found in the chain. -*/ -void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) +static struct vfio_device_info *get_device_info (S390PCIBusDevice *pbdev, +uint32_t argsz) { - g_autofree struct vfio_device_info *info = NULL; +struct vfio_device_info *info =g_malloc0 (argsz);VFIOPCIDevice *vfio_pci; -uint32_t argsz;int fd; - argsz =sizeof (*info); - info =g_malloc0 (argsz); - vfio_pci =container_of (pbdev->pdev, VFIOPCIDevice, pdev); fd = vfio_pci->vbasedev.fd;
@@ -250,7 +259,8 @@ retry:if (ioctl (fd,VFIO_DEVICE_GET_INFO , info)) {trace_s390_pci_clp_dev_info (vfio_pci->vbasedev.name); -return ; +free (info); +return NULL ; }if (info->argsz > argsz) {
@@ -259,6 +269,47 @@ retry: goto retry; } +return info; +} + +/* +* Get the host function handle from the vfio CLP capabilities chain. Returns +* true if a fh value was placed into the provided buffer. Returns false +* if a fh could not be obtained (ioctl failed or capabilitiy version does +* not include the fh) +*/ +bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev,uint32_t *fh) +{ + g_autofree struct vfio_device_info *info = NULL; + +assert (fh); + + info =get_device_info (pbdev,sizeof (*info)); +if (!info) { +return false; + } + +return get_host_fh (pbdev, info, fh); +} + +/* +* This function will issue the VFIO_DEVICE_GET_INFO ioctl and look for +* capabilities that contain information about CLP features provided by the +* underlying host. +* On entry, defaults have already been placed into the guest CLP response +* buffers. On exit, defaults will have been overwritten for any CLP features +* found in the capability chain; defaults will remain for any CLP features not +* found in the chain. +*/ +void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) +{ + g_autofree struct vfio_device_info *info = NULL; + + info =get_device_info (pbdev,sizeof (*info)); +if (!info) { +return ; + } +/* * Find the CLP features provided and fill in the guest CLP responses. * Always call s390_pci_read_base first as information from this could
include/hw/s390x/s390-pci-vfio.h +6 -0
@@ -20,6 +20,7 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail);S390PCIDMACount *s390_pci_start_dma_count (S390pciState *s,S390PCIBusDevice *pbdev);void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt); +bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev,uint32_t *fh);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)
@@ -33,6 +34,11 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, }static inline void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt) { } +static inline bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev, +unsigned int *fh) +{ +return false; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
Signature · 1 line
2.27.0
Re: [PATCH v5 4/9] s390x/pci: add routine to get host function handle from CLP info
Quoted 5 lines · Matthew Rosato
In order to interface with the underlying host zPCI device, we need to know it's function handle. Add a routine to grab this from the vfio CLP capabilities chain.
Signed-off-by Matthew Rosato <[email protected]>
Quoted 149 lines
--- hw/s390x/s390-pci-vfio.c | 83 ++++++++++++++++++++++++++------ include/hw/s390x/s390-pci-vfio.h | 6 +++ 2 files changed, 73 insertions(+), 16 deletions(-)hw/s390x/s390-pci-vfio.c +67
-16@@ -124,6 +124,27 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, pbdev->zpci_fn.pft =0 ; } +static bool get_host_fh (S390PCIBusDevice *pbdev,struct vfio_device_info *info, +uint32_t *fh) +{ +struct vfio_info_cap_header *hdr; +struct vfio_device_info_cap_zpci_base *cap; +VFIOPCIDevice *vpci =container_of (pbdev->pdev, VFIOPCIDevice, pdev); + + hdr =vfio_get_device_info_cap (info,VFIO_DEVICE_INFO_CAP_ZPCI_BASE ); + +/* Can only get the host fh with version 2 or greater */ +if (hdr ==NULL || hdr->version <2 ) { +trace_s390_pci_clp_cap (vpci->vbasedev.name, +VFIO_DEVICE_INFO_CAP_ZPCI_BASE ); +return false; + } + cap = (void *) hdr; + + *fh = cap->fh; +return true; +} +static void s390_pci_read_group (S390PCIBusDevice *pbdev,struct vfio_device_info *info) {@@ -217,25 +238,13 @@ static void s390_pci_read_pfip(S390PCIBusDevice *pbdev,memcpy (pbdev->zpci_fn.pfip, cap->pfip,CLP_PFIP_NR_SEGMENTS ); } -/* -* This function will issue the VFIO_DEVICE_GET_INFO ioctl and look for -* capabilities that contain information about CLP features provided by the -* underlying host. -* On entry, defaults have already been placed into the guest CLP response -* buffers. On exit, defaults will have been overwritten for any CLP features -* found in the capability chain; defaults will remain for any CLP features not -* found in the chain. -*/ -void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) +static struct vfio_device_info *get_device_info (S390PCIBusDevice *pbdev, +uint32_t argsz) { - g_autofree struct vfio_device_info *info = NULL; +struct vfio_device_info *info =g_malloc0 (argsz);VFIOPCIDevice *vfio_pci; -uint32_t argsz;int fd; - argsz =sizeof (*info); - info =g_malloc0 (argsz); - vfio_pci =container_of (pbdev->pdev, VFIOPCIDevice, pdev); fd = vfio_pci->vbasedev.fd;@@ -250,7 +259,8 @@ retry:if (ioctl (fd,VFIO_DEVICE_GET_INFO , info)) {trace_s390_pci_clp_dev_info (vfio_pci->vbasedev.name); -return ; +free (info); +return NULL ; }if (info->argsz > argsz) {@@ -259,6 +269,47 @@ retry: goto retry; } +return info; +} + +/* +* Get the host function handle from the vfio CLP capabilities chain. Returns +* true if a fh value was placed into the provided buffer. Returns false +* if a fh could not be obtained (ioctl failed or capabilitiy version does +* not include the fh) +*/ +bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev,uint32_t *fh) +{ + g_autofree struct vfio_device_info *info = NULL; + +assert (fh); + + info =get_device_info (pbdev,sizeof (*info)); +if (!info) { +return false; + } + +return get_host_fh (pbdev, info, fh); +} + +/* +* This function will issue the VFIO_DEVICE_GET_INFO ioctl and look for +* capabilities that contain information about CLP features provided by the +* underlying host. +* On entry, defaults have already been placed into the guest CLP response +* buffers. On exit, defaults will have been overwritten for any CLP features +* found in the capability chain; defaults will remain for any CLP features not +* found in the chain. +*/ +void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) +{ + g_autofree struct vfio_device_info *info = NULL; + + info =get_device_info (pbdev,sizeof (*info)); +if (!info) { +return ; + } +/* * Find the CLP features provided and fill in the guest CLP responses. * Always call s390_pci_read_base first as information from this could include/hw/s390x/s390-pci-vfio.h +6
-0@@ -20,6 +20,7 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail);S390PCIDMACount *s390_pci_start_dma_count (S390pciState *s,S390PCIBusDevice *pbdev);void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt); +bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev,uint32_t *fh);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)@@ -33,6 +34,11 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, }static inline void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt) { } +static inline bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev, +unsigned int *fh) +{ +return false; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 4/9] s390x/pci: add routine to get host function handle from CLP info
In reply to Matthew RosatoQuoted 9 lines · Matthew Rosato
In order to interface with the underlying host zPCI device, we need to know it's function handle. Add a routine to grab this from the vfio CLP capabilities chain.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-vfio.c | 83 ++++++++++++++++++++++++++------ include/hw/s390x/s390-pci-vfio.h | 6 +++ 2 files changed, 73 insertions(+), 16 deletions(-)
[...]
Quoted 13 lines
include/hw/s390x/s390-pci-vfio.h +1
-0@@ -20,6 +20,7 @@ bool s390_pci_update_dma_avail(int fd, unsigned int *avail);S390PCIDMACount *s390_pci_start_dma_count (S390pciState *s,S390PCIBusDevice *pbdev);void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt); +bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev,uint32_t *fh);void s390_pci_get_clp_info (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_update_dma_avail (int fd,unsigned int *avail)
include/hw/s390x/s390-pci-vfio.h static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, · line 38
Quoted 5 lines
include/hw/s390x/s390-pci-vfio.h +2
-0@@ -33,6 +34,11 @@ static inline S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, }static inline void s390_pci_end_dma_count (S390pciState *s,S390PCIDMACount *cnt) { } +static inline bool s390_pci_get_host_fh (S390PCIBusDevice *pbdev, +unsigned int *fh)
This prototype does not match the one before the else - please replace "unsigned int" with "uint32_t".
Thomas
Quoted 6 lines
include/hw/s390x/s390-pci-vfio.h +3
-0+{ +return false; +}static inline void s390_pci_get_clp_info (S390PCIBusDevice *pbdev) { }#endif
PATCH v5 5/9 s390x/pci: enable for load/store intepretation
In reply to Matthew RosatoIf the appropriate CPU facilty is available as well as the necessary ZPCI_OP ioctl, then the underlying KVM host will enable load/store intepretation for any guest device without a SHM bit in the guest function handle. For a device that will be using interpretation support, ensure the guest function handle matches the host function handle; this value is re-checked every time the guest issues a SET PCI FN to enable the guest device as it is the only opportunity to reflect function handle changes.
By default, unless interpret=off is specified, interpretation support will always be assumed and exploited if the necessary ioctl and features are available on the host kernel. When these are unavailable, we will silently revert to the interception model; this allows existing guest configurations to work unmodified on hosts with and without zPCI interpretation support, allowing QEMU to choose the best support model available.
--- hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 66 ++++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 12 ++++++ hw/s390x/s390-pci-kvm.c | 21 +++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 24 ++++++++++++ target/s390x/kvm/kvm.c | 7 ++++ target/s390x/kvm/kvm_s390x.h | 1 + 8 files changed, 132 insertions(+), 1 deletion(-) create mode 100644 hw/s390x/s390-pci-kvm.c create mode 100644 include/hw/s390x/s390-pci-kvm.h
hw/s390x/meson.build +1 -0
@@ -23,6 +23,7 @@ s390x_ss.add(when: 'CONFIG_KVM', if_true: files( 's390-skeys-kvm.c', 's390-stattrib-kvm.c', 'pv.c', + 's390-pci-kvm.c', )) s390x_ss.add(when: 'CONFIG_TCG', if_true: files( 'tod-tcg.c',
hw/s390x/s390-pci-bus.c +65 -1
101 lines not on this page · raw message
hw/s390x/s390-pci-inst.c +12 -0
@@ -18,6 +18,8 @@#include "sysemu/hw_accel.h" #include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST
@@ -246,6 +248,16 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } +/* +* Take this opportunity to make sure we still have an accurate +* host fh. It's possible part of the handle changed while the +* device was disabled to the guest (e.g. vfio hot reset for +* ISM during plug) +*/ +if (pbdev->interp) { +/* Take this opportunity to make sure we are sync'd with host */ +s390_pci_get_host_fh (pbdev, &pbdev->fh); + } pbdev->fh |=FH_MASK_ENABLE ; pbdev->state =ZPCI_FS_ENABLED ;stl_p (&ressetpci->fh, pbdev->fh);
hw/s390x/s390-pci-kvm.c new file +21 -0
@@ -0,0 +1,21 @@ +/* +* s390 zPCI KVM interfaces +* +* Copyright 2022 IBM Corp. +* Author(s): Matthew Rosato <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#include "qemu/osdep.h" + +#include "kvm/kvm_s390x.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "cpu_models.h" + +bool s390_pci_kvm_interp_allowed (void ) +{ +return s390_has_feat (S390_FEAT_ZPCI_INTERP ) &&kvm_s390_get_zpci_op (); +}
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -350,6 +350,7 @@ struct S390PCIBusDevice {IndAddr *indicator;bool pci_unplug_request_processed;bool unplug_requested; +bool interp;QTAILQ_ENTRY (S390PCIBusDevice) link; };
include/hw/s390x/s390-pci-kvm.h new file +24 -0
@@ -0,0 +1,24 @@ +/* +* s390 PCI KVM interfaces +* +* Copyright 2022 IBM Corp. +* Author(s): Matthew Rosato <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ + +#ifndef HW_S390_PCI_KVM_H +#define HW_S390_PCI_KVM_H + +#ifdef CONFIG_KVM +bool s390_pci_kvm_interp_allowed (void ); +#else +static inline bool s390_pci_kvm_interp_allowed (void ) +{ +return false; +} +#endif + +#endif
target/s390x/kvm/kvm.c +7 -0
@@ -157,6 +157,7 @@ static int cap_ri;static int cap_hpage_1m;static int cap_vcpu_resets;static int cap_protected; +static int cap_zpci_op;static int active_cmma;
@@ -358,6 +359,7 @@ int kvm_arch_init(MachineState *ms, KVMState *s) cap_s390_irq =kvm_check_extension (s,KVM_CAP_S390_INJECT_IRQ ); cap_vcpu_resets =kvm_check_extension (s,KVM_CAP_S390_VCPU_RESETS ); cap_protected =kvm_check_extension (s,KVM_CAP_S390_PROTECTED ); + cap_zpci_op =kvm_check_extension (s,KVM_CAP_S390_ZPCI_OP );kvm_vm_enable_cap (s,KVM_CAP_S390_USER_SIGP ,0 );kvm_vm_enable_cap (s,KVM_CAP_S390_VECTOR_REGISTERS ,0 );
@@ -2567,3 +2569,8 @@ bool kvm_arch_cpu_check_are_resettable(void) {return true; } + +int kvm_s390_get_zpci_op (void ) +{ +return cap_zpci_op; +}
target/s390x/kvm/kvm_s390x.h +1 -0
@@ -27,6 +27,7 @@ void kvm_s390_vcpu_interrupt_pre_save(S390CPU *cpu);int kvm_s390_vcpu_interrupt_post_load (S390CPU *cpu);int kvm_s390_get_hpage_1m (void );int kvm_s390_get_ri (void ); +int kvm_s390_get_zpci_op (void );int kvm_s390_get_clock (uint8_t *tod_high,uint64_t *tod_clock);int kvm_s390_get_clock_ext (uint8_t *tod_high,uint64_t *tod_clock);int kvm_s390_set_clock (uint8_t tod_high,uint64_t tod_clock);
Signature · 1 line
2.27.0
Re: [PATCH v5 5/9] s390x/pci: enable for load/store intepretation
Quoted 30 lines · Matthew Rosato
If the appropriate CPU facilty is available as well as the necessary
ZPCI_OPioctl, then the underlying KVM host will enable load/store intepretation for any guest device without a SHM bit in the guest function handle. For a device that will be using interpretation support, ensure the guest function handle matches the host function handle; this value is re-checked every time the guest issues a SET PCI FN to enable the guest device as it is the only opportunity to reflect function handle changes.By default, unless interpret=off is specified, interpretation support will always be assumed and exploited if the necessary ioctl and features are available on the host kernel. When these are unavailable, we will silently revert to the interception model; this allows existing guest configurations to work unmodified on hosts with and without zPCI interpretation support, allowing QEMU to choose the best support model available.
Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 66 ++++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 12 ++++++ hw/s390x/s390-pci-kvm.c | 21 +++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 24 ++++++++++++ target/s390x/kvm/kvm.c | 7 ++++ target/s390x/kvm/kvm_s390x.h | 1 + 8 files changed, 132 insertions(+), 1 deletion(-)create mode 100644
hw/s390x/s390-pci-kvm.ccreate mode 100644include/hw/s390x/s390-pci-kvm.h
...snip...
Quoted 27 lines
if (s390_pci_msix_init(pbdev)) {hw/s390x/s390-pci-bus.c +1
-0@@ -1360,6 +1423,7 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_UINT16 ("uid" , S390PCIBusDevice, uid,UID_UNDEFINED ),DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target), +DEFINE_PROP_BOOL ("interpret" , S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };hw/s390x/s390-pci-inst.c +3
-0@@ -18,6 +18,8 @@#include "sysemu/hw_accel.h" #include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST @@ -246,6 +248,16 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } +/*
hw/s390x/s390-pci-inst.c int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) · line 261
Quoted 10 lines
hw/s390x/s390-pci-inst.c +9
-0+* Take this opportunity to make sure we still have an accurate +* host fh. It's possible part of the handle changed while the +* device was disabled to the guest (e.g. vfio hot reset for +* ISM during plug) +*/ +if (pbdev->interp) { +/* Take this opportunity to make sure we are sync'd with host */ +s390_pci_get_host_fh (pbdev, &pbdev->fh); + } pbdev->fh |=FH_MASK_ENABLE ;
Are we sure here that the PCI device is always enabled? Shouldn't we check?
Quoted 19 lines
hw/s390x/s390-pci-inst.c +0
-0pbdev->state =ZPCI_FS_ENABLED ;stl_p (&ressetpci->fh, pbdev->fh);hw/s390x/s390-pci-kvm.c new file +11
-0@@ -0,0 +1,21 @@ +/* +* s390 zPCI KVM interfaces +* +* Copyright 2022 IBM Corp. +* Author(s): Matthew Rosato <[email protected]> +* +* This work is licensed under the terms of the GNU GPL, version 2 or (at +* your option) any later version. See the COPYING file in the top-level +* directory. +*/ +
hw/s390x/s390-pci-kvm.c line 21
Quoted 10 lines
hw/s390x/s390-pci-kvm.c new file +10
-0+#include "qemu/osdep.h" + +#include "kvm/kvm_s390x.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "cpu_models.h" + +bool s390_pci_kvm_interp_allowed (void ) +{ +return s390_has_feat (S390_FEAT_ZPCI_INTERP ) &&kvm_s390_get_zpci_op (); +}
ZPCI is not supported by the PV currently but I do not see what could prevent it to be enable in the future. As the code of QEMU zPCI is not PV compatible, I would like to add a check for PV.
... && !s390_is_pv())
Quoted 12 lines
include/hw/s390x/s390-pci-bus.h +1
-0@@ -350,6 +350,7 @@ struct S390PCIBusDevice {IndAddr *indicator;bool pci_unplug_request_processed;bool unplug_requested; +bool interp;QTAILQ_ENTRY (S390PCIBusDevice) link; };
...snip...
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 5/9] s390x/pci: enable for load/store intepretation
Quoted 69 lines · Pierre Morel
Quoted 30 lines · Matthew Rosato
If the appropriate CPU facilty is available as well as the necessary
ZPCI_OPioctl, then the underlying KVM host will enable load/store intepretation for any guest device without a SHM bit in the guest function handle. For a device that will be using interpretation support, ensure the guest function handle matches the host function handle; this value is re-checked every time the guest issues a SET PCI FN to enable the guest device as it is the only opportunity to reflect function handle changes.By default, unless interpret=off is specified, interpretation support will always be assumed and exploited if the necessary ioctl and features are available on the host kernel. When these are unavailable, we will silently revert to the interception model; this allows existing guest configurations to work unmodified on hosts with and without zPCI interpretation support, allowing QEMU to choose the best support model available.
Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 66 ++++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 12 ++++++ hw/s390x/s390-pci-kvm.c | 21 +++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 24 ++++++++++++ target/s390x/kvm/kvm.c | 7 ++++ target/s390x/kvm/kvm_s390x.h | 1 +8 files changed, 132 insertions(+), 1 deletion(-)create mode 100644
hw/s390x/s390-pci-kvm.ccreate mode 100644include/hw/s390x/s390-pci-kvm.h...snip...
Quoted 33 lines
if (
s390_pci_msix_init(pbdev)) {hw/s390x/s390-pci-bus.c +1
-0@@ -1360,6 +1423,7 @@ static Property s390_pci_device_properties[] = { DEFINE_PROP_UINT16("uid", S390PCIBusDevice, uid, UID_UNDEFINED), DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid), DEFINE_PROP_STRING("target", S390PCIBusDevice, target), + DEFINE_PROP_BOOL("interpret", S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };hw/s390x/s390-pci-inst.c +4
-0@@ -18,6 +18,8 @@ #include "sysemu/hw_accel.h" #include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST@@ -246,6 +248,16 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } + /* +* Take this opportunity to make sure we still have an accurate
+ * host fh. It's possible part of the handle changed while the
+ * device was disabled to the guest (e.g. vfio hot reset for + * ISM during plug)
+ */
+ if (pbdev->interp) {
Quoted 8 lines
Quoted 5 lines
+ /* Take this opportunity to make sure we are sync'd with host */
+s390_pci_get_host_fh(pbdev, &pbdev->fh); + }
pbdev->fh|=FH_MASK_ENABLE;Are we sure here that the PCI device is always enabled? Shouldn't we check?
I guess you mean the host device? Interesting thought.
So, to be clear, the idea on setting FH_MASK_ENABLE here is that we are handling a guest CLP SET PCI FN enable so the guest fh should always have FH_MASK_ENABLE set if we return CLP_RC_OK to the guest.
But for interpretation, if we find the host function is disabled, I suppose we could return an error on the guest CLP (not sure which error yet); otherwise, if we return the force-enabled handle and CLP_RC_OK as we do here then the guest will just get errors attempting to use it.
Re: [PATCH v5 5/9] s390x/pci: enable for load/store intepretation
Quoted 66 lines · Matthew Rosato
Quoted 65 lines · Pierre Morel
Quoted 30 lines · Matthew Rosato
If the appropriate CPU facilty is available as well as the necessary
ZPCI_OPioctl, then the underlying KVM host will enable load/store intepretation for any guest device without a SHM bit in the guest function handle. For a device that will be using interpretation support, ensure the guest function handle matches the host function handle; this value is re-checked every time the guest issues a SET PCI FN to enable the guest device as it is the only opportunity to reflect function handle changes.By default, unless interpret=off is specified, interpretation support will always be assumed and exploited if the necessary ioctl and features are available on the host kernel. When these are unavailable, we will silently revert to the interception model; this allows existing guest configurations to work unmodified on hosts with and without zPCI interpretation support, allowing QEMU to choose the best support model available.
Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/meson.build | 1 + hw/s390x/s390-pci-bus.c | 66 ++++++++++++++++++++++++++++++++- hw/s390x/s390-pci-inst.c | 12 ++++++ hw/s390x/s390-pci-kvm.c | 21 +++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 24 ++++++++++++ target/s390x/kvm/kvm.c | 7 ++++ target/s390x/kvm/kvm_s390x.h | 1 +8 files changed, 132 insertions(+), 1 deletion(-)create mode 100644
hw/s390x/s390-pci-kvm.ccreate mode 100644include/hw/s390x/s390-pci-kvm.h...snip...
Quoted 29 lines
if (
s390_pci_msix_init(pbdev)) {hw/s390x/s390-pci-bus.c +1
-0@@ -1360,6 +1423,7 @@ static Property s390_pci_device_properties[] = { DEFINE_PROP_UINT16("uid", S390PCIBusDevice, uid, UID_UNDEFINED), DEFINE_PROP_S390_PCI_FID("fid", S390PCIBusDevice, fid), DEFINE_PROP_STRING("target", S390PCIBusDevice, target), + DEFINE_PROP_BOOL("interpret", S390PCIBusDevice, interp, true), DEFINE_PROP_END_OF_LIST(), };hw/s390x/s390-pci-inst.c +4
-0@@ -18,6 +18,8 @@ #include "sysemu/hw_accel.h" #include "hw/s390x/s390-pci-inst.h" #include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-vfio.h" #include "hw/s390x/tod.h" #ifndef DEBUG_S390PCI_INST@@ -246,6 +248,16 @@ int clp_service_call(S390CPU *cpu, uint8_t r2, uintptr_t ra) goto out; } + /* +* Take this opportunity to make sure we still have an accurate
+ * host fh. It's possible part of the handle changed while the
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
+ * device was disabled to the guest (e.g. vfio hot reset for
+ * ISM during plug)
+ */
+ if (pbdev->interp) {
+ /* Take this opportunity to make sure we are sync'd with host */
+s390_pci_get_host_fh(pbdev, &pbdev->fh);
Here we should check the return value and, AFAIU, assume that the device disappear if it did return false.
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
+ }
pbdev->fh|=FH_MASK_ENABLE;Are we sure here that the PCI device is always enabled? Shouldn't we check?
I guess you mean the host device? Interesting thought.
So, to be clear, the idea on setting
FH_MASK_ENABLEhere is that we are handling a guest CLP SET PCI FN enable so the guest fh should always haveFH_MASK_ENABLEset if we returnCLP_RC_OKto the guest.But for interpretation, if we find the host function is disabled, I suppose we could return an error on the guest CLP (not sure which error yet); otherwise, if we return the force-enabled handle and
CLP_RC_OKas we do here then the guest will just get errors attempting to use it.
hum, in this case can't we have a loop on clp enable->error->clp disable->clp enable->error...
I think we should return an error if what the guest asked for could not be done.
Quoted 3 lines
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
PATCH v5 6/9 s390x/pci: don't fence interpreted devices without MSI-X
In reply to Matthew RosatoLack of MSI-X support is not an issue for interpreted passthrough devices, so let's let these in. This will allow, for example, ISM devices to be passed through -- but only when interpretation is available and being used.
--- hw/s390x/s390-pci-bus.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
hw/s390x/s390-pci-bus.c +1 -1
@@ -1093,7 +1093,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->interp = false; } -if (s390_pci_msix_init (pbdev)) { +if (s390_pci_msix_init (pbdev) && !pbdev->interp) {error_setg (errp,"MSI-X support is mandatory " "in the S390 architecture" );return ;
Signature · 1 line
2.27.0
PATCH v5 7/9 s390x/pci: enable adapter event notification for interpreted devices
In reply to Matthew RosatoUse the associated kvm ioctl operation to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting.
--- hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 14 ++++++++++++ 5 files changed, 100 insertions(+), 5 deletions(-)
hw/s390x/s390-pci-bus.c +17 -3
@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc =SCLP_RC_NO_ACTION_REQUIRED ;break ;default : -if (pbdev->summary_ind) { +if (pbdev->interp && (pbdev->fh &FH_MASK_ENABLE )) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable (pbdev); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
@@ -1078,6 +1081,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, }else {DPRINTF ("zPCI interpretation facilities missing.\n" ); pbdev->interp = false; + pbdev->forwarding_assist = false; } } pbdev->iommu->dma_limit =s390_pci_start_dma_count (s, pbdev);
@@ -1086,11 +1090,13 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev,if (!pbdev->interp) {/* Do vfio passthrough but intercept for I/O */ pbdev->fh |=FH_SHM_VFIO ; + pbdev->forwarding_assist = false; } }else { pbdev->fh |=FH_SHM_EMUL ;/* Always intercept emulated devices */ pbdev->interp = false; + pbdev->forwarding_assist = false; }if (s390_pci_msix_init (pbdev) && !pbdev->interp) {
@@ -1240,7 +1246,10 @@ static void s390_pcihost_reset(DeviceState *dev)/* Process all pending unplug requests */ QTAILQ_FOREACH_SAFE(pbdev, &s->zpci_devs, link, next) {if (pbdev->unplug_requested) { -if (pbdev->summary_ind) { +if (pbdev->interp && (pbdev->fh &FH_MASK_ENABLE )) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable (pbdev); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
@@ -1378,7 +1387,10 @@ static void s390_pci_device_reset(DeviceState *dev)break ; } -if (pbdev->summary_ind) { +if (pbdev->interp && (pbdev->fh &FH_MASK_ENABLE )) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable (pbdev); + }else if (pbdev->summary_ind) {pci_dereg_irqs (pbdev); }if (pbdev->iommu->enabled) {
@@ -1424,6 +1436,8 @@ static Property s390_pci_device_properties[] = {DEFINE_PROP_S390_PCI_FID ("fid" , S390PCIBusDevice, fid),DEFINE_PROP_STRING ("target" , S390PCIBusDevice, target),DEFINE_PROP_BOOL ("interpret" , S390PCIBusDevice, interp, true), +DEFINE_PROP_BOOL ("forwarding_assist" , S390PCIBusDevice, forwarding_assist, + true), DEFINE_PROP_END_OF_LIST(), };
hw/s390x/s390-pci-inst.c +38 -2
@@ -1062,6 +1062,32 @@ static void fmb_update(void *opaque)timer_mod (pbdev->fmb_timer, t + pbdev->pci_group->zpci_group.mui); } +static int mpcifc_reg_int_interp (S390PCIBusDevice *pbdev,ZpciFib *fib) +{ +int rc; + + rc =s390_pci_kvm_aif_enable (pbdev, fib, pbdev->forwarding_assist); +if (rc) { +DPRINTF ("Failed to enable interrupt forwarding\n" ); +return rc; + } + +return 0 ; +} + +static int mpcifc_dereg_int_interp (S390PCIBusDevice *pbdev,ZpciFib *fib) +{ +int rc; + + rc =s390_pci_kvm_aif_disable (pbdev); +if (rc) { +DPRINTF ("Failed to disable interrupt forwarding\n" ); +return rc; + } + +return 0 ; +} +int mpcifc_service_call (S390CPU *cpu,uint8_t r1,uint64_t fiba,uint8_t ar,uintptr_t ra) {
@@ -1116,7 +1142,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar,switch (oc) {case ZPCI_MOD_FC_REG_INT : -if (pbdev->summary_ind) { +if (pbdev->interp) { +if (mpcifc_reg_int_interp (pbdev, &fib)) { + cc =ZPCI_PCI_LS_ERR ; +s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); + } + }else if (pbdev->summary_ind) { cc =ZPCI_PCI_LS_ERR ;s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); }else if (reg_irqs (env, pbdev, fib)) {
@@ -1125,7 +1156,12 @@ int mpcifc_service_call(S390CPU *cpu, uint8_t r1, uint64_t fiba, uint8_t ar, }break ;case ZPCI_MOD_FC_DEREG_INT : -if (!pbdev->summary_ind) { +if (pbdev->interp) { +if (mpcifc_dereg_int_interp (pbdev, &fib)) { + cc =ZPCI_PCI_LS_ERR ; +s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); + } + }else if (!pbdev->summary_ind) { cc =ZPCI_PCI_LS_ERR ;s390_set_status_code (env, r1,ZPCI_MOD_ST_SEQUENCE ); }else {
hw/s390x/s390-pci-kvm.c +30 -0
@@ -11,11 +11,41 @@#include "qemu/osdep.h" +#include <linux/kvm.h> +#include "kvm/kvm_s390x.h" +#include "hw/s390x/s390-pci-bus.h" #include "hw/s390x/s390-pci-kvm.h" +#include "hw/s390x/s390-pci-inst.h" #include "cpu_models.h" bool s390_pci_kvm_interp_allowed (void ) {return s390_has_feat (S390_FEAT_ZPCI_INTERP ) &&kvm_s390_get_zpci_op (); } + +int s390_pci_kvm_aif_enable (S390PCIBusDevice *pbdev,ZpciFib *fib,bool assist) +{ +struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op =KVM_S390_ZPCIOP_REG_AEN , + .u.reg_aen.ibv = fib->aibv, + .u.reg_aen.sb = fib->aisb, + .u.reg_aen.noi =FIB_DATA_NOI (fib->data), + .u.reg_aen.isc =FIB_DATA_ISC (fib->data), + .u.reg_aen.sbo =FIB_DATA_AISBO (fib->data), + .u.reg_aen.flags = (assist) ?0 :KVM_S390_ZPCIOP_REGAEN_HOST + }; + +return kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +} + +int s390_pci_kvm_aif_disable (S390PCIBusDevice *pbdev) +{ +struct kvm_s390_zpci_op args = { + .fh = pbdev->fh, + .op =KVM_S390_ZPCIOP_DEREG_AEN + }; + +return kvm_vm_ioctl (kvm_state,KVM_S390_ZPCI_OP , &args); +}
include/hw/s390x/s390-pci-bus.h +1 -0
@@ -351,6 +351,7 @@ struct S390PCIBusDevice {bool pci_unplug_request_processed;bool unplug_requested;bool interp; +bool forwarding_assist;QTAILQ_ENTRY (S390PCIBusDevice) link; };
include/hw/s390x/s390-pci-kvm.h +14 -0
@@ -12,13 +12,27 @@#ifndef HW_S390_PCI_KVM_H #define HW_S390_PCI_KVM_H +#include "hw/s390x/s390-pci-bus.h" +#include "hw/s390x/s390-pci-inst.h" +#ifdef CONFIG_KVM bool s390_pci_kvm_interp_allowed (void ); +int s390_pci_kvm_aif_enable (S390PCIBusDevice *pbdev,ZpciFib *fib,bool assist); +int s390_pci_kvm_aif_disable (S390PCIBusDevice *pbdev);#else static inline bool s390_pci_kvm_interp_allowed (void ) {return false; } +static inline int s390_pci_kvm_aif_enable (S390PCIBusDevice *pbdev,ZpciFib *fib, +bool assist) +{ +return -EINVAL ; +} +static inline int s390_pci_kvm_aif_disable (S390PCIBusDevice *pbdev) +{ +return -EINVAL ; +}#endif #endif
Signature · 1 line
2.27.0
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 18 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the '
forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 14 ++++++++++++ 5 files changed, 100 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-0
hw/s390x/s390-pci-bus.c void s390_pci_sclp_deconfigure(SCCB *sccb) · line 195
Quoted 7 lines
hw/s390x/s390-pci-bus.c +3
-1@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc =SCLP_RC_NO_ACTION_REQUIRED ;break ;default : -if (pbdev->summary_ind) { +if (pbdev->interp && (pbdev->fh &FH_MASK_ENABLE )) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable (pbdev);
Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?
regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 16 lines · Pierre Morel
Quoted 13 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the '
forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 14 ++++++++++++5 files changed, 100 insertions(+), 5 deletions(-)
Quoted 16 lines
Quoted 7 lines
hw/s390x/s390-pci-bus.c +0
-0@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break;Quoted 5 lines
default:
- if (pbdev->summary_ind) {
+ if (pbdev->interp&& (pbdev->fh&FH_MASK_ENABLE)) { + /* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable(pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?
I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
VFIO_DEVICE_SET_IRQS won't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issue VFIO_DEVICE_SET_IRQS directly for an interpreted device, but I think would also need s390-specific changes to VFIO_DEVICE_SET_IRQS accommodate this (e.g. maybe something like a VFIO_IRQ_SET_DATA_S390AEN where we can then specify the aen information in vfio_irq_set.data -- or something else I haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 34 lines · Matthew Rosato
Quoted 16 lines · Pierre Morel
Quoted 13 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the '
forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 14 ++++++++++++5 files changed, 100 insertions(+), 5 deletions(-)Quoted 16 lines
Quoted 7 lines
hw/s390x/s390-pci-bus.c +0
-0@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break;Quoted 5 lines
default:
- if (pbdev->summary_ind) {
+ if (pbdev->interp&& (pbdev->fh&FH_MASK_ENABLE)) { + /* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable(pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?
Quoted 3 lines
I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
VFIO_DEVICE_SET_IRQSwon't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issueVFIO_DEVICE_SET_IRQSdirectly for an interpreted device, but I think would also need s390-specific changes toVFIO_DEVICE_SET_IRQSaccommodate this (e.g. maybe something like aVFIO_IRQ_SET_DATA_S390AENwhere we can then specify the aen information in vfio_irq_set.data -- or something else I
Hi,
yes this in VFIO_DEVICE_SET_IRQS is what I think should be done.
Quoted 1 line
haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
I understood that the demand was concerning the IOMMU but I may be wrong.
For my opinion, the handling of AEN is not specific to KVM but specific to the device, for example the code should be the same if Z ever decide to use XEN or another hypervizor, except for the GISA part but this part is already implemented in KVM in a way it can be used from a device like in VFIO AP.
@Alex, what do you think?
Regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 67 lines · Pierre Morel
Quoted 41 lines · Matthew Rosato
Quoted 26 lines · Pierre Morel
Quoted 24 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification
and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 14 ++++++++++++ 5 files changed, 100 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-1@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc =SCLP_RC_NO_ACTION_REQUIRED ;break ;default : -if (pbdev->summary_ind) {Quoted 9 lines
hw/s390x/s390-pci-bus.c void s390_pci_sclp_deconfigure(SCCB *sccb) · line 195
Quoted 3 lines
hw/s390x/s390-pci-bus.c +3
-0+if (pbdev->interp && (pbdev->fh &FH_MASK_ENABLE )) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable (pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
Quoted 8 lines
VFIO_DEVICE_SET_IRQSwon't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issueVFIO_DEVICE_SET_IRQSdirectly for an interpreted device, but I think would also need s390-specific changes toVFIO_DEVICE_SET_IRQSaccommodate this (e.g. maybe something like aVFIO_IRQ_SET_DATA_S390AENwhere we can then specify the aen information in vfio_irq_set.data -- or something else IHi,
yes this in
VFIO_DEVICE_SET_IRQSis what I think should be done.Quoted 2 lines
haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
I understood that the demand was concerning the IOMMU but I may be wrong. For my opinion, the handling of AEN is not specific to KVM but specific to the device, for example the code should be the same if Z ever decide to use XEN or another hypervizor, except for the GISA part but this part is already implemented in KVM in a way it can be used from a device like in VFIO AP.
Quoted 5 lines
@Alex, what do you think?
Regards,
Pierre
As I understand it the question isn't if it is specific to KVM but rather if it is specific to virtualization. As vfio-pci is also used for non virtualization purposes such as with DPDK/SPDK or a fully emulating QEMU, it should only be in VFIO if it is relevant for these kinds of user-space PCI accesses too. I'm not an AEN expert but as I understand it, this does forwarding interrupts into a SIE context which only makes sense for virtualization not for general user-space PCI.
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 74 lines · Niklas Schnelle
Quoted 67 lines · Pierre Morel
Quoted 41 lines · Matthew Rosato
Quoted 26 lines · Pierre Morel
Quoted 24 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification
and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>--- hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40 +++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + include/hw/s390x/s390-pci-kvm.h | 14 ++++++++++++ 5 files changed, 100 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-1@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc =SCLP_RC_NO_ACTION_REQUIRED ;break ;default : -if (pbdev->summary_ind) {Quoted 9 lines
hw/s390x/s390-pci-bus.c void s390_pci_sclp_deconfigure(SCCB *sccb) · line 195
Quoted 3 lines
hw/s390x/s390-pci-bus.c +3
-0+if (pbdev->interp && (pbdev->fh &FH_MASK_ENABLE )) { +/* Interpreted devices were using interrupt forwarding */ +s390_pci_kvm_aif_disable (pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
Quoted 8 lines
VFIO_DEVICE_SET_IRQSwon't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issueVFIO_DEVICE_SET_IRQSdirectly for an interpreted device, but I think would also need s390-specific changes toVFIO_DEVICE_SET_IRQSaccommodate this (e.g. maybe something like aVFIO_IRQ_SET_DATA_S390AENwhere we can then specify the aen information in vfio_irq_set.data -- or something else IHi,
yes this in
VFIO_DEVICE_SET_IRQSis what I think should be done.Quoted 2 lines
haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
I understood that the demand was concerning the IOMMU but I may be wrong. For my opinion, the handling of AEN is not specific to KVM but specific to the device, for example the code should be the same if Z ever decide to use XEN or another hypervizor, except for the GISA part but this part is already implemented in KVM in a way it can be used from a device like in VFIO AP.
Quoted 5 lines
@Alex, what do you think?
Regards,
Pierre
Quoted 8 lines
As I understand it the question isn't if it is specific to KVM but rather if it is specific to virtualization. As vfio-pci is also used for non virtualization purposes such as with DPDK/SPDK or a fully emulating QEMU, it should only be in VFIO if it is relevant for these kinds of user-space PCI accesses too. I'm not an AEN expert but as I understand it, this does forwarding interrupts into a SIE context which only makes sense for virtualization not for general user-space PCI.
Being in VFIO kernel part does not mean that this part should be called from any user of VFIO in userland. That is a reason why I did propose an extension and not using the current implementation of VFIO_DEVICE_SET_IRQS as is.
The reason behind is that the AEN hardware handling is device specific: we need the Function Handle to program AEN.
If the API is through KVM which is device agnostic the implementation in KVM has to search through the system to find the device being handled to apply AEN on it.
This not the logical way for me and it is a potential source of problems for future extensions.
Regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 60 lines · Pierre Morel
Quoted 57 lines · Niklas Schnelle
Quoted 56 lines · Pierre Morel
Quoted 43 lines · Matthew Rosato
Quoted 28 lines · Pierre Morel
Quoted 26 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification
and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40
+++++++++++++++++++++++++++++++--
hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 14 ++++++++++++5 files changed, 100 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-0@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break;default:
- if (pbdev->summary_ind) {
+ if (pbdev->interp&& (pbdev->fh&FH_MASK_ENABLE)) {Quoted 9 lines
Quoted 3 lines
+ /* Interpreted devices were using interrupt forwarding */
+s390_pci_kvm_aif_disable(pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
Quoted 8 lines
VFIO_DEVICE_SET_IRQSwon't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issueVFIO_DEVICE_SET_IRQSdirectly for an interpreted device, but I think would also need s390-specific changes toVFIO_DEVICE_SET_IRQSaccommodate this (e.g. maybe something like aVFIO_IRQ_SET_DATA_S390AENwhere we can then specify the aen information in vfio_irq_set.data -- or something else IHi,
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
yes this in
VFIO_DEVICE_SET_IRQSis what I think should be done.Quoted 2 lines
haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
I understood that the demand was concerning the IOMMU but I may be wrong.
The IOMMU was an issue, but the request to move the ioctl out of vfio to kvm was specifically because these ioctl operations were only relevant for VMs and are not applicable to vfio uses cases outside of virtualization.
https://lore.kernel.org/kvm/[email protected]/
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
For my opinion, the handling of AEN is not specific to KVM but specific to the device, for example the code should be the same if Z ever decide to use XEN or another hypervizor, except for the GISA part but this part is already implemented in KVM in a way it can be used from a device like in VFIO AP.
Fundamentally, these operations are valid only when you have _both_ a virtual machine and vfio device. (Yes, you could swap in a new hypervisor with a new GISA implementation, but at the end of it the hypervisor must still provide the GISA designation for this to work)
If fh lookup is a concern, one idea that Jason floated was passing the vfio device fd as an argument to the kvm ioctl (so pass this down on a kvm ioctl from userspace instead of a fh) and then using a new vfio external API to get the relevant device from the provided fd.
https://lore.kernel.org/kvm/[email protected]/
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
@Alex, what do you think?
Regards,
Pierre
Quoted 7 lines
Quoted 7 lines
As I understand it the question isn't if it is specific to KVM but rather if it is specific to virtualization. As vfio-pci is also used for non virtualization purposes such as with DPDK/SPDK or a fully emulating QEMU, it should only be in VFIO if it is relevant for these kinds of user-space PCI accesses too. I'm not an AEN expert but as I understand it, this does forwarding interrupts into a SIE context which only makes sense for virtualization not for general user-space PCI.
Right, AEN forwarding is only relevant for virtual machines.
Quoted 6 lines
Quoted 1 line
Being in VFIO kernel part does not mean that this part should be called from any user of VFIO in userland. That is a reason why I did propose an extension and not using the current implementation of
VFIO_DEVICE_SET_IRQSas is.The reason behind is that the AEN hardware handling is device specific: we need the Function Handle to program AEN.
You also need the GISA designation which is provided by the kvm or you also can't program AEN. So you ultimately need both a function handle that is 'owned' by the device (vfio device fd) and the GISA designation that is 'owned' by kvm (kvm fd). So there are 2 different "owning" fds involved.
Quoted 2 lines
If the API is through KVM which is device agnostic the implementation in KVM has to search through the system to find the device being handled to apply AEN on it.
See comment above about instead passing the vfio device fd.
Quoted 3 lines
This not the logical way for me and it is a potential source of problems for future extensions.
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 67 lines · Matthew Rosato
Quoted 66 lines · Pierre Morel
Quoted 63 lines · Niklas Schnelle
Quoted 62 lines · Pierre Morel
Quoted 43 lines · Matthew Rosato
Quoted 28 lines · Pierre Morel
Quoted 26 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification
and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40
+++++++++++++++++++++++++++++++--
hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 14 ++++++++++++5 files changed, 100 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-0@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break;default:
- if (pbdev->summary_ind) {
+ if (pbdev->interp&& (pbdev->fh&FH_MASK_ENABLE)) {Quoted 9 lines
Quoted 3 lines
+ /* Interpreted devices were using interrupt forwarding */
+s390_pci_kvm_aif_disable(pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
Quoted 8 lines
VFIO_DEVICE_SET_IRQSwon't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issueVFIO_DEVICE_SET_IRQSdirectly for an interpreted device, but I think would also need s390-specific changes toVFIO_DEVICE_SET_IRQSaccommodate this (e.g. maybe something like aVFIO_IRQ_SET_DATA_S390AENwhere we can then specify the aen information in vfio_irq_set.data -- or something else IHi,
yes this in
VFIO_DEVICE_SET_IRQSis what I think should be done.Quoted 2 lines
haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I understood that the demand was concerning the IOMMU but I may be wrong.
The IOMMU was an issue, but the request to move the ioctl out of vfio to kvm was specifically because these ioctl operations were only relevant for VMs and are not applicable to vfio uses cases outside of virtualization.
I absolutely agree that KVM specific handling should go through KVM fd. But as I say here under, AEN is not KVM specific but device specific. Instruction interpretation is KVM specific.
see later---v
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
For my opinion, the handling of AEN is not specific to KVM but specific to the device, for example the code should be the same if Z ever decide to use XEN or another hypervizor, except for the GISA part but this part is already implemented in KVM in a way it can be used from a device like in VFIO AP.
Quoted 5 lines
Fundamentally, these operations are valid only when you have _both_ a virtual machine and vfio device. (Yes, you could swap in a new hypervisor with a new GISA implementation, but at the end of it the hypervisor must still provide the GISA designation for this to work)
If fh lookup is a concern, one idea that Jason floated was passing the vfio device fd as an argument to the kvm ioctl (so pass this down on a kvm ioctl from userspace instead of a fh) and then using a new vfio external API to get the relevant device from the provided fd.
^------
This looks like a wrong architecture to me.
If something is used to virtualize the I/O of a device it should go through the device VFIO fd.
If we need a new VFIO external API why not using an extension of the VFIO_DEVICE_SET_IRQS and use directly the VFIO device to setup interrupts?
see following ----v
Quoted 20 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
@Alex, what do you think?
Regards,
PierreQuoted 7 lines
Quoted 7 lines
As I understand it the question isn't if it is specific to KVM but rather if it is specific to virtualization. As vfio-pci is also used for non virtualization purposes such as with DPDK/SPDK or a fully emulating QEMU, it should only be in VFIO if it is relevant for these kinds of user-space PCI accesses too. I'm not an AEN expert but as I understand it, this does forwarding interrupts into a SIE context which only makes sense for virtualization not for general user-space PCI.
Right, AEN forwarding is only relevant for virtual machines.
Quoted 2 lines
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Being in VFIO kernel part does not mean that this part should be called from any user of VFIO in userland. That is a reason why I did propose an extension and not using the current implementation of
VFIO_DEVICE_SET_IRQSas is.The reason behind is that the AEN hardware handling is device specific: we need the Function Handle to program AEN.
You also need the GISA designation which is provided by the kvm or you also can't program AEN. So you ultimately need both a function handle that is 'owned' by the device (vfio device fd) and the GISA designation that is 'owned' by kvm (kvm fd). So there are 2 different "owning" fds involved.
Yes GISA is a host structure, not device specific but guest specific and exist very soon during the guest creation, there should be no problem to retrieve it from a VFIO device IOTCL.
Quoted 12 lines
Quoted 3 lines
If the API is through KVM which is device agnostic the implementation in KVM has to search through the system to find the device being handled to apply AEN on it.
See comment above about instead passing the vfio device fd.
Quoted 3 lines
This not the logical way for me and it is a potential source of problems for future extensions.
^------
There are three different things to modify for the Z-guest to use VFIO:
- IOMMU
- device IRQ
- instruction interpretation, feature negociation
For my opinion only the last one should go directly through the KVM fd.
This should be possible for all architectures.
If it is not possible for Z, the failing path must be adapted it should not go through another path.
Giving the right IRQ information to the host can be done with a dedicated IOCTL through the VFIO device fd, just like we need an extension in the other direction to retrieve the Z specific capabilities.
I am quite sure that other architectures will need some specificity too for the interrupt or IOMMU handling in the future with increasing implementation of virtualization in the firmware.
Having a dedicated IOCTL command means it can be called from QEMU and for guest virtualizuation only then let unused for other userland access.
Regards,
Pierre
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 7/9] s390x/pci: enable adapter event notification for interpreted devices
Quoted 154 lines · Pierre Morel
Quoted 67 lines · Matthew Rosato
Quoted 66 lines · Pierre Morel
Quoted 63 lines · Niklas Schnelle
Quoted 62 lines · Pierre Morel
Quoted 43 lines · Matthew Rosato
Quoted 28 lines · Pierre Morel
Quoted 26 lines · Matthew Rosato
Use the associated kvm ioctl operation to enable adapter event notification
and forwarding for devices when requested. This feature will be set up with or without firmware assist based upon the 'forwarding_assist' setting.Signed-off-by Matthew Rosato <[email protected]>---
hw/s390x/s390-pci-bus.c | 20 ++++++++++++++--- hw/s390x/s390-pci-inst.c | 40
+++++++++++++++++++++++++++++++--
hw/s390x/s390-pci-kvm.c | 30 +++++++++++++++++++++++++include/hw/s390x/s390-pci-bus.h| 1 +include/hw/s390x/s390-pci-kvm.h| 14 ++++++++++++5 files changed, 100 insertions(+), 5 deletions(-)hw/s390x/s390-pci-bus.c +0
-0@@ -190,7 +190,10 @@ void s390_pci_sclp_deconfigure(SCCB *sccb) rc = SCLP_RC_NO_ACTION_REQUIRED; break;default:
- if (pbdev->summary_ind) {
+ if (pbdev->interp&& (pbdev->fh&FH_MASK_ENABLE)) {Quoted 9 lines
Quoted 3 lines
+ /* Interpreted devices were using interrupt forwarding */
+s390_pci_kvm_aif_disable(pbdev);Same remark as for the kernel part.
The VFIO device is already initialized and the action is on this device, Shouldn't we use the VFIO device interface instead of the KVM interface?I don't necessarily disagree, but in v3 of the kernel series I was told not to use VFIO ioctls to accomplish tasks that are unique to KVM (e.g. AEN interpretation) and to instead use a KVM ioctl.
Quoted 8 lines
VFIO_DEVICE_SET_IRQSwon't work as-is for reasons described in the kernel series (e.g. we don't see any of the config space notifiers because of instruction interpretation) -- as far as I can figure we could add our own s390 code to QEMU to issueVFIO_DEVICE_SET_IRQSdirectly for an interpreted device, but I think would also need s390-specific changes toVFIO_DEVICE_SET_IRQSaccommodate this (e.g. maybe something like aVFIO_IRQ_SET_DATA_S390AENwhere we can then specify the aen information in vfio_irq_set.data -- or something else IHi,
yes this in
VFIO_DEVICE_SET_IRQSis what I think should be done.Quoted 2 lines
haven't though of yet) -- I can try to look at this some more and see if I get a good idea.
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
I understood that the demand was concerning the IOMMU but I may be wrong.
The IOMMU was an issue, but the request to move the ioctl out of vfio to kvm was specifically because these ioctl operations were only relevant for VMs and are not applicable to vfio uses cases outside of virtualization.
I absolutely agree that KVM specific handling should go through KVM fd. But as I say here under, AEN is not KVM specific but device specific. Instruction interpretation is KVM specific.
see later---vQuoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
For my opinion, the handling of AEN is not specific to KVM but specific to the device, for example the code should be the same if Z ever decide to use XEN or another hypervizor, except for the GISA part but this part is already implemented in KVM in a way it can be used from a device like in VFIO AP.
Quoted 5 lines
Fundamentally, these operations are valid only when you have _both_ a virtual machine and vfio device. (Yes, you could swap in a new hypervisor with a new GISA implementation, but at the end of it the hypervisor must still provide the GISA designation for this to work)
If fh lookup is a concern, one idea that Jason floated was passing the vfio device fd as an argument to the kvm ioctl (so pass this down on a kvm ioctl from userspace instead of a fh) and then using a new vfio external API to get the relevant device from the provided fd.
^------
This looks like a wrong architecture to me.If something is used to virtualize the I/O of a device it should go through the device VFIO fd.
If we need a new VFIO external API why not using an extension of the
VFIO_DEVICE_SET_IRQSand use directly the VFIO device to setup interrupts?see following ----v
Quoted 20 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
@Alex, what do you think?
Regards,
PierreQuoted 7 lines
Quoted 7 lines
As I understand it the question isn't if it is specific to KVM but rather if it is specific to virtualization. As vfio-pci is also used for non virtualization purposes such as with DPDK/SPDK or a fully emulating QEMU, it should only be in VFIO if it is relevant for these kinds of user-space PCI accesses too. I'm not an AEN expert but as I understand it, this does forwarding interrupts into a SIE context which only makes sense for virtualization not for general user-space PCI.
Right, AEN forwarding is only relevant for virtual machines.
Quoted 2 lines
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
Being in VFIO kernel part does not mean that this part should be called from any user of VFIO in userland. That is a reason why I did propose an extension and not using the current implementation of
VFIO_DEVICE_SET_IRQSas is.The reason behind is that the AEN hardware handling is device specific: we need the Function Handle to program AEN.
You also need the GISA designation which is provided by the kvm or you also can't program AEN. So you ultimately need both a function handle that is 'owned' by the device (vfio device fd) and the GISA designation that is 'owned' by kvm (kvm fd). So there are 2 different "owning" fds involved.
Yes GISA is a host structure, not device specific but guest specific and exist very soon during the guest creation, there should be no problem to retrieve it from a VFIO device IOTCL.
Quoted 12 lines
Quoted 3 lines
If the API is through KVM which is device agnostic the implementation in KVM has to search through the system to find the device being handled to apply AEN on it.
See comment above about instead passing the vfio device fd.
Quoted 3 lines
This not the logical way for me and it is a potential source of problems for future extensions.
^------
There are three different things to modify for the Z-guest to use VFIO:
- IOMMU
- device IRQ
- instruction interpretation, feature negociationFor my opinion only the last one should go directly through the KVM fd.
Quoted 9 lines
This should be possible for all architectures.
If it is not possible for Z, the failing path must be adapted it should not go through another path.Giving the right IRQ information to the host can be done with a dedicated IOCTL through the VFIO device fd, just like we need an extension in the other direction to retrieve the Z specific capabilities.
I am quite sure that other architectures will need some specificity too for the interrupt or IOMMU handling in the future with increasing implementation of virtualization in the firmware.
Having a dedicated IOCTL command means it can be called from QEMU and for guest virtualizuation only then let unused for other userland access.
Another approach (that I admittedly don't have all the details worked out on yet) would be to do something like add a new type of kvm_irq_routing_entry that can be used specifically for AEN. Then we can establish this route with the following info:
struct kvm_irq_routing_s390_aen {
__u64 ind_addr;
__u64 summary_addr;
__u32 fd; /* vfio device fd */
__u32 noi;
__u8 isc;
__u8 sbo;
};The vfio device fd is required as it would then be used to get the associated zdev and thus its fh, which we need when we go to activate AEN (mpcifc). Our existing adapter-based routes (kvm_irq_routing_s390_adapter) lack this association and I can't think of a way around that besides introducing a different route type.
During interrupt.c:kvm_set_routing_entry() we can stash the guest info + fd. Then during irq_bypass_{add,del}_producer for this new route type we can find the zpci->fh via the vfio device fd and then actually (un)pin guest addresses / issue the host mpcfic using the routing info. vfio will trigger irq_bypass_{add,del}_producer when the virq is enabled/disabled.
From QEMU though, it gets a bit weird -- since we enable load/store interpretation, we will never see any of the config space writes from the guest. So, vfio MSI notifiers never get triggered to call e.g. kvm_irqchip_add_msi_route -- we would have to do this ourselves in s390x-pci, specifying the new route type above. And then also possibly issue our own VFIO_DEVICE_SET_IRQS since again, it will never get tripped via a vfio notifier.
Or alternatively, we can intentionally trigger the MSI notifiers from s390x-pci code (looks like spapr does this via spapr_msi_setmsg) for the number of vectors the guest specifies on the mpcific to create the necessary virq(s) and drive the subsequent VFIO_DEVICE_SET_IRQ call. Actually, we might have to do that anyway to satisfy VFIO_DEVICE_SET_IRQS expectations in the kernel. And, using the above structure, we probably only need to create a single virq since it contains all of the routes in one payload + once AEN is established we are always delivering interrupts to the guest via GISA, not over eventfd.
PATCH v5 8/9 s390x/pci: let intercept devices have separate PCI groups
In reply to Matthew RosatoLet's use the reserved pool of simulated PCI groups to allow intercept devices to have separate groups from interpreted devices as some group values may be different. If we run out of simulated PCI groups, subsequent intercept devices just get the default group.
Furthermore, if we encounter any PCI groups from hostdevs that are marked as simulated, let's just assign them to the default group to avoid conflicts between host simulated groups and our own simulated groups.
--- hw/s390x/s390-pci-bus.c | 19 ++++++++++++++-- hw/s390x/s390-pci-vfio.c | 40 ++++++++++++++++++++++++++++++--- include/hw/s390x/s390-pci-bus.h | 6 ++++- 3 files changed, 59 insertions(+), 6 deletions(-)
hw/s390x/s390-pci-bus.c +17 -2
@@ -748,13 +748,14 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn)object_unref (OBJECT (iommu)); } -S390PCIGroup *s390_group_create (int id) +S390PCIGroup *s390_group_create (int id,int host_id) {S390PCIGroup *group;S390pciState *s =s390_get_phb (); group =g_new0 (S390PCIGroup,1 ); group->id = id; + group->host_id = host_id;QTAILQ_INSERT_TAIL (&s->zpci_groups, group, link);return group; }
@@ -772,12 +773,25 @@ S390PCIGroup *s390_group_find(int id)return NULL ; } +S390PCIGroup *s390_group_find_host_sim (int host_id) +{ +S390PCIGroup *group; +S390pciState *s =s390_get_phb (); + + QTAILQ_FOREACH(group, &s->zpci_groups, link) { +if (group->id >=ZPCI_SIM_GRP_START && group->host_id == host_id) { +return group; + } + } +return NULL ; +} +static void s390_pci_init_default_group (void ) {S390PCIGroup *group;ClpRspQueryPciGrp *resgrp; - group =s390_group_create (ZPCI_DEFAULT_FN_GRP ); + group =s390_group_create (ZPCI_DEFAULT_FN_GRP ,ZPCI_DEFAULT_FN_GRP ); resgrp = &group->zpci_group; resgrp->fr =1 ; resgrp->dasm =0 ;
@@ -825,6 +839,7 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp) NULL, g_free); s->zpci_table =g_hash_table_new_full (g_int_hash, g_int_equal,NULL ,NULL ); s->bus_no =0 ; + s->next_sim_grp =ZPCI_SIM_GRP_START ;QTAILQ_INIT (&s->pending_sei);QTAILQ_INIT (&s->zpci_devs);QTAILQ_INIT (&s->zpci_dma_limit);
hw/s390x/s390-pci-vfio.c +37 -3
@@ -150,13 +150,18 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, {struct vfio_info_cap_header *hdr;struct vfio_device_info_cap_zpci_group *cap; +S390pciState *s =s390_get_phb ();ClpRspQueryPciGrp *resgrp;VFIOPCIDevice *vpci =container_of (pbdev->pdev, VFIOPCIDevice, pdev); +uint8_t start_gid = pbdev->zpci_fn.pfgid; hdr =vfio_get_device_info_cap (info,VFIO_DEVICE_INFO_CAP_ZPCI_GROUP ); -/* If capability not provided, just use the default group */ -if (hdr ==NULL ) { +/* +* If capability not provided or the underlying hostdev is simulated, just +* use the default group. +*/ +if (hdr ==NULL || pbdev->zpci_fn.pfgid >=ZPCI_SIM_GRP_START ) {trace_s390_pci_clp_cap (vpci->vbasedev.name,VFIO_DEVICE_INFO_CAP_ZPCI_GROUP ); pbdev->zpci_fn.pfgid =ZPCI_DEFAULT_FN_GRP ;
@@ -165,11 +170,40 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, } cap = (void *) hdr; +/* +* For an intercept device, let's use an existing simulated group if one +* one was already created for other intercept devices in this group. +* If not, create a new simulated group if any are still available. +* If all else fails, just fall back on the default group. +*/ +if (!pbdev->interp) { + pbdev->pci_group =s390_group_find_host_sim (pbdev->zpci_fn.pfgid); +if (pbdev->pci_group) { +/* Use existing simulated group */ + pbdev->zpci_fn.pfgid = pbdev->pci_group->id; +return ; + }else { +if (s->next_sim_grp ==ZPCI_DEFAULT_FN_GRP ) { +/* All out of simulated groups, use default */ +trace_s390_pci_clp_cap (vpci->vbasedev.name, +VFIO_DEVICE_INFO_CAP_ZPCI_GROUP ); + pbdev->zpci_fn.pfgid =ZPCI_DEFAULT_FN_GRP ; + pbdev->pci_group =s390_group_find (ZPCI_DEFAULT_FN_GRP ); +return ; + }else { +/* We can assign a new simulated group */ + pbdev->zpci_fn.pfgid = s->next_sim_grp; + s->next_sim_grp++; +/* Fall through to create the new sim group using CLP info */ + } + } + } +/* See if the PCI group is already defined, create if not */ pbdev->pci_group =s390_group_find (pbdev->zpci_fn.pfgid);if (!pbdev->pci_group) { - pbdev->pci_group =s390_group_create (pbdev->zpci_fn.pfgid); + pbdev->pci_group =s390_group_create (pbdev->zpci_fn.pfgid, start_gid); resgrp = &pbdev->pci_group->zpci_group;if (cap->flags &VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH ) {
include/hw/s390x/s390-pci-bus.h +5 -1
@@ -315,13 +315,16 @@ typedef struct ZpciFmb {QEMU_BUILD_BUG_MSG (offsetof(ZpciFmb , fmt0) !=48 ,"padding in ZpciFmb" );#define ZPCI_DEFAULT_FN_GRP 0xFF +#define ZPCI_SIM_GRP_START 0xF0typedef struct S390PCIGroup {ClpRspQueryPciGrp zpci_group;int id; +int host_id;QTAILQ_ENTRY (S390PCIGroup) link; }S390PCIGroup ; -S390PCIGroup *s390_group_create (int id); +S390PCIGroup *s390_group_create (int id,int host_id);S390PCIGroup *s390_group_find (int id); +S390PCIGroup *s390_group_find_host_sim (int host_id);struct S390PCIBusDevice {DeviceState qdev;
@@ -370,6 +373,7 @@ struct S390pciState { QTAILQ_HEAD(, S390PCIBusDevice) zpci_devs; QTAILQ_HEAD(, S390PCIDMACount) zpci_dma_limit; QTAILQ_HEAD(, S390PCIGroup) zpci_groups; +uint8_t next_sim_grp; };S390pciState *s390_get_phb (void );
Signature · 1 line
2.27.0
PATCH v5 9/9 s390x/pci: reflect proper maxstbl for groups of interpreted devices
In reply to Matthew RosatoThe maximum supported store block length might be different depending on whether the instruction is interpretively executed (firmware-reported maximum) or handled via userspace intercept (host kernel API maximum). Choose the best available value during group creation.
--- hw/s390x/s390-pci-vfio.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
hw/s390x/s390-pci-vfio.c +5 -1
@@ -213,7 +213,11 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, resgrp->msia = cap->msi_addr; resgrp->mui = cap->mui; resgrp->i = cap->noi; - resgrp->maxstbl = cap->maxstbl; +if (pbdev->interp && hdr->version >=2 ) { + resgrp->maxstbl = cap->imaxstbl; + }else { + resgrp->maxstbl = cap->maxstbl; + } resgrp->version = cap->version; resgrp->dtsm =ZPCI_DTSM ; }
Signature · 1 line
2.27.0
Re: [PATCH v5 9/9] s390x/pci: reflect proper maxstbl for groups of interpreted devices
Quoted 6 lines · Matthew Rosato
The maximum supported store block length might be different depending on whether the instruction is interpretively executed (firmware-reported maximum) or handled via userspace intercept (host kernel API maximum). Choose the best available value during group creation.
Signed-off-by Matthew Rosato <[email protected]>
Quoted 22 lines
--- hw/s390x/s390-pci-vfio.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)hw/s390x/s390-pci-vfio.c +5
-1@@ -213,7 +213,11 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, resgrp->msia = cap->msi_addr; resgrp->mui = cap->mui; resgrp->i = cap->noi; - resgrp->maxstbl = cap->maxstbl; +if (pbdev->interp && hdr->version >=2 ) { + resgrp->maxstbl = cap->imaxstbl; + }else { + resgrp->maxstbl = cap->maxstbl; + } resgrp->version = cap->version; resgrp->dtsm =ZPCI_DTSM ; }
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 9/9] s390x/pci: reflect proper maxstbl for groups of interpreted devices
In reply to Matthew RosatoQuoted 6 lines · Matthew Rosato
The maximum supported store block length might be different depending on whether the instruction is interpretively executed (firmware-reported maximum) or handled via userspace intercept (host kernel API maximum). Choose the best available value during group creation.
Signed-off-by Matthew Rosato <[email protected]>
Quoted 22 lines
--- hw/s390x/s390-pci-vfio.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)hw/s390x/s390-pci-vfio.c +5
-1@@ -213,7 +213,11 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, resgrp->msia = cap->msi_addr; resgrp->mui = cap->mui; resgrp->i = cap->noi; - resgrp->maxstbl = cap->maxstbl; +if (pbdev->interp && hdr->version >=2 ) { + resgrp->maxstbl = cap->imaxstbl; + }else { + resgrp->maxstbl = cap->maxstbl; + } resgrp->version = cap->version; resgrp->dtsm =ZPCI_DTSM ; }
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen
Re: [PATCH v5 9/9] s390x/pci: reflect proper maxstbl for groups of interpreted devices
In reply to Matthew RosatoQuoted 6 lines · Matthew Rosato
The maximum supported store block length might be different depending on whether the instruction is interpretively executed (firmware-reported maximum) or handled via userspace intercept (host kernel API maximum). Choose the best available value during group creation.
Signed-off-by Matthew Rosato <[email protected]>
Quoted 22 lines
--- hw/s390x/s390-pci-vfio.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)hw/s390x/s390-pci-vfio.c +5
-1@@ -213,7 +213,11 @@ static void s390_pci_read_group(S390PCIBusDevice *pbdev, resgrp->msia = cap->msi_addr; resgrp->mui = cap->mui; resgrp->i = cap->noi; - resgrp->maxstbl = cap->maxstbl; +if (pbdev->interp && hdr->version >=2 ) { + resgrp->maxstbl = cap->imaxstbl; + }else { + resgrp->maxstbl = cap->maxstbl; + } resgrp->version = cap->version; resgrp->dtsm =ZPCI_DTSM ; }
Signature · 2 lines
Pierre Morel IBM Lab Boeblingen