vfio based pci pass-through for qemu/KVM on s390
Subsystems (7)
cover vfio based pci pass-through for qemu/KVM on s390
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platform
I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390
qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform
Thx for feedback and review comments
Frank
RFC PATCH 1/6 KVM: s390: Enable PCI instructions
Enable PCI instructions for s390 KVM.
--- arch/s390/kvm/kvm-s390.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/s390/kvm/kvm-s390.c +1 -1
@@ -1787,7 +1787,7 @@ static int __init kvm_s390_init(void) }memcpy (vfacilities, S390_lowcore.stfle_fac_list,16 ); vfacilities[0 ] &=0xff82fff3f4fc2000UL ; - vfacilities[1 ] &=0x005c000000000000UL ; + vfacilities[1 ] &=0x07dc000000000000UL ;return 0 ; }
RFC PATCH 2/6 iommu: add iommu for s390 platform
In reply to [email protected]From: Frank Blaschka <[email protected]>
Add a basic iommu for the s390 platform. The code is pretty simple since on s390 each PCI device has its own virtual io address space starting at the same vio address. For this a domain could hold only one pci device. Also there is no relation between pci devices so each device belongs to a separate iommu group.
--- arch/s390/include/asm/pci.h | 3 arch/s390/pci/pci_dma.c | 21 ++++- drivers/iommu/Kconfig | 9 ++ drivers/iommu/Makefile | 1 drivers/iommu/s390-iommu.c | 181 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 213 insertions(+), 2 deletions(-)
arch/s390/include/asm/pci.h +3 -0
@@ -177,6 +177,9 @@ struct zpci_dev *get_zdev_by_fid(u32);/* DMA */ int zpci_dma_init (void );void zpci_dma_exit (void ); +int dma_update_trans (struct zpci_dev *zdev,unsigned long pa, +dma_addr_t dma_addr,size_t size,int flags); +void dma_purge_rto_entries (struct zpci_dev *zdev);/* FMB */ int zpci_fmb_enable_device (struct zpci_dev *);
arch/s390/pci/pci_dma.c +19 -2
@@ -139,8 +139,8 @@ static void dma_update_cpu_trans(structentry_clr_protected (entry); } -static int dma_update_trans (struct zpci_dev *zdev,unsigned long pa, -dma_addr_t dma_addr,size_t size,int flags) +int dma_update_trans (struct zpci_dev *zdev,unsigned long pa, +dma_addr_t dma_addr,size_t size,int flags) {unsigned int nr_pages =PAGE_ALIGN (size) >>PAGE_SHIFT ;u8 *page_addr = (u8 *) (pa &PAGE_MASK );
@@ -180,6 +180,7 @@ no_refresh:spin_unlock_irqrestore (&zdev->dma_table_lock, irq_flags);return rc; } +EXPORT_SYMBOL_GPL (dma_update_trans);static void dma_free_seg_table (unsigned long entry) {
@@ -210,6 +211,22 @@ static void dma_cleanup_tables(struct zp zdev->dma_table =NULL ; } +void dma_purge_rto_entries (struct zpci_dev *zdev) +{ +unsigned long *table; +int rtx; + +if (!zdev || !zdev->dma_table) +return ; + table = zdev->dma_table; +for (rtx =0 ; rtx <ZPCI_TABLE_ENTRIES ; rtx++) +if (reg_entry_isvalid (table[rtx])) { +dma_free_seg_table (table[rtx]); +invalidate_table_entry (&table[rtx]); + } +} +EXPORT_SYMBOL_GPL (dma_purge_rto_entries); +static unsigned long __dma_alloc_iommu (struct zpci_dev *zdev,unsigned long start,int size) {
drivers/iommu/Kconfig +9 -0
@@ -302,4 +302,13 @@ config ARM_SMMU Say Y here if your SoC includes an IOMMU device implementing the ARM SMMU architecture. +config S390_IOMMU + bool "s390 IOMMU Support" + depends on S390 + select IOMMU_API + help + Support for the IBM s/390 IOMMU + + If unsure, say N here. + endif # IOMMU_SUPPORT
drivers/iommu/Makefile +1 -0
@@ -19,3 +19,4 @@ obj-$(CONFIG_EXYNOS_IOMMU) += exynos-iom obj-$(CONFIG_SHMOBILE_IOMMU) += shmobile-iommu.o obj-$(CONFIG_SHMOBILE_IPMMU) += shmobile-ipmmu.o obj-$(CONFIG_FSL_PAMU) += fsl_pamu.o fsl_pamu_domain.o +obj-$(CONFIG_S390_IOMMU) += s390-iommu.o
drivers/iommu/s390-iommu.c new file +181 -0
181 lines not on this page · raw message
RFC PATCH 3/6 vfio: make vfio build on s390
In reply to [email protected]From: Frank Blaschka <[email protected]>
This patch adds some small changes to make vfio build on s390.
--- drivers/vfio/Kconfig | 2 +- drivers/vfio/pci/vfio_pci_rdwr.c | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-)
drivers/vfio/Kconfig +1 -1
@@ -16,7 +16,7 @@ config VFIO_SPAPR_EEH menuconfig VFIO tristate "VFIO Non-Privileged userspace driver framework" depends on IOMMU_API - select VFIO_IOMMU_TYPE1 if X86 + select VFIO_IOMMU_TYPE1 if (X86 || S390) select VFIO_IOMMU_SPAPR_TCE if (PPC_POWERNV || PPC_PSERIES) select VFIO_SPAPR_EEH if (PPC_POWERNV || PPC_PSERIES) select ANON_INODES
drivers/vfio/pci/vfio_pci_rdwr.c +8 -0
@@ -177,6 +177,13 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_return done; } +#ifdef CONFIG_NO_IOPORT_MAP +ssize_t vfio_pci_vga_rw (struct vfio_pci_device *vdev,char __user *buf, +size_t count,loff_t *ppos,bool iswrite) +{ +return -EINVAL ; +} +#else ssize_t vfio_pci_vga_rw (struct vfio_pci_device *vdev,char __user *buf,size_t count,loff_t *ppos,bool iswrite) {
@@ -236,3 +243,4 @@ ssize_t vfio_pci_vga_rw(struct vfio_pci_ return done; } +#endif
RFC PATCH 4/6 s390: Add PCI bus support
In reply to [email protected]From: Frank Blaschka <[email protected]>
This patch implements a pci bus for s390x together with some infrastructure to generate and handle hotplug events. It also provides device configuration/unconfiguration via sclp instruction interception.
--- default-configs/s390x-softmmu.mak | 1 hw/s390x/Makefile.objs | 1 hw/s390x/css.c | 5 hw/s390x/css.h | 1 hw/s390x/s390-pci-bus.c | 404 ++++++++++++++++++++++++++++++++++++++ hw/s390x/s390-pci-bus.h | 166 +++++++++++++++ hw/s390x/s390-virtio-ccw.c | 2 hw/s390x/sclp.c | 10 include/hw/s390x/sclp.h | 8 target-s390x/ioinst.c | 52 ++++ target-s390x/ioinst.h | 1 11 files changed, 650 insertions(+), 1 deletion(-)
default-configs/s390x-softmmu.mak +1 -0
@@ -1,3 +1,4 @@ +include pci.mak CONFIG_VIRTIO=y CONFIG_SCLPCONSOLE=y CONFIG_S390_FLIC=y
hw/s390x/Makefile.objs +1 -0
@@ -8,3 +8,4 @@ obj-y += ipl.o obj-y += css.o obj-y += s390-virtio-ccw.o obj-y += virtio-ccw.o +obj-$(CONFIG_KVM) += s390-pci-bus.o
hw/s390x/css.c +5 -0
@@ -1281,6 +1281,11 @@ void css_generate_chp_crws(uint8_t cssid/* TODO */ } +void css_generate_css_crws (uint8_t cssid) +{ +css_queue_crw (CRW_RSC_CSS ,0 ,0 ,0 ); +} +int css_enable_mcsse (void ) {trace_css_enable_facility ("mcsse" );
hw/s390x/css.h +1 -0
@@ -99,6 +99,7 @@ void css_queue_crw(uint8_t rsc, uint8_tvoid css_generate_sch_crws (uint8_t cssid,uint8_t ssid,uint16_t schid,int hotplugged,int add);void css_generate_chp_crws (uint8_t cssid,uint8_t chpid); +void css_generate_css_crws (uint8_t cssid);void css_adapter_interrupt (uint8_t isc);#define CSS_IO_ADAPTER_VIRTIO 1
hw/s390x/s390-pci-bus.c new file +404 -0
404 lines not on this page · raw message
hw/s390x/s390-pci-bus.h new file +166 -0
166 lines not on this page · raw message
hw/s390x/s390-virtio-ccw.c +2 -0
@@ -18,6 +18,7 @@#include "css.h" #include "virtio-ccw.h" #include "qemu/config-file.h" +#include "s390-pci-bus.h" #define TYPE_S390_CCW_MACHINE "s390-ccw-machine"
@@ -126,6 +127,7 @@ static void ccw_init(MachineState *machis390_init_ipl_dev (machine->kernel_filename, machine->kernel_cmdline, machine->initrd_filename,"s390-ccw.img" );s390_flic_init (); +s390_pci_bus_init ();/* register hypercalls */ virtio_ccw_register_hcalls ();
hw/s390x/sclp.c +9 -1
@@ -20,6 +20,7 @@#include "qemu/config-file.h" #include "hw/s390x/sclp.h" #include "hw/s390x/event-facility.h" +#include "hw/s390x/s390-pci-bus.h" static inline SCLPEventFacility *get_event_facility (void ) {
@@ -62,7 +63,8 @@ static void read_SCP_info(SCCB *sccb) read_info->entries[i].type =0 ; } - read_info->facilities =cpu_to_be64 (SCLP_HAS_CPU_INFO ); + read_info->facilities =cpu_to_be64 (SCLP_HAS_CPU_INFO | +SCLP_HAS_PCI_RECONFIG );/* * The storage increment size is a multiple of 1M and is a power of 2.
@@ -350,6 +352,12 @@ static void sclp_execute(SCCB *sccb, uincase SCLP_UNASSIGN_STORAGE :unassign_storage (sccb);break ; +case SCLP_CMDW_CONFIGURE_PCI : +s390_pci_sclp_configure (1 , sccb); +break ; +case SCLP_CMDW_DECONFIGURE_PCI : +s390_pci_sclp_configure (0 , sccb); +break ;default : efc->command_handler (ef, sccb, code);break ;
include/hw/s390x/sclp.h +8 -0
@@ -45,14 +45,22 @@#define SCLP_CMDW_CONFIGURE_CPU 0x00110001#define SCLP_CMDW_DECONFIGURE_CPU 0x00100001 +/* SCLP PCI codes */ +#define SCLP_HAS_PCI_RECONFIG 0x0000000040000000ULL +#define SCLP_CMDW_CONFIGURE_PCI 0x001a0001 +#define SCLP_CMDW_DECONFIGURE_PCI 0x001b0001 +#define SCLP_RECONFIG_PCI_ATPYE 2 +/* SCLP response codes */ #define SCLP_RC_NORMAL_READ_COMPLETION 0x0010#define SCLP_RC_NORMAL_COMPLETION 0x0020#define SCLP_RC_SCCB_BOUNDARY_VIOLATION 0x0100 +#define SCLP_RC_NO_ACTION_REQUIRED 0x0120#define SCLP_RC_INVALID_SCLP_COMMAND 0x01f0#define SCLP_RC_CONTAINED_EQUIPMENT_CHECK 0x0340#define SCLP_RC_INSUFFICIENT_SCCB_LENGTH 0x0300#define SCLP_RC_STANDBY_READ_COMPLETION 0x0410 +#define SCLP_RC_ADAPTER_ID_NOT_RECOGNIZED 0x09f0#define SCLP_RC_INVALID_FUNCTION 0x40f0#define SCLP_RC_NO_EVENT_BUFFERS_STORED 0x60f0#define SCLP_RC_INVALID_SELECTION_MASK 0x70f0
target-s390x/ioinst.c +52 -0
76 lines not on this page · raw message
target-s390x/ioinst.h +1 -0
@@ -194,6 +194,7 @@ typedef struct CRW {#define CRW_RSC_SUBCH 0x3#define CRW_RSC_CHP 0x4 +#define CRW_RSC_CSS 0xb/* I/O interruption code */ typedef struct IOIntCode {
RFC PATCH 5/6 s390: implement pci instruction
In reply to [email protected]From: Frank Blaschka <[email protected]>
This patch implements the s390 pci instructions in qemu. This allows to attach qemu pci devices including vfio. This does not mean the devices are functional but at least detection and config/memory space access is working.
--- target-s390x/Makefile.objs | 2 target-s390x/kvm.c | 52 +++ target-s390x/pci_ic.c | 621 +++++++++++++++++++++++++++++++++++++++++++++ target-s390x/pci_ic.h | 425 ++++++++++++++++++++++++++++++ 4 files changed, 1099 insertions(+), 1 deletion(-)
target-s390x/Makefile.objs +1 -1
@@ -2,4 +2,4 @@ obj-y += translate.o helper.o cpu.o inte obj-y += int_helper.o fpu_helper.o cc_helper.o mem_helper.o misc_helper.o obj-y += gdbstub.o obj-$(CONFIG_SOFTMMU) += ioinst.o arch_dump.o -obj-$(CONFIG_KVM) += kvm.o +obj-$(CONFIG_KVM) += kvm.o pci_ic.o
target-s390x/kvm.c +52 -0
96 lines not on this page · raw message
target-s390x/pci_ic.c new file +621 -0
621 lines not on this page · raw message
target-s390x/pci_ic.h new file +425 -0
425 lines not on this page · raw message
Re: [RFC patch 5/6] s390: implement pci instruction
Hi Frank,
Quoted 81 lines · On Fri, 19 Sep 2014 13:54:34 +0200 [email protected] wrote:
From: Frank Blaschka <[email protected]>
This patch implements the s390 pci instructions in qemu. This allows to attach qemu pci devices including vfio. This does not mean the devices are functional but at least detection and config/memory space access is working.
Signed-off-by Frank Blaschka <[email protected]>--- target-s390x/Makefile.objs | 2 target-s390x/kvm.c | 52 +++ target-s390x/pci_ic.c | 621 +++++++++++++++++++++++++++++++++++++++++++++ target-s390x/pci_ic.h | 425 ++++++++++++++++++++++++++++++ 4 files changed, 1099 insertions(+), 1 deletion(-)target-s390x/Makefile.objs +1
-1@@ -2,4 +2,4 @@ obj-y += translate.o helper.o cpu.o inte obj-y += int_helper.o fpu_helper.o cc_helper.o mem_helper.o misc_helper.o obj-y += gdbstub.o obj-$(CONFIG_SOFTMMU) += ioinst.o arch_dump.o -obj-$(CONFIG_KVM) += kvm.o +obj-$(CONFIG_KVM) += kvm.o pci_ic.otarget-s390x/kvm.c +23
-0@@ -40,6 +40,7 @@#include "exec/gdbstub.h" #include "trace.h" #include "qapi-event.h" +#include "pci_ic.h" /* #define DEBUG_KVM */ @@ -56,6 +57,7 @@#define IPA0_B2 0xb200#define IPA0_B9 0xb900#define IPA0_EB 0xeb00 +#define IPA0_E3 0xe300#define PRIV_B2_SCLP_CALL 0x20#define PRIV_B2_CSCH 0x30@@ -76,8 +78,17 @@#define PRIV_B2_XSCH 0x76#define PRIV_EB_SQBS 0x8a +#define PRIV_EB_PCISTB 0xd0 +#define PRIV_EB_SIC 0xd1#define PRIV_B9_EQBS 0x9c +#define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PCISTG 0xd0 +#define PRIV_B9_PCILG 0xd2 +#define PRIV_B9_RPCIT 0xd3 + +#define PRIV_E3_MPCIFC 0xd0 +#define PRIV_E3_STPCIFC 0xd4#define DIAG_IPL 0x308#define DIAG_KVM_HYPERCALL 0x500@@ -813,6 +824,18 @@ static int handle_b9(S390CPU *cpu, strucint r =0 ;switch (ipa1) { +case PRIV_B9_CLP : + r =kvm_clp_service_call (cpu, run); +break ; +case PRIV_B9_PCISTG : + r =kvm_pcistg_service_call (cpu, run); +break ; +case PRIV_B9_PCILG : + r =kvm_pcilg_service_call (cpu, run); +break ; +case PRIV_B9_RPCIT : + r =kvm_rpcit_service_call (cpu, run); +break ;case PRIV_B9_EQBS :/* just inject exception */ r =-1 ;@@ -831,6 +854,12 @@ static int handle_eb(S390CPU *cpu, strucint r =0 ;
target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 865
Quoted 10 lines
target-s390x/kvm.c +6
-0switch (ipa1) { +case PRIV_EB_PCISTB : + r =kvm_pcistb_service_call (cpu, run); +break ; +case PRIV_EB_SIC : + r =kvm_sic_service_call (cpu, run); +break ;case PRIV_EB_SQBS :/* just inject exception */ r =-1 ;
I'm not sure, but I think the handler for the eb instructions is wrong:
The second byte of the opcode is encoded in the lowest byte of the ipb field, not the lowest byte of the ipa field (just like with the e3 handler). Did you verify that your handlers get called correctly?
Quoted 13 lines
target-s390x/kvm.c +9
-0@@ -844,6 +873,26 @@ static int handle_eb(S390CPU *cpu, strucreturn r; } +static int handle_e3 (S390CPU *cpu,struct kvm_run *run,uint8_t ipa1) +{ +int r =0 ; + +switch (ipa1) { +case PRIV_E3_MPCIFC : + r =kvm_mpcifc_service_call (cpu, run); +break ; +case PRIV_E3_STPCIFC :
target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 894
Quoted 10 lines
target-s390x/kvm.c +10
-0+ r =kvm_stpcifc_service_call (cpu, run); +break ; +default : + r =-1 ; +DPRINTF ("KVM: unhandled PRIV: 0xe3%x\n" , ipa1); +break ; + } + +return r; +}
Could you please replace "ipa1" with "ipb1" to avoid confusion here?
Quoted 16 lines
target-s390x/kvm.c +3
-0static int handle_hypercall (S390CPU *cpu,struct kvm_run *run) {CPUS390XState *env = &cpu->env;@@ -1038,6 +1087,9 @@ static int handle_instruction(S390CPU *ccase IPA0_EB : r =handle_eb (cpu, run, ipa1);break ; +case IPA0_E3 : + r =handle_e3 (cpu, run, run->s390_sieic.ipb &0xff ); +break ;case IPA0_DIAG : r =handle_diag (cpu, run, run->s390_sieic.ipb);break ;target-s390x/pci_ic.c new file +0
-0@@ -0,0 +1,621 @@
[...]
target-s390x/pci_ic.c line 308
Quoted 13 lines
target-s390x/pci_ic.c new file +13
-0+ +int kvm_pcilg_service_call (S390CPU *cpu,struct kvm_run *run) +{ +CPUS390XState *env = &cpu->env; +S390PCIBusDevice *pbdev; +uint8_t r1 = (run->s390_sieic.ipb &0x00f00000 ) >>20 ; +uint8_t r2 = (run->s390_sieic.ipb &0x000f0000 ) >>16 ; +PciLgStg *rp; +uint64_t offset; +uint64_t data; + +cpu_synchronize_state (CPU (cpu)); + rp = (PciLgStg *)&env->regs[r2];
I think you have to check for r2 to be even here (and inject a specification exception otherwise)
You should also check rp->len for a valid value and inject an operand exception otherwise.
Quoted 1 line
+ offset = env->regs[r2 + 1];
You should also check for a valid offset value here (and inject an operand exception otherwise).
... and while you're at it - it also seems like the pci instructions are priviledged, so you should check for not being in the problem state here, too, just to be sure ;-)
target-s390x/pci_ic.c line 319
Quoted 9 lines
target-s390x/pci_ic.c new file +9
-0+ pbdev =s390_pci_find_dev_by_fh (rp->fh); +if (!pbdev) { +DPRINTF ("pcilg no pci dev\n" ); +return -EIO ; + } + +if (rp->pcias <6 ) { +MemoryRegion *mr = pbdev->pdev->io_regions[rp->pcias].memory; +io_mem_read (mr, offset, &data, rp->len);
Does this also work if len != 8 ? I think the data should be put in the rightmost byte positions, with the unused leftmost bytes set to zero... so some more logic might be needed here.
Quoted 11 lines
target-s390x/pci_ic.c new file +11
-0+ }else if (rp->pcias ==15 ) { + data =pci_host_config_read_common ( + pbdev->pdev, offset,pci_config_size (pbdev->pdev), rp->len); + +switch (rp->len) { +case 1 : +break ; +case 2 : + data =cpu_to_le16 (data); +break ; +case 4 :
target-s390x/pci_ic.c line 340
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+ data =cpu_to_le32 (data); +break ; +case 8 : + data =cpu_to_le64 (data); +break ; +default : +abort (); + } + }else { +DPRINTF ("invalid space\n" );
Set condition code 1 in this case?
Quoted 9 lines
target-s390x/pci_ic.c new file +9
-0+ } + + env->regs[r1] = data; +setcc (cpu,0 ); +return 0 ; +} + +int kvm_pcistg_service_call (S390CPU *cpu,struct kvm_run *run) +{
target-s390x/pci_ic.c line 359
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+CPUS390XState *env = &cpu->env; +uint8_t r1 = (run->s390_sieic.ipb &0x00f00000 ) >>20 ; +uint8_t r2 = (run->s390_sieic.ipb &0x000f0000 ) >>16 ; +PciLgStg *rp; +uint64_t offset, data; +S390PCIBusDevice *pbdev; + +cpu_synchronize_state (CPU (cpu)); + rp = (PciLgStg *)&env->regs[r2]; + offset = env->regs[r2 +1 ];
You likely need the same sanity checks here as with the pcilg function.
Quoted 11 lines
target-s390x/pci_ic.c new file +11
-0+ pbdev =s390_pci_find_dev_by_fh (rp->fh); +if (!pbdev) { +DPRINTF ("pcistg no pci dev\n" ); +return -EIO ; + } + + data = env->regs[r1]; + +if (rp->pcias <6 ) { +MemoryRegion *mr; + if (pbdev->msix_table_bar == rp->pcias &&
target-s390x/pci_ic.c line 381
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+ offset >= pbdev->msix_table_offset && + offset <= pbdev->msix_table_offset + + (pbdev->msix_entries - 1) * PCI_MSIX_ENTRY_SIZE) { + offset = offset - pbdev->msix_table_offset; + mr = &pbdev->pdev->msix_table_mmio; + } else { + mr = pbdev->pdev->io_regions[rp->pcias].memory; + } + +io_mem_write (mr, offset, data, rp->len);
Same concerns as with the pcilg - this likely only works for len == 8 ?
Quoted 12 lines
target-s390x/pci_ic.c new file +12
-0+ }else if (rp->pcias ==15 ) { +switch (rp->len) { +case 1 : +break ; +case 2 : + data =le16_to_cpu (data); +break ; +case 4 : + data =le32_to_cpu (data); +break ; +case 8 : + data =le64_to_cpu (data);
target-s390x/pci_ic.c line 403
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+break ; +default : +abort (); + } + +pci_host_config_write_common (pbdev->pdev, offset, +pci_config_size (pbdev->pdev), + data, rp->len); + }else { +DPRINTF ("pcistg invalid space\n" );
Set condition code 1 in this case?
Quoted 37 lines
target-s390x/pci_ic.c new file +37
-0+ } + +setcc (cpu,0 ); +return 0 ; +} + +static uint64_t guest_io_table_walk (uint64_t guest_iota, +uint64_t guest_dma_address) +{ +uint64_t sto_a, pto_a, px_a; +uint64_t sto, pto, pte; +uint32_t rtx, sx, px; + + rtx =calc_rtx (guest_dma_address); + sx =calc_sx (guest_dma_address); + px =calc_px (guest_dma_address); + + sto_a = guest_iota + rtx *sizeof (uint64_t ); +cpu_physical_memory_rw (sto_a, (uint8_t *)&sto,sizeof (uint64_t ),0 ); + sto = (uint64_t )get_rt_sto (sto); + + pto_a = sto + sx *sizeof (uint64_t ); +cpu_physical_memory_rw (pto_a, (uint8_t *)&pto,sizeof (uint64_t ),0 ); + pto = (uint64_t )get_st_pto (pto); + + px_a = pto + px *sizeof (uint64_t ); +cpu_physical_memory_rw (px_a, (uint8_t *)&pte,sizeof (uint64_t ),0 ); + +return pte; +} + +int kvm_rpcit_service_call (S390CPU *cpu,struct kvm_run *run) +{ +CPUS390XState *env = &cpu->env; +uint8_t r1 = (run->s390_sieic.ipb &0x00f00000 ) >>20 ; +uint8_t r2 = (run->s390_sieic.ipb &0x000f0000 ) >>16 ; +uint32_t fh;
target-s390x/pci_ic.c line 450
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+uint64_t pte; +S390PCIBusDevice *pbdev; +ram_addr_t size; +int flags; +IOMMUTLBEntry entry; + +cpu_synchronize_state (CPU (cpu)); + + fh = env->regs[r1] >>32 ; + size = env->regs[r2 +1 ];
Missing two checks again:
- r2 should be even
- CPU should not be in problem state
target-s390x/pci_ic.c line 456
Quoted 5 lines
target-s390x/pci_ic.c new file +5
-0+ pbdev =s390_pci_find_dev_by_fh (fh); + +if (!pbdev) { +DPRINTF ("rpcit no pci dev\n" ); +return -EIO ;
I think it would be better to set condition code 3 instead.
Quoted 36 lines
target-s390x/pci_ic.c new file +36
-0+ } + + pte =guest_io_table_walk (pbdev->g_iota, env->regs[r2]); + flags = pte &ZPCI_PTE_FLAG_MASK ; + entry.target_as = &address_space_memory; + entry.iova = env->regs[r2]; + entry.translated_addr = pte &ZPCI_PTE_ADDR_MASK ; + entry.addr_mask = size -1 ; + +if (flags &ZPCI_PTE_INVALID ) { + entry.perm =IOMMU_NONE ; + }else { + entry.perm =IOMMU_RW ; + } + +memory_region_notify_iommu (pci_device_iommu_address_space ( + pbdev->pdev)->root, entry); + +setcc (cpu,0 ); +return 0 ; +} + +int kvm_sic_service_call (S390CPU *cpu,struct kvm_run *run) +{ +DPRINTF ("sic\n" ); +return 0 ; +} + +int kvm_pcistb_service_call (S390CPU *cpu,struct kvm_run *run) +{ +CPUS390XState *env = &cpu->env; +uint8_t r1 = (run->s390_sieic.ipa &0x00f0 ) >>4 ; +uint8_t r3 = run->s390_sieic.ipa &0x000f ; +PciStb *rp; +uint64_t gaddr; +uint64_t *uaddr, *pu;
target-s390x/pci_ic.c line 502
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+hwaddr len; +S390PCIBusDevice *pbdev; +MemoryRegion *mr; +int i; + +cpu_synchronize_state (CPU (cpu)); + + rp = (PciStb *)&env->regs[r1]; + gaddr =get_base_disp_rsy (cpu, run); + len = rp->len;
Not sure, but don't you also have to check the length and offset here for valid ranges?
At least you should check for problem state again.
target-s390x/pci_ic.c line 507
Quoted 4 lines
target-s390x/pci_ic.c new file +4
-0+ pbdev =s390_pci_find_dev_by_fh (rp->fh); +if (!pbdev) { +DPRINTF ("pcistb no pci dev fh 0x%x\n" , rp->fh); +return -EIO ;
User cc3 instead?
Quoted 5 lines
target-s390x/pci_ic.c new file +5
-0+ } + + uaddr =cpu_physical_memory_map (gaddr, &len,0 ); + mr = pbdev->pdev->io_regions[rp->pcias].memory; +
target-s390x/pci_ic.c line 522
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+ pu = uaddr; +for (i =0 ; i < rp->len /8 ; i++) { +io_mem_write (mr, env->regs[r3] + i *8 , *pu,8 ); + pu++; + } + +cpu_physical_memory_unmap (uaddr, len,0 , len); +setcc (cpu,0 ); +return 0 ; +}
[...]
Thomas
Re: [Qemu-devel] [RFC patch 5/6] s390: implement pci instruction
Quoted 87 lines · Thomas Huth
Hi Frank,
Quoted 81 lines · On Fri, 19 Sep 2014 13:54:34 +0200 [email protected] wrote:
From: Frank Blaschka <[email protected]>
This patch implements the s390 pci instructions in qemu. This allows to attach qemu pci devices including vfio. This does not mean the devices are functional but at least detection and config/memory space access is working.
Signed-off-by Frank Blaschka <[email protected]>--- target-s390x/Makefile.objs | 2 target-s390x/kvm.c | 52 +++ target-s390x/pci_ic.c | 621 +++++++++++++++++++++++++++++++++++++++++++++ target-s390x/pci_ic.h | 425 ++++++++++++++++++++++++++++++ 4 files changed, 1099 insertions(+), 1 deletion(-)target-s390x/Makefile.objs +1
-1@@ -2,4 +2,4 @@ obj-y += translate.o helper.o cpu.o inte obj-y += int_helper.o fpu_helper.o cc_helper.o mem_helper.o misc_helper.o obj-y += gdbstub.o obj-$(CONFIG_SOFTMMU) += ioinst.o arch_dump.o -obj-$(CONFIG_KVM) += kvm.o +obj-$(CONFIG_KVM) += kvm.o pci_ic.otarget-s390x/kvm.c +23
-0@@ -40,6 +40,7 @@#include "exec/gdbstub.h" #include "trace.h" #include "qapi-event.h" +#include "pci_ic.h" /* #define DEBUG_KVM */ @@ -56,6 +57,7 @@#define IPA0_B2 0xb200#define IPA0_B9 0xb900#define IPA0_EB 0xeb00 +#define IPA0_E3 0xe300#define PRIV_B2_SCLP_CALL 0x20#define PRIV_B2_CSCH 0x30@@ -76,8 +78,17 @@#define PRIV_B2_XSCH 0x76#define PRIV_EB_SQBS 0x8a +#define PRIV_EB_PCISTB 0xd0 +#define PRIV_EB_SIC 0xd1#define PRIV_B9_EQBS 0x9c +#define PRIV_B9_CLP 0xa0 +#define PRIV_B9_PCISTG 0xd0 +#define PRIV_B9_PCILG 0xd2 +#define PRIV_B9_RPCIT 0xd3 + +#define PRIV_E3_MPCIFC 0xd0 +#define PRIV_E3_STPCIFC 0xd4#define DIAG_IPL 0x308#define DIAG_KVM_HYPERCALL 0x500@@ -813,6 +824,18 @@ static int handle_b9(S390CPU *cpu, strucint r =0 ;switch (ipa1) { +case PRIV_B9_CLP : + r =kvm_clp_service_call (cpu, run); +break ; +case PRIV_B9_PCISTG : + r =kvm_pcistg_service_call (cpu, run); +break ; +case PRIV_B9_PCILG : + r =kvm_pcilg_service_call (cpu, run); +break ; +case PRIV_B9_RPCIT : + r =kvm_rpcit_service_call (cpu, run); +break ;case PRIV_B9_EQBS :/* just inject exception */ r =-1 ;@@ -831,6 +854,12 @@ static int handle_eb(S390CPU *cpu, strucint r =0 ;
Quoted 16 lines
target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 865
Quoted 10 lines
target-s390x/kvm.c +6
-0switch (ipa1) { +case PRIV_EB_PCISTB : + r =kvm_pcistb_service_call (cpu, run); +break ; +case PRIV_EB_SIC : + r =kvm_sic_service_call (cpu, run); +break ;case PRIV_EB_SQBS :/* just inject exception */ r =-1 ;I'm not sure, but I think the handler for the eb instructions is wrong:
The second byte of the opcode is encoded in the lowest byte of the ipb field, not the lowest byte of the ipa field (just like with the e3 handler). Did you verify that your handlers get called correctly?
Hi Thomas, you are absolutely right. I already have a patch available for this issue but did not append it to this RFC post (since it is basically a bug fix). To the next posting I will add this patch as well.
Will also fix the remaining issues thx for your review.
Quoted 323 lines
Quoted 13 lines
target-s390x/kvm.c +9
-0@@ -844,6 +873,26 @@ static int handle_eb(S390CPU *cpu, strucreturn r; } +static int handle_e3 (S390CPU *cpu,struct kvm_run *run,uint8_t ipa1) +{ +int r =0 ; + +switch (ipa1) { +case PRIV_E3_MPCIFC : + r =kvm_mpcifc_service_call (cpu, run); +break ; +case PRIV_E3_STPCIFC :target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 894
Quoted 10 lines
target-s390x/kvm.c +10
-0+ r =kvm_stpcifc_service_call (cpu, run); +break ; +default : + r =-1 ; +DPRINTF ("KVM: unhandled PRIV: 0xe3%x\n" , ipa1); +break ; + } + +return r; +}Could you please replace "ipa1" with "ipb1" to avoid confusion here?
Quoted 16 lines
target-s390x/kvm.c +3
-0static int handle_hypercall (S390CPU *cpu,struct kvm_run *run) {CPUS390XState *env = &cpu->env;@@ -1038,6 +1087,9 @@ static int handle_instruction(S390CPU *ccase IPA0_EB : r =handle_eb (cpu, run, ipa1);break ; +case IPA0_E3 : + r =handle_e3 (cpu, run, run->s390_sieic.ipb &0xff ); +break ;case IPA0_DIAG : r =handle_diag (cpu, run, run->s390_sieic.ipb);break ;target-s390x/pci_ic.c new file +0
-0@@ -0,0 +1,621 @@[...]
target-s390x/pci_ic.c line 308
Quoted 13 lines
target-s390x/pci_ic.c new file +13
-0+ +int kvm_pcilg_service_call (S390CPU *cpu,struct kvm_run *run) +{ +CPUS390XState *env = &cpu->env; +S390PCIBusDevice *pbdev; +uint8_t r1 = (run->s390_sieic.ipb &0x00f00000 ) >>20 ; +uint8_t r2 = (run->s390_sieic.ipb &0x000f0000 ) >>16 ; +PciLgStg *rp; +uint64_t offset; +uint64_t data; + +cpu_synchronize_state (CPU (cpu)); + rp = (PciLgStg *)&env->regs[r2];I think you have to check for r2 to be even here (and inject a specification exception otherwise)
You should also check
rp->lenfor a valid value and inject an operand exception otherwise.Quoted 1 line
+ offset = env->regs[r2 + 1];
You should also check for a valid offset value here (and inject an operand exception otherwise).
... and while you're at it - it also seems like the pci instructions are priviledged, so you should check for not being in the problem state here, too, just to be sure ;-)
target-s390x/pci_ic.c line 319
Quoted 9 lines
target-s390x/pci_ic.c new file +9
-0+ pbdev =s390_pci_find_dev_by_fh (rp->fh); +if (!pbdev) { +DPRINTF ("pcilg no pci dev\n" ); +return -EIO ; + } + +if (rp->pcias <6 ) { +MemoryRegion *mr = pbdev->pdev->io_regions[rp->pcias].memory; +io_mem_read (mr, offset, &data, rp->len);Does this also work if len != 8 ? I think the data should be put in the rightmost byte positions, with the unused leftmost bytes set to zero... so some more logic might be needed here.
Quoted 11 lines
target-s390x/pci_ic.c new file +11
-0+ }else if (rp->pcias ==15 ) { + data =pci_host_config_read_common ( + pbdev->pdev, offset,pci_config_size (pbdev->pdev), rp->len); + +switch (rp->len) { +case 1 : +break ; +case 2 : + data =cpu_to_le16 (data); +break ; +case 4 :target-s390x/pci_ic.c line 340
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+ data =cpu_to_le32 (data); +break ; +case 8 : + data =cpu_to_le64 (data); +break ; +default : +abort (); + } + }else { +DPRINTF ("invalid space\n" );Set condition code 1 in this case?
Quoted 9 lines
target-s390x/pci_ic.c new file +9
-0+ } + + env->regs[r1] = data; +setcc (cpu,0 ); +return 0 ; +} + +int kvm_pcistg_service_call (S390CPU *cpu,struct kvm_run *run) +{target-s390x/pci_ic.c line 359
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+CPUS390XState *env = &cpu->env; +uint8_t r1 = (run->s390_sieic.ipb &0x00f00000 ) >>20 ; +uint8_t r2 = (run->s390_sieic.ipb &0x000f0000 ) >>16 ; +PciLgStg *rp; +uint64_t offset, data; +S390PCIBusDevice *pbdev; + +cpu_synchronize_state (CPU (cpu)); + rp = (PciLgStg *)&env->regs[r2]; + offset = env->regs[r2 +1 ];You likely need the same sanity checks here as with the pcilg function.
Quoted 11 lines
target-s390x/pci_ic.c new file +11
-0+ pbdev =s390_pci_find_dev_by_fh (rp->fh); +if (!pbdev) { +DPRINTF ("pcistg no pci dev\n" ); +return -EIO ; + } + + data = env->regs[r1]; + +if (rp->pcias <6 ) { +MemoryRegion *mr; + if (pbdev->msix_table_bar == rp->pcias &&target-s390x/pci_ic.c line 381
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+ offset >= pbdev->msix_table_offset && + offset <= pbdev->msix_table_offset + + (pbdev->msix_entries - 1) * PCI_MSIX_ENTRY_SIZE) { + offset = offset - pbdev->msix_table_offset; + mr = &pbdev->pdev->msix_table_mmio; + } else { + mr = pbdev->pdev->io_regions[rp->pcias].memory; + } + +io_mem_write (mr, offset, data, rp->len);Same concerns as with the pcilg - this likely only works for len == 8 ?
Quoted 12 lines
target-s390x/pci_ic.c new file +12
-0+ }else if (rp->pcias ==15 ) { +switch (rp->len) { +case 1 : +break ; +case 2 : + data =le16_to_cpu (data); +break ; +case 4 : + data =le32_to_cpu (data); +break ; +case 8 : + data =le64_to_cpu (data);target-s390x/pci_ic.c line 403
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+break ; +default : +abort (); + } + +pci_host_config_write_common (pbdev->pdev, offset, +pci_config_size (pbdev->pdev), + data, rp->len); + }else { +DPRINTF ("pcistg invalid space\n" );Set condition code 1 in this case?
Quoted 37 lines
target-s390x/pci_ic.c new file +37
-0+ } + +setcc (cpu,0 ); +return 0 ; +} + +static uint64_t guest_io_table_walk (uint64_t guest_iota, +uint64_t guest_dma_address) +{ +uint64_t sto_a, pto_a, px_a; +uint64_t sto, pto, pte; +uint32_t rtx, sx, px; + + rtx =calc_rtx (guest_dma_address); + sx =calc_sx (guest_dma_address); + px =calc_px (guest_dma_address); + + sto_a = guest_iota + rtx *sizeof (uint64_t ); +cpu_physical_memory_rw (sto_a, (uint8_t *)&sto,sizeof (uint64_t ),0 ); + sto = (uint64_t )get_rt_sto (sto); + + pto_a = sto + sx *sizeof (uint64_t ); +cpu_physical_memory_rw (pto_a, (uint8_t *)&pto,sizeof (uint64_t ),0 ); + pto = (uint64_t )get_st_pto (pto); + + px_a = pto + px *sizeof (uint64_t ); +cpu_physical_memory_rw (px_a, (uint8_t *)&pte,sizeof (uint64_t ),0 ); + +return pte; +} + +int kvm_rpcit_service_call (S390CPU *cpu,struct kvm_run *run) +{ +CPUS390XState *env = &cpu->env; +uint8_t r1 = (run->s390_sieic.ipb &0x00f00000 ) >>20 ; +uint8_t r2 = (run->s390_sieic.ipb &0x000f0000 ) >>16 ; +uint32_t fh;target-s390x/pci_ic.c line 450
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+uint64_t pte; +S390PCIBusDevice *pbdev; +ram_addr_t size; +int flags; +IOMMUTLBEntry entry; + +cpu_synchronize_state (CPU (cpu)); + + fh = env->regs[r1] >>32 ; + size = env->regs[r2 +1 ];Missing two checks again:
- r2 should be even
- CPU should not be in problem statetarget-s390x/pci_ic.c line 456
Quoted 5 lines
target-s390x/pci_ic.c new file +5
-0+ pbdev =s390_pci_find_dev_by_fh (fh); + +if (!pbdev) { +DPRINTF ("rpcit no pci dev\n" ); +return -EIO ;I think it would be better to set condition code 3 instead.
Quoted 36 lines
target-s390x/pci_ic.c new file +36
-0+ } + + pte =guest_io_table_walk (pbdev->g_iota, env->regs[r2]); + flags = pte &ZPCI_PTE_FLAG_MASK ; + entry.target_as = &address_space_memory; + entry.iova = env->regs[r2]; + entry.translated_addr = pte &ZPCI_PTE_ADDR_MASK ; + entry.addr_mask = size -1 ; + +if (flags &ZPCI_PTE_INVALID ) { + entry.perm =IOMMU_NONE ; + }else { + entry.perm =IOMMU_RW ; + } + +memory_region_notify_iommu (pci_device_iommu_address_space ( + pbdev->pdev)->root, entry); + +setcc (cpu,0 ); +return 0 ; +} + +int kvm_sic_service_call (S390CPU *cpu,struct kvm_run *run) +{ +DPRINTF ("sic\n" ); +return 0 ; +} + +int kvm_pcistb_service_call (S390CPU *cpu,struct kvm_run *run) +{ +CPUS390XState *env = &cpu->env; +uint8_t r1 = (run->s390_sieic.ipa &0x00f0 ) >>4 ; +uint8_t r3 = run->s390_sieic.ipa &0x000f ; +PciStb *rp; +uint64_t gaddr; +uint64_t *uaddr, *pu;target-s390x/pci_ic.c line 502
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+hwaddr len; +S390PCIBusDevice *pbdev; +MemoryRegion *mr; +int i; + +cpu_synchronize_state (CPU (cpu)); + + rp = (PciStb *)&env->regs[r1]; + gaddr =get_base_disp_rsy (cpu, run); + len = rp->len;Not sure, but don't you also have to check the length and offset here for valid ranges?
At least you should check for problem state again.target-s390x/pci_ic.c line 507
Quoted 4 lines
target-s390x/pci_ic.c new file +4
-0+ pbdev =s390_pci_find_dev_by_fh (rp->fh); +if (!pbdev) { +DPRINTF ("pcistb no pci dev fh 0x%x\n" , rp->fh); +return -EIO ;User cc3 instead?
Quoted 5 lines
target-s390x/pci_ic.c new file +5
-0+ } + + uaddr =cpu_physical_memory_map (gaddr, &len,0 ); + mr = pbdev->pdev->io_regions[rp->pcias].memory; +target-s390x/pci_ic.c line 522
Quoted 10 lines
target-s390x/pci_ic.c new file +10
-0+ pu = uaddr; +for (i =0 ; i < rp->len /8 ; i++) { +io_mem_write (mr, env->regs[r3] + i *8 , *pu,8 ); + pu++; + } + +cpu_physical_memory_unmap (uaddr, len,0 , len); +setcc (cpu,0 ); +return 0 ; +}[...]
Thomas
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
RFC PATCH 6/6 vfio: make vfio run on s390 platform
In reply to [email protected]From: Frank Blaschka <[email protected]>
Following changes are made because of platform differences:
1) s390 does not support mmap'ing of PCI BARs so we have to go via slow path
2) no intx support
3) no classic MSIX interrupts. The pci hw understands the concept of requesting MSIX irqs but irqs are delivered as s390 adapter irqs. Introduce s390 specific functions for msix notification (slow path) and msi routes (kvm fast path).
4) Use type1 iommu but register only for iommu address space
--- hw/misc/vfio.c | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+)
hw/misc/vfio.c +24 -0
@@ -40,6 +40,9 @@#include "sysemu/kvm.h" #include "sysemu/sysemu.h" #include "hw/misc/vfio.h" +#ifdef TARGET_S390X +#include "hw/s390x/s390-pci-bus.h" +#endif /* #define DEBUG_VFIO */ #ifdef DEBUG_VFIO
@@ -51,7 +54,11 @@#endif /* Extra debugging, trap acceleration paths for more logging */ +#ifdef TARGET_S390X +#define VFIO_ALLOW_MMAP 0 +#else #define VFIO_ALLOW_MMAP 1 +#endif #define VFIO_ALLOW_KVM_INTX 1#define VFIO_ALLOW_KVM_MSI 1#define VFIO_ALLOW_KVM_MSIX 1
@@ -554,6 +561,10 @@ static int vfio_enable_intx(VFIODevice *struct vfio_irq_set *irq_set;int32_t *pfd; +#ifdef TARGET_S390X +return 0 ; +#endif +if (!pin) {return 0 ; }
@@ -664,7 +675,11 @@ static void vfio_msi_interrupt(void *opa#endif if (vdev->interrupt ==VFIO_INT_MSIX ) { +#ifdef TARGET_S390X +s390_msix_notify (&vdev->pdev, nr); +#else msix_notify (&vdev->pdev, nr); +#endif }else if (vdev->interrupt ==VFIO_INT_MSI ) {msi_notify (&vdev->pdev, nr); }else {
@@ -730,7 +745,11 @@ static void vfio_add_kvm_msi_virq(VFIOMSreturn ; } +#ifdef TARGET_S390X + virq =s390_irqchip_add_msi_route (&vector->vdev->pdev, kvm_state, *msg); +#else virq =kvm_irqchip_add_msi_route (kvm_state, *msg); +#endif if (virq <0 ) {event_notifier_cleanup (&vector->kvm_interrupt);return ;
@@ -3702,8 +3721,13 @@ static int vfio_connect_container(VFIOGr container->iommu_data.type1.listener = vfio_memory_listener; container->iommu_data.release = vfio_listener_release; +#ifdef TARGET_S390X +memory_listener_register (&container->iommu_data.type1.listener, + container->space->as); +#else memory_listener_register (&container->iommu_data.type1.listener, &address_space_memory); +#endif if (container->iommu_data.type1.error) { ret = container->iommu_data.type1.error;
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390
Quoted 7 lines
qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformThx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.
- s390_msix_notify() vs msix_notify() should be abstracted somewhere
else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route
- We can probably come up with a better way to determine which address
space to connect to the memory listener.
Looks like a reasonable first pass, good re-use of vfio code. Thanks,
Alex
Quoted 41 lines · Alex Williamson
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390Quoted 7 lines
qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformThx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.c
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.
Can we emulate MMIO on mmap'ed regions by routing every memory access via the kernel? It'd be slow, but at least make existing VFIO code compatible.
Quoted 6 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.
-s390_msix_notify()vsmsix_notify()should be abstracted somewhere
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_route
Yes, please implement emulated PCI device support first, then do VFIO.
Alex
Quoted 43 lines · Alexander Graf
Quoted 41 lines · Alex Williamson
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390Quoted 7 lines
qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformThx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.c
Quoted 8 lines
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Can we emulate MMIO on mmap'ed regions by routing every memory access via the kernel? It'd be slow, but at least make existing VFIO code compatible.
Isn't that effectively what we do when we use memory_region_init_io() vs memory_region_init_ram_ptr() or are you suggesting something that can handle the MMIO without bouncing out to QEMU? VFIO is already compatible with regions that cannot be mmap'd, the kernel just needs to report it as such. Thanks,
Alex
Quoted 53 lines · Alex Williamson
Quoted 43 lines · Alexander Graf
Quoted 41 lines · Alex Williamson
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390Quoted 7 lines
qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformThx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cQuoted 8 lines
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Can we emulate MMIO on mmap'ed regions by routing every memory access via the kernel? It'd be slow, but at least make existing VFIO code compatible.
Quoted 5 lines
Isn't that effectively what we do when we use
memory_region_init_io()vsmemory_region_init_ram_ptr()or are you suggesting something that can handle the MMIO without bouncing out to QEMU? VFIO is already compatible with regions that cannot be mmap'd, the kernel just needs to report it as such. Thanks,
Ah, cool. I guess I missed that part :). Then all is well.
Alex
Quoted 27 lines · Alex Williamson
Quoted 26 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
Quoted 8 lines
Quoted 5 lines
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformThx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.c
Ok, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.
Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +2 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_ info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.
The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewhere
Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.
Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_route
Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX
- in reality currently there is only a PCI network card available
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.
Any suggestion or idea for that?
Quoted 5 lines
Looks like a reasonable first pass, good re-use of vfio code. Thanks,
Alex
Thx,
Frank
Quoted 1 line
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Thx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
The patches in each email are showing up as attachments in my mail client. Is it just me?
Quoted 21 lines
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cOk, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +0
-0@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2
-0info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {
Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.
Quoted 6 lines
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.The current code is fine. Problem is the card reports an interrupt index (
PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.
Yep, let's try to do something consistent with the MMAP testing.
Quoted 10 lines
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewherePlatform does not have have an apic so there is nothing we could emulate in qemu to make the existing
msix_notify()work.Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_routeSame here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Quoted 6 lines
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX
What about MSI (non-X)?
Quoted 1 line
- in reality currently there is only a PCI network card available
On the physical hardware?
Quoted 3 lines
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.
Quoted 3 lines
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.Any suggestion or idea for that?
I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,
Alex
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformQuoted 6 lines
Quoted 4 lines
Quoted 1 line
Thx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
The patches in each email are showing up as attachments in my mail client. Is it just me?
Quoted 21 lines
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cOk, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +0
-0@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
Quoted 16 lines
drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2
-0info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.
Ok, understand this. There is no capability of feature so I will go with the function.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.The current code is fine. Problem is the card reports an interrupt index (
PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.Yep, let's try to do something consistent with the MMAP testing.
Do you mean let the kernel announce this also?
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewherePlatform does not have have an apic so there is nothing we could emulate in qemu to make the existing
msix_notify()work.Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_routeSame here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Quoted 8 lines
Quoted 6 lines
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIXWhat about MSI (non-X)?
In theory MSI should work also but I have not seen in reality.
Quoted 5 lines
Quoted 2 lines
- in reality currently there is only a PCI network card available
On the physical hardware?
yes
Quoted 10 lines
Quoted 3 lines
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.
Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.Any suggestion or idea for that?
I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,
I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?
Thx Frank
Quoted 3 lines
Alex
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformQuoted 6 lines
Quoted 4 lines
Quoted 1 line
Thx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
The patches in each email are showing up as attachments in my mail client. Is it just me?
Quoted 21 lines
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cOk, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +0
-0@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_Quoted 16 lines
drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2
-0info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.
Ok, understand this. There is no capability of feature so I will go with the function.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The current code is fine. Problem is the card reports an interrupt index (
PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.Yep, let's try to do something consistent with the MMAP testing.
Do you mean let the kernel announce this also?
Yes, the kernel reports a count of 0 in vfio_irq_info when the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.
Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewherePlatform does not have have an apic so there is nothing we could emulate in qemu to make the existing
msix_notify()work.Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_routeSame here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Quoted 8 lines
Quoted 6 lines
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIXWhat about MSI (non-X)?
In theory MSI should work also but I have not seen in reality.
Quoted 5 lines
Quoted 2 lines
- in reality currently there is only a PCI network card available
On the physical hardware?
yes
Quoted 10 lines
Quoted 3 lines
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.
Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.Any suggestion or idea for that?
Quoted 7 lines
Quoted 4 lines
I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,
I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?
Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,
Alex
Quoted 158 lines · Alex Williamson
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformQuoted 6 lines
Quoted 4 lines
Quoted 1 line
Thx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
The patches in each email are showing up as attachments in my mail client. Is it just me?
Quoted 21 lines
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cOk, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +0
-0@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_Quoted 16 lines
drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2
-0info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.
Ok, understand this. There is no capability of feature so I will go with the function.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The current code is fine. Problem is the card reports an interrupt index (
PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.Yep, let's try to do something consistent with the MMAP testing.
Do you mean let the kernel announce this also?
Yes, the kernel reports a count of 0 in
vfio_irq_infowhen the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewherePlatform does not have have an apic so there is nothing we could emulate in qemu to make the existing
msix_notify()work.Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_routeSame here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Quoted 8 lines
Quoted 6 lines
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIXWhat about MSI (non-X)?
In theory MSI should work also but I have not seen in reality.
Quoted 5 lines
Quoted 2 lines
- in reality currently there is only a PCI network card available
On the physical hardware?
yes
Quoted 10 lines
Quoted 3 lines
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.
Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.Any suggestion or idea for that?
Quoted 7 lines
Quoted 4 lines
I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,
I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?
Quoted 8 lines
Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,
Do you think this would be sufficient?
@@ -3689,8 +3701,13 @@ static int vfio_connect_container(VFIOGr
container->iommu_data.type1.listener = vfio_memory_listener;
container->iommu_data.release = vfio_listener_release;- memory_listener_register(&container->iommu_data.type1.listener,
- &address_space_memory);
+ if (memory_region_is_iommu(as->root)) {
+ memory_listener_register(&container->iommu_data.type1.listener,
+ container->space->as);
+ } else {
+ memory_listener_register(&container->iommu_data.type1.listener,
+ &address_space_memory);
+ }
if (container->iommu_data.type1.error) {
ret = container->iommu_data.type1.error;If not what else has to be checked? What are the indications to add the memory listener to container address space or to address_space_memory? Thx for your help.
Quoted 3 lines
Alex
Quoted 187 lines · Frank Blaschka
Quoted 158 lines · Alex Williamson
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformQuoted 6 lines
Quoted 4 lines
Quoted 1 line
Thx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
The patches in each email are showing up as attachments in my mail client. Is it just me?
Quoted 21 lines
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cOk, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +0
-0@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_Quoted 16 lines
drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2
-0info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.
Ok, understand this. There is no capability of feature so I will go with the function.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The current code is fine. Problem is the card reports an interrupt index (
PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.Yep, let's try to do something consistent with the MMAP testing.
Do you mean let the kernel announce this also?
Yes, the kernel reports a count of 0 in
vfio_irq_infowhen the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewherePlatform does not have have an apic so there is nothing we could emulate in qemu to make the existing
msix_notify()work.Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_routeSame here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Quoted 8 lines
Quoted 6 lines
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIXWhat about MSI (non-X)?
In theory MSI should work also but I have not seen in reality.
Quoted 5 lines
Quoted 2 lines
- in reality currently there is only a PCI network card available
On the physical hardware?
yes
Quoted 10 lines
Quoted 3 lines
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.
Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.Any suggestion or idea for that?
Quoted 7 lines
Quoted 4 lines
I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,
I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?
Quoted 8 lines
Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,
Do you think this would be sufficient?
(quoted diff) +7
-2@@ -3689,8 +3701,13 @@ static int vfio_connect_container(VFIOGr container->iommu_data.type1.listener = vfio_memory_listener; container->iommu_data.release = vfio_listener_release; - memory_listener_register(&container->iommu_data.type1.listener, - &address_space_memory); + if (memory_region_is_iommu(as->root)) { + memory_listener_register(&container->iommu_data.type1.listener, + container->space->as); + } else { + memory_listener_register(&container->iommu_data.type1.listener, + &address_space_memory); + } if (container->iommu_data.type1.error) { ret = container->iommu_data.type1.error;
Quoted 3 lines
If not what else has to be checked? What are the indications to add the memory listener to container address space or to
address_space_memory? Thx for your help.
Sure, that's what I was asking for, but as I'm looking at is, shouldn't we be able to use container->space->as regardless? It seems like an oversight that address_space_memory wasn't replaced with container->space->as when Alexey added support for multiple address spaces.
The container->space->as value comes from pci_device_iommu_address_space() which walks the PCI bus up from the device looking for an IOMMU address space. If it doesn't find one (likely) it uses address_space_memory. So I suspect if we fix the code, there's no need for any sort of switch. Thanks,
Alex
Quoted 198 lines · Alex Williamson
Quoted 187 lines · Frank Blaschka
Quoted 158 lines · Alex Williamson
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:
This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.
Most interesting patch is:
vfio: make vfio run on s390 platformI hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.
The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.
kernel patches apply to linux-kvm-next
KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390qemu patches apply to qemu-master
s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platformQuoted 6 lines
Quoted 4 lines
Quoted 1 line
Thx for feedback and review comments
Sending patches as attachments makes it difficult to comment inline.
Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?
The patches in each email are showing up as attachments in my mail client. Is it just me?
Quoted 21 lines
Quoted 6 lines
2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call froms390-iommu.cOk, will change this.
Quoted 4 lines
6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...
drivers/vfio/pci/vfio_pci.c +0
-0@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_Quoted 16 lines
drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387
Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2
-0info.flags =VFIO_REGION_INFO_FLAG_READ |VFIO_REGION_INFO_FLAG_WRITE ; +#ifndef CONFIG_S390 if (pci_resource_flags (pdev, info.index) &IORESOURCE_MEM && info.size >=PAGE_SIZE ) info.flags |=VFIO_REGION_INFO_FLAG_MMAP ; +#endif break ;case VFIO_PCI_ROM_REGION_INDEX : {Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.
Ok, understand this. There is no capability of feature so I will go with the function.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The current code is fine. Problem is the card reports an interrupt index (
PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.Yep, let's try to do something consistent with the MMAP testing.
Do you mean let the kernel announce this also?
Yes, the kernel reports a count of 0 in
vfio_irq_infowhen the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line
-
s390_msix_notify()vsmsix_notify()should be abstracted somewherePlatform does not have have an apic so there is nothing we could emulate in qemu to make the existing
msix_notify()work.Quoted 2 lines
else. How would an emulated PCI device with MSI-X support work?
- same foradd_msi_routeSame here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.
Quoted 8 lines
Quoted 6 lines
Any suggestion or idea how a better abstraction could look like?
With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIXWhat about MSI (non-X)?
In theory MSI should work also but I have not seen in reality.
Quoted 5 lines
Quoted 2 lines
- in reality currently there is only a PCI network card available
On the physical hardware?
yes
Quoted 10 lines
Quoted 3 lines
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.
Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.
Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.
Quoted 4 lines
Quoted 3 lines
Quoted 2 lines
- We can probably come up with a better way to determine which address
space to connect to the memory listener.Any suggestion or idea for that?
Quoted 7 lines
Quoted 4 lines
I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,
I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?
Quoted 8 lines
Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,
Do you think this would be sufficient?
(quoted diff) +7
-2@@ -3689,8 +3701,13 @@ static int vfio_connect_container(VFIOGr container->iommu_data.type1.listener = vfio_memory_listener; container->iommu_data.release = vfio_listener_release; - memory_listener_register(&container->iommu_data.type1.listener, - &address_space_memory); + if (memory_region_is_iommu(as->root)) { + memory_listener_register(&container->iommu_data.type1.listener, + container->space->as); + } else { + memory_listener_register(&container->iommu_data.type1.listener, + &address_space_memory); + } if (container->iommu_data.type1.error) { ret = container->iommu_data.type1.error;Quoted 3 lines
If not what else has to be checked? What are the indications to add the memory listener to container address space or to
address_space_memory? Thx for your help.Sure, that's what I was asking for, but as I'm looking at is, shouldn't we be able to use
container->space->asregardless? It seems like an oversight thataddress_space_memorywasn't replaced withcontainer->space->aswhen Alexey added support for multiple address spaces.
Quoted 6 lines
The
container->space->asvalue comes frompci_device_iommu_address_space()which walks the PCI bus up from the device looking for an IOMMU address space. If it doesn't find one (likely) it usesaddress_space_memory. So I suspect if we fix the code, there's no need for any sort of switch. Thanks,
This was my impression too, but I was not completely sure about this. Thx for confirmation I will chance this. Now with all that work my changes in vfio are minimal and without any platform #ifdef. I will post the complete patch set again if I have a solution (or at least an idea) for the MSI-X stuff.
Thanks again for your help and patience,
Frank
Quoted 7 lines
Alex
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html