Skip to contentSkip to conversation

vfio based pci pass-through for qemu/KVM on s390

RFC PATCH In review [email protected] 5 participants 20 messages Sep 19, 2014 – Oct 2, 2014

Subsystems (7)
KVM s390 IOMMU VFIO driver +3 more subsystems
lore
Share & tools
lore
#0 lore raw

cover vfio based pci pass-through for qemu/KVM on s390

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Thx for feedback and review comments

Frank

#1 lore raw

RFC PATCH 1/6 KVM: s390: Enable PCI instructions

Enable PCI instructions for s390 KVM.

---
 arch/s390/kvm/kvm-s390.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/s390/kvm/kvm-s390.c +1 -1
@@ -1787,7 +1787,7 @@ static int __init kvm_s390_init(void)
 	}
 	memcpy(vfacilities, S390_lowcore.stfle_fac_list, 16);
 	vfacilities[0] &= 0xff82fff3f4fc2000UL;
-	vfacilities[1] &= 0x005c000000000000UL;
+	vfacilities[1] &= 0x07dc000000000000UL;
 	return 0;
 }
 
#2 lore raw

RFC PATCH 2/6 iommu: add iommu for s390 platform

In reply to [email protected]

From: Frank Blaschka <[email protected]>

Add a basic iommu for the s390 platform. The code is pretty simple since on s390 each PCI device has its own virtual io address space starting at the same vio address. For this a domain could hold only one pci device. Also there is no relation between pci devices so each device belongs to a separate iommu group.

---
 arch/s390/include/asm/pci.h |    3 
 arch/s390/pci/pci_dma.c     |   21 ++++-
 drivers/iommu/Kconfig       |    9 ++
 drivers/iommu/Makefile      |    1 
 drivers/iommu/s390-iommu.c  |  181 ++++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 213 insertions(+), 2 deletions(-)
arch/s390/include/asm/pci.h +3 -0
@@ -177,6 +177,9 @@ struct zpci_dev *get_zdev_by_fid(u32);
 /* DMA */
 int zpci_dma_init(void);
 void zpci_dma_exit(void);
+int dma_update_trans(struct zpci_dev *zdev, unsigned long pa,
+		     dma_addr_t dma_addr, size_t size, int flags);
+void dma_purge_rto_entries(struct zpci_dev *zdev);
 
 /* FMB */
 int zpci_fmb_enable_device(struct zpci_dev *);
arch/s390/pci/pci_dma.c +19 -2
@@ -139,8 +139,8 @@ static void dma_update_cpu_trans(struct
 		entry_clr_protected(entry);
 }
 
-static int dma_update_trans(struct zpci_dev *zdev, unsigned long pa,
-			    dma_addr_t dma_addr, size_t size, int flags)
+int dma_update_trans(struct zpci_dev *zdev, unsigned long pa,
+		     dma_addr_t dma_addr, size_t size, int flags)
 {
 	unsigned int nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
 	u8 *page_addr = (u8 *) (pa & PAGE_MASK);
@@ -180,6 +180,7 @@ no_refresh:
 	spin_unlock_irqrestore(&zdev->dma_table_lock, irq_flags);
 	return rc;
 }
+EXPORT_SYMBOL_GPL(dma_update_trans);
 
 static void dma_free_seg_table(unsigned long entry)
 {
@@ -210,6 +211,22 @@ static void dma_cleanup_tables(struct zp
 	zdev->dma_table = NULL;
 }
 
+void dma_purge_rto_entries(struct zpci_dev *zdev)
+{
+	unsigned long *table;
+	int rtx;
+
+	if (!zdev || !zdev->dma_table)
+		return;
+	table = zdev->dma_table;
+	for (rtx = 0; rtx < ZPCI_TABLE_ENTRIES; rtx++)
+		if (reg_entry_isvalid(table[rtx])) {
+			dma_free_seg_table(table[rtx]);
+			invalidate_table_entry(&table[rtx]);
+		}
+}
+EXPORT_SYMBOL_GPL(dma_purge_rto_entries);
+
 static unsigned long __dma_alloc_iommu(struct zpci_dev *zdev,
 				       unsigned long start, int size)
 {
drivers/iommu/Kconfig +9 -0
@@ -302,4 +302,13 @@ config ARM_SMMU
 	  Say Y here if your SoC includes an IOMMU device implementing
 	  the ARM SMMU architecture.
 
+config S390_IOMMU
+        bool "s390 IOMMU Support"
+        depends on S390
+        select IOMMU_API
+        help
+          Support for the IBM s/390 IOMMU
+
+          If unsure, say N here.
+
 endif # IOMMU_SUPPORT
drivers/iommu/Makefile +1 -0
@@ -19,3 +19,4 @@ obj-$(CONFIG_EXYNOS_IOMMU) += exynos-iom
 obj-$(CONFIG_SHMOBILE_IOMMU) += shmobile-iommu.o
 obj-$(CONFIG_SHMOBILE_IPMMU) += shmobile-ipmmu.o
 obj-$(CONFIG_FSL_PAMU) += fsl_pamu.o fsl_pamu_domain.o
+obj-$(CONFIG_S390_IOMMU) += s390-iommu.o
drivers/iommu/s390-iommu.c new file +181 -0

181 lines not on this page · raw message

#3 lore raw

RFC PATCH 3/6 vfio: make vfio build on s390

In reply to [email protected]

From: Frank Blaschka <[email protected]>

This patch adds some small changes to make vfio build on s390.

---
 drivers/vfio/Kconfig             |    2 +-
 drivers/vfio/pci/vfio_pci_rdwr.c |    8 ++++++++
 2 files changed, 9 insertions(+), 1 deletion(-)
drivers/vfio/Kconfig +1 -1
@@ -16,7 +16,7 @@ config VFIO_SPAPR_EEH
 menuconfig VFIO
 	tristate "VFIO Non-Privileged userspace driver framework"
 	depends on IOMMU_API
-	select VFIO_IOMMU_TYPE1 if X86
+	select VFIO_IOMMU_TYPE1 if (X86 || S390)
 	select VFIO_IOMMU_SPAPR_TCE if (PPC_POWERNV || PPC_PSERIES)
 	select VFIO_SPAPR_EEH if (PPC_POWERNV || PPC_PSERIES)
 	select ANON_INODES
drivers/vfio/pci/vfio_pci_rdwr.c +8 -0
@@ -177,6 +177,13 @@ ssize_t vfio_pci_bar_rw(struct vfio_pci_
 	return done;
 }
 
+#ifdef CONFIG_NO_IOPORT_MAP
+ssize_t vfio_pci_vga_rw(struct vfio_pci_device *vdev, char __user *buf,
+			size_t count, loff_t *ppos, bool iswrite)
+{
+	return -EINVAL;
+}
+#else
 ssize_t vfio_pci_vga_rw(struct vfio_pci_device *vdev, char __user *buf,
 			       size_t count, loff_t *ppos, bool iswrite)
 {
@@ -236,3 +243,4 @@ ssize_t vfio_pci_vga_rw(struct vfio_pci_
 
 	return done;
 }
+#endif
#4 lore raw

RFC PATCH 4/6 s390: Add PCI bus support

In reply to [email protected]

From: Frank Blaschka <[email protected]>

This patch implements a pci bus for s390x together with some infrastructure to generate and handle hotplug events. It also provides device configuration/unconfiguration via sclp instruction interception.

---
 default-configs/s390x-softmmu.mak |    1 
 hw/s390x/Makefile.objs            |    1 
 hw/s390x/css.c                    |    5 
 hw/s390x/css.h                    |    1 
 hw/s390x/s390-pci-bus.c           |  404 ++++++++++++++++++++++++++++++++++++++
 hw/s390x/s390-pci-bus.h           |  166 +++++++++++++++
 hw/s390x/s390-virtio-ccw.c        |    2 
 hw/s390x/sclp.c                   |   10 
 include/hw/s390x/sclp.h           |    8 
 target-s390x/ioinst.c             |   52 ++++
 target-s390x/ioinst.h             |    1 
 11 files changed, 650 insertions(+), 1 deletion(-)
default-configs/s390x-softmmu.mak +1 -0
@@ -1,3 +1,4 @@
+include pci.mak
 CONFIG_VIRTIO=y
 CONFIG_SCLPCONSOLE=y
 CONFIG_S390_FLIC=y
hw/s390x/Makefile.objs +1 -0
@@ -8,3 +8,4 @@ obj-y += ipl.o
 obj-y += css.o
 obj-y += s390-virtio-ccw.o
 obj-y += virtio-ccw.o
+obj-$(CONFIG_KVM) += s390-pci-bus.o
hw/s390x/css.c +5 -0
@@ -1281,6 +1281,11 @@ void css_generate_chp_crws(uint8_t cssid
     /* TODO */
 }
 
+void css_generate_css_crws(uint8_t cssid)
+{
+    css_queue_crw(CRW_RSC_CSS, 0, 0, 0);
+}
+
 int css_enable_mcsse(void)
 {
     trace_css_enable_facility("mcsse");
hw/s390x/css.h +1 -0
@@ -99,6 +99,7 @@ void css_queue_crw(uint8_t rsc, uint8_t
 void css_generate_sch_crws(uint8_t cssid, uint8_t ssid, uint16_t schid,
                            int hotplugged, int add);
 void css_generate_chp_crws(uint8_t cssid, uint8_t chpid);
+void css_generate_css_crws(uint8_t cssid);
 void css_adapter_interrupt(uint8_t isc);
 
 #define CSS_IO_ADAPTER_VIRTIO 1
hw/s390x/s390-pci-bus.c new file +404 -0

404 lines not on this page · raw message

hw/s390x/s390-pci-bus.h new file +166 -0

166 lines not on this page · raw message

hw/s390x/s390-virtio-ccw.c +2 -0
@@ -18,6 +18,7 @@
 #include "css.h"
 #include "virtio-ccw.h"
 #include "qemu/config-file.h"
+#include "s390-pci-bus.h"
 
 #define TYPE_S390_CCW_MACHINE               "s390-ccw-machine"
 
@@ -126,6 +127,7 @@ static void ccw_init(MachineState *machi
     s390_init_ipl_dev(machine->kernel_filename, machine->kernel_cmdline,
                       machine->initrd_filename, "s390-ccw.img");
     s390_flic_init();
+    s390_pci_bus_init();
 
     /* register hypercalls */
     virtio_ccw_register_hcalls();
hw/s390x/sclp.c +9 -1
@@ -20,6 +20,7 @@
 #include "qemu/config-file.h"
 #include "hw/s390x/sclp.h"
 #include "hw/s390x/event-facility.h"
+#include "hw/s390x/s390-pci-bus.h"
 
 static inline SCLPEventFacility *get_event_facility(void)
 {
@@ -62,7 +63,8 @@ static void read_SCP_info(SCCB *sccb)
         read_info->entries[i].type = 0;
     }
 
-    read_info->facilities = cpu_to_be64(SCLP_HAS_CPU_INFO);
+    read_info->facilities = cpu_to_be64(SCLP_HAS_CPU_INFO |
+                                        SCLP_HAS_PCI_RECONFIG);
 
     /*
      * The storage increment size is a multiple of 1M and is a power of 2.
@@ -350,6 +352,12 @@ static void sclp_execute(SCCB *sccb, uin
     case SCLP_UNASSIGN_STORAGE:
         unassign_storage(sccb);
         break;
+    case SCLP_CMDW_CONFIGURE_PCI:
+        s390_pci_sclp_configure(1, sccb);
+        break;
+    case SCLP_CMDW_DECONFIGURE_PCI:
+        s390_pci_sclp_configure(0, sccb);
+        break;
     default:
         efc->command_handler(ef, sccb, code);
         break;
include/hw/s390x/sclp.h +8 -0
@@ -45,14 +45,22 @@
 #define SCLP_CMDW_CONFIGURE_CPU                 0x00110001
 #define SCLP_CMDW_DECONFIGURE_CPU               0x00100001
 
+/* SCLP PCI codes */
+#define SCLP_HAS_PCI_RECONFIG                   0x0000000040000000ULL
+#define SCLP_CMDW_CONFIGURE_PCI                 0x001a0001
+#define SCLP_CMDW_DECONFIGURE_PCI               0x001b0001
+#define SCLP_RECONFIG_PCI_ATPYE                 2
+
 /* SCLP response codes */
 #define SCLP_RC_NORMAL_READ_COMPLETION          0x0010
 #define SCLP_RC_NORMAL_COMPLETION               0x0020
 #define SCLP_RC_SCCB_BOUNDARY_VIOLATION         0x0100
+#define SCLP_RC_NO_ACTION_REQUIRED              0x0120
 #define SCLP_RC_INVALID_SCLP_COMMAND            0x01f0
 #define SCLP_RC_CONTAINED_EQUIPMENT_CHECK       0x0340
 #define SCLP_RC_INSUFFICIENT_SCCB_LENGTH        0x0300
 #define SCLP_RC_STANDBY_READ_COMPLETION         0x0410
+#define SCLP_RC_ADAPTER_ID_NOT_RECOGNIZED       0x09f0
 #define SCLP_RC_INVALID_FUNCTION                0x40f0
 #define SCLP_RC_NO_EVENT_BUFFERS_STORED         0x60f0
 #define SCLP_RC_INVALID_SELECTION_MASK          0x70f0
target-s390x/ioinst.c +52 -0

76 lines not on this page · raw message

target-s390x/ioinst.h +1 -0
@@ -194,6 +194,7 @@ typedef struct CRW {
 
 #define CRW_RSC_SUBCH 0x3
 #define CRW_RSC_CHP   0x4
+#define CRW_RSC_CSS   0xb
 
 /* I/O interruption code */
 typedef struct IOIntCode {
#5 lore raw

RFC PATCH 5/6 s390: implement pci instruction

In reply to [email protected]

From: Frank Blaschka <[email protected]>

This patch implements the s390 pci instructions in qemu. This allows to attach qemu pci devices including vfio. This does not mean the devices are functional but at least detection and config/memory space access is working.

---
 target-s390x/Makefile.objs |    2 
 target-s390x/kvm.c         |   52 +++
 target-s390x/pci_ic.c      |  621 +++++++++++++++++++++++++++++++++++++++++++++
 target-s390x/pci_ic.h      |  425 ++++++++++++++++++++++++++++++
 4 files changed, 1099 insertions(+), 1 deletion(-)
target-s390x/Makefile.objs +1 -1
@@ -2,4 +2,4 @@ obj-y += translate.o helper.o cpu.o inte
 obj-y += int_helper.o fpu_helper.o cc_helper.o mem_helper.o misc_helper.o
 obj-y += gdbstub.o
 obj-$(CONFIG_SOFTMMU) += ioinst.o arch_dump.o
-obj-$(CONFIG_KVM) += kvm.o
+obj-$(CONFIG_KVM) += kvm.o pci_ic.o
target-s390x/kvm.c +52 -0

96 lines not on this page · raw message

target-s390x/pci_ic.c new file +621 -0

621 lines not on this page · raw message

target-s390x/pci_ic.h new file +425 -0

425 lines not on this page · raw message

#6 lore raw

Re: [RFC patch 5/6] s390: implement pci instruction

Hi Frank,

Quoted 81 lines · On Fri, 19 Sep 2014 13:54:34 +0200 [email protected] wrote:

From: Frank Blaschka <[email protected]>

This patch implements the s390 pci instructions in qemu. This allows to attach qemu pci devices including vfio. This does not mean the devices are functional but at least detection and config/memory space access is working.

---
 target-s390x/Makefile.objs |    2 
 target-s390x/kvm.c         |   52 +++
 target-s390x/pci_ic.c      |  621 +++++++++++++++++++++++++++++++++++++++++++++
 target-s390x/pci_ic.h      |  425 ++++++++++++++++++++++++++++++
 4 files changed, 1099 insertions(+), 1 deletion(-)
target-s390x/Makefile.objs +1 -1
@@ -2,4 +2,4 @@ obj-y += translate.o helper.o cpu.o inte
 obj-y += int_helper.o fpu_helper.o cc_helper.o mem_helper.o misc_helper.o
 obj-y += gdbstub.o
 obj-$(CONFIG_SOFTMMU) += ioinst.o arch_dump.o
-obj-$(CONFIG_KVM) += kvm.o
+obj-$(CONFIG_KVM) += kvm.o pci_ic.o
target-s390x/kvm.c +23 -0
@@ -40,6 +40,7 @@
 #include "exec/gdbstub.h"
 #include "trace.h"
 #include "qapi-event.h"
+#include "pci_ic.h"
 
 /* #define DEBUG_KVM */
 
@@ -56,6 +57,7 @@
 #define IPA0_B2                         0xb200
 #define IPA0_B9                         0xb900
 #define IPA0_EB                         0xeb00
+#define IPA0_E3                         0xe300
 
 #define PRIV_B2_SCLP_CALL               0x20
 #define PRIV_B2_CSCH                    0x30
@@ -76,8 +78,17 @@
 #define PRIV_B2_XSCH                    0x76
 
 #define PRIV_EB_SQBS                    0x8a
+#define PRIV_EB_PCISTB                  0xd0
+#define PRIV_EB_SIC                     0xd1
 
 #define PRIV_B9_EQBS                    0x9c
+#define PRIV_B9_CLP                     0xa0
+#define PRIV_B9_PCISTG                  0xd0
+#define PRIV_B9_PCILG                   0xd2
+#define PRIV_B9_RPCIT                   0xd3
+
+#define PRIV_E3_MPCIFC                  0xd0
+#define PRIV_E3_STPCIFC                 0xd4
 
 #define DIAG_IPL                        0x308
 #define DIAG_KVM_HYPERCALL              0x500
@@ -813,6 +824,18 @@ static int handle_b9(S390CPU *cpu, struc
     int r = 0;
 
     switch (ipa1) {
+    case PRIV_B9_CLP:
+        r = kvm_clp_service_call(cpu, run);
+        break;
+    case PRIV_B9_PCISTG:
+        r = kvm_pcistg_service_call(cpu, run);
+        break;
+    case PRIV_B9_PCILG:
+        r = kvm_pcilg_service_call(cpu, run);
+        break;
+    case PRIV_B9_RPCIT:
+        r = kvm_rpcit_service_call(cpu, run);
+        break;
     case PRIV_B9_EQBS:
         /* just inject exception */
         r = -1;
@@ -831,6 +854,12 @@ static int handle_eb(S390CPU *cpu, struc
     int r = 0;
 

target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 865

Quoted 10 lines
target-s390x/kvm.c +6 -0
     switch (ipa1) {
+    case PRIV_EB_PCISTB:
+        r = kvm_pcistb_service_call(cpu, run);
+        break;
+    case PRIV_EB_SIC:
+        r = kvm_sic_service_call(cpu, run);
+        break;
     case PRIV_EB_SQBS:
         /* just inject exception */
         r = -1;

I'm not sure, but I think the handler for the eb instructions is wrong:
The second byte of the opcode is encoded in the lowest byte of the ipb field, not the lowest byte of the ipa field (just like with the e3 handler). Did you verify that your handlers get called correctly?

Quoted 13 lines
target-s390x/kvm.c +9 -0
@@ -844,6 +873,26 @@ static int handle_eb(S390CPU *cpu, struc
     return r;
 }
 
+static int handle_e3(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
+{
+    int r = 0;
+
+    switch (ipa1) {
+    case PRIV_E3_MPCIFC:
+        r = kvm_mpcifc_service_call(cpu, run);
+        break;
+    case PRIV_E3_STPCIFC:

target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 894

Quoted 10 lines
target-s390x/kvm.c +10 -0
+        r = kvm_stpcifc_service_call(cpu, run);
+        break;
+    default:
+        r = -1;
+        DPRINTF("KVM: unhandled PRIV: 0xe3%x\n", ipa1);
+        break;
+    }
+
+    return r;
+}

Could you please replace "ipa1" with "ipb1" to avoid confusion here?

Quoted 16 lines
target-s390x/kvm.c +3 -0
 static int handle_hypercall(S390CPU *cpu, struct kvm_run *run)
 {
     CPUS390XState *env = &cpu->env;
@@ -1038,6 +1087,9 @@ static int handle_instruction(S390CPU *c
     case IPA0_EB:
         r = handle_eb(cpu, run, ipa1);
         break;
+    case IPA0_E3:
+        r = handle_e3(cpu, run, run->s390_sieic.ipb & 0xff);
+        break;
     case IPA0_DIAG:
         r = handle_diag(cpu, run, run->s390_sieic.ipb);
         break;
target-s390x/pci_ic.c new file +0 -0
@@ -0,0 +1,621 @@

[...]

target-s390x/pci_ic.c line 308

Quoted 13 lines
target-s390x/pci_ic.c new file +13 -0
+
+int kvm_pcilg_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    CPUS390XState *env = &cpu->env;
+    S390PCIBusDevice *pbdev;
+    uint8_t r1 = (run->s390_sieic.ipb & 0x00f00000) >> 20;
+    uint8_t r2 = (run->s390_sieic.ipb & 0x000f0000) >> 16;
+    PciLgStg *rp;
+    uint64_t offset;
+    uint64_t data;
+
+    cpu_synchronize_state(CPU(cpu));
+    rp = (PciLgStg *)&env->regs[r2];

I think you have to check for r2 to be even here (and inject a specification exception otherwise)

You should also check rp->len for a valid value and inject an operand exception otherwise.

Quoted 1 line

+ offset = env->regs[r2 + 1];

You should also check for a valid offset value here (and inject an operand exception otherwise).

... and while you're at it - it also seems like the pci instructions are priviledged, so you should check for not being in the problem state here, too, just to be sure ;-)

target-s390x/pci_ic.c line 319

Quoted 9 lines
target-s390x/pci_ic.c new file +9 -0
+    pbdev = s390_pci_find_dev_by_fh(rp->fh);
+    if (!pbdev) {
+        DPRINTF("pcilg no pci dev\n");
+        return -EIO;
+    }
+
+    if (rp->pcias < 6) {
+        MemoryRegion *mr = pbdev->pdev->io_regions[rp->pcias].memory;
+        io_mem_read(mr, offset, &data, rp->len);

Does this also work if len != 8 ? I think the data should be put in the rightmost byte positions, with the unused leftmost bytes set to zero... so some more logic might be needed here.

Quoted 11 lines
target-s390x/pci_ic.c new file +11 -0
+    } else if (rp->pcias == 15) {
+        data =  pci_host_config_read_common(
+                   pbdev->pdev, offset, pci_config_size(pbdev->pdev), rp->len);
+
+        switch (rp->len) {
+        case 1:
+            break;
+        case 2:
+            data = cpu_to_le16(data);
+            break;
+        case 4:

target-s390x/pci_ic.c line 340

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+            data = cpu_to_le32(data);
+            break;
+        case 8:
+            data = cpu_to_le64(data);
+            break;
+        default:
+            abort();
+        }
+    } else {
+        DPRINTF("invalid space\n");

Set condition code 1 in this case?

Quoted 9 lines
target-s390x/pci_ic.c new file +9 -0
+    }
+
+    env->regs[r1] = data;
+    setcc(cpu, 0);
+    return 0;
+}
+
+int kvm_pcistg_service_call(S390CPU *cpu, struct kvm_run *run)
+{

target-s390x/pci_ic.c line 359

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    CPUS390XState *env = &cpu->env;
+    uint8_t r1 = (run->s390_sieic.ipb & 0x00f00000) >> 20;
+    uint8_t r2 = (run->s390_sieic.ipb & 0x000f0000) >> 16;
+    PciLgStg *rp;
+    uint64_t offset, data;
+    S390PCIBusDevice *pbdev;
+
+    cpu_synchronize_state(CPU(cpu));
+    rp = (PciLgStg *)&env->regs[r2];
+    offset = env->regs[r2 + 1];

You likely need the same sanity checks here as with the pcilg function.

Quoted 11 lines
target-s390x/pci_ic.c new file +11 -0
+    pbdev = s390_pci_find_dev_by_fh(rp->fh);
+    if (!pbdev) {
+        DPRINTF("pcistg no pci dev\n");
+        return -EIO;
+    }
+
+    data = env->regs[r1];
+
+    if (rp->pcias < 6) {
+        MemoryRegion *mr;
+        if (pbdev->msix_table_bar == rp->pcias &&

target-s390x/pci_ic.c line 381

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+            offset >= pbdev->msix_table_offset &&
+            offset <= pbdev->msix_table_offset +
+                      (pbdev->msix_entries - 1) * PCI_MSIX_ENTRY_SIZE) {
+            offset = offset - pbdev->msix_table_offset;
+            mr = &pbdev->pdev->msix_table_mmio;
+        } else {
+            mr = pbdev->pdev->io_regions[rp->pcias].memory;
+        }
+
+        io_mem_write(mr, offset, data, rp->len);

Same concerns as with the pcilg - this likely only works for len == 8 ?

Quoted 12 lines
target-s390x/pci_ic.c new file +12 -0
+    } else if (rp->pcias == 15) {
+        switch (rp->len) {
+        case 1:
+            break;
+        case 2:
+            data = le16_to_cpu(data);
+            break;
+        case 4:
+            data = le32_to_cpu(data);
+            break;
+        case 8:
+            data = le64_to_cpu(data);

target-s390x/pci_ic.c line 403

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+            break;
+        default:
+            abort();
+        }
+
+        pci_host_config_write_common(pbdev->pdev, offset,
+                                     pci_config_size(pbdev->pdev),
+                                     data, rp->len);
+    } else {
+        DPRINTF("pcistg invalid space\n");

Set condition code 1 in this case?

Quoted 37 lines
target-s390x/pci_ic.c new file +37 -0
+    }
+
+    setcc(cpu, 0);
+    return 0;
+}
+
+static uint64_t guest_io_table_walk(uint64_t guest_iota,
+                                    uint64_t guest_dma_address)
+{
+    uint64_t sto_a, pto_a, px_a;
+    uint64_t sto, pto, pte;
+    uint32_t rtx, sx, px;
+
+    rtx = calc_rtx(guest_dma_address);
+    sx = calc_sx(guest_dma_address);
+    px = calc_px(guest_dma_address);
+
+    sto_a = guest_iota + rtx * sizeof(uint64_t);
+    cpu_physical_memory_rw(sto_a, (uint8_t *)&sto, sizeof(uint64_t), 0);
+    sto = (uint64_t)get_rt_sto(sto);
+
+    pto_a = sto + sx * sizeof(uint64_t);
+    cpu_physical_memory_rw(pto_a, (uint8_t *)&pto, sizeof(uint64_t), 0);
+    pto = (uint64_t)get_st_pto(pto);
+
+    px_a = pto + px * sizeof(uint64_t);
+    cpu_physical_memory_rw(px_a, (uint8_t *)&pte, sizeof(uint64_t), 0);
+
+    return pte;
+}
+
+int kvm_rpcit_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    CPUS390XState *env = &cpu->env;
+    uint8_t r1 = (run->s390_sieic.ipb & 0x00f00000) >> 20;
+    uint8_t r2 = (run->s390_sieic.ipb & 0x000f0000) >> 16;
+    uint32_t fh;

target-s390x/pci_ic.c line 450

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    uint64_t pte;
+    S390PCIBusDevice *pbdev;
+    ram_addr_t size;
+    int flags;
+    IOMMUTLBEntry entry;
+
+    cpu_synchronize_state(CPU(cpu));
+
+    fh = env->regs[r1] >> 32;
+    size = env->regs[r2 + 1];

Missing two checks again:
- r2 should be even
- CPU should not be in problem state

target-s390x/pci_ic.c line 456

Quoted 5 lines
target-s390x/pci_ic.c new file +5 -0
+    pbdev = s390_pci_find_dev_by_fh(fh);
+
+    if (!pbdev) {
+        DPRINTF("rpcit no pci dev\n");
+        return -EIO;

I think it would be better to set condition code 3 instead.

Quoted 36 lines
target-s390x/pci_ic.c new file +36 -0
+    }
+
+    pte = guest_io_table_walk(pbdev->g_iota, env->regs[r2]);
+    flags = pte & ZPCI_PTE_FLAG_MASK;
+    entry.target_as = &address_space_memory;
+    entry.iova = env->regs[r2];
+    entry.translated_addr = pte & ZPCI_PTE_ADDR_MASK;
+    entry.addr_mask = size - 1;
+
+    if (flags & ZPCI_PTE_INVALID) {
+        entry.perm = IOMMU_NONE;
+    } else {
+        entry.perm = IOMMU_RW;
+    }
+
+    memory_region_notify_iommu(pci_device_iommu_address_space(
+                               pbdev->pdev)->root, entry);
+
+    setcc(cpu, 0);
+    return 0;
+}
+
+int kvm_sic_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    DPRINTF("sic\n");
+    return 0;
+}
+
+int kvm_pcistb_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    CPUS390XState *env = &cpu->env;
+    uint8_t r1 = (run->s390_sieic.ipa & 0x00f0) >> 4;
+    uint8_t r3 = run->s390_sieic.ipa & 0x000f;
+    PciStb *rp;
+    uint64_t gaddr;
+    uint64_t *uaddr, *pu;

target-s390x/pci_ic.c line 502

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    hwaddr len;
+    S390PCIBusDevice *pbdev;
+    MemoryRegion *mr;
+    int i;
+
+    cpu_synchronize_state(CPU(cpu));
+
+    rp = (PciStb *)&env->regs[r1];
+    gaddr = get_base_disp_rsy(cpu, run);
+    len = rp->len;

Not sure, but don't you also have to check the length and offset here for valid ranges?
At least you should check for problem state again.

target-s390x/pci_ic.c line 507

Quoted 4 lines
target-s390x/pci_ic.c new file +4 -0
+    pbdev = s390_pci_find_dev_by_fh(rp->fh);
+    if (!pbdev) {
+        DPRINTF("pcistb no pci dev fh 0x%x\n", rp->fh);
+        return -EIO;

User cc3 instead?

Quoted 5 lines
target-s390x/pci_ic.c new file +5 -0
+    }
+
+    uaddr = cpu_physical_memory_map(gaddr, &len, 0);
+    mr = pbdev->pdev->io_regions[rp->pcias].memory;
+

target-s390x/pci_ic.c line 522

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    pu = uaddr;
+    for (i = 0; i < rp->len / 8; i++) {
+        io_mem_write(mr, env->regs[r3] + i * 8, *pu, 8);
+        pu++;
+    }
+
+    cpu_physical_memory_unmap(uaddr, len, 0, len);
+    setcc(cpu, 0);
+    return 0;
+}

[...]

Thomas

#7 lore raw

Re: [Qemu-devel] [RFC patch 5/6] s390: implement pci instruction

Quoted 87 lines · Thomas Huth

Hi Frank,

Quoted 81 lines · On Fri, 19 Sep 2014 13:54:34 +0200 [email protected] wrote:

From: Frank Blaschka <[email protected]>

This patch implements the s390 pci instructions in qemu. This allows to attach qemu pci devices including vfio. This does not mean the devices are functional but at least detection and config/memory space access is working.

---
 target-s390x/Makefile.objs |    2 
 target-s390x/kvm.c         |   52 +++
 target-s390x/pci_ic.c      |  621 +++++++++++++++++++++++++++++++++++++++++++++
 target-s390x/pci_ic.h      |  425 ++++++++++++++++++++++++++++++
 4 files changed, 1099 insertions(+), 1 deletion(-)
target-s390x/Makefile.objs +1 -1
@@ -2,4 +2,4 @@ obj-y += translate.o helper.o cpu.o inte
 obj-y += int_helper.o fpu_helper.o cc_helper.o mem_helper.o misc_helper.o
 obj-y += gdbstub.o
 obj-$(CONFIG_SOFTMMU) += ioinst.o arch_dump.o
-obj-$(CONFIG_KVM) += kvm.o
+obj-$(CONFIG_KVM) += kvm.o pci_ic.o
target-s390x/kvm.c +23 -0
@@ -40,6 +40,7 @@
 #include "exec/gdbstub.h"
 #include "trace.h"
 #include "qapi-event.h"
+#include "pci_ic.h"
 
 /* #define DEBUG_KVM */
 
@@ -56,6 +57,7 @@
 #define IPA0_B2                         0xb200
 #define IPA0_B9                         0xb900
 #define IPA0_EB                         0xeb00
+#define IPA0_E3                         0xe300
 
 #define PRIV_B2_SCLP_CALL               0x20
 #define PRIV_B2_CSCH                    0x30
@@ -76,8 +78,17 @@
 #define PRIV_B2_XSCH                    0x76
 
 #define PRIV_EB_SQBS                    0x8a
+#define PRIV_EB_PCISTB                  0xd0
+#define PRIV_EB_SIC                     0xd1
 
 #define PRIV_B9_EQBS                    0x9c
+#define PRIV_B9_CLP                     0xa0
+#define PRIV_B9_PCISTG                  0xd0
+#define PRIV_B9_PCILG                   0xd2
+#define PRIV_B9_RPCIT                   0xd3
+
+#define PRIV_E3_MPCIFC                  0xd0
+#define PRIV_E3_STPCIFC                 0xd4
 
 #define DIAG_IPL                        0x308
 #define DIAG_KVM_HYPERCALL              0x500
@@ -813,6 +824,18 @@ static int handle_b9(S390CPU *cpu, struc
     int r = 0;
 
     switch (ipa1) {
+    case PRIV_B9_CLP:
+        r = kvm_clp_service_call(cpu, run);
+        break;
+    case PRIV_B9_PCISTG:
+        r = kvm_pcistg_service_call(cpu, run);
+        break;
+    case PRIV_B9_PCILG:
+        r = kvm_pcilg_service_call(cpu, run);
+        break;
+    case PRIV_B9_RPCIT:
+        r = kvm_rpcit_service_call(cpu, run);
+        break;
     case PRIV_B9_EQBS:
         /* just inject exception */
         r = -1;
@@ -831,6 +854,12 @@ static int handle_eb(S390CPU *cpu, struc
     int r = 0;
 
Quoted 16 lines

target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 865

Quoted 10 lines
target-s390x/kvm.c +6 -0
     switch (ipa1) {
+    case PRIV_EB_PCISTB:
+        r = kvm_pcistb_service_call(cpu, run);
+        break;
+    case PRIV_EB_SIC:
+        r = kvm_sic_service_call(cpu, run);
+        break;
     case PRIV_EB_SQBS:
         /* just inject exception */
         r = -1;

I'm not sure, but I think the handler for the eb instructions is wrong:
The second byte of the opcode is encoded in the lowest byte of the ipb field, not the lowest byte of the ipa field (just like with the e3 handler). Did you verify that your handlers get called correctly?

Hi Thomas, you are absolutely right. I already have a patch available for this issue but did not append it to this RFC post (since it is basically a bug fix). To the next posting I will add this patch as well.

Will also fix the remaining issues thx for your review.

Quoted 323 lines
Quoted 13 lines
target-s390x/kvm.c +9 -0
@@ -844,6 +873,26 @@ static int handle_eb(S390CPU *cpu, struc
     return r;
 }
 
+static int handle_e3(S390CPU *cpu, struct kvm_run *run, uint8_t ipa1)
+{
+    int r = 0;
+
+    switch (ipa1) {
+    case PRIV_E3_MPCIFC:
+        r = kvm_mpcifc_service_call(cpu, run);
+        break;
+    case PRIV_E3_STPCIFC:

target-s390x/kvm.c static int handle_eb(S390CPU *cpu, struc · line 894

Quoted 10 lines
target-s390x/kvm.c +10 -0
+        r = kvm_stpcifc_service_call(cpu, run);
+        break;
+    default:
+        r = -1;
+        DPRINTF("KVM: unhandled PRIV: 0xe3%x\n", ipa1);
+        break;
+    }
+
+    return r;
+}

Could you please replace "ipa1" with "ipb1" to avoid confusion here?

Quoted 16 lines
target-s390x/kvm.c +3 -0
 static int handle_hypercall(S390CPU *cpu, struct kvm_run *run)
 {
     CPUS390XState *env = &cpu->env;
@@ -1038,6 +1087,9 @@ static int handle_instruction(S390CPU *c
     case IPA0_EB:
         r = handle_eb(cpu, run, ipa1);
         break;
+    case IPA0_E3:
+        r = handle_e3(cpu, run, run->s390_sieic.ipb & 0xff);
+        break;
     case IPA0_DIAG:
         r = handle_diag(cpu, run, run->s390_sieic.ipb);
         break;
target-s390x/pci_ic.c new file +0 -0
@@ -0,0 +1,621 @@

[...]

target-s390x/pci_ic.c line 308

Quoted 13 lines
target-s390x/pci_ic.c new file +13 -0
+
+int kvm_pcilg_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    CPUS390XState *env = &cpu->env;
+    S390PCIBusDevice *pbdev;
+    uint8_t r1 = (run->s390_sieic.ipb & 0x00f00000) >> 20;
+    uint8_t r2 = (run->s390_sieic.ipb & 0x000f0000) >> 16;
+    PciLgStg *rp;
+    uint64_t offset;
+    uint64_t data;
+
+    cpu_synchronize_state(CPU(cpu));
+    rp = (PciLgStg *)&env->regs[r2];

I think you have to check for r2 to be even here (and inject a specification exception otherwise)

You should also check rp->len for a valid value and inject an operand exception otherwise.

Quoted 1 line

+ offset = env->regs[r2 + 1];

You should also check for a valid offset value here (and inject an operand exception otherwise).

... and while you're at it - it also seems like the pci instructions are priviledged, so you should check for not being in the problem state here, too, just to be sure ;-)

target-s390x/pci_ic.c line 319

Quoted 9 lines
target-s390x/pci_ic.c new file +9 -0
+    pbdev = s390_pci_find_dev_by_fh(rp->fh);
+    if (!pbdev) {
+        DPRINTF("pcilg no pci dev\n");
+        return -EIO;
+    }
+
+    if (rp->pcias < 6) {
+        MemoryRegion *mr = pbdev->pdev->io_regions[rp->pcias].memory;
+        io_mem_read(mr, offset, &data, rp->len);

Does this also work if len != 8 ? I think the data should be put in the rightmost byte positions, with the unused leftmost bytes set to zero... so some more logic might be needed here.

Quoted 11 lines
target-s390x/pci_ic.c new file +11 -0
+    } else if (rp->pcias == 15) {
+        data =  pci_host_config_read_common(
+                   pbdev->pdev, offset, pci_config_size(pbdev->pdev), rp->len);
+
+        switch (rp->len) {
+        case 1:
+            break;
+        case 2:
+            data = cpu_to_le16(data);
+            break;
+        case 4:

target-s390x/pci_ic.c line 340

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+            data = cpu_to_le32(data);
+            break;
+        case 8:
+            data = cpu_to_le64(data);
+            break;
+        default:
+            abort();
+        }
+    } else {
+        DPRINTF("invalid space\n");

Set condition code 1 in this case?

Quoted 9 lines
target-s390x/pci_ic.c new file +9 -0
+    }
+
+    env->regs[r1] = data;
+    setcc(cpu, 0);
+    return 0;
+}
+
+int kvm_pcistg_service_call(S390CPU *cpu, struct kvm_run *run)
+{

target-s390x/pci_ic.c line 359

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    CPUS390XState *env = &cpu->env;
+    uint8_t r1 = (run->s390_sieic.ipb & 0x00f00000) >> 20;
+    uint8_t r2 = (run->s390_sieic.ipb & 0x000f0000) >> 16;
+    PciLgStg *rp;
+    uint64_t offset, data;
+    S390PCIBusDevice *pbdev;
+
+    cpu_synchronize_state(CPU(cpu));
+    rp = (PciLgStg *)&env->regs[r2];
+    offset = env->regs[r2 + 1];

You likely need the same sanity checks here as with the pcilg function.

Quoted 11 lines
target-s390x/pci_ic.c new file +11 -0
+    pbdev = s390_pci_find_dev_by_fh(rp->fh);
+    if (!pbdev) {
+        DPRINTF("pcistg no pci dev\n");
+        return -EIO;
+    }
+
+    data = env->regs[r1];
+
+    if (rp->pcias < 6) {
+        MemoryRegion *mr;
+        if (pbdev->msix_table_bar == rp->pcias &&

target-s390x/pci_ic.c line 381

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+            offset >= pbdev->msix_table_offset &&
+            offset <= pbdev->msix_table_offset +
+                      (pbdev->msix_entries - 1) * PCI_MSIX_ENTRY_SIZE) {
+            offset = offset - pbdev->msix_table_offset;
+            mr = &pbdev->pdev->msix_table_mmio;
+        } else {
+            mr = pbdev->pdev->io_regions[rp->pcias].memory;
+        }
+
+        io_mem_write(mr, offset, data, rp->len);

Same concerns as with the pcilg - this likely only works for len == 8 ?

Quoted 12 lines
target-s390x/pci_ic.c new file +12 -0
+    } else if (rp->pcias == 15) {
+        switch (rp->len) {
+        case 1:
+            break;
+        case 2:
+            data = le16_to_cpu(data);
+            break;
+        case 4:
+            data = le32_to_cpu(data);
+            break;
+        case 8:
+            data = le64_to_cpu(data);

target-s390x/pci_ic.c line 403

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+            break;
+        default:
+            abort();
+        }
+
+        pci_host_config_write_common(pbdev->pdev, offset,
+                                     pci_config_size(pbdev->pdev),
+                                     data, rp->len);
+    } else {
+        DPRINTF("pcistg invalid space\n");

Set condition code 1 in this case?

Quoted 37 lines
target-s390x/pci_ic.c new file +37 -0
+    }
+
+    setcc(cpu, 0);
+    return 0;
+}
+
+static uint64_t guest_io_table_walk(uint64_t guest_iota,
+                                    uint64_t guest_dma_address)
+{
+    uint64_t sto_a, pto_a, px_a;
+    uint64_t sto, pto, pte;
+    uint32_t rtx, sx, px;
+
+    rtx = calc_rtx(guest_dma_address);
+    sx = calc_sx(guest_dma_address);
+    px = calc_px(guest_dma_address);
+
+    sto_a = guest_iota + rtx * sizeof(uint64_t);
+    cpu_physical_memory_rw(sto_a, (uint8_t *)&sto, sizeof(uint64_t), 0);
+    sto = (uint64_t)get_rt_sto(sto);
+
+    pto_a = sto + sx * sizeof(uint64_t);
+    cpu_physical_memory_rw(pto_a, (uint8_t *)&pto, sizeof(uint64_t), 0);
+    pto = (uint64_t)get_st_pto(pto);
+
+    px_a = pto + px * sizeof(uint64_t);
+    cpu_physical_memory_rw(px_a, (uint8_t *)&pte, sizeof(uint64_t), 0);
+
+    return pte;
+}
+
+int kvm_rpcit_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    CPUS390XState *env = &cpu->env;
+    uint8_t r1 = (run->s390_sieic.ipb & 0x00f00000) >> 20;
+    uint8_t r2 = (run->s390_sieic.ipb & 0x000f0000) >> 16;
+    uint32_t fh;

target-s390x/pci_ic.c line 450

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    uint64_t pte;
+    S390PCIBusDevice *pbdev;
+    ram_addr_t size;
+    int flags;
+    IOMMUTLBEntry entry;
+
+    cpu_synchronize_state(CPU(cpu));
+
+    fh = env->regs[r1] >> 32;
+    size = env->regs[r2 + 1];

Missing two checks again:
- r2 should be even
- CPU should not be in problem state

target-s390x/pci_ic.c line 456

Quoted 5 lines
target-s390x/pci_ic.c new file +5 -0
+    pbdev = s390_pci_find_dev_by_fh(fh);
+
+    if (!pbdev) {
+        DPRINTF("rpcit no pci dev\n");
+        return -EIO;

I think it would be better to set condition code 3 instead.

Quoted 36 lines
target-s390x/pci_ic.c new file +36 -0
+    }
+
+    pte = guest_io_table_walk(pbdev->g_iota, env->regs[r2]);
+    flags = pte & ZPCI_PTE_FLAG_MASK;
+    entry.target_as = &address_space_memory;
+    entry.iova = env->regs[r2];
+    entry.translated_addr = pte & ZPCI_PTE_ADDR_MASK;
+    entry.addr_mask = size - 1;
+
+    if (flags & ZPCI_PTE_INVALID) {
+        entry.perm = IOMMU_NONE;
+    } else {
+        entry.perm = IOMMU_RW;
+    }
+
+    memory_region_notify_iommu(pci_device_iommu_address_space(
+                               pbdev->pdev)->root, entry);
+
+    setcc(cpu, 0);
+    return 0;
+}
+
+int kvm_sic_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    DPRINTF("sic\n");
+    return 0;
+}
+
+int kvm_pcistb_service_call(S390CPU *cpu, struct kvm_run *run)
+{
+    CPUS390XState *env = &cpu->env;
+    uint8_t r1 = (run->s390_sieic.ipa & 0x00f0) >> 4;
+    uint8_t r3 = run->s390_sieic.ipa & 0x000f;
+    PciStb *rp;
+    uint64_t gaddr;
+    uint64_t *uaddr, *pu;

target-s390x/pci_ic.c line 502

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    hwaddr len;
+    S390PCIBusDevice *pbdev;
+    MemoryRegion *mr;
+    int i;
+
+    cpu_synchronize_state(CPU(cpu));
+
+    rp = (PciStb *)&env->regs[r1];
+    gaddr = get_base_disp_rsy(cpu, run);
+    len = rp->len;

Not sure, but don't you also have to check the length and offset here for valid ranges?
At least you should check for problem state again.

target-s390x/pci_ic.c line 507

Quoted 4 lines
target-s390x/pci_ic.c new file +4 -0
+    pbdev = s390_pci_find_dev_by_fh(rp->fh);
+    if (!pbdev) {
+        DPRINTF("pcistb no pci dev fh 0x%x\n", rp->fh);
+        return -EIO;

User cc3 instead?

Quoted 5 lines
target-s390x/pci_ic.c new file +5 -0
+    }
+
+    uaddr = cpu_physical_memory_map(gaddr, &len, 0);
+    mr = pbdev->pdev->io_regions[rp->pcias].memory;
+

target-s390x/pci_ic.c line 522

Quoted 10 lines
target-s390x/pci_ic.c new file +10 -0
+    pu = uaddr;
+    for (i = 0; i < rp->len / 8; i++) {
+        io_mem_write(mr, env->regs[r3] + i * 8, *pu, 8);
+        pu++;
+    }
+
+    cpu_physical_memory_unmap(uaddr, len, 0, len);
+    setcc(cpu, 0);
+    return 0;
+}

[...]

Thomas

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

RFC PATCH 6/6 vfio: make vfio run on s390 platform

In reply to [email protected]

From: Frank Blaschka <[email protected]>

Following changes are made because of platform differences:

1) s390 does not support mmap'ing of PCI BARs so we have to go via slow path
2) no intx support
3) no classic MSIX interrupts. The pci hw understands the concept of requesting MSIX irqs but irqs are delivered as s390 adapter irqs. Introduce s390 specific functions for msix notification (slow path) and msi routes (kvm fast path).
4) Use type1 iommu but register only for iommu address space

---
 hw/misc/vfio.c |   24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
hw/misc/vfio.c +24 -0
@@ -40,6 +40,9 @@
 #include "sysemu/kvm.h"
 #include "sysemu/sysemu.h"
 #include "hw/misc/vfio.h"
+#ifdef TARGET_S390X
+#include "hw/s390x/s390-pci-bus.h"
+#endif
 
 /* #define DEBUG_VFIO */
 #ifdef DEBUG_VFIO
@@ -51,7 +54,11 @@
 #endif
 
 /* Extra debugging, trap acceleration paths for more logging */
+#ifdef TARGET_S390X
+#define VFIO_ALLOW_MMAP 0
+#else
 #define VFIO_ALLOW_MMAP 1
+#endif
 #define VFIO_ALLOW_KVM_INTX 1
 #define VFIO_ALLOW_KVM_MSI 1
 #define VFIO_ALLOW_KVM_MSIX 1
@@ -554,6 +561,10 @@ static int vfio_enable_intx(VFIODevice *
     struct vfio_irq_set *irq_set;
     int32_t *pfd;
 
+#ifdef TARGET_S390X
+    return 0;
+#endif
+
     if (!pin) {
         return 0;
     }
@@ -664,7 +675,11 @@ static void vfio_msi_interrupt(void *opa
 #endif
 
     if (vdev->interrupt == VFIO_INT_MSIX) {
+#ifdef TARGET_S390X
+        s390_msix_notify(&vdev->pdev, nr);
+#else
         msix_notify(&vdev->pdev, nr);
+#endif
     } else if (vdev->interrupt == VFIO_INT_MSI) {
         msi_notify(&vdev->pdev, nr);
     } else {
@@ -730,7 +745,11 @@ static void vfio_add_kvm_msi_virq(VFIOMS
         return;
     }
 
+#ifdef TARGET_S390X
+    virq = s390_irqchip_add_msi_route(&vector->vdev->pdev, kvm_state, *msg);
+#else
     virq = kvm_irqchip_add_msi_route(kvm_state, *msg);
+#endif
     if (virq < 0) {
         event_notifier_cleanup(&vector->kvm_interrupt);
         return;
@@ -3702,8 +3721,13 @@ static int vfio_connect_container(VFIOGr
         container->iommu_data.type1.listener = vfio_memory_listener;
         container->iommu_data.release = vfio_listener_release;
 
+#ifdef TARGET_S390X
+        memory_listener_register(&container->iommu_data.type1.listener,
+                                 container->space->as);
+#else
         memory_listener_register(&container->iommu_data.type1.listener,
                                  &address_space_memory);
+#endif
 
         if (container->iommu_data.type1.error) {
             ret = container->iommu_data.type1.error;
#9 lore raw
In reply to [email protected]
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

Quoted 7 lines

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.
- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.
- s390_msix_notify() vs msix_notify() should be abstracted somewhere
else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route
- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Looks like a reasonable first pass, good re-use of vfio code. Thanks,

Alex

#10 lore raw
Quoted 41 lines · Alex Williamson
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

Quoted 7 lines

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Can we emulate MMIO on mmap'ed regions by routing every memory access via the kernel? It'd be slow, but at least make existing VFIO code compatible.

Quoted 6 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.
- s390_msix_notify() vs msix_notify() should be abstracted somewhere
else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Yes, please implement emulated PCI device support first, then do VFIO.

Alex

#11 lore raw
Quoted 43 lines · Alexander Graf
Quoted 41 lines · Alex Williamson
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

Quoted 7 lines

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Quoted 8 lines
Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Can we emulate MMIO on mmap'ed regions by routing every memory access via the kernel? It'd be slow, but at least make existing VFIO code compatible.

Isn't that effectively what we do when we use memory_region_init_io() vs memory_region_init_ram_ptr() or are you suggesting something that can handle the MMIO without bouncing out to QEMU? VFIO is already compatible with regions that cannot be mmap'd, the kernel just needs to report it as such. Thanks,

Alex

#12 lore raw
Quoted 53 lines · Alex Williamson
Quoted 43 lines · Alexander Graf
Quoted 41 lines · Alex Williamson
Quoted 24 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

Quoted 7 lines

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Quoted 8 lines
Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Can we emulate MMIO on mmap'ed regions by routing every memory access via the kernel? It'd be slow, but at least make existing VFIO code compatible.

Quoted 5 lines

Isn't that effectively what we do when we use memory_region_init_io() vs memory_region_init_ram_ptr() or are you suggesting something that can handle the MMIO without bouncing out to QEMU? VFIO is already compatible with regions that cannot be mmap'd, the kernel just needs to report it as such. Thanks,

Ah, cool. I guess I missed that part :). Then all is well.

Alex

#13 lore raw
In reply to Alex Williamson
Quoted 27 lines · Alex Williamson
Quoted 26 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

Quoted 8 lines
Quoted 5 lines

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +2 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX
- in reality currently there is only a PCI network card available
- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

Quoted 5 lines

Looks like a reasonable first pass, good re-use of vfio code. Thanks,

Alex

Thx,

Frank

Quoted 1 line
#14 lore raw
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

The patches in each email are showing up as attachments in my mail client. Is it just me?

Quoted 21 lines
Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +0 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 

drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2 -0
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {

Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.

Quoted 6 lines
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Yep, let's try to do something consistent with the MMAP testing.

Quoted 10 lines
Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Quoted 6 lines

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX

What about MSI (non-X)?

Quoted 1 line

- in reality currently there is only a PCI network card available

On the physical hardware?

Quoted 3 lines

- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.

Quoted 3 lines
Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,

Alex

#15 lore raw
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

The patches in each email are showing up as attachments in my mail client. Is it just me?

Quoted 21 lines
Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +0 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 
Quoted 16 lines

drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2 -0
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {

Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.

Ok, understand this. There is no capability of feature so I will go with the function.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Yep, let's try to do something consistent with the MMAP testing.

Do you mean let the kernel announce this also?

Quoted 10 lines
Quoted 10 lines
Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Quoted 8 lines
Quoted 6 lines

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX

What about MSI (non-X)?

In theory MSI should work also but I have not seen in reality.

Quoted 5 lines
Quoted 2 lines

- in reality currently there is only a PCI network card available

On the physical hardware?

yes

Quoted 10 lines
Quoted 3 lines

- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.

Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,

I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?

Thx Frank

Quoted 3 lines

Alex

#16 lore raw
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

The patches in each email are showing up as attachments in my mail client. Is it just me?

Quoted 21 lines
Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +0 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 
Quoted 16 lines

drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2 -0
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {

Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.

Ok, understand this. There is no capability of feature so I will go with the function.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Yep, let's try to do something consistent with the MMAP testing.

Do you mean let the kernel announce this also?

Yes, the kernel reports a count of 0 in vfio_irq_info when the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.

Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Quoted 8 lines
Quoted 6 lines

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX

What about MSI (non-X)?

In theory MSI should work also but I have not seen in reality.

Quoted 5 lines
Quoted 2 lines

- in reality currently there is only a PCI network card available

On the physical hardware?

yes

Quoted 10 lines
Quoted 3 lines

- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.

Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.

Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

Quoted 7 lines
Quoted 4 lines

I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,

I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?

Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,

Alex

#17 lore raw
Quoted 158 lines · Alex Williamson
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

The patches in each email are showing up as attachments in my mail client. Is it just me?

Quoted 21 lines
Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +0 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 
Quoted 16 lines

drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2 -0
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {

Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.

Ok, understand this. There is no capability of feature so I will go with the function.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Yep, let's try to do something consistent with the MMAP testing.

Do you mean let the kernel announce this also?

Yes, the kernel reports a count of 0 in vfio_irq_info when the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.

Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Quoted 8 lines
Quoted 6 lines

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX

What about MSI (non-X)?

In theory MSI should work also but I have not seen in reality.

Quoted 5 lines
Quoted 2 lines

- in reality currently there is only a PCI network card available

On the physical hardware?

yes

Quoted 10 lines
Quoted 3 lines

- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.

Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.

Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

Quoted 7 lines
Quoted 4 lines

I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,

I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?

Quoted 8 lines

Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,

Do you think this would be sufficient?

@@ -3689,8 +3701,13 @@ static int vfio_connect_container(VFIOGr

         container->iommu_data.type1.listener = vfio_memory_listener;
         container->iommu_data.release = vfio_listener_release;

- memory_listener_register(&container->iommu_data.type1.listener,
- &address_space_memory);
+ if (memory_region_is_iommu(as->root)) {
+ memory_listener_register(&container->iommu_data.type1.listener,
+ container->space->as);
+ } else {
+ memory_listener_register(&container->iommu_data.type1.listener,
+ &address_space_memory);
+ }

         if (container->iommu_data.type1.error) {
             ret = container->iommu_data.type1.error;

If not what else has to be checked? What are the indications to add the memory listener to container address space or to address_space_memory? Thx for your help.

Quoted 3 lines

Alex

#18 lore raw
Quoted 187 lines · Frank Blaschka
Quoted 158 lines · Alex Williamson
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

The patches in each email are showing up as attachments in my mail client. Is it just me?

Quoted 21 lines
Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +0 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 
Quoted 16 lines

drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2 -0
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {

Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.

Ok, understand this. There is no capability of feature so I will go with the function.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Yep, let's try to do something consistent with the MMAP testing.

Do you mean let the kernel announce this also?

Yes, the kernel reports a count of 0 in vfio_irq_info when the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.

Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Quoted 8 lines
Quoted 6 lines

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX

What about MSI (non-X)?

In theory MSI should work also but I have not seen in reality.

Quoted 5 lines
Quoted 2 lines

- in reality currently there is only a PCI network card available

On the physical hardware?

yes

Quoted 10 lines
Quoted 3 lines

- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.

Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.

Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

Quoted 7 lines
Quoted 4 lines

I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,

I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?

Quoted 8 lines

Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,

Do you think this would be sufficient?

(quoted diff) +7 -2
@@ -3689,8 +3701,13 @@ static int vfio_connect_container(VFIOGr
         container->iommu_data.type1.listener = vfio_memory_listener;
         container->iommu_data.release = vfio_listener_release;
 
-        memory_listener_register(&container->iommu_data.type1.listener,
-                                 &address_space_memory);
+        if (memory_region_is_iommu(as->root)) {
+            memory_listener_register(&container->iommu_data.type1.listener,
+                                     container->space->as);
+        } else {
+            memory_listener_register(&container->iommu_data.type1.listener,
+                                     &address_space_memory);
+        }
 
         if (container->iommu_data.type1.error) {
             ret = container->iommu_data.type1.error;
Quoted 3 lines

If not what else has to be checked? What are the indications to add the memory listener to container address space or to address_space_memory? Thx for your help.

Sure, that's what I was asking for, but as I'm looking at is, shouldn't we be able to use container->space->as regardless? It seems like an oversight that address_space_memory wasn't replaced with container->space->as when Alexey added support for multiple address spaces.

The container->space->as value comes from pci_device_iommu_address_space() which walks the PCI bus up from the device looking for an IOMMU address space. If it doesn't find one (likely) it uses address_space_memory. So I suspect if we fix the code, there's no need for any sort of switch. Thanks,

Alex

#19 lore raw
Quoted 198 lines · Alex Williamson
Quoted 187 lines · Frank Blaschka
Quoted 158 lines · Alex Williamson
Quoted 88 lines · Frank Blaschka
Quoted 64 lines · Alex Williamson
Quoted 32 lines · Frank Blaschka
Quoted 31 lines · Alex Williamson
Quoted 30 lines · On Fri, 2014-09-19 at 13:54 +0200, [email protected] wrote:

This set of patches implements a vfio based solution for pci pass-through on the s390 platform. The kernel stuff is pretty much straight forward, but qemu needs more work.

Most interesting patch is:
vfio: make vfio run on s390 platform

I hope Alex & Alex can give me some guidance how to do the changes in an appropriate way. After creating a separate iommmu address space for each attached PCI device I can successfully run the vfio type1 iommu. So If we could extend type1 not registering all guest memory (see patch) I think we do not need a special vfio iommu for s390 for the moment.

The patches implement the base pass-through support. s390 specific virtualization functions are currently not included. This would be a second step after the base support is done.

kernel patches apply to linux-kvm-next

KVM: s390: Enable PCI instructions
iommu: add iommu for s390 platform
vfio: make vfio build on s390

qemu patches apply to qemu-master

s390: Add PCI bus support
s390: implement pci instruction
vfio: make vfio run on s390 platform

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Thx for feedback and review comments

Sending patches as attachments makes it difficult to comment inline.

Sorry, don't understand this. I sent every patch as separate email so you can comment directly on the patch. What do you prefer?

The patches in each email are showing up as attachments in my mail client. Is it just me?

Quoted 21 lines
Quoted 6 lines

2/6
- careful of the namespace as you're changing functions from static and
exporting them
- doesn't seem like functions need to be exported, just non-static to
call from s390-iommu.c

Ok, will change this.

Quoted 4 lines

6/6
- We shouldn't need to globally disable mmap, each VFIO region reports
whether it supports mmap and vfio-pci on s390 should indicate mmap is not supported on the platform.

Yes, this is even better to let the kernel announce a BAR can not be mmap'ed. Checking the kernel code I realized the BARs are valid for mmap'ing but the s390 platform does simply not allow this. So I feal we have to introduce a platform switch in kernel. How about this ...

drivers/vfio/pci/vfio_pci.c +0 -0
@@ -377,9 +377,11 @@ static long vfio_pci_ioctl(void *device_
 
Quoted 16 lines

drivers/vfio/pci/vfio_pci.c static long vfio_pci_ioctl(void *device_ · line 387

Quoted 10 lines
drivers/vfio/pci/vfio_pci.c +2 -0
                        info.flags = VFIO_REGION_INFO_FLAG_READ |
                                     VFIO_REGION_INFO_FLAG_WRITE;
+#ifndef CONFIG_S390
                        if (pci_resource_flags(pdev, info.index) &
                            IORESOURCE_MEM && info.size >= PAGE_SIZE)
                                info.flags |= VFIO_REGION_INFO_FLAG_MMAP;
+#endif
                        break;
                case VFIO_PCI_ROM_REGION_INDEX:
                {

Maybe pull it out into a function. Also, is there some capability or feature we can test rather than just the architecture? I'd prefer it to be excluded because of a platform feature that prevents it rather than the overall architecture itself.

Ok, understand this. There is no capability of feature so I will go with the function.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

- INTx should be done the same way, the interrupt index for INTx should
report 0 count. The current code likely doesn't handle this, but it should be easy to fix.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The current code is fine. Problem is the card reports an interrupt index (PCI_INTERRUPT_PIN) but again the platform does not support INTx at all. So we need a platform switch as well.

Yep, let's try to do something consistent with the MMAP testing.

Do you mean let the kernel announce this also?

Yes, the kernel reports a count of 0 in vfio_irq_info when the interrupt type is not supported. We do this for MSI/X already, but it's assumed that INTx is always present since it's part of what most platforms would consider the minimal feature set.

Quoted 47 lines
Quoted 10 lines
Quoted 10 lines
Quoted 1 line

- s390_msix_notify() vs msix_notify() should be abstracted somewhere

Platform does not have have an apic so there is nothing we could emulate in qemu to make the existing msix_notify() work.

Quoted 2 lines

else. How would an emulated PCI device with MSI-X support work?
- same for add_msi_route

Same here, we have to setup an adapter route due to the fact MSIX notifications are delivered as adapter/thin IRQs on the platform.

Quoted 8 lines
Quoted 6 lines

Any suggestion or idea how a better abstraction could look like?

With all the platform constraints I was not able to find a suitable emulated device. Remember s390:
- does not support IO BARs
- does not support INTx only MSIX

What about MSI (non-X)?

In theory MSI should work also but I have not seen in reality.

Quoted 5 lines
Quoted 2 lines

- in reality currently there is only a PCI network card available

On the physical hardware?

yes

Quoted 10 lines
Quoted 3 lines

- platform does not support fancy I/O like usb or audio :-) So we don't even have kernel (host and guest) support for this kind of devices.

Does that mean you couldn't? What about virtio-net-pci with MSI-X interrupts or emulated xhci with MSI-X interrupts, couldn't those be supported if s390 MSI-X were properly integrated into the QEMU MSI-X API? vfio-pci isn't the right level to be switching between the standard API and the s390 API.

Yes, I also think vfio might not be the best place to switch API. Will try to move s390 specifics to MSI-X level.

Quoted 4 lines
Quoted 3 lines
Quoted 2 lines

- We can probably come up with a better way to determine which address
space to connect to the memory listener.

Any suggestion or idea for that?

Quoted 7 lines
Quoted 4 lines

I imagine you can tell by the address space of the device whether it lives behind an emulated IOMMU or not and therefore pick the closest address space for the notifier, the IOMMU or the system. Thanks,

I do not undertand this in detail, can you elaborate a little bit more on this? Or maybe provide a code snip?

Quoted 8 lines

Well, I'm mostly making things up, but my assumption is that the device appears behind an IOMMU in the guest and by walking through address spaces from the device, we should be able to figure that out and avoid using a platform #ifdef. IOW, it's not s390 that makes us need to use a different address space, it's the guest topology of having an emulated IOMMU for the device, and that's what we should be keying on rather than the arch. Thanks,

Do you think this would be sufficient?

(quoted diff) +7 -2
@@ -3689,8 +3701,13 @@ static int vfio_connect_container(VFIOGr
         container->iommu_data.type1.listener = vfio_memory_listener;
         container->iommu_data.release = vfio_listener_release;
 
-        memory_listener_register(&container->iommu_data.type1.listener,
-                                 &address_space_memory);
+        if (memory_region_is_iommu(as->root)) {
+            memory_listener_register(&container->iommu_data.type1.listener,
+                                     container->space->as);
+        } else {
+            memory_listener_register(&container->iommu_data.type1.listener,
+                                     &address_space_memory);
+        }
 
         if (container->iommu_data.type1.error) {
             ret = container->iommu_data.type1.error;
Quoted 3 lines

If not what else has to be checked? What are the indications to add the memory listener to container address space or to address_space_memory? Thx for your help.

Sure, that's what I was asking for, but as I'm looking at is, shouldn't we be able to use container->space->as regardless? It seems like an oversight that address_space_memory wasn't replaced with container->space->as when Alexey added support for multiple address spaces.

Quoted 6 lines

The container->space->as value comes from pci_device_iommu_address_space() which walks the PCI bus up from the device looking for an IOMMU address space. If it doesn't find one (likely) it uses address_space_memory. So I suspect if we fix the code, there's no need for any sort of switch. Thanks,

This was my impression too, but I was not completely sure about this. Thx for confirmation I will chance this. Now with all that work my changes in vfio are minimal and without any platform #ifdef. I will post the complete patch set again if I have a solution (or at least an idea) for the MSI-X stuff.

Thanks again for your help and patience,

Frank

Quoted 7 lines

Alex

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html