Skip to contentSkip to conversation

KVM: s390: enable zPCI for interpretive execution

PATCH In review Matthew Rosato 6 participants 43 messages Jun 6, 2022 – Jul 8, 2022

Subsystems (10)
Documentation KVM s390 VFIO driver +6 more subsystems
lore
Share & tools
lore

Matthew Rosato

author maintainer
#0 lore raw

v9 cover KVM: s390: enable zPCI for interpretive execution

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci. This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device. Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle). Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' based on discussion in QEMU thread.

Matthew Rosato (21):
s390/sclp: detect the zPCI load/store interpretation facility s390/sclp: detect the AISII facility
s390/sclp: detect the AENI facility
s390/sclp: detect the AISI facility
s390/airq: pass more TPI info to airq handlers
s390/airq: allow for airq structure that uses an input vector s390/pci: externalize the SIC operation controls and routine s390/pci: stash associated GISA designation
s390/pci: stash dtsm and maxstbl
vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM
KVM: s390: pci: add basic kvm_zdev structure
KVM: s390: pci: do initial setup for AEN interpretation
KVM: s390: pci: enable host forwarding of Adapter Event Notifications
KVM: s390: mechanism to enable guest zPCI Interpretation
KVM: s390: pci: provide routines for enabling/disabling interrupt
forwarding
KVM: s390: pci: add routines to start/stop interpretive execution vfio-pci/zdev: add open/close device hooks
vfio-pci/zdev: add function handle to clp base capability vfio-pci/zdev: different maxstbl for interpreted devices
KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
MAINTAINERS: additional files related kvm s390 pci passthrough

 Documentation/virt/kvm/api.rst   |  47 +++
 MAINTAINERS                      |   1 +
 arch/s390/include/asm/airq.h     |   7 +-
 arch/s390/include/asm/kvm_host.h |  23 ++
 arch/s390/include/asm/pci.h      |  11 +
 arch/s390/include/asm/pci_clp.h  |   9 +-
 arch/s390/include/asm/pci_insn.h |  29 +-
 arch/s390/include/asm/sclp.h     |   4 +
 arch/s390/include/asm/tpi.h      |  13 +
 arch/s390/kvm/Makefile           |   1 +
 arch/s390/kvm/interrupt.c        |  96 ++++-
 arch/s390/kvm/kvm-s390.c         |  83 +++-
 arch/s390/kvm/kvm-s390.h         |  10 +
 arch/s390/kvm/pci.c              | 690 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |  88 ++++
 arch/s390/pci/pci.c              |  16 +
 arch/s390/pci/pci_clp.c          |   7 +
 arch/s390/pci/pci_insn.c         |   4 +-
 arch/s390/pci/pci_irq.c          |  48 ++-
 drivers/s390/char/sclp_early.c   |   4 +
 drivers/s390/cio/airq.c          |  12 +-
 drivers/s390/cio/qdio_thinint.c  |   6 +-
 drivers/s390/crypto/ap_bus.c     |   9 +-
 drivers/s390/virtio/virtio_ccw.c |   6 +-
 drivers/vfio/pci/Kconfig         |  11 +
 drivers/vfio/pci/Makefile        |   2 +-
 drivers/vfio/pci/vfio_pci_core.c |  10 +-
 drivers/vfio/pci/vfio_pci_zdev.c |  35 +-
 include/linux/sched/user.h       |   3 +-
 include/linux/vfio_pci_core.h    |  12 +-
 include/uapi/linux/kvm.h         |  31 ++
 include/uapi/linux/vfio_zdev.h   |   7 +
 32 files changed, 1279 insertions(+), 56 deletions(-)
 create mode 100644 arch/s390/kvm/pci.c
 create mode 100644 arch/s390/kvm/pci.h
Signature · 1 line
2.27.0
author maintainer
#1 lore raw

PATCH v9 1/21 s390/sclp: detect the zPCI load/store interpretation facility

Detect the zPCI Load/Store Interpretation facility.

---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)
arch/s390/include/asm/sclp.h +1 -0
index 236b34b75ddb..68ef0ddf6400 100644
@@ -88,6 +88,7 @@ struct sclp_info {
 	unsigned char has_sipl : 1;
 	unsigned char has_dirq : 1;
 	unsigned char has_iplcc : 1;
+	unsigned char has_zpci_lsi : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
drivers/s390/char/sclp_early.c +1 -0
index dd313ff57df3..d5671c03d5cc 100644
@@ -45,6 +45,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_gisaf = !!(sccb->fac118 & 0x08);
 	sclp.has_hvs = !!(sccb->fac119 & 0x80);
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
+	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
 	if (sccb->fac91 & 0x40)
Signature · 1 line
2.27.0
author maintainer
#2 lore raw

PATCH v9 2/21 s390/sclp: detect the AISII facility

In reply to Matthew Rosato

Detect the Adapter Interruption Source ID Interpretation facility.

---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)
arch/s390/include/asm/sclp.h +1 -0
index 68ef0ddf6400..d05fbffa7692 100644
@@ -89,6 +89,7 @@ struct sclp_info {
 	unsigned char has_dirq : 1;
 	unsigned char has_iplcc : 1;
 	unsigned char has_zpci_lsi : 1;
+	unsigned char has_aisii : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
drivers/s390/char/sclp_early.c +1 -0
index d5671c03d5cc..39c9bd7e1c26 100644
@@ -45,6 +45,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_gisaf = !!(sccb->fac118 & 0x08);
 	sclp.has_hvs = !!(sccb->fac119 & 0x80);
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
+	sclp.has_aisii = !!(sccb->fac118 & 0x40);
 	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
Signature · 1 line
2.27.0
author maintainer
#3 lore raw

PATCH v9 3/21 s390/sclp: detect the AENI facility

In reply to Matthew Rosato

Detect the Adapter Event Notification Interpretation facility.

---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)
arch/s390/include/asm/sclp.h +1 -0
index d05fbffa7692..b0e67414c492 100644
@@ -90,6 +90,7 @@ struct sclp_info {
 	unsigned char has_iplcc : 1;
 	unsigned char has_zpci_lsi : 1;
 	unsigned char has_aisii : 1;
+	unsigned char has_aeni : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
drivers/s390/char/sclp_early.c +1 -0
index 39c9bd7e1c26..aefdf0b17dbe 100644
@@ -46,6 +46,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_hvs = !!(sccb->fac119 & 0x80);
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
 	sclp.has_aisii = !!(sccb->fac118 & 0x40);
+	sclp.has_aeni = !!(sccb->fac118 & 0x20);
 	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
Signature · 1 line
2.27.0
author maintainer
#4 lore raw

PATCH v9 4/21 s390/sclp: detect the AISI facility

In reply to Matthew Rosato

Detect the Adapter Interruption Suppression Interpretation facility.

---
 arch/s390/include/asm/sclp.h   | 1 +
 drivers/s390/char/sclp_early.c | 1 +
 2 files changed, 2 insertions(+)
arch/s390/include/asm/sclp.h +1 -0
index b0e67414c492..addefe8ccdba 100644
@@ -91,6 +91,7 @@ struct sclp_info {
 	unsigned char has_zpci_lsi : 1;
 	unsigned char has_aisii : 1;
 	unsigned char has_aeni : 1;
+	unsigned char has_aisi : 1;
 	unsigned int ibc;
 	unsigned int mtid;
 	unsigned int mtid_cp;
drivers/s390/char/sclp_early.c +1 -0
index aefdf0b17dbe..d15b0d541de3 100644
@@ -47,6 +47,7 @@ static void __init sclp_early_facilities_detect(void)
 	sclp.has_kss = !!(sccb->fac98 & 0x01);
 	sclp.has_aisii = !!(sccb->fac118 & 0x40);
 	sclp.has_aeni = !!(sccb->fac118 & 0x20);
+	sclp.has_aisi = !!(sccb->fac118 & 0x10);
 	sclp.has_zpci_lsi = !!(sccb->fac118 & 0x01);
 	if (sccb->fac85 & 0x02)
 		S390_lowcore.machine_flags |= MACHINE_FLAG_ESOP;
Signature · 1 line
2.27.0
author maintainer
#5 lore raw

PATCH v9 5/21 s390/airq: pass more TPI info to airq handlers

In reply to Matthew Rosato

A subsequent patch will introduce an airq handler that requires additional TPI information beyond directed vs floating, so pass the entire tpi_info structure via the handler. Only pci actually uses this information today, for the other airq handlers this is effectively a no-op.

---
 arch/s390/include/asm/airq.h     | 3 ++-
 arch/s390/kvm/interrupt.c        | 4 +++-
 arch/s390/pci/pci_irq.c          | 9 +++++++--
 drivers/s390/cio/airq.c          | 2 +-
 drivers/s390/cio/qdio_thinint.c  | 6 ++++--
 drivers/s390/crypto/ap_bus.c     | 9 ++++++---
 drivers/s390/virtio/virtio_ccw.c | 4 +++-
 7 files changed, 26 insertions(+), 11 deletions(-)
arch/s390/include/asm/airq.h +2 -1
index 01936fdfaddb..7918a7d09028 100644
@@ -12,10 +12,11 @@
 
 #include <linux/bit_spinlock.h>
 #include <linux/dma-mapping.h>
+#include <asm/tpi.h>
 
 struct airq_struct {
 	struct hlist_node list;		/* Handler queueing. */
-	void (*handler)(struct airq_struct *airq, bool floating);
+	void (*handler)(struct airq_struct *airq, struct tpi_info *tpi_info);
 	u8 *lsi_ptr;			/* Local-Summary-Indicator pointer */
 	u8 lsi_mask;			/* Local-Summary-Indicator mask */
 	u8 isc;				/* Interrupt-subclass */
arch/s390/kvm/interrupt.c +3 -1
index af96dc0549a4..17ff475157d8 100644
@@ -28,6 +28,7 @@
 #include <asm/switch_to.h>
 #include <asm/nmi.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
 #include "trace-s390.h"
@@ -3311,7 +3312,8 @@ int kvm_s390_gisc_unregister(struct kvm *kvm, u32 gisc)
 }
 EXPORT_SYMBOL_GPL(kvm_s390_gisc_unregister);
 
-static void gib_alert_irq_handler(struct airq_struct *airq, bool floating)
+static void gib_alert_irq_handler(struct airq_struct *airq,
+				  struct tpi_info *tpi_info)
 {
 	inc_irq_stat(IRQIO_GAL);
 	process_gib_alert_list();
arch/s390/pci/pci_irq.c +7 -2
index 500cd2dbdf53..b805c75252ed 100644
@@ -11,6 +11,7 @@
 
 #include <asm/isc.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 
 static enum {FLOATING, DIRECTED} irq_delivery;
 
@@ -216,8 +217,11 @@ static void zpci_handle_fallback_irq(void)
 	}
 }
 
-static void zpci_directed_irq_handler(struct airq_struct *airq, bool floating)
+static void zpci_directed_irq_handler(struct airq_struct *airq,
+				      struct tpi_info *tpi_info)
 {
+	bool floating = !tpi_info->directed_irq;
+
 	if (floating) {
 		inc_irq_stat(IRQIO_PCF);
 		zpci_handle_fallback_irq();
@@ -227,7 +231,8 @@ static void zpci_directed_irq_handler(struct airq_struct *airq, bool floating)
 	}
 }
 
-static void zpci_floating_irq_handler(struct airq_struct *airq, bool floating)
+static void zpci_floating_irq_handler(struct airq_struct *airq,
+				      struct tpi_info *tpi_info)
 {
 	unsigned long si, ai;
 	struct airq_iv *aibv;
drivers/s390/cio/airq.c +1 -1
index c0ed364bf446..230eb5b5b64e 100644
@@ -99,7 +99,7 @@ static irqreturn_t do_airq_interrupt(int irq, void *dummy)
 	rcu_read_lock();
 	hlist_for_each_entry_rcu(airq, head, list)
 		if ((*airq->lsi_ptr & airq->lsi_mask) != 0)
-			airq->handler(airq, !tpi_info->directed_irq);
+			airq->handler(airq, tpi_info);
 	rcu_read_unlock();
 
 	return IRQ_HANDLED;
drivers/s390/cio/qdio_thinint.c +4 -2
index 8e09bf3a2fcd..9b9335dd06db 100644
@@ -15,6 +15,7 @@
 #include <asm/qdio.h>
 #include <asm/airq.h>
 #include <asm/isc.h>
+#include <asm/tpi.h>
 
 #include "cio.h"
 #include "ioasm.h"
@@ -93,9 +94,10 @@ static inline u32 clear_shared_ind(void)
 /**
  * tiqdio_thinint_handler - thin interrupt handler for qdio
  * @airq: pointer to adapter interrupt descriptor
- * @floating: flag to recognize floating vs. directed interrupts (unused)
+ * @tpi_info: interrupt information (e.g. floating vs directed -- unused)
  */
-static void tiqdio_thinint_handler(struct airq_struct *airq, bool floating)
+static void tiqdio_thinint_handler(struct airq_struct *airq,
+				   struct tpi_info *tpi_info)
 {
 	u64 irq_time = S390_lowcore.int_clock;
 	u32 si_used = clear_shared_ind();
drivers/s390/crypto/ap_bus.c +6 -3
index 5c13d2079d96..1b85f21c151b 100644
@@ -27,6 +27,7 @@
 #include <linux/kthread.h>
 #include <linux/mutex.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 #include <linux/atomic.h>
 #include <asm/isc.h>
 #include <linux/hrtimer.h>
@@ -131,7 +132,8 @@ static int ap_max_adapter_id = 63;
 static struct bus_type ap_bus_type;
 
 /* Adapter interrupt definitions */
-static void ap_interrupt_handler(struct airq_struct *airq, bool floating);
+static void ap_interrupt_handler(struct airq_struct *airq,
+				 struct tpi_info *tpi_info);
 
 static bool ap_irq_flag;
 
@@ -452,9 +454,10 @@ static enum hrtimer_restart ap_poll_timeout(struct hrtimer *unused)
 /**
  * ap_interrupt_handler() - Schedule ap_tasklet on interrupt
  * @airq: pointer to adapter interrupt descriptor
- * @floating: ignored
+ * @tpi_info: ignored
  */
-static void ap_interrupt_handler(struct airq_struct *airq, bool floating)
+static void ap_interrupt_handler(struct airq_struct *airq,
+				 struct tpi_info *tpi_info)
 {
 	inc_irq_stat(IRQIO_APB);
 	tasklet_schedule(&ap_tasklet);
drivers/s390/virtio/virtio_ccw.c +3 -1
index 97e51c34e6cf..58cca50996ee 100644
@@ -33,6 +33,7 @@
 #include <asm/virtio-ccw.h>
 #include <asm/isc.h>
 #include <asm/airq.h>
+#include <asm/tpi.h>
 
 /*
  * virtio related functions
@@ -204,7 +205,8 @@ static void drop_airq_indicator(struct virtqueue *vq, struct airq_info *info)
 	write_unlock_irqrestore(&info->lock, flags);
 }
 
-static void virtio_airq_handler(struct airq_struct *airq, bool floating)
+static void virtio_airq_handler(struct airq_struct *airq,
+				struct tpi_info *tpi_info)
 {
 	struct airq_info *info = container_of(airq, struct airq_info, airq);
 	unsigned long ai;
Signature · 1 line
2.27.0
author maintainer
#6 lore raw

PATCH v9 6/21 s390/airq: allow for airq structure that uses an input vector

In reply to Matthew Rosato

When doing device passthrough where interrupts are being forwarded from host to guest, we wish to use a pinned section of guest memory as the vector (the same memory used by the guest as the vector). To accomplish this, add a new parameter for airq_iv_create which allows passing an existing vector to be used instead of allocating a new one. The caller is responsible for ensuring the vector is pinned in memory as well as for unpinning the memory when the vector is no longer needed.

A subsequent patch will use this new parameter for zPCI interpretation.

---
 arch/s390/include/asm/airq.h     |  4 +++-
 arch/s390/pci/pci_irq.c          |  8 ++++----
 drivers/s390/cio/airq.c          | 10 +++++++---
 drivers/s390/virtio/virtio_ccw.c |  2 +-
 4 files changed, 15 insertions(+), 9 deletions(-)
arch/s390/include/asm/airq.h +3 -1
index 7918a7d09028..e82e5626e139 100644
@@ -47,8 +47,10 @@ struct airq_iv {
 #define AIRQ_IV_PTR		4	/* Allocate the ptr array */
 #define AIRQ_IV_DATA		8	/* Allocate the data array */
 #define AIRQ_IV_CACHELINE	16	/* Cacheline alignment for the vector */
+#define AIRQ_IV_GUESTVEC	32	/* Vector is a pinned guest page */
 
-struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags);
+struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags,
+			       unsigned long *vec);
 void airq_iv_release(struct airq_iv *iv);
 unsigned long airq_iv_alloc(struct airq_iv *iv, unsigned long num);
 void airq_iv_free(struct airq_iv *iv, unsigned long bit, unsigned long num);
arch/s390/pci/pci_irq.c +4 -4
index b805c75252ed..87c7d121c255 100644
@@ -296,7 +296,7 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
 		zdev->aisb = bit;
 
 		/* Create adapter interrupt vector */
-		zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | AIRQ_IV_BITLOCK);
+		zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA | AIRQ_IV_BITLOCK, NULL);
 		if (!zdev->aibv)
 			return -ENOMEM;
 
@@ -419,7 +419,7 @@ static int __init zpci_directed_irq_init(void)
 	union zpci_sic_iib iib = {{0}};
 	unsigned int cpu;
 
-	zpci_sbv = airq_iv_create(num_possible_cpus(), 0);
+	zpci_sbv = airq_iv_create(num_possible_cpus(), 0, NULL);
 	if (!zpci_sbv)
 		return -ENOMEM;
 
@@ -441,7 +441,7 @@ static int __init zpci_directed_irq_init(void)
 		zpci_ibv[cpu] = airq_iv_create(cache_line_size() * BITS_PER_BYTE,
 					       AIRQ_IV_DATA |
 					       AIRQ_IV_CACHELINE |
-					       (!cpu ? AIRQ_IV_ALLOC : 0));
+					       (!cpu ? AIRQ_IV_ALLOC : 0), NULL);
 		if (!zpci_ibv[cpu])
 			return -ENOMEM;
 	}
@@ -458,7 +458,7 @@ static int __init zpci_floating_irq_init(void)
 	if (!zpci_ibv)
 		return -ENOMEM;
 
-	zpci_sbv = airq_iv_create(ZPCI_NR_DEVICES, AIRQ_IV_ALLOC);
+	zpci_sbv = airq_iv_create(ZPCI_NR_DEVICES, AIRQ_IV_ALLOC, NULL);
 	if (!zpci_sbv)
 		goto out_free;
 
drivers/s390/cio/airq.c +7 -3
index 230eb5b5b64e..34967e67249e 100644
@@ -122,10 +122,12 @@ static inline unsigned long iv_size(unsigned long bits)
  * airq_iv_create - create an interrupt vector
  * @bits: number of bits in the interrupt vector
  * @flags: allocation flags
+ * @vec: pointer to pinned guest memory if AIRQ_IV_GUESTVEC
  *
  * Returns a pointer to an interrupt vector structure
  */
-struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags)
+struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags,
+			       unsigned long *vec)
 {
 	struct airq_iv *iv;
 	unsigned long size;
@@ -146,6 +148,8 @@ struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags)
 					     &iv->vector_dma);
 		if (!iv->vector)
 			goto out_free;
+	} else if (flags & AIRQ_IV_GUESTVEC) {
+		iv->vector = vec;
 	} else {
 		iv->vector = cio_dma_zalloc(size);
 		if (!iv->vector)
@@ -185,7 +189,7 @@ struct airq_iv *airq_iv_create(unsigned long bits, unsigned long flags)
 	kfree(iv->avail);
 	if (iv->flags & AIRQ_IV_CACHELINE && iv->vector)
 		dma_pool_free(airq_iv_cache, iv->vector, iv->vector_dma);
-	else
+	else if (!(iv->flags & AIRQ_IV_GUESTVEC))
 		cio_dma_free(iv->vector, size);
 	kfree(iv);
 out:
@@ -204,7 +208,7 @@ void airq_iv_release(struct airq_iv *iv)
 	kfree(iv->bitlock);
 	if (iv->flags & AIRQ_IV_CACHELINE)
 		dma_pool_free(airq_iv_cache, iv->vector, iv->vector_dma);
-	else
+	else if (!(iv->flags & AIRQ_IV_GUESTVEC))
 		cio_dma_free(iv->vector, iv_size(iv->bits));
 	kfree(iv->avail);
 	kfree(iv);
drivers/s390/virtio/virtio_ccw.c +1 -1
index 58cca50996ee..51bef226d33f 100644
@@ -242,7 +242,7 @@ static struct airq_info *new_airq_info(int index)
 		return NULL;
 	rwlock_init(&info->lock);
 	info->aiv = airq_iv_create(VIRTIO_IV_BITS, AIRQ_IV_ALLOC | AIRQ_IV_PTR
-				   | AIRQ_IV_CACHELINE);
+				   | AIRQ_IV_CACHELINE, NULL);
 	if (!info->aiv) {
 		kfree(info);
 		return NULL;
Signature · 1 line
2.27.0
author maintainer
#7 lore raw

PATCH v9 7/21 s390/pci: externalize the SIC operation controls and routine

In reply to Matthew Rosato

A subsequent patch will be issuing SIC from KVM -- export the necessary routine and make the operation control definitions available from a header. Because the routine will now be exported, let's rename __zpci_set_irq_ctrl to zpci_set_irq_ctrl and get rid of the zero'd iib wrapper function of the same name.

---
 arch/s390/include/asm/pci_insn.h | 17 +++++++++--------
 arch/s390/pci/pci_insn.c         |  3 ++-
 arch/s390/pci/pci_irq.c          | 26 ++++++++++++--------------
 3 files changed, 23 insertions(+), 23 deletions(-)
arch/s390/include/asm/pci_insn.h +9 -8
index 61cf9531f68f..5331082fa516 100644
@@ -98,6 +98,14 @@ struct zpci_fib {
 	u32 gd;
 } __packed __aligned(8);
 
+/* Set Interruption Controls Operation Controls  */
+#define	SIC_IRQ_MODE_ALL		0
+#define	SIC_IRQ_MODE_SINGLE		1
+#define	SIC_IRQ_MODE_DIRECT		4
+#define	SIC_IRQ_MODE_D_ALL		16
+#define	SIC_IRQ_MODE_D_SINGLE		17
+#define	SIC_IRQ_MODE_SET_CPU		18
+
 /* directed interruption information block */
 struct zpci_diib {
 	u32 : 1;
@@ -134,13 +142,6 @@ int __zpci_store(u64 data, u64 req, u64 offset);
 int zpci_store(const volatile void __iomem *addr, u64 data, unsigned long len);
 int __zpci_store_block(const u64 *data, u64 req, u64 offset);
 void zpci_barrier(void);
-int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
-
-static inline int zpci_set_irq_ctrl(u16 ctl, u8 isc)
-{
-	union zpci_sic_iib iib = {{0}};
-
-	return __zpci_set_irq_ctrl(ctl, isc, &iib);
-}
+int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib);
 
 #endif
arch/s390/pci/pci_insn.c +2 -1
index 1a822b7799f8..5798178aec9d 100644
@@ -138,7 +138,7 @@ int zpci_refresh_trans(u64 fn, u64 addr, u64 range)
 }
 
 /* Set Interruption Controls */
-int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
+int zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
 {
 	if (!test_facility(72))
 		return -EIO;
@@ -149,6 +149,7 @@ int __zpci_set_irq_ctrl(u16 ctl, u8 isc, union zpci_sic_iib *iib)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(zpci_set_irq_ctrl);
 
 /* PCI Load */
 static inline int ____pcilg(u64 *data, u64 req, u64 offset, u8 *status)
arch/s390/pci/pci_irq.c +12 -14
index 87c7d121c255..f2b3145b6697 100644
@@ -15,13 +15,6 @@
 
 static enum {FLOATING, DIRECTED} irq_delivery;
 
-#define	SIC_IRQ_MODE_ALL		0
-#define	SIC_IRQ_MODE_SINGLE		1
-#define	SIC_IRQ_MODE_DIRECT		4
-#define	SIC_IRQ_MODE_D_ALL		16
-#define	SIC_IRQ_MODE_D_SINGLE		17
-#define	SIC_IRQ_MODE_SET_CPU		18
-
 /*
  * summary bit vector
  * FLOATING - summary bit per function
@@ -154,6 +147,7 @@ static struct irq_chip zpci_irq_chip = {
 static void zpci_handle_cpu_local_irq(bool rescan)
 {
 	struct airq_iv *dibv = zpci_ibv[smp_processor_id()];
+	union zpci_sic_iib iib = {{0}};
 	unsigned long bit;
 	int irqs_on = 0;
 
@@ -165,7 +159,7 @@ static void zpci_handle_cpu_local_irq(bool rescan)
 				/* End of second scan with interrupts on. */
 				break;
 			/* First scan complete, reenable interrupts. */
-			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC))
+			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC, &iib))
 				break;
 			bit = 0;
 			continue;
@@ -193,6 +187,7 @@ static void zpci_handle_remote_irq(void *data)
 static void zpci_handle_fallback_irq(void)
 {
 	struct cpu_irq_data *cpu_data;
+	union zpci_sic_iib iib = {{0}};
 	unsigned long cpu;
 	int irqs_on = 0;
 
@@ -203,7 +198,7 @@ static void zpci_handle_fallback_irq(void)
 				/* End of second scan with interrupts on. */
 				break;
 			/* First scan complete, reenable interrupts. */
-			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC))
+			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC, &iib))
 				break;
 			cpu = 0;
 			continue;
@@ -234,6 +229,7 @@ static void zpci_directed_irq_handler(struct airq_struct *airq,
 static void zpci_floating_irq_handler(struct airq_struct *airq,
 				      struct tpi_info *tpi_info)
 {
+	union zpci_sic_iib iib = {{0}};
 	unsigned long si, ai;
 	struct airq_iv *aibv;
 	int irqs_on = 0;
@@ -247,7 +243,7 @@ static void zpci_floating_irq_handler(struct airq_struct *airq,
 				/* End of second scan with interrupts on. */
 				break;
 			/* First scan complete, reenable interrupts. */
-			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC))
+			if (zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC, &iib))
 				break;
 			si = 0;
 			continue;
@@ -407,11 +403,12 @@ static struct airq_struct zpci_airq = {
 static void __init cpu_enable_directed_irq(void *unused)
 {
 	union zpci_sic_iib iib = {{0}};
+	union zpci_sic_iib ziib = {{0}};
 
 	iib.cdiib.dibv_addr = (u64) zpci_ibv[smp_processor_id()]->vector;
 
-	__zpci_set_irq_ctrl(SIC_IRQ_MODE_SET_CPU, 0, &iib);
-	zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_SET_CPU, 0, &iib);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC, &ziib);
 }
 
 static int __init zpci_directed_irq_init(void)
@@ -426,7 +423,7 @@ static int __init zpci_directed_irq_init(void)
 	iib.diib.isc = PCI_ISC;
 	iib.diib.nr_cpus = num_possible_cpus();
 	iib.diib.disb_addr = virt_to_phys(zpci_sbv->vector);
-	__zpci_set_irq_ctrl(SIC_IRQ_MODE_DIRECT, 0, &iib);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_DIRECT, 0, &iib);
 
 	zpci_ibv = kcalloc(num_possible_cpus(), sizeof(*zpci_ibv),
 			   GFP_KERNEL);
@@ -471,6 +468,7 @@ static int __init zpci_floating_irq_init(void)
 
 int __init zpci_irq_init(void)
 {
+	union zpci_sic_iib iib = {{0}};
 	int rc;
 
 	irq_delivery = sclp.has_dirq ? DIRECTED : FLOATING;
@@ -502,7 +500,7 @@ int __init zpci_irq_init(void)
 	 * Enable floating IRQs (with suppression after one IRQ). When using
 	 * directed IRQs this enables the fallback path.
 	 */
-	zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC);
+	zpci_set_irq_ctrl(SIC_IRQ_MODE_SINGLE, PCI_ISC, &iib);
 
 	return 0;
 out_airq:
Signature · 1 line
2.27.0
author maintainer
#8 lore raw

PATCH v9 8/21 s390/pci: stash associated GISA designation

In reply to Matthew Rosato

For passthrough devices, we will need to know the GISA designation of the guest if interpretation facilities are to be used. Setup to stash this in the zdev and set a default of 0 (no GISA designation) for now; a subsequent patch will set a valid GISA designation for passthrough devices. Also, extend mpcific routines to specify this stashed designation as part of the mpcific command.

---
 arch/s390/include/asm/pci.h     | 1 +
 arch/s390/include/asm/pci_clp.h | 3 ++-
 arch/s390/pci/pci.c             | 6 ++++++
 arch/s390/pci/pci_clp.c         | 5 +++++
 arch/s390/pci/pci_irq.c         | 5 +++++
 5 files changed, 19 insertions(+), 1 deletion(-)
arch/s390/include/asm/pci.h +1 -0
index fdb9745ee998..42a4a312a6dd 100644
@@ -123,6 +123,7 @@ struct zpci_dev {
 	enum zpci_state state;
 	u32		fid;		/* function ID, used by sclp */
 	u32		fh;		/* function handle, used by insn's */
+	u32		gisa;		/* GISA designation for passthrough */
 	u16		vfn;		/* virtual function number */
 	u16		pchid;		/* physical channel ID */
 	u8		pfgid;		/* function group ID */
arch/s390/include/asm/pci_clp.h +2 -1
index 1f4b666e85ee..f3286bc5ba6e 100644
@@ -173,7 +173,8 @@ struct clp_req_set_pci {
 	u16 reserved2;
 	u8 oc;				/* operation controls */
 	u8 ndas;			/* number of dma spaces */
-	u64 reserved3;
+	u32 reserved3;
+	u32 gisa;			/* GISA designation */
 } __packed;
 
 /* Set PCI function response */
arch/s390/pci/pci.c +6 -0
index bc980fd313d5..c8b9d866434c 100644
@@ -120,6 +120,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
 	fib.pba = base;
 	fib.pal = limit;
 	fib.iota = iota | ZPCI_IOTA_RTTO_FLAG;
+	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc)
 		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
@@ -133,6 +134,8 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
 	struct zpci_fib fib = {0};
 	u8 cc, status;
 
+	fib.gd = zdev->gisa;
+
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc)
 		zpci_dbg(3, "unreg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
@@ -160,6 +163,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev)
 	atomic64_set(&zdev->unmapped_pages, 0);
 
 	fib.fmb_addr = virt_to_phys(zdev->fmb);
+	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc) {
 		kmem_cache_free(zdev_fmb_cache, zdev->fmb);
@@ -178,6 +182,8 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev)
 	if (!zdev->fmb)
 		return -EINVAL;
 
+	fib.gd = zdev->gisa;
+
 	/* Function measurement is disabled if fmb address is zero */
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc == 3) /* Function already gone. */
arch/s390/pci/pci_clp.c +5 -0
index 375e0a5120bc..d058c83467ef 100644
@@ -229,12 +229,16 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
 {
 	struct clp_req_rsp_set_pci *rrb;
 	int rc, retries = 100;
+	u32 gisa = 0;
 
 	*fh = 0;
 	rrb = clp_alloc_block(GFP_KERNEL);
 	if (!rrb)
 		return -ENOMEM;
 
+	if (command != CLP_SET_DISABLE_PCI_FN)
+		gisa = zdev->gisa;
+
 	do {
 		memset(rrb, 0, sizeof(*rrb));
 		rrb->request.hdr.len = sizeof(rrb->request);
@@ -243,6 +247,7 @@ static int clp_set_pci_fn(struct zpci_dev *zdev, u32 *fh, u8 nr_dma_as, u8 comma
 		rrb->request.fh = zdev->fh;
 		rrb->request.oc = command;
 		rrb->request.ndas = nr_dma_as;
+		rrb->request.gisa = gisa;
 
 		rc = clp_req(rrb, CLP_LPS_PCI);
 		if (rrb->response.hdr.rsp == CLP_RC_SETPCIFN_BUSY) {
arch/s390/pci/pci_irq.c +5 -0
index f2b3145b6697..a2b42a63a53b 100644
@@ -43,6 +43,7 @@ static int zpci_set_airq(struct zpci_dev *zdev)
 	fib.fmt0.aibvo = 0;	/* each zdev has its own interrupt vector */
 	fib.fmt0.aisb = virt_to_phys(zpci_sbv->vector) + (zdev->aisb / 64) * 8;
 	fib.fmt0.aisbo = zdev->aisb & 63;
+	fib.gd = zdev->gisa;
 
 	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
 }
@@ -54,6 +55,8 @@ static int zpci_clear_airq(struct zpci_dev *zdev)
 	struct zpci_fib fib = {0};
 	u8 cc, status;
 
+	fib.gd = zdev->gisa;
+
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc == 3 || (cc == 1 && status == 24))
 		/* Function already gone or IRQs already deregistered. */
@@ -72,6 +75,7 @@ static int zpci_set_directed_irq(struct zpci_dev *zdev)
 	fib.fmt = 1;
 	fib.fmt1.noi = zdev->msi_nr_irqs;
 	fib.fmt1.dibvo = zdev->msi_first_bit;
+	fib.gd = zdev->gisa;
 
 	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
 }
@@ -84,6 +88,7 @@ static int zpci_clear_directed_irq(struct zpci_dev *zdev)
 	u8 cc, status;
 
 	fib.fmt = 1;
+	fib.gd = zdev->gisa;
 	cc = zpci_mod_fc(req, &fib, &status);
 	if (cc == 3 || (cc == 1 && status == 24))
 		/* Function already gone or IRQs already deregistered. */
Signature · 1 line
2.27.0
author maintainer
#9 lore raw

PATCH v9 9/21 s390/pci: stash dtsm and maxstbl

In reply to Matthew Rosato

Store information about what IOAT designation types are supported by underlying hardware as well as the largest store block size allowed. These values will be needed by passthrough.

---
 arch/s390/include/asm/pci.h     | 2 ++
 arch/s390/include/asm/pci_clp.h | 6 ++++--
 arch/s390/pci/pci_clp.c         | 2 ++
 3 files changed, 8 insertions(+), 2 deletions(-)
arch/s390/include/asm/pci.h +2 -0
index 42a4a312a6dd..4c5b8fbc2079 100644
@@ -126,9 +126,11 @@ struct zpci_dev {
 	u32		gisa;		/* GISA designation for passthrough */
 	u16		vfn;		/* virtual function number */
 	u16		pchid;		/* physical channel ID */
+	u16		maxstbl;	/* Maximum store block size */
 	u8		pfgid;		/* function group ID */
 	u8		pft;		/* pci function type */
 	u8		port;
+	u8		dtsm;		/* Supported DT mask */
 	u8		rid_available	: 1;
 	u8		has_hp_slot	: 1;
 	u8		has_resources	: 1;
arch/s390/include/asm/pci_clp.h +4 -2
index f3286bc5ba6e..d6189ed14f84 100644
@@ -153,9 +153,11 @@ struct clp_rsp_query_pci_grp {
 	u8			:  6;
 	u8 frame		:  1;
 	u8 refresh		:  1;	/* TLB refresh mode */
-	u16 reserved2;
+	u16			:  3;
+	u16 maxstbl		: 13;	/* Maximum store block size */
 	u16 mui;
-	u16			: 16;
+	u8 dtsm;			/* Supported DT mask */
+	u8 reserved3;
 	u16 maxfaal;
 	u16			:  4;
 	u16 dnoi		: 12;
arch/s390/pci/pci_clp.c +2 -0
index d058c83467ef..ee367798e388 100644
@@ -106,6 +106,8 @@ static void clp_store_query_pci_fngrp(struct zpci_dev *zdev,
 	zdev->max_msi = response->noi;
 	zdev->fmb_update = response->mui;
 	zdev->version = response->version;
+	zdev->maxstbl = response->maxstbl;
+	zdev->dtsm = response->dtsm;
 
 	switch (response->version) {
 	case 1:
Signature · 1 line
2.27.0
author maintainer
#10 lore raw

PATCH v9 10/21 vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM

In reply to Matthew Rosato

The current contents of vfio-pci-zdev are today only useful in a KVM environment; let's tie everything currently under vfio-pci-zdev to this Kconfig statement and require KVM in this case, reducing complexity (e.g. symbol lookups).

---
 drivers/vfio/pci/Kconfig      | 11 +++++++++++
 drivers/vfio/pci/Makefile     |  2 +-
 include/linux/vfio_pci_core.h |  2 +-
 3 files changed, 13 insertions(+), 2 deletions(-)
drivers/vfio/pci/Kconfig +11 -0
index 4da1914425e1..f9d0c908e738 100644
@@ -44,6 +44,17 @@ config VFIO_PCI_IGD
 	  To enable Intel IGD assignment through vfio-pci, say Y.
 endif
 
+config VFIO_PCI_ZDEV_KVM
+	bool "VFIO PCI extensions for s390x KVM passthrough"
+	depends on S390 && KVM
+	default y
+	help
+	  Support s390x-specific extensions to enable support for enhancements
+	  to KVM passthrough capabilities, such as interpretive execution of
+	  zPCI instructions.
+
+	  To enable s390x KVM vfio-pci extensions, say Y.
+
 source "drivers/vfio/pci/mlx5/Kconfig"
 
 source "drivers/vfio/pci/hisilicon/Kconfig"
drivers/vfio/pci/Makefile +1 -1
index 7052ebd893e0..24c524224da5 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
-vfio-pci-core-$(CONFIG_S390) += vfio_pci_zdev.o
+vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
 vfio-pci-y := vfio_pci.o
include/linux/vfio_pci_core.h +1 -1
index 23c176d4b073..63af2897939c 100644
@@ -206,7 +206,7 @@ static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev)
 }
 #endif
 
-#ifdef CONFIG_S390
+#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
 extern int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 				       struct vfio_info_cap *caps);
 #else
Signature · 1 line
2.27.0
#11 lore raw

Re: [PATCH v9 10/21] vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM

Quoted 20 lines · Matthew Rosato

The current contents of vfio-pci-zdev are today only useful in a KVM environment; let's tie everything currently under vfio-pci-zdev to this Kconfig statement and require KVM in this case, reducing complexity (e.g. symbol lookups).

---
  drivers/vfio/pci/Kconfig      | 11 +++++++++++
  drivers/vfio/pci/Makefile     |  2 +-
  include/linux/vfio_pci_core.h |  2 +-
  3 files changed, 13 insertions(+), 2 deletions(-)
drivers/vfio/pci/Kconfig +0 -0
index 4da1914425e1..f9d0c908e738 100644
@@ -44,6 +44,17 @@ config VFIO_PCI_IGD
  	  To enable Intel IGD assignment through vfio-pci, say Y.
  endif
 

drivers/vfio/pci/Kconfig config VFIO_PCI_IGD · line 56

Quoted 10 lines
drivers/vfio/pci/Kconfig +10 -0
+config VFIO_PCI_ZDEV_KVM
+	bool "VFIO PCI extensions for s390x KVM passthrough"
+	depends on S390 && KVM
+	default y
+	help
+	  Support s390x-specific extensions to enable support for enhancements
+	  to KVM passthrough capabilities, such as interpretive execution of
+	  zPCI instructions.
+
+	  To enable s390x KVM vfio-pci extensions, say Y.

Is it still possible to disable CONFIG_VFIO_PCI_ZDEV_KVM ? Looking at the later patches (e.g. 20/21 where you call kvm_s390_pci_zpci_op() from kvm-s390.c), it rather seems to me that it currently cannot be disabled independently (as long as KVM is enabled).

So if you want to make this selectable by the user, I think you have to put some more #ifdefs in the following patches.
But if this was not meant to be selectable by the user, I think it should not get a help text and rather be selected by the KVM switch in arch/s390/kvm/Kconfig instead of having a "default y".

Thomas

author maintainer
#12 lore raw

Re: [PATCH v9 10/21] vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM

Quoted 25 lines · Thomas Huth
Quoted 24 lines · Matthew Rosato

The current contents of vfio-pci-zdev are today only useful in a KVM environment; let's tie everything currently under vfio-pci-zdev to this Kconfig statement and require KVM in this case, reducing complexity (e.g. symbol lookups).

---
  drivers/vfio/pci/Kconfig      | 11 +++++++++++
  drivers/vfio/pci/Makefile     |  2 +-
  include/linux/vfio_pci_core.h |  2 +-

  3 files changed, 13 insertions(+), 2 deletions(-)
drivers/vfio/pci/Kconfig +0 -0
index 4da1914425e1..f9d0c908e738 100644
@@ -44,6 +44,17 @@ config VFIO_PCI_IGD

        To enable Intel IGD assignment through vfio-pci, say Y. endif
+config VFIO_PCI_ZDEV_KVM
+    bool "VFIO PCI extensions for s390x KVM passthrough" +    depends on S390 && KVM
+    default y
+    help

Quoted 8 lines
Quoted 6 lines

+      Support s390x-specific extensions to enable support for enhancements
+      to KVM passthrough capabilities, such as interpretive execution of +      zPCI instructions.
+
+      To enable s390x KVM vfio-pci extensions, say Y.

Is it still possible to disable CONFIG_VFIO_PCI_ZDEV_KVM ? Looking at the later patches (e.g. 20/21 where you call kvm_s390_pci_zpci_op() from kvm-s390.c), it rather seems to me that it currently cannot be disabled independently (as long as KVM is enabled).

Yes, you can build with, for example, CONFIG_VFIO_PCI_ZDEV_KVM=n and CONFIG_KVM=m -- I tested it again just now. The result is kvm and vfio-pci are built and vfio-pci works, but none of the vfio-pci-zdev extensions are available (including zPCI interpretation).

This is accomplished via the placement of some IS_ENABLED checks. Some calls (e.g. AEN init) are fenced by IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM). There are also some areas that are fenced off via a call to kvm_s390_pci_interp_allowed() which also includes an IS_ENABLED check along with checks for facility and cpu id.

Using patch 20 as an example, KVM_CAP_S390_ZPCI_OP will always be reported as unavailable to userspace if CONFIG_VFIO_PCI_ZDEV_KVM=n due to the call to kvm_s390_pci_interp_allowed(). If userspace sends us the ioctl anyway, we will return -EINVAL because there is again a IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) check before we read the ioctl args from userspace.

Quoted 6 lines

So if you want to make this selectable by the user, I think you have to put some more #ifdefs in the following patches.
But if this was not meant to be selectable by the user, I think it should not get a help text and rather be selected by the KVM switch in arch/s390/kvm/Kconfig instead of having a "default y".

 Thomas

Reviewed-by
#13 lore raw

Re: [PATCH v9 10/21] vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM

Quoted 8 lines · Matthew Rosato
Quoted 7 lines · Thomas Huth
Quoted 6 lines · Matthew Rosato

The current contents of vfio-pci-zdev are today only useful in a KVM environment; let's tie everything currently under vfio-pci-zdev to this Kconfig statement and require KVM in this case, reducing complexity (e.g. symbol lookups).

Quoted 28 lines
Quoted 19 lines
Quoted 19 lines

---
  drivers/vfio/pci/Kconfig      | 11 +++++++++++
  drivers/vfio/pci/Makefile     |  2 +-
  include/linux/vfio_pci_core.h |  2 +-

  3 files changed, 13 insertions(+), 2 deletions(-)
drivers/vfio/pci/Kconfig +0 -0
index 4da1914425e1..f9d0c908e738 100644
@@ -44,6 +44,17 @@ config VFIO_PCI_IGD

        To enable Intel IGD assignment through vfio-pci, say Y. endif
+config VFIO_PCI_ZDEV_KVM
+    bool "VFIO PCI extensions for s390x KVM passthrough" +    depends on S390 && KVM
+    default y
+    help
+      Support s390x-specific extensions to enable support for

Quoted 8 lines
Quoted 6 lines

enhancements
+      to KVM passthrough capabilities, such as interpretive execution of
+      zPCI instructions.
+
+      To enable s390x KVM vfio-pci extensions, say Y.

Is it still possible to disable CONFIG_VFIO_PCI_ZDEV_KVM ? Looking at the later patches (e.g. 20/21 where you call kvm_s390_pci_zpci_op() from kvm-s390.c), it rather seems to me that it currently cannot be disabled independently (as long as KVM is enabled).

Quoted 5 lines

Yes, you can build with, for example, CONFIG_VFIO_PCI_ZDEV_KVM=n and CONFIG_KVM=m -- I tested it again just now.  The result is kvm and vfio-pci are built and vfio-pci works, but none of the vfio-pci-zdev extensions are available (including zPCI interpretation).

This is accomplished via the placement of some IS_ENABLED checks.  Some calls (e.g. AEN init) are fenced by IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM).  There are also some areas that are fenced off via a call to kvm_s390_pci_interp_allowed() which also includes an IS_ENABLED check along with checks for facility and cpu id.

Using patch 20 as an example, KVM_CAP_S390_ZPCI_OP will always be reported as unavailable to userspace if CONFIG_VFIO_PCI_ZDEV_KVM=n due to the call to kvm_s390_pci_interp_allowed().  If userspace sends us the ioctl anyway, we will return -EINVAL because there is again a IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) check before we read the ioctl args from userspace.

Yes and the code will not be generated by the compiler in patch 20 after the break if CONFIG_VFIO_PCI_ZDEV_KVM is not enabled.

+ case KVM_S390_ZPCI_OP: {
+ struct kvm_s390_zpci_op args;
+
+ r = -EINVAL;
+ if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
+ break;

Code not generated----v

+ if (copy_from_user(&args, argp, sizeof(args))) {
+ r = -EFAULT;
+ break;
+ }
+ r = kvm_s390_pci_zpci_op(kvm, &args);
+ break;

----------^

+ }

Quoted 8 lines
Quoted 7 lines

So if you want to make this selectable by the user, I think you have to put some more #ifdefs in the following patches.
But if this was not meant to be selectable by the user, I think it should not get a help text and rather be selected by the KVM switch in arch/s390/kvm/Kconfig instead of having a "default y".

  Thomas

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
#14 lore raw

Re: [PATCH v9 10/21] vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM

Quoted 58 lines · Pierre Morel
Quoted 8 lines · Matthew Rosato
Quoted 7 lines · Thomas Huth
Quoted 6 lines · Matthew Rosato

The current contents of vfio-pci-zdev are today only useful in a KVM environment; let's tie everything currently under vfio-pci-zdev to this Kconfig statement and require KVM in this case, reducing complexity (e.g. symbol lookups).

Quoted 26 lines
Quoted 17 lines
Quoted 17 lines

---
  drivers/vfio/pci/Kconfig      | 11 +++++++++++
  drivers/vfio/pci/Makefile     |  2 +-
  include/linux/vfio_pci_core.h |  2 +-

  3 files changed, 13 insertions(+), 2 deletions(-)
drivers/vfio/pci/Kconfig +0 -0
index 4da1914425e1..f9d0c908e738 100644
@@ -44,6 +44,17 @@ config VFIO_PCI_IGD

        To enable Intel IGD assignment through vfio-pci, say Y. endif
+config VFIO_PCI_ZDEV_KVM
+    bool "VFIO PCI extensions for s390x KVM passthrough" +    depends on S390 && KVM
+    default y

Quoted 8 lines
Quoted 6 lines

+    help
+      Support s390x-specific extensions to enable support for enhancements +      to KVM passthrough capabilities, such as interpretive execution of +      zPCI instructions.
+
+      To enable s390x KVM vfio-pci extensions, say Y.

Is it still possible to disable CONFIG_VFIO_PCI_ZDEV_KVM ? Looking at the later patches (e.g. 20/21 where you call kvm_s390_pci_zpci_op() from kvm-s390.c), it rather seems to me that it currently cannot be disabled independently (as long as KVM is enabled).

Quoted 5 lines

Yes, you can build with, for example, CONFIG_VFIO_PCI_ZDEV_KVM=n and CONFIG_KVM=m -- I tested it again just now.  The result is kvm and vfio-pci are built and vfio-pci works, but none of the vfio-pci-zdev extensions are available (including zPCI interpretation).

This is accomplished via the placement of some IS_ENABLED checks.  Some calls (e.g. AEN init) are fenced by IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM). There are also some areas that are fenced off via a call to kvm_s390_pci_interp_allowed() which also includes an IS_ENABLED check along with checks for facility and cpu id.

Using patch 20 as an example, KVM_CAP_S390_ZPCI_OP will always be reported as unavailable to userspace if CONFIG_VFIO_PCI_ZDEV_KVM=n due to the call to kvm_s390_pci_interp_allowed().  If userspace sends us the ioctl anyway, we will return -EINVAL because there is again a IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) check before we read the ioctl args from userspace.

Yes and the code will not be generated by the compiler in patch 20 after the break if CONFIG_VFIO_PCI_ZDEV_KVM is not enabled.

+    case KVM_S390_ZPCI_OP: {
+        struct kvm_s390_zpci_op args;
+
+        r = -EINVAL;
+        if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) +            break;

Code not generated----v

Quoted 8 lines

+        if (copy_from_user(&args, argp, sizeof(args))) { +            r = -EFAULT;
+            break;
+        }
+        r = kvm_s390_pci_zpci_op(kvm, &args);
+        break;

----------^

OK, good to know, thanks for the clarification!

Thomas

Acked-by on v9 10/21
#15 lore raw

Re: [PATCH v9 10/21] vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM

In reply to Matthew Rosato
Quoted 11 lines · On Mon, 6 Jun 2022 16:33:14 -0400 Matthew Rosato <[email protected]>…

The current contents of vfio-pci-zdev are today only useful in a KVM environment; let's tie everything currently under vfio-pci-zdev to this Kconfig statement and require KVM in this case, reducing complexity (e.g. symbol lookups).

---
 drivers/vfio/pci/Kconfig      | 11 +++++++++++
 drivers/vfio/pci/Makefile     |  2 +-
 include/linux/vfio_pci_core.h |  2 +-
 3 files changed, 13 insertions(+), 2 deletions(-)
Quoted 48 lines
drivers/vfio/pci/Kconfig +11 -0
index 4da1914425e1..f9d0c908e738 100644
@@ -44,6 +44,17 @@ config VFIO_PCI_IGD
 	  To enable Intel IGD assignment through vfio-pci, say Y.
 endif
 
+config VFIO_PCI_ZDEV_KVM
+	bool "VFIO PCI extensions for s390x KVM passthrough"
+	depends on S390 && KVM
+	default y
+	help
+	  Support s390x-specific extensions to enable support for enhancements
+	  to KVM passthrough capabilities, such as interpretive execution of
+	  zPCI instructions.
+
+	  To enable s390x KVM vfio-pci extensions, say Y.
+
 source "drivers/vfio/pci/mlx5/Kconfig"
 
 source "drivers/vfio/pci/hisilicon/Kconfig"
drivers/vfio/pci/Makefile +1 -1
index 7052ebd893e0..24c524224da5 100644
@@ -1,7 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0-only
 
 vfio-pci-core-y := vfio_pci_core.o vfio_pci_intrs.o vfio_pci_rdwr.o vfio_pci_config.o
-vfio-pci-core-$(CONFIG_S390) += vfio_pci_zdev.o
+vfio-pci-core-$(CONFIG_VFIO_PCI_ZDEV_KVM) += vfio_pci_zdev.o
 obj-$(CONFIG_VFIO_PCI_CORE) += vfio-pci-core.o
 
 vfio-pci-y := vfio_pci.o
include/linux/vfio_pci_core.h +1 -1
index 23c176d4b073..63af2897939c 100644
@@ -206,7 +206,7 @@ static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev)
 }
 #endif
 
-#ifdef CONFIG_S390
+#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
 extern int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 				       struct vfio_info_cap *caps);
 #else
author maintainer
#16 lore raw

PATCH v9 11/21 KVM: s390: pci: add basic kvm_zdev structure

In reply to Matthew Rosato

This structure will be used to carry kvm passthrough information related to zPCI devices.

---
 arch/s390/include/asm/pci.h |  3 +++
 arch/s390/kvm/Makefile      |  1 +
 arch/s390/kvm/pci.c         | 36 ++++++++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h         | 21 +++++++++++++++++++++
 4 files changed, 61 insertions(+)
 create mode 100644 arch/s390/kvm/pci.c
 create mode 100644 arch/s390/kvm/pci.h
arch/s390/include/asm/pci.h +3 -0
index 4c5b8fbc2079..50f1851edfbe 100644
@@ -97,6 +97,7 @@ struct zpci_bar_struct {
 };
 
 struct s390_domain;
+struct kvm_zdev;
 
 #define ZPCI_FUNCTIONS_PER_BUS 256
 struct zpci_bus {
@@ -189,7 +190,9 @@ struct zpci_dev {
 
 	struct dentry	*debugfs_dev;
 
+	/* IOMMU and passthrough */
 	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
+	struct kvm_zdev *kzdev;
 };
 
 static inline bool zdev_enabled(struct zpci_dev *zdev)
arch/s390/kvm/Makefile +1 -0
index 26f4a74e5ce4..02217fb4ae10 100644
@@ -10,4 +10,5 @@ ccflags-y := -Ivirt/kvm -Iarch/s390/kvm
 kvm-y += kvm-s390.o intercept.o interrupt.o priv.o sigp.o
 kvm-y += diag.o gaccess.o guestdbg.o vsie.o pv.o
 
+kvm-$(CONFIG_VFIO_PCI_ZDEV_KVM) += pci.o
 obj-$(CONFIG_KVM) += kvm.o
arch/s390/kvm/pci.c new file +36 -0
new file mode 100644
index 000000000000..e27191ea27c4
@@ -0,0 +1,36 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * s390 kvm PCI passthrough support
+ *
+ * Copyright IBM Corp. 2022
+ *
+ *    Author(s): Matthew Rosato <[email protected]>
+ */
+
+#include <linux/kvm_host.h>
+#include <linux/pci.h>
+#include "pci.h"
+
+static int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
+{
+	struct kvm_zdev *kzdev;
+
+	kzdev = kzalloc(sizeof(struct kvm_zdev), GFP_KERNEL);
+	if (!kzdev)
+		return -ENOMEM;
+
+	kzdev->zdev = zdev;
+	zdev->kzdev = kzdev;
+
+	return 0;
+}
+
+static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
+{
+	struct kvm_zdev *kzdev;
+
+	kzdev = zdev->kzdev;
+	WARN_ON(kzdev->zdev != zdev);
+	zdev->kzdev = NULL;
+	kfree(kzdev);
+}
arch/s390/kvm/pci.h new file +21 -0
new file mode 100644
index 000000000000..ce93978e8913
@@ -0,0 +1,21 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * s390 kvm PCI passthrough support
+ *
+ * Copyright IBM Corp. 2022
+ *
+ *    Author(s): Matthew Rosato <[email protected]>
+ */
+
+#ifndef __KVM_S390_PCI_H
+#define __KVM_S390_PCI_H
+
+#include <linux/kvm_host.h>
+#include <linux/pci.h>
+
+struct kvm_zdev {
+	struct zpci_dev *zdev;
+	struct kvm *kvm;
+};
+
+#endif /* __KVM_S390_PCI_H */
Signature · 1 line
2.27.0
author maintainer
#17 lore raw

PATCH v9 12/21 KVM: s390: pci: do initial setup for AEN interpretation

In reply to Matthew Rosato

Initial setup for Adapter Event Notification Interpretation for zPCI passthrough devices. Specifically, allocate a structure for forwarding of adapter events and pass the address of this structure to firmware.

---
 arch/s390/include/asm/pci.h      |   4 +
 arch/s390/include/asm/pci_insn.h |  12 +++
 arch/s390/kvm/interrupt.c        |  14 +++
 arch/s390/kvm/kvm-s390.c         |  11 +++
 arch/s390/kvm/pci.c              | 160 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |  49 ++++++++++
 arch/s390/pci/pci.c              |   6 ++
 7 files changed, 256 insertions(+)
arch/s390/include/asm/pci.h +4 -0
index 50f1851edfbe..322060a75d9f 100644
@@ -9,6 +9,7 @@
 #include <asm-generic/pci.h>
 #include <asm/pci_clp.h>
 #include <asm/pci_debug.h>
+#include <asm/pci_insn.h>
 #include <asm/sclp.h>
 
 #define PCIBIOS_MIN_IO		0x1000
@@ -204,6 +205,9 @@ extern const struct attribute_group *zpci_attr_groups[];
 extern unsigned int s390_pci_force_floating __initdata;
 extern unsigned int s390_pci_no_rid;
 
+extern union zpci_sic_iib *zpci_aipb;
+extern struct airq_iv *zpci_aif_sbv;
+
 /* -----------------------------------------------------------------------------
   Prototypes
 ----------------------------------------------------------------------------- */
arch/s390/include/asm/pci_insn.h +12 -0
index 5331082fa516..e5f57cfe1d45 100644
@@ -101,6 +101,7 @@ struct zpci_fib {
 /* Set Interruption Controls Operation Controls  */
 #define	SIC_IRQ_MODE_ALL		0
 #define	SIC_IRQ_MODE_SINGLE		1
+#define	SIC_SET_AENI_CONTROLS		2
 #define	SIC_IRQ_MODE_DIRECT		4
 #define	SIC_IRQ_MODE_D_ALL		16
 #define	SIC_IRQ_MODE_D_SINGLE		17
@@ -127,9 +128,20 @@ struct zpci_cdiib {
 	u64 : 64;
 } __packed __aligned(8);
 
+/* adapter interruption parameters block */
+struct zpci_aipb {
+	u64 faisb;
+	u64 gait;
+	u16 : 13;
+	u16 afi : 3;
+	u32 : 32;
+	u16 faal;
+} __packed __aligned(8);
+
 union zpci_sic_iib {
 	struct zpci_diib diib;
 	struct zpci_cdiib cdiib;
+	struct zpci_aipb aipb;
 };
 
 DECLARE_STATIC_KEY_FALSE(have_mio);
arch/s390/kvm/interrupt.c +14 -0
index 17ff475157d8..37ff4358121a 100644
@@ -32,6 +32,7 @@
 #include "kvm-s390.h"
 #include "gaccess.h"
 #include "trace-s390.h"
+#include "pci.h"
 
 #define PFAULT_INIT 0x0600
 #define PFAULT_DONE 0x0680
@@ -3328,6 +3329,11 @@ void kvm_s390_gib_destroy(void)
 {
 	if (!gib)
 		return;
+	if (kvm_s390_pci_interp_allowed() && aift) {
+		mutex_lock(&aift->aift_lock);
+		kvm_s390_pci_aen_exit();
+		mutex_unlock(&aift->aift_lock);
+	}
 	chsc_sgib(0);
 	unregister_adapter_interrupt(&gib_alert_irq);
 	free_page((unsigned long)gib);
@@ -3365,6 +3371,14 @@ int kvm_s390_gib_init(u8 nisc)
 		goto out_unreg_gal;
 	}
 
+	if (kvm_s390_pci_interp_allowed()) {
+		if (kvm_s390_pci_aen_init(nisc)) {
+			pr_err("Initializing AEN for PCI failed\n");
+			rc = -EIO;
+			goto out_unreg_gal;
+		}
+	}
+
 	KVM_EVENT(3, "gib 0x%pK (nisc=%d) initialized", gib, gib->nisc);
 	goto out;
 
arch/s390/kvm/kvm-s390.c +11 -0
index 8fcb56141689..251eaaff9c67 100644
@@ -47,6 +47,7 @@
 #include <asm/fpu/api.h>
 #include "kvm-s390.h"
 #include "gaccess.h"
+#include "pci.h"
 
 #define CREATE_TRACE_POINTS
 #include "trace.h"
@@ -502,6 +503,14 @@ int kvm_arch_init(void *opaque)
 		goto out;
 	}
 
+	if (kvm_s390_pci_interp_allowed()) {
+		rc = kvm_s390_pci_init();
+		if (rc) {
+			pr_err("Unable to allocate AIFT for PCI\n");
+			goto out;
+		}
+	}
+
 	rc = kvm_s390_gib_init(GAL_ISC);
 	if (rc)
 		goto out;
@@ -516,6 +525,8 @@ int kvm_arch_init(void *opaque)
 void kvm_arch_exit(void)
 {
 	kvm_s390_gib_destroy();
+	if (kvm_s390_pci_interp_allowed())
+		kvm_s390_pci_exit();
 	debug_unregister(kvm_s390_dbf);
 	debug_unregister(kvm_s390_dbf_uv);
 }
arch/s390/kvm/pci.c +160 -0

171 lines not on this page · raw message

arch/s390/kvm/pci.h +49 -0

59 lines not on this page · raw message

arch/s390/pci/pci.c +6 -0
index c8b9d866434c..86cd4d8446b1 100644
@@ -61,6 +61,12 @@ DEFINE_STATIC_KEY_FALSE(have_mio);
 
 static struct kmem_cache *zdev_fmb_cache;
 
+/* AEN structures that must be preserved over KVM module re-insertion */
+union zpci_sic_iib *zpci_aipb;
+EXPORT_SYMBOL_GPL(zpci_aipb);
+struct airq_iv *zpci_aif_sbv;
+EXPORT_SYMBOL_GPL(zpci_aif_sbv);
+
 struct zpci_dev *get_zdev_by_fid(u32 fid)
 {
 	struct zpci_dev *tmp, *zdev = NULL;
Signature · 1 line
2.27.0
author maintainer
#18 lore raw

PATCH v9 13/21 KVM: s390: pci: enable host forwarding of Adapter Event Notifications

In reply to Matthew Rosato

In cases where interrupts are not forwarded to the guest via firmware, KVM is responsible for ensuring delivery. When an interrupt presents with the forwarding bit, we must process the forwarding tables until all interrupts are delivered.

---
 arch/s390/include/asm/kvm_host.h |  1 +
 arch/s390/include/asm/tpi.h      | 13 ++++++
 arch/s390/kvm/interrupt.c        | 78 +++++++++++++++++++++++++++++++-
 arch/s390/kvm/kvm-s390.c         |  3 +-
 arch/s390/kvm/pci.h              | 10 ++++
 5 files changed, 103 insertions(+), 2 deletions(-)
arch/s390/include/asm/kvm_host.h +1 -0
index 766028d54a3e..c1518a505060 100644
@@ -759,6 +759,7 @@ struct kvm_vm_stat {
 	u64 inject_pfault_done;
 	u64 inject_service_signal;
 	u64 inject_virtio;
+	u64 aen_forward;
 };
 
 struct kvm_arch_memory_slot {
arch/s390/include/asm/tpi.h +13 -0
index 1ac538b8cbf5..f76e5fdff23a 100644
@@ -19,6 +19,19 @@ struct tpi_info {
 	u32 :12;
 } __packed __aligned(4);
 
+/* I/O-Interruption Code as stored by TPI for an Adapter I/O */
+struct tpi_adapter_info {
+	u32 aism:8;
+	u32 :22;
+	u32 error:1;
+	u32 forward:1;
+	u32 reserved;
+	u32 adapter_IO:1;
+	u32 directed_irq:1;
+	u32 isc:3;
+	u32 :27;
+} __packed __aligned(4);
+
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_S390_TPI_H */
arch/s390/kvm/interrupt.c +77 -1

88 lines not on this page · raw message

arch/s390/kvm/kvm-s390.c +2 -1
index 251eaaff9c67..d96c9be0480b 100644
@@ -64,7 +64,8 @@ const struct _kvm_stats_desc kvm_vm_stats_desc[] = {
 	STATS_DESC_COUNTER(VM, inject_float_mchk),
 	STATS_DESC_COUNTER(VM, inject_pfault_done),
 	STATS_DESC_COUNTER(VM, inject_service_signal),
-	STATS_DESC_COUNTER(VM, inject_virtio)
+	STATS_DESC_COUNTER(VM, inject_virtio),
+	STATS_DESC_COUNTER(VM, aen_forward)
 };
 
 const struct kvm_stats_header kvm_vm_stats_header = {
arch/s390/kvm/pci.h +10 -0
index c357d900f8b0..9f7828d97605 100644
@@ -13,6 +13,7 @@
 #include <linux/kvm_host.h>
 #include <linux/pci.h>
 #include <linux/mutex.h>
+#include <linux/kvm_host.h>
 #include <asm/airq.h>
 #include <asm/cpu.h>
 
@@ -40,6 +41,15 @@ struct zpci_aift {
 
 extern struct zpci_aift *aift;
 
+static inline struct kvm *kvm_s390_pci_si_to_kvm(struct zpci_aift *aift,
+						 unsigned long si)
+{
+	if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM) || aift->kzdev == 0 ||
+	    aift->kzdev[si] == 0)
+		return 0;
+	return aift->kzdev[si]->kvm;
+};
+
 int kvm_s390_pci_aen_init(u8 nisc);
 void kvm_s390_pci_aen_exit(void);
 
Signature · 1 line
2.27.0
author maintainer
#19 lore raw

PATCH v9 14/21 KVM: s390: mechanism to enable guest zPCI Interpretation

In reply to Matthew Rosato

The guest must have access to certain facilities in order to allow interpretive execution of zPCI instructions and adapter event notifications. However, there are some cases where a guest might disable interpretation -- provide a mechanism via which we can defer enabling the associated zPCI interpretation facilities until the guest indicates it wishes to use them.

---
 arch/s390/include/asm/kvm_host.h |  4 ++++
 arch/s390/kvm/kvm-s390.c         | 38 ++++++++++++++++++++++++++++++++
 arch/s390/kvm/kvm-s390.h         | 10 +++++++++
 3 files changed, 52 insertions(+)
arch/s390/include/asm/kvm_host.h +4 -0
index c1518a505060..8e381603b6a7 100644
@@ -254,7 +254,10 @@ struct kvm_s390_sie_block {
 #define ECB2_IEP	0x20
 #define ECB2_PFMFI	0x08
 #define ECB2_ESCA	0x04
+#define ECB2_ZPCI_LSI	0x02
 	__u8    ecb2;                   /* 0x0062 */
+#define ECB3_AISI	0x20
+#define ECB3_AISII	0x10
 #define ECB3_DEA 0x08
 #define ECB3_AES 0x04
 #define ECB3_RI  0x01
@@ -940,6 +943,7 @@ struct kvm_arch{
 	int use_cmma;
 	int use_pfmfi;
 	int use_skf;
+	int use_zpci_interp;
 	int user_cpu_state_ctrl;
 	int user_sigp;
 	int user_stsi;
arch/s390/kvm/kvm-s390.c +38 -0
index d96c9be0480b..a66da3f66114 100644
@@ -1031,6 +1031,42 @@ static int kvm_s390_vm_set_crypto(struct kvm *kvm, struct kvm_device_attr *attr)
 	return 0;
 }
 
+static void kvm_s390_vcpu_pci_setup(struct kvm_vcpu *vcpu)
+{
+	/* Only set the ECB bits after guest requests zPCI interpretation */
+	if (!vcpu->kvm->arch.use_zpci_interp)
+		return;
+
+	vcpu->arch.sie_block->ecb2 |= ECB2_ZPCI_LSI;
+	vcpu->arch.sie_block->ecb3 |= ECB3_AISII + ECB3_AISI;
+}
+
+void kvm_s390_vcpu_pci_enable_interp(struct kvm *kvm)
+{
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	lockdep_assert_held(&kvm->lock);
+
+	if (!kvm_s390_pci_interp_allowed())
+		return;
+
+	/*
+	 * If host is configured for PCI and the necessary facilities are
+	 * available, turn on interpretation for the life of this guest
+	 */
+	kvm->arch.use_zpci_interp = 1;
+
+	kvm_s390_vcpu_block_all(kvm);
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		kvm_s390_vcpu_pci_setup(vcpu);
+		kvm_s390_sync_request(KVM_REQ_VSIE_RESTART, vcpu);
+	}
+
+	kvm_s390_vcpu_unblock_all(kvm);
+}
+
 static void kvm_s390_sync_request_broadcast(struct kvm *kvm, int req)
 {
 	unsigned long cx;
@@ -3336,6 +3372,8 @@ static int kvm_s390_vcpu_setup(struct kvm_vcpu *vcpu)
 
 	kvm_s390_vcpu_crypto_setup(vcpu);
 
+	kvm_s390_vcpu_pci_setup(vcpu);
+
 	mutex_lock(&vcpu->kvm->lock);
 	if (kvm_s390_pv_is_protected(vcpu->kvm)) {
 		rc = kvm_s390_pv_create_cpu(vcpu, &uvrc, &uvrrc);
arch/s390/kvm/kvm-s390.h +10 -0
index 497d52a83c78..975cffaf13de 100644
@@ -507,6 +507,16 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
  */
 void kvm_s390_vcpu_crypto_reset_all(struct kvm *kvm);
 
+/**
+ * kvm_s390_vcpu_pci_enable_interp
+ *
+ * Set the associated PCI attributes for each vcpu to allow for zPCI Load/Store
+ * interpretation as well as adapter interruption forwarding.
+ *
+ * @kvm: the KVM guest
+ */
+void kvm_s390_vcpu_pci_enable_interp(struct kvm *kvm);
+
 /**
  * diag9c_forwarding_hz
  *
Signature · 1 line
2.27.0
author maintainer
#20 lore raw

PATCH v9 15/21 KVM: s390: pci: provide routines for enabling/disabling interrupt forwarding

In reply to Matthew Rosato

These routines will be wired into a kvm ioctl in order to respond to requests to enable / disable a device for Adapter Event Notifications / Adapter Interuption Forwarding.

---
 arch/s390/kvm/pci.c        | 247 +++++++++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h        |   1 +
 arch/s390/pci/pci_insn.c   |   1 +
 include/linux/sched/user.h |   3 +-
 4 files changed, 251 insertions(+), 1 deletion(-)
arch/s390/kvm/pci.c +247 -0
index d566551d3018..b232c8cbaa81 100644
@@ -11,6 +11,7 @@
 #include <linux/pci.h>
 #include <asm/pci.h>
 #include <asm/pci_insn.h>
+#include <asm/pci_io.h>
 #include "pci.h"
 
 struct zpci_aift *aift;
@@ -152,6 +153,252 @@ int kvm_s390_pci_aen_init(u8 nisc)
 	return rc;
 }
 
+/* Modify PCI: Register floating adapter interruption forwarding */
+static int kvm_zpci_set_airq(struct zpci_dev *zdev)
+{
+	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_REG_INT);
+	struct zpci_fib fib = {};
+	u8 status;
+
+	fib.fmt0.isc = zdev->kzdev->fib.fmt0.isc;
+	fib.fmt0.sum = 1;       /* enable summary notifications */
+	fib.fmt0.noi = airq_iv_end(zdev->aibv);
+	fib.fmt0.aibv = virt_to_phys(zdev->aibv->vector);
+	fib.fmt0.aibvo = 0;
+	fib.fmt0.aisb = virt_to_phys(aift->sbv->vector + (zdev->aisb / 64) * 8);
+	fib.fmt0.aisbo = zdev->aisb & 63;
+	fib.gd = zdev->gisa;
+
+	return zpci_mod_fc(req, &fib, &status) ? -EIO : 0;
+}
+
+/* Modify PCI: Unregister floating adapter interruption forwarding */
+static int kvm_zpci_clear_airq(struct zpci_dev *zdev)
+{
+	u64 req = ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_DEREG_INT);
+	struct zpci_fib fib = {};
+	u8 cc, status;
+
+	fib.gd = zdev->gisa;
+
+	cc = zpci_mod_fc(req, &fib, &status);
+	if (cc == 3 || (cc == 1 && status == 24))
+		/* Function already gone or IRQs already deregistered. */
+		cc = 0;
+
+	return cc ? -EIO : 0;
+}
+
+static inline void unaccount_mem(unsigned long nr_pages)
+{
+	struct user_struct *user = get_uid(current_user());
+
+	if (user)
+		atomic_long_sub(nr_pages, &user->locked_vm);
+	if (current->mm)
+		atomic64_sub(nr_pages, &current->mm->pinned_vm);
+}
+
+static inline int account_mem(unsigned long nr_pages)
+{
+	struct user_struct *user = get_uid(current_user());
+	unsigned long page_limit, cur_pages, new_pages;
+
+	page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+	do {
+		cur_pages = atomic_long_read(&user->locked_vm);
+		new_pages = cur_pages + nr_pages;
+		if (new_pages > page_limit)
+			return -ENOMEM;
+	} while (atomic_long_cmpxchg(&user->locked_vm, cur_pages,
+					new_pages) != cur_pages);
+
+	atomic64_add(nr_pages, &current->mm->pinned_vm);
+
+	return 0;
+}
+
+static int kvm_s390_pci_aif_enable(struct zpci_dev *zdev, struct zpci_fib *fib,
+				   bool assist)
+{
+	struct page *pages[1], *aibv_page, *aisb_page = NULL;
+	unsigned int msi_vecs, idx;
+	struct zpci_gaite *gaite;
+	unsigned long hva, bit;
+	struct kvm *kvm;
+	phys_addr_t gaddr;
+	int rc = 0, gisc, npages, pcount = 0;
+
+	/*
+	 * Interrupt forwarding is only applicable if the device is already
+	 * enabled for interpretation
+	 */
+	if (zdev->gisa == 0)
+		return -EINVAL;
+
+	kvm = zdev->kzdev->kvm;
+	msi_vecs = min_t(unsigned int, fib->fmt0.noi, zdev->max_msi);
+
+	/* Get the associated forwarding ISC - if invalid, return the error */
+	gisc = kvm_s390_gisc_register(kvm, fib->fmt0.isc);
+	if (gisc < 0)
+		return gisc;
+
+	/* Replace AIBV address */
+	idx = srcu_read_lock(&kvm->srcu);
+	hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aibv));
+	npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM, pages);
+	srcu_read_unlock(&kvm->srcu, idx);
+	if (npages < 1) {
+		rc = -EIO;
+		goto out;
+	}
+	aibv_page = pages[0];
+	pcount++;
+	gaddr = page_to_phys(aibv_page) + (fib->fmt0.aibv & ~PAGE_MASK);
+	fib->fmt0.aibv = gaddr;
+
+	/* Pin the guest AISB if one was specified */
+	if (fib->fmt0.sum == 1) {
+		idx = srcu_read_lock(&kvm->srcu);
+		hva = gfn_to_hva(kvm, gpa_to_gfn((gpa_t)fib->fmt0.aisb));
+		npages = pin_user_pages_fast(hva, 1, FOLL_WRITE | FOLL_LONGTERM,
+					     pages);
+		srcu_read_unlock(&kvm->srcu, idx);
+		if (npages < 1) {
+			rc = -EIO;
+			goto unpin1;
+		}
+		aisb_page = pages[0];
+		pcount++;
+	}
+
+	/* Account for pinned pages, roll back on failure */
+	if (account_mem(pcount))
+		goto unpin2;
+
+	/* AISB must be allocated before we can fill in GAITE */
+	mutex_lock(&aift->aift_lock);
+	bit = airq_iv_alloc_bit(aift->sbv);
+	if (bit == -1UL)
+		goto unlock;
+	zdev->aisb = bit; /* store the summary bit number */
+	zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_DATA |
+				    AIRQ_IV_BITLOCK |
+				    AIRQ_IV_GUESTVEC,
+				    phys_to_virt(fib->fmt0.aibv));
+
+	spin_lock_irq(&aift->gait_lock);
+	gaite = (struct zpci_gaite *)aift->gait + (zdev->aisb *
+						   sizeof(struct zpci_gaite));
+
+	/* If assist not requested, host will get all alerts */
+	if (assist)
+		gaite->gisa = (u32)virt_to_phys(&kvm->arch.sie_page2->gisa);
+	else
+		gaite->gisa = 0;
+
+	gaite->gisc = fib->fmt0.isc;
+	gaite->count++;
+	gaite->aisbo = fib->fmt0.aisbo;
+	gaite->aisb = virt_to_phys(page_address(aisb_page) + (fib->fmt0.aisb &
+							      ~PAGE_MASK));
+	aift->kzdev[zdev->aisb] = zdev->kzdev;
+	spin_unlock_irq(&aift->gait_lock);
+
+	/* Update guest FIB for re-issue */
+	fib->fmt0.aisbo = zdev->aisb & 63;
+	fib->fmt0.aisb = virt_to_phys(aift->sbv->vector + (zdev->aisb / 64) * 8);
+	fib->fmt0.isc = gisc;
+
+	/* Save some guest fib values in the host for later use */
+	zdev->kzdev->fib.fmt0.isc = fib->fmt0.isc;
+	zdev->kzdev->fib.fmt0.aibv = fib->fmt0.aibv;
+	mutex_unlock(&aift->aift_lock);
+
+	/* Issue the clp to setup the irq now */
+	rc = kvm_zpci_set_airq(zdev);
+	return rc;
+
+unlock:
+	mutex_unlock(&aift->aift_lock);
+unpin2:
+	if (fib->fmt0.sum == 1)
+		unpin_user_page(aisb_page);
+unpin1:
+	unpin_user_page(aibv_page);
+out:
+	return rc;
+}
+
+static int kvm_s390_pci_aif_disable(struct zpci_dev *zdev, bool force)
+{
+	struct kvm_zdev *kzdev = zdev->kzdev;
+	struct zpci_gaite *gaite;
+	struct page *vpage = NULL, *spage = NULL;
+	int rc, pcount = 0;
+	u8 isc;
+
+	if (zdev->gisa == 0)
+		return -EINVAL;
+
+	mutex_lock(&aift->aift_lock);
+
+	/*
+	 * If the clear fails due to an error, leave now unless we know this
+	 * device is about to go away (force) -- In that case clear the GAITE
+	 * regardless.
+	 */
+	rc = kvm_zpci_clear_airq(zdev);
+	if (rc && !force)
+		goto out;
+
+	if (zdev->kzdev->fib.fmt0.aibv == 0)
+		goto out;
+	spin_lock_irq(&aift->gait_lock);
+	gaite = (struct zpci_gaite *)aift->gait + (zdev->aisb *
+						   sizeof(struct zpci_gaite));
+	isc = gaite->gisc;
+	gaite->count--;
+	if (gaite->count == 0) {
+		/* Release guest AIBV and AISB */
+		vpage = phys_to_page(kzdev->fib.fmt0.aibv);
+		if (gaite->aisb != 0)
+			spage = phys_to_page(gaite->aisb);
+		/* Clear the GAIT entry */
+		gaite->aisb = 0;
+		gaite->gisc = 0;
+		gaite->aisbo = 0;
+		gaite->gisa = 0;
+		aift->kzdev[zdev->aisb] = 0;
+		/* Clear zdev info */
+		airq_iv_free_bit(aift->sbv, zdev->aisb);
+		airq_iv_release(zdev->aibv);
+		zdev->aisb = 0;
+		zdev->aibv = NULL;
+	}
+	spin_unlock_irq(&aift->gait_lock);
+	kvm_s390_gisc_unregister(kzdev->kvm, isc);
+	kzdev->fib.fmt0.isc = 0;
+	kzdev->fib.fmt0.aibv = 0;
+
+	if (vpage) {
+		unpin_user_page(vpage);
+		pcount++;
+	}
+	if (spage) {
+		unpin_user_page(spage);
+		pcount++;
+	}
+	if (pcount > 0)
+		unaccount_mem(pcount);
+out:
+	mutex_unlock(&aift->aift_lock);
+
+	return rc;
+}
+
 static int kvm_s390_pci_dev_open(struct zpci_dev *zdev)
 {
 	struct kvm_zdev *kzdev;
arch/s390/kvm/pci.h +1 -0
index 9f7828d97605..9d091033fc02 100644
@@ -20,6 +20,7 @@
 struct kvm_zdev {
 	struct zpci_dev *zdev;
 	struct kvm *kvm;
+	struct zpci_fib fib;
 };
 
 struct zpci_gaite {
arch/s390/pci/pci_insn.c +1 -0
index 5798178aec9d..56480be48244 100644
@@ -92,6 +92,7 @@ u8 zpci_mod_fc(u64 req, struct zpci_fib *fib, u8 *status)
 
 	return cc;
 }
+EXPORT_SYMBOL_GPL(zpci_mod_fc);
 
 /* Refresh PCI Translations */
 static inline u8 __rpcit(u64 fn, u64 addr, u64 range, u8 *status)
include/linux/sched/user.h +2 -1
index 00ed419dd464..f054d0360a75 100644
@@ -24,7 +24,8 @@ struct user_struct {
 	kuid_t uid;
 
 #if defined(CONFIG_PERF_EVENTS) || defined(CONFIG_BPF_SYSCALL) || \
-    defined(CONFIG_NET) || defined(CONFIG_IO_URING)
+	defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \
+	defined(CONFIG_VFIO_PCI_ZDEV_KVM)
 	atomic_long_t locked_vm;
 #endif
 #ifdef CONFIG_WATCH_QUEUE
Signature · 1 line
2.27.0
author maintainer
#21 lore raw

PATCH v9 16/21 KVM: s390: pci: add routines to start/stop interpretive execution

In reply to Matthew Rosato

These routines will be invoked at the time an s390x vfio-pci device is associated with a KVM (or when the association is removed), allowing the zPCI device to enable or disable load/store intepretation mode; this requires the host zPCI device to inform firmware of the unique token (GISA designation) that is associated with the owning KVM.

---
 arch/s390/include/asm/kvm_host.h |  18 ++++
 arch/s390/include/asm/pci.h      |   1 +
 arch/s390/kvm/kvm-s390.c         |  15 +++
 arch/s390/kvm/pci.c              | 162 +++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h              |   5 +
 arch/s390/pci/pci.c              |   4 +
 6 files changed, 205 insertions(+)
arch/s390/include/asm/kvm_host.h +18 -0
index 8e381603b6a7..6e83d746bae2 100644
@@ -19,6 +19,7 @@
 #include <linux/kvm.h>
 #include <linux/seqlock.h>
 #include <linux/module.h>
+#include <linux/pci.h>
 #include <asm/debug.h>
 #include <asm/cpu.h>
 #include <asm/fpu/api.h>
@@ -967,6 +968,8 @@ struct kvm_arch{
 	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
 	struct kvm_s390_gisa_interrupt gisa_int;
 	struct kvm_s390_pv pv;
+	struct list_head kzdev_list;
+	spinlock_t kzdev_list_lock;
 };
 
 #define KVM_HVA_ERR_BAD		(-1UL)
@@ -1017,4 +1020,19 @@ static inline void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
 static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
+#define __KVM_HAVE_ARCH_VM_FREE
+void kvm_arch_free_vm(struct kvm *kvm);
+
+#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
+void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
+#else
+static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
+					    struct kvm *kvm)
+{
+	return -EPERM;
+}
+static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
+#endif
+
 #endif
arch/s390/include/asm/pci.h +1 -0
index 322060a75d9f..85eb0ef9d4c3 100644
@@ -194,6 +194,7 @@ struct zpci_dev {
 	/* IOMMU and passthrough */
 	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
 	struct kvm_zdev *kzdev;
+	struct mutex kzdev_lock;
 };
 
 static inline bool zdev_enabled(struct zpci_dev *zdev)
arch/s390/kvm/kvm-s390.c +15 -0
index a66da3f66114..4758bb731199 100644
@@ -2790,6 +2790,14 @@ static void sca_dispose(struct kvm *kvm)
 	kvm->arch.sca = NULL;
 }
 
+void kvm_arch_free_vm(struct kvm *kvm)
+{
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
+		kvm_s390_pci_clear_list(kvm);
+
+	__kvm_arch_free_vm(kvm);
+}
+
 int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 {
 	gfp_t alloc_flags = GFP_KERNEL_ACCOUNT;
@@ -2872,6 +2880,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	kvm_s390_crypto_init(kvm);
 
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {
+		mutex_lock(&kvm->lock);
+		kvm_s390_pci_init_list(kvm);
+		kvm_s390_vcpu_pci_enable_interp(kvm);
+		mutex_unlock(&kvm->lock);
+	}
+
 	mutex_init(&kvm->arch.float_int.ais_lock);
 	spin_lock_init(&kvm->arch.float_int.lock);
 	for (i = 0; i < FIRQ_LIST_COUNT; i++)
arch/s390/kvm/pci.c +162 -0

175 lines not on this page · raw message

arch/s390/kvm/pci.h +5 -0
index 9d091033fc02..fb2b91b76e0c 100644
@@ -13,6 +13,7 @@
 #include <linux/kvm_host.h>
 #include <linux/pci.h>
 #include <linux/mutex.h>
+#include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <asm/airq.h>
 #include <asm/cpu.h>
@@ -21,6 +22,7 @@ struct kvm_zdev {
 	struct zpci_dev *zdev;
 	struct kvm *kvm;
 	struct zpci_fib fib;
+	struct list_head entry;
 };
 
 struct zpci_gaite {
@@ -54,6 +56,9 @@ static inline struct kvm *kvm_s390_pci_si_to_kvm(struct zpci_aift *aift,
 int kvm_s390_pci_aen_init(u8 nisc);
 void kvm_s390_pci_aen_exit(void);
 
+void kvm_s390_pci_init_list(struct kvm *kvm);
+void kvm_s390_pci_clear_list(struct kvm *kvm);
+
 int kvm_s390_pci_init(void);
 void kvm_s390_pci_exit(void);
 
arch/s390/pci/pci.c +4 -0
index 86cd4d8446b1..73cdc5539384 100644
@@ -132,6 +132,7 @@ int zpci_register_ioat(struct zpci_dev *zdev, u8 dmaas,
 		zpci_dbg(3, "reg ioat fid:%x, cc:%d, status:%d\n", zdev->fid, cc, status);
 	return cc;
 }
+EXPORT_SYMBOL_GPL(zpci_register_ioat);
 
 /* Modify PCI: Unregister I/O address translation parameters */
 int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dmaas)
@@ -712,6 +713,7 @@ int zpci_enable_device(struct zpci_dev *zdev)
 		zpci_update_fh(zdev, fh);
 	return rc;
 }
+EXPORT_SYMBOL_GPL(zpci_enable_device);
 
 int zpci_disable_device(struct zpci_dev *zdev)
 {
@@ -735,6 +737,7 @@ int zpci_disable_device(struct zpci_dev *zdev)
 	}
 	return rc;
 }
+EXPORT_SYMBOL_GPL(zpci_disable_device);
 
 /**
  * zpci_hot_reset_device - perform a reset of the given zPCI function
@@ -828,6 +831,7 @@ struct zpci_dev *zpci_create_device(u32 fid, u32 fh, enum zpci_state state)
 
 	kref_init(&zdev->kref);
 	mutex_init(&zdev->lock);
+	mutex_init(&zdev->kzdev_lock);
 
 	rc = zpci_init_iommu(zdev);
 	if (rc)
Signature · 1 line
2.27.0
Acked-by on v9 16/21
#22 lore raw

Re: [PATCH v9 16/21] KVM: s390: pci: add routines to start/stop interpretive execution

Quoted 62 lines · Matthew Rosato

These routines will be invoked at the time an s390x vfio-pci device is associated with a KVM (or when the association is removed), allowing the zPCI device to enable or disable load/store intepretation mode; this requires the host zPCI device to inform firmware of the unique token (GISA designation) that is associated with the owning KVM.

---
  arch/s390/include/asm/kvm_host.h |  18 ++++
  arch/s390/include/asm/pci.h      |   1 +
  arch/s390/kvm/kvm-s390.c         |  15 +++
  arch/s390/kvm/pci.c              | 162 +++++++++++++++++++++++++++++++
  arch/s390/kvm/pci.h              |   5 +
  arch/s390/pci/pci.c              |   4 +
  6 files changed, 205 insertions(+)
arch/s390/include/asm/kvm_host.h +18 -0
index 8e381603b6a7..6e83d746bae2 100644
@@ -19,6 +19,7 @@
  #include <linux/kvm.h>
  #include <linux/seqlock.h>
  #include <linux/module.h>
+#include <linux/pci.h>
  #include <asm/debug.h>
  #include <asm/cpu.h>
  #include <asm/fpu/api.h>
@@ -967,6 +968,8 @@ struct kvm_arch{
  	DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
  	struct kvm_s390_gisa_interrupt gisa_int;
  	struct kvm_s390_pv pv;
+	struct list_head kzdev_list;
+	spinlock_t kzdev_list_lock;
  };
 
  #define KVM_HVA_ERR_BAD		(-1UL)
@@ -1017,4 +1020,19 @@ static inline void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
  static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
  static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
 
+#define __KVM_HAVE_ARCH_VM_FREE
+void kvm_arch_free_vm(struct kvm *kvm);
+
+#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
+void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
+#else
+static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
+					    struct kvm *kvm)
+{
+	return -EPERM;
+}
+static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
+#endif
+
  #endif
arch/s390/include/asm/pci.h +0 -0
index 322060a75d9f..85eb0ef9d4c3 100644

arch/s390/include/asm/pci.h struct zpci_dev { · line 197

Quoted 4 lines
arch/s390/include/asm/pci.h +1 -0
@@ -194,6 +194,7 @@ struct zpci_dev {
  	/* IOMMU and passthrough */
  	struct s390_domain *s390_domain; /* s390 IOMMU domain data */
  	struct kvm_zdev *kzdev;
+	struct mutex kzdev_lock;

I guess that since it did not exist before the lock is not there to protect the zpci_dev struct. May be add a comment to say what it is protecting.

Quoted 67 lines
arch/s390/include/asm/pci.h +0 -0
  };
 
  static inline bool zdev_enabled(struct zpci_dev *zdev)
arch/s390/kvm/kvm-s390.c +15 -0
index a66da3f66114..4758bb731199 100644
@@ -2790,6 +2790,14 @@ static void sca_dispose(struct kvm *kvm)
  	kvm->arch.sca = NULL;
  }
 
+void kvm_arch_free_vm(struct kvm *kvm)
+{
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
+		kvm_s390_pci_clear_list(kvm);
+
+	__kvm_arch_free_vm(kvm);
+}
+
  int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
  {
  	gfp_t alloc_flags = GFP_KERNEL_ACCOUNT;
@@ -2872,6 +2880,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
  	kvm_s390_crypto_init(kvm);
 
+	if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {
+		mutex_lock(&kvm->lock);
+		kvm_s390_pci_init_list(kvm);
+		kvm_s390_vcpu_pci_enable_interp(kvm);
+		mutex_unlock(&kvm->lock);
+	}
+
  	mutex_init(&kvm->arch.float_int.ais_lock);
  	spin_lock_init(&kvm->arch.float_int.lock);
  	for (i = 0; i < FIRQ_LIST_COUNT; i++)
arch/s390/kvm/pci.c +15 -0
index b232c8cbaa81..24211741deb0 100644
@@ -12,7 +12,9 @@
  #include <asm/pci.h>
  #include <asm/pci_insn.h>
  #include <asm/pci_io.h>
+#include <asm/sclp.h>
  #include "pci.h"
+#include "kvm-s390.h"
 
  struct zpci_aift *aift;
 
@@ -423,6 +425,166 @@ static void kvm_s390_pci_dev_release(struct zpci_dev *zdev)
  	kfree(kzdev);
  }
 
+
+/*
+ * Register device with the specified KVM. If interpetation facilities are
+ * available, enable them and let userspace indicate whether or not they will
+ * be used (specify SHM bit to disable).
+ */
+int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+{
+	int rc;
+
+	if (!zdev)
+		return -EINVAL;
+

arch/s390/kvm/pci.c static void kvm_s390_pci_dev_release(struct zpci_dev *zdev) · line 450

Quoted 10 lines
arch/s390/kvm/pci.c +10 -0
+	mutex_lock(&zdev->kzdev_lock);
+
+	if (zdev->kzdev || zdev->gisa != 0 || !kvm) {
+		mutex_unlock(&zdev->kzdev_lock);
+		return -EINVAL;
+	}
+
+	kvm_get_kvm(kvm);
+
+	mutex_lock(&kvm->lock);

Why do we need to lock KVM here?

just a question, I do not think it is a big problem.

Quoted 64 lines
arch/s390/kvm/pci.c +64 -0
+
+	rc = kvm_s390_pci_dev_open(zdev);
+	if (rc)
+		goto err;
+
+	/*
+	 * If interpretation facilities aren't available, add the device to
+	 * the kzdev list but don't enable for interpretation.
+	 */
+	if (!kvm_s390_pci_interp_allowed())
+		goto out;
+
+	/*
+	 * If this is the first request to use an interpreted device, make the
+	 * necessary vcpu changes
+	 */
+	if (!kvm->arch.use_zpci_interp)
+		kvm_s390_vcpu_pci_enable_interp(kvm);
+
+	if (zdev_enabled(zdev)) {
+		rc = zpci_disable_device(zdev);
+		if (rc)
+			goto err;
+	}
+
+	/*
+	 * Store information about the identity of the kvm guest allowed to
+	 * access this device via interpretation to be used by host CLP
+	 */
+	zdev->gisa = (u32)virt_to_phys(&kvm->arch.sie_page2->gisa);
+
+	rc = zpci_enable_device(zdev);
+	if (rc)
+		goto clear_gisa;
+
+	/* Re-register the IOMMU that was already created */
+	rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+				virt_to_phys(zdev->dma_table));
+	if (rc)
+		goto clear_gisa;
+
+out:
+	zdev->kzdev->kvm = kvm;
+
+	spin_lock(&kvm->arch.kzdev_list_lock);
+	list_add_tail(&zdev->kzdev->entry, &kvm->arch.kzdev_list);
+	spin_unlock(&kvm->arch.kzdev_list_lock);
+
+	mutex_unlock(&kvm->lock);
+	mutex_unlock(&zdev->kzdev_lock);
+	return 0;
+
+clear_gisa:
+	zdev->gisa = 0;
+err:
+	if (zdev->kzdev)
+		kvm_s390_pci_dev_release(zdev);
+	mutex_unlock(&kvm->lock);
+	mutex_unlock(&zdev->kzdev_lock);
+	kvm_put_kvm(kvm);
+	return rc;
+}
+EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
+

arch/s390/kvm/pci.c static void kvm_s390_pci_dev_release(struct zpci_dev *zdev) · line 524

Quoted 10 lines
arch/s390/kvm/pci.c +10 -0
+void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+{
+	struct kvm *kvm;
+
+	if (!zdev)
+		return;
+
+	mutex_lock(&zdev->kzdev_lock);
+
+	if (WARN_ON(!zdev->kzdev)) {

When can this happen ?

Quoted 14 lines
arch/s390/kvm/pci.c +14 -0
+		mutex_unlock(&zdev->kzdev_lock);
+		return;
+	}
+
+	kvm = zdev->kzdev->kvm;
+	mutex_lock(&kvm->lock);
+
+	/*
+	 * A 0 gisa means interpretation was never enabled, just remove the
+	 * device from the list.
+	 */
+	if (zdev->gisa == 0)
+		goto out;
+

arch/s390/kvm/pci.c static void kvm_s390_pci_dev_release(struct zpci_dev *zdev) · line 548

Quoted 10 lines
arch/s390/kvm/pci.c +10 -0
+	/* Forwarding must be turned off before interpretation */
+	if (zdev->kzdev->fib.fmt0.aibv != 0)
+		kvm_s390_pci_aif_disable(zdev, true);
+
+	/* Remove the host CLP guest designation */
+	zdev->gisa = 0;
+
+	if (zdev_enabled(zdev)) {
+		if (zpci_disable_device(zdev))
+			goto out;

NIT debug trace ?

arch/s390/kvm/pci.c static void kvm_s390_pci_dev_release(struct zpci_dev *zdev) · line 552

Quoted 4 lines
arch/s390/kvm/pci.c +4 -0
+	}
+
+	if (zpci_enable_device(zdev))
+		goto out;

NIT debug trace?

Only some questions, otherwise, LGTM

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
author maintainer
#23 lore raw

Re: [PATCH v9 16/21] KVM: s390: pci: add routines to start/stop interpretive execution

Quoted 62 lines · Pierre Morel
Quoted 59 lines · Matthew Rosato

These routines will be invoked at the time an s390x vfio-pci device is associated with a KVM (or when the association is removed), allowing the zPCI device to enable or disable load/store intepretation mode; this requires the host zPCI device to inform firmware of the unique token (GISA designation) that is associated with the owning KVM.

---
  arch/s390/include/asm/kvm_host.h |  18 ++++
  arch/s390/include/asm/pci.h      |   1 +
  arch/s390/kvm/kvm-s390.c         |  15 +++

  arch/s390/kvm/pci.c              | 162 +++++++++++++++++++++++++++++++
  arch/s390/kvm/pci.h              |   5 +
  arch/s390/pci/pci.c              |   4 +
  6 files changed, 205 insertions(+)
arch/s390/include/asm/kvm_host.h +18 -0
index 8e381603b6a7..6e83d746bae2 100644
@@ -19,6 +19,7 @@
   #include <linux/kvm.h>
   #include <linux/seqlock.h>
   #include <linux/module.h>
+#include <linux/pci.h>
   #include <asm/debug.h>
   #include <asm/cpu.h>
   #include <asm/fpu/api.h>
@@ -967,6 +968,8 @@ struct kvm_arch{
       DECLARE_BITMAP(idle_mask, KVM_MAX_VCPUS);
       struct kvm_s390_gisa_interrupt gisa_int;
       struct kvm_s390_pv pv;
+    struct list_head kzdev_list;
+    spinlock_t kzdev_list_lock;
   };
   #define KVM_HVA_ERR_BAD        (-1UL)
@@ -1017,4 +1020,19 @@ static inline void 
 kvm_arch_flush_shadow_memslot(struct kvm *kvm,
   static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) {}
   static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu) {}
+#define __KVM_HAVE_ARCH_VM_FREE
+void kvm_arch_free_vm(struct kvm *kvm);
+
+#ifdef CONFIG_VFIO_PCI_ZDEV_KVM
+int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm);
+void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev);
+#else
+static inline int kvm_s390_pci_register_kvm(struct zpci_dev *dev,
+                        struct kvm *kvm)
+{
+    return -EPERM;
+}
+static inline void kvm_s390_pci_unregister_kvm(struct zpci_dev *dev) {}
+#endif
+
   #endif
arch/s390/include/asm/pci.h +0 -1
index 322060a75d9f..85eb0ef9d4c3 100644
--- a/arch/s390/include/asm/pci.h
Quoted 8 lines
Quoted 6 lines

+++ b/arch/s390/include/asm/pci.h

arch/s390/include/asm/pci.h +0 -0
@@ -194,6 +194,7 @@ struct zpci_dev {
       /* IOMMU and passthrough */

      struct s390_domain *s390_domain; /* s390 IOMMU domain data */ struct kvm_zdev *kzdev;
+    struct mutex kzdev_lock;

I guess that since it did not exist before the lock is not there to protect the zpci_dev struct.

Right, not the zpci_dev itself but it is protecting the contents of the kzdev (including the pointer to the zdev e.g. kzdev->zdev)

Quoted 1 line

May be add a comment to say what it is protecting.

Sure

Quoted 70 lines
Quoted 70 lines

  };
  static inline bool zdev_enabled(struct zpci_dev *zdev)

arch/s390/kvm/kvm-s390.c +15 -0
index a66da3f66114..4758bb731199 100644
@@ -2790,6 +2790,14 @@ static void sca_dispose(struct kvm *kvm)
       kvm->arch.sca = NULL;
   }
+void kvm_arch_free_vm(struct kvm *kvm)
+{
+    if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
+        kvm_s390_pci_clear_list(kvm);
+
+    __kvm_arch_free_vm(kvm);
+}
+
   int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
   {
       gfp_t alloc_flags = GFP_KERNEL_ACCOUNT;
@@ -2872,6 +2880,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned 
 long type)
       kvm_s390_crypto_init(kvm);
+    if (IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM)) {
+        mutex_lock(&kvm->lock);
+        kvm_s390_pci_init_list(kvm);
+        kvm_s390_vcpu_pci_enable_interp(kvm);
+        mutex_unlock(&kvm->lock);
+    }
+
       mutex_init(&kvm->arch.float_int.ais_lock);
       spin_lock_init(&kvm->arch.float_int.lock);
       for (i = 0; i < FIRQ_LIST_COUNT; i++)
arch/s390/kvm/pci.c +5 -0
index b232c8cbaa81..24211741deb0 100644
@@ -12,7 +12,9 @@
   #include <asm/pci.h>
   #include <asm/pci_insn.h>
   #include <asm/pci_io.h>
+#include <asm/sclp.h>
   #include "pci.h"
+#include "kvm-s390.h"
   struct zpci_aift *aift;
@@ -423,6 +425,166 @@ static void kvm_s390_pci_dev_release(struct 
 zpci_dev *zdev)
       kfree(kzdev);
   }
+
+/*
+ * Register device with the specified KVM. If interpetation 

facilities are
+ * available, enable them and let userspace indicate whether or not they will
+ * be used (specify SHM bit to disable).
+ */

arch/s390/kvm/pci.c +11 -0
+int kvm_s390_pci_register_kvm(struct zpci_dev *zdev, struct kvm *kvm)
+{
+    int rc;
+
+    if (!zdev)
+        return -EINVAL;
+
+    mutex_lock(&zdev->kzdev_lock);
+
+    if (zdev->kzdev || zdev->gisa != 0 || !kvm) {
+        mutex_unlock(&zdev->kzdev_lock);
Quoted 8 lines

arch/s390/kvm/pci.c static void kvm_s390_pci_dev_release(struct zpci_dev *zdev) · line 450

Quoted 6 lines

+        return -EINVAL;
+    }

arch/s390/kvm/pci.c +4 -0
+
+    kvm_get_kvm(kvm);
+
+    mutex_lock(&kvm->lock);

Why do we need to lock KVM here?

Hmm, good point, now that we get a reference this seems unnecessary

Quoted 72 lines

just a question, I do not think it is a big problem.

Quoted 69 lines
arch/s390/kvm/pci.c +14 -0
+
+    rc = kvm_s390_pci_dev_open(zdev);
+    if (rc)
+        goto err;
+
+    /*
+     * If interpretation facilities aren't available, add the device to
+     * the kzdev list but don't enable for interpretation.
+     */
+    if (!kvm_s390_pci_interp_allowed())
+        goto out;
+
+    /*
+     * If this is the first request to use an interpreted device, 

make the
+     * necessary vcpu changes
+     */
+    if (!kvm->arch.use_zpci_interp)
+        kvm_s390_vcpu_pci_enable_interp(kvm);

arch/s390/kvm/pci.c +50 -0
+
+    if (zdev_enabled(zdev)) {
+        rc = zpci_disable_device(zdev);
+        if (rc)
+            goto err;
+    }
+
+    /*
+     * Store information about the identity of the kvm guest allowed to
+     * access this device via interpretation to be used by host CLP
+     */
+    zdev->gisa = (u32)virt_to_phys(&kvm->arch.sie_page2->gisa);
+
+    rc = zpci_enable_device(zdev);
+    if (rc)
+        goto clear_gisa;
+
+    /* Re-register the IOMMU that was already created */
+    rc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma,
+                virt_to_phys(zdev->dma_table));
+    if (rc)
+        goto clear_gisa;
+
+out:
+    zdev->kzdev->kvm = kvm;
+
+    spin_lock(&kvm->arch.kzdev_list_lock);
+    list_add_tail(&zdev->kzdev->entry, &kvm->arch.kzdev_list);
+    spin_unlock(&kvm->arch.kzdev_list_lock);
+
+    mutex_unlock(&kvm->lock);
+    mutex_unlock(&zdev->kzdev_lock);
+    return 0;
+
+clear_gisa:
+    zdev->gisa = 0;
+err:
+    if (zdev->kzdev)
+        kvm_s390_pci_dev_release(zdev);
+    mutex_unlock(&kvm->lock);
+    mutex_unlock(&zdev->kzdev_lock);
+    kvm_put_kvm(kvm);
+    return rc;
+}
+EXPORT_SYMBOL_GPL(kvm_s390_pci_register_kvm);
+
+void kvm_s390_pci_unregister_kvm(struct zpci_dev *zdev)
+{
+    struct kvm *kvm;
+
Quoted 9 lines

arch/s390/kvm/pci.c static void kvm_s390_pci_dev_release(struct zpci_dev *zdev) · line 524

Quoted 6 lines

+    if (!zdev)
+        return;

arch/s390/kvm/pci.c +4 -0
+
+    mutex_lock(&zdev->kzdev_lock);
+
+    if (WARN_ON(!zdev->kzdev)) {

When can this happen ?

It cannot today, nor should it ever (hence the WARN_ON) -- if we do, it's a case of programming error introduced somewhere (vfio has a KVM reference but we never built a kzdev via kvm_s390_pci_register_kvm or lost it somehow)

Quoted 18 lines
Quoted 18 lines

+        mutex_unlock(&zdev->kzdev_lock);
+        return;
+    }

arch/s390/kvm/pci.c +15 -0
+
+    kvm = zdev->kzdev->kvm;
+    mutex_lock(&kvm->lock);
+
+    /*
+     * A 0 gisa means interpretation was never enabled, just remove the
+     * device from the list.
+     */
+    if (zdev->gisa == 0)
+        goto out;
+
+    /* Forwarding must be turned off before interpretation */
+    if (zdev->kzdev->fib.fmt0.aibv != 0)
+        kvm_s390_pci_aif_disable(zdev, true);
+
Quoted 8 lines
Quoted 6 lines

+    /* Remove the host CLP guest designation */
+    zdev->gisa = 0;
+
+    if (zdev_enabled(zdev)) {
+        if (zpci_disable_device(zdev))
+            goto out;

NIT debug trace ?

We should at least get a trace entry in from clp_disable_fh() if something goes wrong here.

Quoted 7 lines
Quoted 5 lines

+    }
+
+    if (zpci_enable_device(zdev))
+        goto out;

NIT debug trace?

And similarly, a trace entry from clp_enable_fh() here. So I think these are OK for now.

I am consdering a follow-on to add new s390dbf entries for 'kvm-pci' or so, these might make sense there for additional context, but let's leave that for after this series.

Quoted 5 lines

Only some questions, otherwise, LGTM

Thanks!

author maintainer
#24 lore raw

PATCH v9 17/21 vfio-pci/zdev: add open/close device hooks

In reply to Matthew Rosato

During vfio-pci open_device, pass the KVM associated with the vfio group (if one exists). This is needed in order to pass a special indicator (GISA) to firmware to allow zPCI interpretation facilities to be used for only the specific KVM associated with the vfio-pci device. During vfio-pci close_device, unregister the notifier.

---
 drivers/vfio/pci/vfio_pci_core.c | 10 +++++++++-
 drivers/vfio/pci/vfio_pci_zdev.c | 24 ++++++++++++++++++++++++
 include/linux/vfio_pci_core.h    | 10 ++++++++++
 3 files changed, 43 insertions(+), 1 deletion(-)
drivers/vfio/pci/vfio_pci_core.c +9 -1
index a0d69ddaf90d..b1e5cfbadf38 100644
@@ -316,10 +316,14 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 		pci_write_config_word(pdev, PCI_COMMAND, cmd);
 	}
 
-	ret = vfio_config_init(vdev);
+	ret = vfio_pci_zdev_open_device(vdev);
 	if (ret)
 		goto out_free_state;
 
+	ret = vfio_config_init(vdev);
+	if (ret)
+		goto out_free_zdev;
+
 	msix_pos = pdev->msix_cap;
 	if (msix_pos) {
 		u16 flags;
@@ -340,6 +344,8 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 
 	return 0;
 
+out_free_zdev:
+	vfio_pci_zdev_close_device(vdev);
 out_free_state:
 	kfree(vdev->pci_saved_state);
 	vdev->pci_saved_state = NULL;
@@ -418,6 +424,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 
 	vdev->needs_reset = true;
 
+	vfio_pci_zdev_close_device(vdev);
+
 	/*
 	 * If we have saved state, restore it.  If we can reset the device,
 	 * even better.  Resetting with current state seems better than
drivers/vfio/pci/vfio_pci_zdev.c +24 -0
index ea4c0d2b0663..686f2e75e392 100644
@@ -11,6 +11,7 @@
 #include <linux/uaccess.h>
 #include <linux/vfio.h>
 #include <linux/vfio_zdev.h>
+#include <linux/kvm_host.h>
 #include <asm/pci_clp.h>
 #include <asm/pci_io.h>
 
@@ -136,3 +137,26 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 
 	return ret;
 }
+
+int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
+{
+	struct zpci_dev *zdev = to_zpci(vdev->pdev);
+
+	if (!zdev)
+		return -ENODEV;
+
+	if (!vdev->vdev.kvm)
+		return 0;
+
+	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+}
+
+void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
+{
+	struct zpci_dev *zdev = to_zpci(vdev->pdev);
+
+	if (!zdev || !vdev->vdev.kvm)
+		return;
+
+	kvm_s390_pci_unregister_kvm(zdev);
+}
include/linux/vfio_pci_core.h +10 -0
index 63af2897939c..d5d9e17f0156 100644
@@ -209,12 +209,22 @@ static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev)
 #ifdef CONFIG_VFIO_PCI_ZDEV_KVM
 extern int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 				       struct vfio_info_cap *caps);
+int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev);
+void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev);
 #else
 static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 					      struct vfio_info_cap *caps)
 {
 	return -ENODEV;
 }
+
+static inline int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+static inline void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
+{}
 #endif
 
 /* Will be exported for vfio pci drivers usage */
Signature · 1 line
2.27.0
Reviewed-by on v9 17/21
#25 lore raw

Re: [PATCH v9 17/21] vfio-pci/zdev: add open/close device hooks

Quoted 7 lines · Matthew Rosato

During vfio-pci open_device, pass the KVM associated with the vfio group (if one exists). This is needed in order to pass a special indicator (GISA) to firmware to allow zPCI interpretation facilities to be used for only the specific KVM associated with the vfio-pci device. During vfio-pci close_device, unregister the notifier.

Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
Acked-by on v9 17/21
#26 lore raw

Re: [PATCH v9 17/21] vfio-pci/zdev: add open/close device hooks

In reply to Matthew Rosato
Quoted 6 lines · On Mon, 6 Jun 2022 16:33:21 -0400 Matthew Rosato <[email protected]>…

During vfio-pci open_device, pass the KVM associated with the vfio group (if one exists). This is needed in order to pass a special indicator (GISA) to firmware to allow zPCI interpretation facilities to be used for only the specific KVM associated with the vfio-pci device. During vfio-pci close_device, unregister the notifier.

Quoted 6 lines
---
 drivers/vfio/pci/vfio_pci_core.c | 10 +++++++++-
 drivers/vfio/pci/vfio_pci_zdev.c | 24 ++++++++++++++++++++++++
 include/linux/vfio_pci_core.h    | 10 ++++++++++
 3 files changed, 43 insertions(+), 1 deletion(-)
Quoted 104 lines
drivers/vfio/pci/vfio_pci_core.c +9 -1
index a0d69ddaf90d..b1e5cfbadf38 100644
@@ -316,10 +316,14 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 		pci_write_config_word(pdev, PCI_COMMAND, cmd);
 	}
 
-	ret = vfio_config_init(vdev);
+	ret = vfio_pci_zdev_open_device(vdev);
 	if (ret)
 		goto out_free_state;
 
+	ret = vfio_config_init(vdev);
+	if (ret)
+		goto out_free_zdev;
+
 	msix_pos = pdev->msix_cap;
 	if (msix_pos) {
 		u16 flags;
@@ -340,6 +344,8 @@ int vfio_pci_core_enable(struct vfio_pci_core_device *vdev)
 
 	return 0;
 
+out_free_zdev:
+	vfio_pci_zdev_close_device(vdev);
 out_free_state:
 	kfree(vdev->pci_saved_state);
 	vdev->pci_saved_state = NULL;
@@ -418,6 +424,8 @@ void vfio_pci_core_disable(struct vfio_pci_core_device *vdev)
 
 	vdev->needs_reset = true;
 
+	vfio_pci_zdev_close_device(vdev);
+
 	/*
 	 * If we have saved state, restore it.  If we can reset the device,
 	 * even better.  Resetting with current state seems better than
drivers/vfio/pci/vfio_pci_zdev.c +24 -0
index ea4c0d2b0663..686f2e75e392 100644
@@ -11,6 +11,7 @@
 #include <linux/uaccess.h>
 #include <linux/vfio.h>
 #include <linux/vfio_zdev.h>
+#include <linux/kvm_host.h>
 #include <asm/pci_clp.h>
 #include <asm/pci_io.h>
 
@@ -136,3 +137,26 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 
 	return ret;
 }
+
+int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
+{
+	struct zpci_dev *zdev = to_zpci(vdev->pdev);
+
+	if (!zdev)
+		return -ENODEV;
+
+	if (!vdev->vdev.kvm)
+		return 0;
+
+	return kvm_s390_pci_register_kvm(zdev, vdev->vdev.kvm);
+}
+
+void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
+{
+	struct zpci_dev *zdev = to_zpci(vdev->pdev);
+
+	if (!zdev || !vdev->vdev.kvm)
+		return;
+
+	kvm_s390_pci_unregister_kvm(zdev);
+}
include/linux/vfio_pci_core.h +10 -0
index 63af2897939c..d5d9e17f0156 100644
@@ -209,12 +209,22 @@ static inline int vfio_pci_igd_init(struct vfio_pci_core_device *vdev)
 #ifdef CONFIG_VFIO_PCI_ZDEV_KVM
 extern int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 				       struct vfio_info_cap *caps);
+int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev);
+void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev);
 #else
 static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device *vdev,
 					      struct vfio_info_cap *caps)
 {
 	return -ENODEV;
 }
+
+static inline int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev)
+{
+	return 0;
+}
+
+static inline void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev)
+{}
 #endif
 
 /* Will be exported for vfio pci drivers usage */
author maintainer
#27 lore raw

PATCH v9 18/21 vfio-pci/zdev: add function handle to clp base capability

In reply to Matthew Rosato

The function handle is a system-wide unique identifier for a zPCI device. With zPCI instruction interpretation, the host will no longer be executing the zPCI instructions on behalf of the guest. As a result, the guest needs to use the real function handle in order for firmware to associate the instruction with the proper PCI function. Let's provide that handle to the guest.

---
 drivers/vfio/pci/vfio_pci_zdev.c | 5 +++--
 include/uapi/linux/vfio_zdev.h   | 3 +++
 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_zdev.c +3 -2
index 686f2e75e392..4f28cdd7ecd1 100644
@@ -24,14 +24,15 @@ static int zpci_base_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
 {
 	struct vfio_device_info_cap_zpci_base cap = {
 		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_BASE,
-		.header.version = 1,
+		.header.version = 2,
 		.start_dma = zdev->start_dma,
 		.end_dma = zdev->end_dma,
 		.pchid = zdev->pchid,
 		.vfn = zdev->vfn,
 		.fmb_length = zdev->fmb_length,
 		.pft = zdev->pft,
-		.gid = zdev->pfgid
+		.gid = zdev->pfgid,
+		.fh = zdev->fh
 	};
 
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
include/uapi/linux/vfio_zdev.h +3 -0
index b4309397b6b2..78c022af3d29 100644
@@ -29,6 +29,9 @@ struct vfio_device_info_cap_zpci_base {
 	__u16 fmb_length;	/* Measurement Block Length (in bytes) */
 	__u8 pft;		/* PCI Function Type */
 	__u8 gid;		/* PCI function group ID */
+	/* End of version 1 */
+	__u32 fh;		/* PCI function handle */
+	/* End of version 2 */
 };
 
 /**
Signature · 1 line
2.27.0
Acked-by on v9 18/21
#28 lore raw

Re: [PATCH v9 18/21] vfio-pci/zdev: add function handle to clp base capability

Quoted 7 lines · On Mon, 6 Jun 2022 16:33:22 -0400 Matthew Rosato <[email protected]>…

The function handle is a system-wide unique identifier for a zPCI device. With zPCI instruction interpretation, the host will no longer be executing the zPCI instructions on behalf of the guest. As a result, the guest needs to use the real function handle in order for firmware to associate the instruction with the proper PCI function. Let's provide that handle to the guest.

Quoted 7 lines
---
 drivers/vfio/pci/vfio_pci_zdev.c | 5 +++--
 include/uapi/linux/vfio_zdev.h   | 3 +++
 2 files changed, 6 insertions(+), 2 deletions(-)
Quoted 36 lines
drivers/vfio/pci/vfio_pci_zdev.c +3 -2
index 686f2e75e392..4f28cdd7ecd1 100644
@@ -24,14 +24,15 @@ static int zpci_base_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
 {
 	struct vfio_device_info_cap_zpci_base cap = {
 		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_BASE,
-		.header.version = 1,
+		.header.version = 2,
 		.start_dma = zdev->start_dma,
 		.end_dma = zdev->end_dma,
 		.pchid = zdev->pchid,
 		.vfn = zdev->vfn,
 		.fmb_length = zdev->fmb_length,
 		.pft = zdev->pft,
-		.gid = zdev->pfgid
+		.gid = zdev->pfgid,
+		.fh = zdev->fh
 	};
 
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
include/uapi/linux/vfio_zdev.h +3 -0
index b4309397b6b2..78c022af3d29 100644
@@ -29,6 +29,9 @@ struct vfio_device_info_cap_zpci_base {
 	__u16 fmb_length;	/* Measurement Block Length (in bytes) */
 	__u8 pft;		/* PCI Function Type */
 	__u8 gid;		/* PCI function group ID */
+	/* End of version 1 */
+	__u32 fh;		/* PCI function handle */
+	/* End of version 2 */
 };
 
 /**
author maintainer
#29 lore raw

PATCH v9 19/21 vfio-pci/zdev: different maxstbl for interpreted devices

In reply to Matthew Rosato

When doing load/store interpretation, the maximum store block length is determined by the underlying firmware, not the host kernel API. Reflect that in the associated Query PCI Function Group clp capability and let userspace decide which is appropriate to present to the guest.

---
 drivers/vfio/pci/vfio_pci_zdev.c | 6 ++++--
 include/uapi/linux/vfio_zdev.h   | 4 ++++
 2 files changed, 8 insertions(+), 2 deletions(-)
drivers/vfio/pci/vfio_pci_zdev.c +4 -2
index 4f28cdd7ecd1..e163aa9f6144 100644
@@ -45,14 +45,16 @@ static int zpci_group_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
 {
 	struct vfio_device_info_cap_zpci_group cap = {
 		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_GROUP,
-		.header.version = 1,
+		.header.version = 2,
 		.dasm = zdev->dma_mask,
 		.msi_addr = zdev->msi_addr,
 		.flags = VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH,
 		.mui = zdev->fmb_update,
 		.noi = zdev->max_msi,
 		.maxstbl = ZPCI_MAX_WRITE_SIZE,
-		.version = zdev->version
+		.version = zdev->version,
+		.reserved = 0,
+		.imaxstbl = zdev->maxstbl
 	};
 
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
include/uapi/linux/vfio_zdev.h +4 -0
index 78c022af3d29..77f2aff1f27e 100644
@@ -50,6 +50,10 @@ struct vfio_device_info_cap_zpci_group {
 	__u16 noi;		/* Maximum number of MSIs */
 	__u16 maxstbl;		/* Maximum Store Block Length */
 	__u8 version;		/* Supported PCI Version */
+	/* End of version 1 */
+	__u8 reserved;
+	__u16 imaxstbl;		/* Maximum Interpreted Store Block Length */
+	/* End of version 2 */
 };
 
 /**
Signature · 1 line
2.27.0
Acked-by on v9 19/21
#30 lore raw

Re: [PATCH v9 19/21] vfio-pci/zdev: different maxstbl for interpreted devices

Quoted 11 lines · On Mon, 6 Jun 2022 16:33:23 -0400 Matthew Rosato <[email protected]>…

When doing load/store interpretation, the maximum store block length is determined by the underlying firmware, not the host kernel API. Reflect that in the associated Query PCI Function Group clp capability and let userspace decide which is appropriate to present to the guest.

---
 drivers/vfio/pci/vfio_pci_zdev.c | 6 ++++--
 include/uapi/linux/vfio_zdev.h   | 4 ++++
 2 files changed, 8 insertions(+), 2 deletions(-)
Quoted 38 lines
drivers/vfio/pci/vfio_pci_zdev.c +4 -2
index 4f28cdd7ecd1..e163aa9f6144 100644
@@ -45,14 +45,16 @@ static int zpci_group_cap(struct zpci_dev *zdev, struct vfio_info_cap *caps)
 {
 	struct vfio_device_info_cap_zpci_group cap = {
 		.header.id = VFIO_DEVICE_INFO_CAP_ZPCI_GROUP,
-		.header.version = 1,
+		.header.version = 2,
 		.dasm = zdev->dma_mask,
 		.msi_addr = zdev->msi_addr,
 		.flags = VFIO_DEVICE_INFO_ZPCI_FLAG_REFRESH,
 		.mui = zdev->fmb_update,
 		.noi = zdev->max_msi,
 		.maxstbl = ZPCI_MAX_WRITE_SIZE,
-		.version = zdev->version
+		.version = zdev->version,
+		.reserved = 0,
+		.imaxstbl = zdev->maxstbl
 	};
 
 	return vfio_info_add_capability(caps, &cap.header, sizeof(cap));
include/uapi/linux/vfio_zdev.h +4 -0
index 78c022af3d29..77f2aff1f27e 100644
@@ -50,6 +50,10 @@ struct vfio_device_info_cap_zpci_group {
 	__u16 noi;		/* Maximum number of MSIs */
 	__u16 maxstbl;		/* Maximum Store Block Length */
 	__u8 version;		/* Supported PCI Version */
+	/* End of version 1 */
+	__u8 reserved;
+	__u16 imaxstbl;		/* Maximum Interpreted Store Block Length */
+	/* End of version 2 */
 };
 
 /**
author maintainer
#31 lore raw

PATCH v9 20/21 KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

In reply to Matthew Rosato

The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing hardware-assisted virtualization features for s390x zPCI passthrough. Add the first 2 operations, which can be used to enable/disable the specified device for Adapter Event Notification interpretation.

---
 Documentation/virt/kvm/api.rst | 47 +++++++++++++++++++
 arch/s390/kvm/kvm-s390.c       | 16 +++++++
 arch/s390/kvm/pci.c            | 85 ++++++++++++++++++++++++++++++++++
 arch/s390/kvm/pci.h            |  2 +
 include/uapi/linux/kvm.h       | 31 +++++++++++++
 5 files changed, 181 insertions(+)
Documentation/virt/kvm/api.rst +47 -0
index 11e00a46c610..2eb604769dce 100644
@@ -5802,6 +5802,53 @@ of CPUID leaf 0xD on the host.
 
 This ioctl injects an event channel interrupt directly to the guest vCPU.
 
+4.136 KVM_S390_ZPCI_OP
+--------------------
+
+:Capability: KVM_CAP_S390_ZPCI_OP
+:Architectures: s390
+:Type: vm ioctl
+:Parameters: struct kvm_s390_zpci_op (in)
+:Returns: 0 on success, <0 on error
+
+Used to manage hardware-assisted virtualization features for zPCI devices.
+
+Parameters are specified via the following structure::
+
+  struct kvm_s390_zpci_op {
+	/* in */
+	__u32 fh;		/* target device */
+	__u8  op;		/* operation to perform */
+	__u8  pad[3];
+	union {
+		/* for KVM_S390_ZPCIOP_REG_AEN */
+		struct {
+			__u64 ibv;	/* Guest addr of interrupt bit vector */
+			__u64 sb;	/* Guest addr of summary bit */
+			__u32 flags;
+			__u32 noi;	/* Number of interrupts */
+			__u8 isc;	/* Guest interrupt subclass */
+			__u8 sbo;	/* Offset of guest summary bit vector */
+			__u16 pad;
+		} reg_aen;
+		__u64 reserved[8];
+	} u;
+  };
+
+The type of operation is specified in the "op" field.
+KVM_S390_ZPCIOP_REG_AEN is used to register the VM for adapter event
+notification interpretation, which will allow firmware delivery of adapter
+events directly to the vm, with KVM providing a backup delivery mechanism;
+KVM_S390_ZPCIOP_DEREG_AEN is used to subsequently disable interpretation of
+adapter event notifications.
+
+The target zPCI function must also be specified via the "fh" field.  For the
+KVM_S390_ZPCIOP_REG_AEN operation, additional information to establish firmware
+delivery must be provided via the "reg_aen" struct.
+
+The "pad" and "reserved" fields may be used for future extensions and should be
+set to 0s by userspace.
+
 5. The kvm_run structure
 ========================
 
arch/s390/kvm/kvm-s390.c +16 -0
index 4758bb731199..f214e0fc62ed 100644
@@ -618,6 +618,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_S390_PROTECTED:
 		r = is_prot_virt_host();
 		break;
+	case KVM_CAP_S390_ZPCI_OP:
+		r = kvm_s390_pci_interp_allowed();
+		break;
 	default:
 		r = 0;
 	}
@@ -2629,6 +2632,19 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			r = -EFAULT;
 		break;
 	}
+	case KVM_S390_ZPCI_OP: {
+		struct kvm_s390_zpci_op args;
+
+		r = -EINVAL;
+		if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
+			break;
+		if (copy_from_user(&args, argp, sizeof(args))) {
+			r = -EFAULT;
+			break;
+		}
+		r = kvm_s390_pci_zpci_op(kvm, &args);
+		break;
+	}
 	default:
 		r = -ENOTTY;
 	}
arch/s390/kvm/pci.c +85 -0

91 lines not on this page · raw message

arch/s390/kvm/pci.h +2 -0
index fb2b91b76e0c..0351382e990f 100644
@@ -59,6 +59,8 @@ void kvm_s390_pci_aen_exit(void);
 void kvm_s390_pci_init_list(struct kvm *kvm);
 void kvm_s390_pci_clear_list(struct kvm *kvm);
 
+int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);
+
 int kvm_s390_pci_init(void);
 void kvm_s390_pci_exit(void);
 
include/uapi/linux/kvm.h +31 -0
index 5088bd9f1922..52cc12d53022 100644
@@ -1157,6 +1157,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_VM_TSC_CONTROL 214
 #define KVM_CAP_SYSTEM_EVENT_DATA 215
 #define KVM_CAP_ARM_SYSTEM_SUSPEND 216
+#define KVM_CAP_S390_ZPCI_OP 217
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -2118,4 +2119,34 @@ struct kvm_stats_desc {
 /* Available with KVM_CAP_XSAVE2 */
 #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
 
+/* Available with KVM_CAP_S390_ZPCI_OP */
+#define KVM_S390_ZPCI_OP         _IOW(KVMIO,  0xd1, struct kvm_s390_zpci_op)
+
+struct kvm_s390_zpci_op {
+	/* in */
+	__u32 fh;               /* target device */
+	__u8  op;               /* operation to perform */
+	__u8  pad[3];
+	union {
+		/* for KVM_S390_ZPCIOP_REG_AEN */
+		struct {
+			__u64 ibv;      /* Guest addr of interrupt bit vector */
+			__u64 sb;       /* Guest addr of summary bit */
+			__u32 flags;
+			__u32 noi;      /* Number of interrupts */
+			__u8 isc;       /* Guest interrupt subclass */
+			__u8 sbo;       /* Offset of guest summary bit vector */
+			__u16 pad;
+		} reg_aen;
+		__u64 reserved[8];
+	} u;
+};
+
+/* types for kvm_s390_zpci_op->op */
+#define KVM_S390_ZPCIOP_REG_AEN                0
+#define KVM_S390_ZPCIOP_DEREG_AEN      1
+
+/* flags for kvm_s390_zpci_op->u.reg_aen.flags */
+#define KVM_S390_ZPCIOP_REGAEN_HOST    (1 << 0)
+
 #endif /* __LINUX_KVM_H */
Signature · 1 line
2.27.0
Reviewed-by on v9 20/21
#32 lore raw

Re: [PATCH v9 20/21] KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

Quoted 5 lines · Matthew Rosato

The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing hardware-assisted virtualization features for s390x zPCI passthrough. Add the first 2 operations, which can be used to enable/disable the specified device for Adapter Event Notification interpretation.

Quoted 8 lines
---
  Documentation/virt/kvm/api.rst | 47 +++++++++++++++++++
  arch/s390/kvm/kvm-s390.c       | 16 +++++++
  arch/s390/kvm/pci.c            | 85 ++++++++++++++++++++++++++++++++++
  arch/s390/kvm/pci.h            |  2 +
  include/uapi/linux/kvm.h       | 31 +++++++++++++
  5 files changed, 181 insertions(+)
Acked-by on v9 20/21
#33 lore raw

Re: [PATCH v9 20/21] KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices

In reply to Matthew Rosato
Quoted 6 lines · Matthew Rosato

The KVM_S390_ZPCI_OP ioctl provides a mechanism for managing hardware-assisted virtualization features for s390x zPCI passthrough. Add the first 2 operations, which can be used to enable/disable the specified device for Adapter Event Notification interpretation.

Despite I really do not like passing the FH in the structure. I think that if I admit that then this LGTM.

Quoted 257 lines
---
  Documentation/virt/kvm/api.rst | 47 +++++++++++++++++++
  arch/s390/kvm/kvm-s390.c       | 16 +++++++
  arch/s390/kvm/pci.c            | 85 ++++++++++++++++++++++++++++++++++
  arch/s390/kvm/pci.h            |  2 +
  include/uapi/linux/kvm.h       | 31 +++++++++++++
  5 files changed, 181 insertions(+)
Documentation/virt/kvm/api.rst +47 -0
index 11e00a46c610..2eb604769dce 100644
@@ -5802,6 +5802,53 @@ of CPUID leaf 0xD on the host.
 
  This ioctl injects an event channel interrupt directly to the guest vCPU.
 
+4.136 KVM_S390_ZPCI_OP
+--------------------
+
+:Capability: KVM_CAP_S390_ZPCI_OP
+:Architectures: s390
+:Type: vm ioctl
+:Parameters: struct kvm_s390_zpci_op (in)
+:Returns: 0 on success, <0 on error
+
+Used to manage hardware-assisted virtualization features for zPCI devices.
+
+Parameters are specified via the following structure::
+
+  struct kvm_s390_zpci_op {
+	/* in */
+	__u32 fh;		/* target device */
+	__u8  op;		/* operation to perform */
+	__u8  pad[3];
+	union {
+		/* for KVM_S390_ZPCIOP_REG_AEN */
+		struct {
+			__u64 ibv;	/* Guest addr of interrupt bit vector */
+			__u64 sb;	/* Guest addr of summary bit */
+			__u32 flags;
+			__u32 noi;	/* Number of interrupts */
+			__u8 isc;	/* Guest interrupt subclass */
+			__u8 sbo;	/* Offset of guest summary bit vector */
+			__u16 pad;
+		} reg_aen;
+		__u64 reserved[8];
+	} u;
+  };
+
+The type of operation is specified in the "op" field.
+KVM_S390_ZPCIOP_REG_AEN is used to register the VM for adapter event
+notification interpretation, which will allow firmware delivery of adapter
+events directly to the vm, with KVM providing a backup delivery mechanism;
+KVM_S390_ZPCIOP_DEREG_AEN is used to subsequently disable interpretation of
+adapter event notifications.
+
+The target zPCI function must also be specified via the "fh" field.  For the
+KVM_S390_ZPCIOP_REG_AEN operation, additional information to establish firmware
+delivery must be provided via the "reg_aen" struct.
+
+The "pad" and "reserved" fields may be used for future extensions and should be
+set to 0s by userspace.
+
  5. The kvm_run structure
  ========================
 
arch/s390/kvm/kvm-s390.c +16 -0
index 4758bb731199..f214e0fc62ed 100644
@@ -618,6 +618,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
  	case KVM_CAP_S390_PROTECTED:
  		r = is_prot_virt_host();
  		break;
+	case KVM_CAP_S390_ZPCI_OP:
+		r = kvm_s390_pci_interp_allowed();
+		break;
  	default:
  		r = 0;
  	}
@@ -2629,6 +2632,19 @@ long kvm_arch_vm_ioctl(struct file *filp,
  			r = -EFAULT;
  		break;
  	}
+	case KVM_S390_ZPCI_OP: {
+		struct kvm_s390_zpci_op args;
+
+		r = -EINVAL;
+		if (!IS_ENABLED(CONFIG_VFIO_PCI_ZDEV_KVM))
+			break;
+		if (copy_from_user(&args, argp, sizeof(args))) {
+			r = -EFAULT;
+			break;
+		}
+		r = kvm_s390_pci_zpci_op(kvm, &args);
+		break;
+	}
  	default:
  		r = -ENOTTY;
  	}
arch/s390/kvm/pci.c +85 -0
index 24211741deb0..4946fb7757d6 100644
@@ -585,6 +585,91 @@ void kvm_s390_pci_clear_list(struct kvm *kvm)
  	spin_unlock(&kvm->arch.kzdev_list_lock);
  }
 
+static struct zpci_dev *get_zdev_from_kvm_by_fh(struct kvm *kvm, u32 fh)
+{
+	struct zpci_dev *zdev = NULL;
+	struct kvm_zdev *kzdev;
+
+	spin_lock(&kvm->arch.kzdev_list_lock);
+	list_for_each_entry(kzdev, &kvm->arch.kzdev_list, entry) {
+		if (kzdev->zdev->fh == fh) {
+			zdev = kzdev->zdev;
+			break;
+		}
+	}
+	spin_unlock(&kvm->arch.kzdev_list_lock);
+
+	return zdev;
+}
+
+static int kvm_s390_pci_zpci_reg_aen(struct zpci_dev *zdev,
+				     struct kvm_s390_zpci_op *args)
+{
+	struct zpci_fib fib = {};
+	bool hostflag;
+
+	fib.fmt0.aibv = args->u.reg_aen.ibv;
+	fib.fmt0.isc = args->u.reg_aen.isc;
+	fib.fmt0.noi = args->u.reg_aen.noi;
+	if (args->u.reg_aen.sb != 0) {
+		fib.fmt0.aisb = args->u.reg_aen.sb;
+		fib.fmt0.aisbo = args->u.reg_aen.sbo;
+		fib.fmt0.sum = 1;
+	} else {
+		fib.fmt0.aisb = 0;
+		fib.fmt0.aisbo = 0;
+		fib.fmt0.sum = 0;
+	}
+
+	hostflag = !(args->u.reg_aen.flags & KVM_S390_ZPCIOP_REGAEN_HOST);
+	return kvm_s390_pci_aif_enable(zdev, &fib, hostflag);
+}
+
+int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args)
+{
+	struct kvm_zdev *kzdev;
+	struct zpci_dev *zdev;
+	int r;
+
+	zdev = get_zdev_from_kvm_by_fh(kvm, args->fh);
+	if (!zdev)
+		return -ENODEV;
+
+	mutex_lock(&zdev->kzdev_lock);
+	mutex_lock(&kvm->lock);
+
+	kzdev = zdev->kzdev;
+	if (!kzdev) {
+		r = -ENODEV;
+		goto out;
+	}
+	if (kzdev->kvm != kvm) {
+		r = -EPERM;
+		goto out;
+	}
+
+	switch (args->op) {
+	case KVM_S390_ZPCIOP_REG_AEN:
+		/* Fail on unknown flags */
+		if (args->u.reg_aen.flags & ~KVM_S390_ZPCIOP_REGAEN_HOST) {
+			r = -EINVAL;
+			break;
+		}
+		r = kvm_s390_pci_zpci_reg_aen(zdev, args);
+		break;
+	case KVM_S390_ZPCIOP_DEREG_AEN:
+		r = kvm_s390_pci_aif_disable(zdev, false);
+		break;
+	default:
+		r = -EINVAL;
+	}
+
+out:
+	mutex_unlock(&kvm->lock);
+	mutex_unlock(&zdev->kzdev_lock);
+	return r;
+}
+
  int kvm_s390_pci_init(void)
  {
  	aift = kzalloc(sizeof(struct zpci_aift), GFP_KERNEL);
arch/s390/kvm/pci.h +2 -0
index fb2b91b76e0c..0351382e990f 100644
@@ -59,6 +59,8 @@ void kvm_s390_pci_aen_exit(void);
  void kvm_s390_pci_init_list(struct kvm *kvm);
  void kvm_s390_pci_clear_list(struct kvm *kvm);
 
+int kvm_s390_pci_zpci_op(struct kvm *kvm, struct kvm_s390_zpci_op *args);
+
  int kvm_s390_pci_init(void);
  void kvm_s390_pci_exit(void);
 
include/uapi/linux/kvm.h +31 -0
index 5088bd9f1922..52cc12d53022 100644
@@ -1157,6 +1157,7 @@ struct kvm_ppc_resize_hpt {
  #define KVM_CAP_VM_TSC_CONTROL 214
  #define KVM_CAP_SYSTEM_EVENT_DATA 215
  #define KVM_CAP_ARM_SYSTEM_SUSPEND 216
+#define KVM_CAP_S390_ZPCI_OP 217
 
  #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -2118,4 +2119,34 @@ struct kvm_stats_desc {
  /* Available with KVM_CAP_XSAVE2 */
  #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
 
+/* Available with KVM_CAP_S390_ZPCI_OP */
+#define KVM_S390_ZPCI_OP         _IOW(KVMIO,  0xd1, struct kvm_s390_zpci_op)
+
+struct kvm_s390_zpci_op {
+	/* in */
+	__u32 fh;               /* target device */
+	__u8  op;               /* operation to perform */
+	__u8  pad[3];
+	union {
+		/* for KVM_S390_ZPCIOP_REG_AEN */
+		struct {
+			__u64 ibv;      /* Guest addr of interrupt bit vector */
+			__u64 sb;       /* Guest addr of summary bit */
+			__u32 flags;
+			__u32 noi;      /* Number of interrupts */
+			__u8 isc;       /* Guest interrupt subclass */
+			__u8 sbo;       /* Offset of guest summary bit vector */
+			__u16 pad;
+		} reg_aen;
+		__u64 reserved[8];
+	} u;
+};
+
+/* types for kvm_s390_zpci_op->op */
+#define KVM_S390_ZPCIOP_REG_AEN                0
+#define KVM_S390_ZPCIOP_DEREG_AEN      1
+
+/* flags for kvm_s390_zpci_op->u.reg_aen.flags */
+#define KVM_S390_ZPCIOP_REGAEN_HOST    (1 << 0)
+
  #endif /* __LINUX_KVM_H */
Signature · 2 lines
Pierre Morel
IBM Lab Boeblingen
author maintainer
#34 lore raw

PATCH v9 21/21 MAINTAINERS: additional files related kvm s390 pci passthrough

In reply to Matthew Rosato

Add entries from the s390 kvm subdirectory related to pci passthrough.

---
 MAINTAINERS | 1 +
 1 file changed, 1 insertion(+)
MAINTAINERS +1 -0
index a6d3bd9d2a8d..3dd8657f5482 100644
@@ -17454,6 +17454,7 @@ M:	Eric Farman <[email protected]>
 L:	[email protected]
 L:	[email protected]
 S:	Supported
+F:	arch/s390/kvm/pci*
 F:	drivers/vfio/pci/vfio_pci_zdev.c
 F:	include/uapi/linux/vfio_zdev.h
 
Signature · 1 line
2.27.0
Reviewed-by on v9 21/21
#35 lore raw

Re: [PATCH v9 21/21] MAINTAINERS: additional files related kvm s390 pci passthrough

Quoted 13 lines · Matthew Rosato

Add entries from the s390 kvm subdirectory related to pci passthrough.

---
  MAINTAINERS | 1 +
  1 file changed, 1 insertion(+)
MAINTAINERS +0 -0
index a6d3bd9d2a8d..3dd8657f5482 100644

MAINTAINERS M: Eric Farman <[email protected]> · line 17460

Quoted 7 lines
MAINTAINERS +1 -0
@@ -17454,6 +17454,7 @@ M:	Eric Farman <[email protected]>
  L:	[email protected]
  L:	[email protected]
  S:	Supported
+F:	arch/s390/kvm/pci*
  F:	drivers/vfio/pci/vfio_pci_zdev.c
  F:	include/uapi/linux/vfio_zdev.h
 
author maintainer
#36 lore raw
In reply to Matthew Rosato
Quoted 14 lines · Matthew Rosato

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci. This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device. Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle). Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

Quoted 6 lines

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Latest QEMU series:

https://lore.kernel.org/kvm/[email protected]/

author maintainer
#37 lore raw
In reply to Matthew Rosato
Quoted 20 lines · Matthew Rosato

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci. This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device. Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle). Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Quoted 12 lines

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' based on discussion in QEMU thread.

Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17).

#38 lore raw

Am 27.06.22 um 22:57 schrieb Matthew Rosato:

Quoted 34 lines · Matthew Rosato
Quoted 20 lines

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci.  This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device.  Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle).  Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

 From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Quoted 12 lines

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' based on discussion in QEMU thread.

Quoted 1 line

Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17).

Yes, I will queue this (as is). Ideally you would rebase this on top of kvm/next but I can also do while applying. Let me know if you want to respin with the Nits from Pierre.

author maintainer
#39 lore raw
Quoted 36 lines · Christian Borntraeger

Am 27.06.22 um 22:57 schrieb Matthew Rosato:

Quoted 35 lines · Matthew Rosato
Quoted 33 lines

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci.  This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device.  Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle).  Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

 From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation'
   based on discussion in QEMU thread.

Quoted 4 lines
Quoted 1 line

Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17).

Yes, I will queue this (as is). Ideally you would rebase this on top of kvm/next but I can also do while applying. Let me know if you want to respin with the Nits from Pierre.

Ah, sorry -- I assume you mean Paolo's kvm/next? I tried now and see some conflicts with the ioctl patch.

Why don't I rebase on top of kvm/next along with these couple of changes from Pierre and send this as a v10 for you to queue.

While at it, there's one other issue to be aware of -- There will also be small merge conflicts with a patch that just hit vfio-next, "vfio: de-extern-ify function prototypes" - My series already avoids adding externs to new prototypes, but adjacent code changes will cause a conflict with patches 10 and 17.

Not sure what the best way to proceed there is.

https://lore.kernel.org/kvm/165471414407.203056.474032786990662279.stgit@omen/

#40 lore raw
Quoted 54 lines · Matthew Rosato
Quoted 40 lines · Christian Borntraeger

Am 27.06.22 um 22:57 schrieb Matthew Rosato:

Quoted 39 lines · Matthew Rosato
Quoted 37 lines

Enable interpretive execution of zPCI instructions + adapter interruption
forwarding for s390x KVM vfio-pci.  This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest
for interpretive execution on that zPCI device.  Load/store interpreation
enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle).  Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

 From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation'
   based on discussion in QEMU thread.

Quoted 6 lines
Quoted 2 lines

Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17).

Yes, I will queue this (as is). Ideally you would rebase this on top of kvm/next but I can also do while applying.
Let me know if you want to respin with the Nits from Pierre.

Ah, sorry -- I assume you mean Paolo's kvm/next? I tried now and see some conflicts with the ioctl patch.

Why don't I rebase on top of kvm/next along with these couple of changes from Pierre and send this as a v10 for you to queue.

Quoted 7 lines

While at it, there's one other issue to be aware of -- There will also be small merge conflicts with a patch that just hit vfio-next, "vfio:
de-extern-ify function prototypes" - My series already avoids adding externs to new prototypes, but adjacent code changes will cause a conflict with patches 10 and 17.

Not sure what the best way to proceed there is.

You should use a branch based on vfio-next and send a Git PR to Christian and Alex

Jason

#41 lore raw
In reply to Matthew Rosato

Am 28.06.22 um 15:40 schrieb Matthew Rosato:

Quoted 43 lines · Christian Borntraeger
Quoted 35 lines

Am 27.06.22 um 22:57 schrieb Matthew Rosato:

Quoted 34 lines · Matthew Rosato
Quoted 32 lines

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci.  This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device.  Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle).  Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

 From the perspective of guest configuration, you passthrough zPCI devices in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' based on discussion in QEMU thread.

Quoted 4 lines
Quoted 1 line

Ping -- I'm hoping this can make the next merge window, but there are still 2 patches left without any review tag (16 & 17).

Yes, I will queue this (as is). Ideally you would rebase this on top of kvm/next but I can also do while applying. Let me know if you want to respin with the Nits from Pierre.

Ah, sorry -- I assume you mean Paolo's kvm/next?  I tried now and see some conflicts with the ioctl patch.

Quoted 7 lines

Why don't I rebase on top of kvm/next along with these couple of changes from Pierre and send this as a v10 for you to queue.

While at it, there's one other issue to be aware of -- There will also be small merge conflicts with a patch that just hit vfio-next, "vfio: de-extern-ify function prototypes" - My series already avoids adding externs to new prototypes, but adjacent code changes will cause a conflict with patches 10 and 17.

Not sure what the best way to proceed there is.

https://lore.kernel.org/kvm/165471414407.203056.474032786990662279.stgit@omen/

I think Linus can sort out if the conflicts are trivial. As an alternative Alex could carry these patches, but then we have a merge conflict between him and KVM. Alex/Paolo, shall I do a topic branch that you both can merge?

#42 lore raw
In reply to Matthew Rosato

Am 06.06.22 um 22:33 schrieb Matthew Rosato:

Quoted 89 lines

Enable interpretive execution of zPCI instructions + adapter interruption forwarding for s390x KVM vfio-pci. This is done by triggering a routine when the VFIO group is associated with the KVM guest, transmitting to firmware a special token (GISA designation) to enable that specific guest for interpretive execution on that zPCI device. Load/store interpreation enablement is then controlled by userspace (based upon whether or not a SHM bit is placed in the virtual function handle). Adapter Event Notification interpretation is controlled from userspace via a new KVM ioctl.

By allowing intepretation of zPCI instructions and firmware delivery of interrupts to guests, we can reduce the frequency of guest SIE exits for zPCI.

From the perspective of guest configuration, you passthrough zPCI devices
in the same manner as before, with intepretation support being used by default if available in kernel+qemu.

Will follow up with a link the most recent QEMU series.

Changelog v8->v9:
- Rebase on top of 5.19-rc1, adjust ioctl and capability defines
- s/kzdev = 0/kzdev = NULL/ (Alex)
- rename vfio_pci_zdev_open to vfio_pci_zdev_open_device (Jason)
- rename vfio_pci_zdev_release to vfio_pci_zdev_close_device (Jason)
- make vfio_pci_zdev_close_device return void, instead WARN_ON or ignore errors in lower level function (kvm_s390_pci_unregister_kvm) (Jason)
- remove notifier accidentally left in struct zpci_dev + associated include statment (Jason)
- Remove patch 'KVM: s390: introduce CPU feature for zPCI Interpretation' based on discussion in QEMU thread.

Matthew Rosato (21):
s390/sclp: detect the zPCI load/store interpretation facility s390/sclp: detect the AISII facility
s390/sclp: detect the AENI facility
s390/sclp: detect the AISI facility
s390/airq: pass more TPI info to airq handlers
s390/airq: allow for airq structure that uses an input vector s390/pci: externalize the SIC operation controls and routine s390/pci: stash associated GISA designation
s390/pci: stash dtsm and maxstbl
vfio/pci: introduce CONFIG_VFIO_PCI_ZDEV_KVM
KVM: s390: pci: add basic kvm_zdev structure
KVM: s390: pci: do initial setup for AEN interpretation
KVM: s390: pci: enable host forwarding of Adapter Event Notifications
KVM: s390: mechanism to enable guest zPCI Interpretation
KVM: s390: pci: provide routines for enabling/disabling interrupt
forwarding
KVM: s390: pci: add routines to start/stop interpretive execution vfio-pci/zdev: add open/close device hooks
vfio-pci/zdev: add function handle to clp base capability vfio-pci/zdev: different maxstbl for interpreted devices
KVM: s390: add KVM_S390_ZPCI_OP to manage guest zPCI devices
MAINTAINERS: additional files related kvm s390 pci passthrough

  Documentation/virt/kvm/api.rst   |  47 +++
  MAINTAINERS                      |   1 +
  arch/s390/include/asm/airq.h     |   7 +-
  arch/s390/include/asm/kvm_host.h |  23 ++
  arch/s390/include/asm/pci.h      |  11 +
  arch/s390/include/asm/pci_clp.h  |   9 +-
  arch/s390/include/asm/pci_insn.h |  29 +-
  arch/s390/include/asm/sclp.h     |   4 +
  arch/s390/include/asm/tpi.h      |  13 +
  arch/s390/kvm/Makefile           |   1 +
  arch/s390/kvm/interrupt.c        |  96 ++++-
  arch/s390/kvm/kvm-s390.c         |  83 +++-
  arch/s390/kvm/kvm-s390.h         |  10 +
  arch/s390/kvm/pci.c              | 690 +++++++++++++++++++++++++++++++
  arch/s390/kvm/pci.h              |  88 ++++
  arch/s390/pci/pci.c              |  16 +
  arch/s390/pci/pci_clp.c          |   7 +
  arch/s390/pci/pci_insn.c         |   4 +-
  arch/s390/pci/pci_irq.c          |  48 ++-
  drivers/s390/char/sclp_early.c   |   4 +
  drivers/s390/cio/airq.c          |  12 +-
  drivers/s390/cio/qdio_thinint.c  |   6 +-
  drivers/s390/crypto/ap_bus.c     |   9 +-
  drivers/s390/virtio/virtio_ccw.c |   6 +-
  drivers/vfio/pci/Kconfig         |  11 +
  drivers/vfio/pci/Makefile        |   2 +-
  drivers/vfio/pci/vfio_pci_core.c |  10 +-
  drivers/vfio/pci/vfio_pci_zdev.c |  35 +-
  include/linux/sched/user.h       |   3 +-
  include/linux/vfio_pci_core.h    |  12 +-
  include/uapi/linux/kvm.h         |  31 ++
  include/uapi/linux/vfio_zdev.h   |   7 +
  32 files changed, 1279 insertions(+), 56 deletions(-)
Quoted 2 lines

create mode 100644 arch/s390/kvm/pci.c
create mode 100644 arch/s390/kvm/pci.h

So I pulled this into a topic branch and will merge that into kvms390/next. We can merge this topic branch into vfio-next and/or s390-next when the conflicts get to complicated.

While pulling I fixed up the numbers for the capability to

#define KVM_CAP_S390_ZPCI_OP 221

and the doc number to

4.137 KVM_S390_ZPCI_OP

to minize struggle when doing backports.